Application test method and device, electronic equipment and storage medium

Abstract

The invention relates to an application testing method and device, electronic equipment and a storage medium. The method comprises the steps that an interface image associated with a to-be-tested application is acquired; wherein the interface image comprises at least one operation object; determining the probability that each operation object responds to the operation in the interface image; and determining a to-be-operated target operation object based on the probability that each operation object responds to the operation. Through the scheme, the matching degree of the test application and the user operation preference can be improved, and the manual maintenance workload is reduced on the basis that the application test covers more operation scenes.

Description

Technical Field

[0001] The present disclosure relates to the field of application testing technology, and in particular to an application testing method and device, electronic equipment, and storage medium. Background Art

[0002] As the diverse applications (Application, APP) in electronic devices have greatly facilitated people's daily lives, the testing of applications has also ushered in rapid development. Traditional application testing methods include UI testing (User interface testing) and Monkey automated testing. Among them, UI testing has problems such as difficulty in identifying controls and maintenance, and the test cases are use cases with a fixed test sequence for controls, which cannot cover complex user operation scenarios; Monkey testing is a pseudo-random test with a certain degree of randomness. This random scenario has a certain deviation from the real user operation scenario. How to improve the intelligence of application testing to cover more user operation scenarios needs to be solved urgently. Summary of the invention

[0003] The present invention provides an application testing method and device, an electronic device, and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an application testing method, including:

[0005] Acquire an interface image associated with the application to be tested; wherein the interface image includes at least one operation object;

[0006] Determining the probability of each operation object in the interface image responding to the operation;

[0007] Determine the target operation object to be operated based on the probability of each operation object responding to the operation;

[0008] A test result of performing an operation simulation on the target operation object is obtained.

[0009] In some embodiments, the at least one operation object in the interface image includes: at least one application icon of an application to be tested;

[0010] The step of determining the target operation object to be operated based on the probability of each operation object responding to the operation includes:

[0011] Based on the probability of the application icons of each application responding to the operation, the application icons that meet the preset first probability value are determined as the target operation objects to be operated.

[0012] In some embodiments, the at least one operation object in the interface image includes: a control icon of at least one operation control included in the interface image of the application to be tested;

[0013] The step of determining the target operation object to be operated based on the probability of each operation object responding to the operation includes:

[0014] Based on the probability of the control icon of each operation control responding to the operation, the control icon that meets the preset second probability value is determined as the target operation object to be operated.

[0015] In some embodiments, determining the target operation object to be operated based on the probability of each operation object responding to the operation includes:

[0016] In the case where the interface image includes multiple operation objects, based on the probability of each operation object responding to the operation, multiple target operation objects to be operated that meet a preset probability range are determined, and according to the probability ranking results of the multiple target operation objects to be operated responding to the operation, the operation order of the multiple target operation objects to be operated is determined.

[0017] In some embodiments, determining the probability of each operation object in the interface image responding to the operation includes:

[0018] Performing target detection on the interface image to obtain coordinates of each operation object in the interface image;

[0019] For each operation object, determining an object identifier of the operation object based on the coordinates of the operation object in the interface image;

[0020] Based on the object identification and coordinates of each operation object, the probability of each operation object responding to the operation is determined.

[0021] In some embodiments, performing target detection on the interface image to obtain coordinates of each operation object in the interface image includes:

[0022] The interface image is subjected to target detection using a preset target detection model to obtain coordinates of each operation object in the interface image; wherein the target detection model is obtained based on training of a first neural network model.

[0023] In some embodiments, the method further comprises:

[0024] Acquire a sample interface image set; wherein the sample interface images in the sample interface image set include at least one sample operation object and a coordinate label value of the sample operation object;

[0025] Using the feature extraction layer of the first neural network model to extract features from each sample interface image in the sample interface image set, to obtain a first feature of the sample interface image;

[0026] Using the feature optimization layer of the first neural network model to perform feature optimization on the first feature of the sample interface image to obtain an optimized second feature;

[0027] Performing target detection based on the second feature of the sample interface image to determine the coordinate prediction value of each sample operation object in the sample interface image;

[0028] According to the coordinate label values ​​of each sample operation object included in each sample interface image, the coordinate prediction value of each sample operation object and the loss function, the parameters of the first neural network model are adjusted to obtain the target detection model.

[0029] In some embodiments, the coordinate label value of the sample operation object includes a region label value of a region where the sample operation object is located; the coordinate prediction value of the sample operation object includes a region prediction value of a region where the sample operation object is located;

[0030] The adjusting the parameters of the first neural network model according to the coordinate prediction value of the sample operation object, the coordinate label value of the sample operation object and the loss function includes:

[0031] Determining, according to the region label value of the region where the sample operation object is located and the region prediction value, the degree of overlap between the region indicated by the region label value and the region indicated by the region prediction value;

[0032] According to the overlap and the loss function, the parameters of the first neural network model are adjusted.

[0033] In some embodiments, determining the probability of each operation object responding to the operation based on the object identification and coordinates of each operation object includes:

[0034] The object identification and coordinates of each operation object in the interface image are processed using a preset probability prediction model to obtain the probability of each operation object responding to the operation; wherein the operation prediction model is obtained based on the training of the second neural network model.

[0035] In some embodiments, the method further comprises:

[0036] Acquire a set of sample operation objects; wherein the set of sample operation objects includes the coordinates of the sample operation objects in each sample interface image and the identifier of the sample operation object; each sample operation object is associated with an operation probability label value;

[0037] Using a second neural network model, the coordinates of each sample operation object in the sample operation object set and the identifier of the sample operation object are processed to determine an operation probability prediction value of the sample operation object;

[0038] According to the operation probability prediction value of each sample operation object and the operation probability label value of the sample operation object, the parameters of the second neural network model are adjusted to obtain the probability prediction model.

[0039] In some embodiments, the application testing method is applied to a first device, and an application to be tested is installed in a second device;

[0040] The method further comprises:

[0041] sending a control instruction indicating a target object to be operated to the second device;

[0042] The obtaining of the test result of the operation simulation on the target operation object includes:

[0043] A test result of the second device performing an operation simulation on the target operation object based on the control instruction is obtained.

[0044] In some embodiments, obtaining a test result of the second device performing an operation simulation on the target operation object based on the control instruction includes:

[0045] In response to the second device detecting an abnormality during the operation simulation process on the target operation object, a test result including log data and operation video during the operation simulation process on the target operation object is obtained.

[0046] According to a second aspect of an embodiment of the present disclosure, there is provided an application testing device, including:

[0047] A first acquisition module is used to acquire an interface image associated with the application to be tested; wherein the interface image includes at least one operation object;

[0048] A first determination module, used to determine the probability of each operation object in the interface image responding to the operation;

[0049] A second determination module is used to determine a target operation object to be operated based on the probability of each operation object responding to the operation;

[0050] The second acquisition module is used to obtain the test result of the operation simulation performed on the target operation object.

[0051] In some embodiments, the at least one operation object in the interface image includes: at least one application icon of an application to be tested;

[0052] The second determination module is used to determine the application icon that meets a preset first probability value as the target operation object to be operated based on the probability of the application icon of each application responding to the operation.

[0053] In some embodiments, the at least one operation object in the interface image includes: a control icon of at least one operation control included in the interface image of the application to be tested;

[0054] The second determination module is used to determine the control icon that meets the preset second probability value as the target operation object to be operated based on the probability of the control icon of each operation control responding to the operation.

[0055] In some embodiments, the second determination module is used to determine, when the interface image includes multiple operation objects, multiple target operation objects to be operated that meet a preset probability range based on the probability of each operation object responding to the operation, and determine the operation order of the multiple target operation objects to be operated according to the probability ranking results of the multiple target operation objects to be operated responding to the operation.

[0056] In some embodiments, the first determination module is used to perform target detection on the interface image to obtain the coordinates of each operation object in the interface image; for each operation object, determine the object identification of the operation object based on the coordinates of the operation object in the interface image; based on the object identification and coordinates of each operation object, determine the probability of each operation object responding to the operation.

[0057] In some embodiments, the first determination module is used to perform target detection on the interface image using a preset target detection model to obtain the coordinates of each operation object in the interface image; wherein the target detection model is obtained based on training of a first neural network model.

[0058] In some embodiments, the apparatus further comprises:

[0059] A third acquisition module is used to acquire a sample interface image set; wherein the sample interface image in the sample interface image set includes at least one sample operation object and a coordinate label value of the sample operation object;

[0060] A feature extraction module, used to extract features from each sample interface image in the sample interface image set using the feature extraction layer of the first neural network model to obtain a first feature of the sample interface image;

[0061] A feature optimization module, used for optimizing the first feature of the sample interface image by using the feature optimization layer of the first neural network model to obtain an optimized second feature;

[0062] A first prediction module, used for performing target detection based on the second feature of the sample interface image, and determining a coordinate prediction value of each sample operation object in the sample interface image;

[0063] The first adjustment module is used to adjust the parameters of the first neural network model according to the coordinate label values ​​of each sample operation object included in each sample interface image, the coordinate prediction value of each sample operation object and the loss function to obtain the target detection model.

[0064] In some embodiments, the coordinate label value of the sample operation object includes a region label value of a region where the sample operation object is located; the coordinate prediction value of the sample operation object includes a region prediction value of a region where the sample operation object is located;

[0065] The first adjustment module is used to determine the overlap between the area indicated by the area label value and the area indicated by the area prediction value according to the area label value of the area where the sample operation object is located and the area prediction value; and adjust the parameters of the first neural network model according to the overlap and the loss function.

[0066] In some embodiments, the first determination module is used to process the object identification and coordinates of each operation object in the interface image using a preset probability prediction model to obtain the probability of each operation object responding to the operation; wherein the operation prediction model is obtained based on training of the second neural network model.

[0067] In some embodiments, the apparatus further comprises:

[0068] A fourth acquisition module is used to acquire a sample operation object set; wherein the sample operation object set includes the coordinates of the sample operation object in each sample interface image and the identifier of the sample operation object; each sample operation object is associated with an operation probability label value;

[0069] A second prediction module is used to process the coordinates of each sample operation object in the sample operation object set and the identifier of the sample operation object by using a second neural network model to determine an operation probability prediction value of the sample operation object;

[0070] The second adjustment module is used to adjust the parameters of the second neural network model according to the operation probability prediction value of each sample operation object and the operation probability label value of the sample operation object to obtain the probability prediction model.

[0071] In some embodiments, the application testing method is applied to a first device, and an application to be tested is installed in a second device;

[0072] The device also includes:

[0073] A sending module, used for sending a control instruction indicating a target object to be operated to the second device;

[0074] The second acquisition module is used to obtain a test result of the second device performing an operation simulation on the target operation object based on the control instruction.

[0075] In some embodiments, the second acquisition module is used to acquire test results including log data and operation video during the operation simulation process on the target operation object in response to the second device detecting an abnormality during the operation simulation process on the target operation object.

[0076] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0077] A processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the method as described in the first aspect above.

[0078] According to a fourth aspect of an embodiment of the present disclosure, there is provided a storage medium, including:

[0079] When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as described in the first aspect above.

[0080] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0081] The test device of the embodiment of the present disclosure determines the probability of each operation object in the interface image of the application to be tested responding to the operation, and determines the target operation object based on the probability of each operation object responding to the operation, thereby simulating the operation of the target operation object to obtain the test result. Compared with the test based on the preset fixed operation sequence in the related art, the method of determining the target operation object in the embodiment of the present disclosure is more flexible, can cover more operation scenarios, and does not require manual maintenance of the operation sequence, thereby greatly reducing the labor cost while improving the intelligence of the test. On the other hand, compared with the test based on the pseudo-random operation sequence in the related art, the probability of the operation object responding to the operation in the embodiment of the present disclosure is more correlated with the user's operation preference, so that the target operation object to be operated based on the current interface image is more matched with the user's operation preference, and on the basis of improving the accuracy of the test results of the simulated operation based on the target operation object, the waste of resources caused by testing the scene that deviates from the user's operation preference is reduced.

[0082] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0084] Figure 1 This is an example diagram of a Monkey test method process in the related technology.

[0085] Figure 2 This is an example of an application testing method flow provided by the embodiment of the present disclosure Figure 1 .

[0086] Figure 3 This is an example diagram of a target detection model provided by an embodiment of the present disclosure.

[0087] Figure 4 This is an example diagram of the input end of a target detection model provided in an embodiment of the present disclosure.

[0088] Figure 5 This is an example diagram of the Yolov5 architecture of a target detection model provided in an embodiment of the present disclosure.

[0089] Figure 6 This is an example diagram of a C3 module provided in an embodiment of the present disclosure.

[0090] Figure 7 This is an example diagram of a Conv module provided in an embodiment of the present disclosure.

[0091] Figure 8 This is an example diagram of a Focus module provided in an embodiment of the present disclosure.

[0092] Fig. 9 This is an example diagram of an SPP module provided by an embodiment of the present disclosure.

[0093] Fig.10 This is an example diagram of an iterative training log of a probability prediction model provided by an embodiment of the present disclosure.

[0094] Fig.11 This is an example diagram of an application test provided by an embodiment of the present disclosure.

[0095] Fig.12 This is an example of a test method flow provided by the embodiment of the present disclosure. Figure 2 .

[0096] Fig.13 is a diagram of an application testing device shown in an embodiment of the present disclosure.

[0097] Fig.14 It is a block diagram of an electronic device shown in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0098] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0099] Figure 1 This is a flow chart of a Monkey test method in the related art, which is applied to an electronic device equipped with an application. Figure 1 It can be seen that the following steps are included:

[0100] S101, executing an action;

[0101] S1011, voice;

[0102] S1012, action;

[0103] In an embodiment of the present disclosure, the electronic device starts to execute the test based on the received instructions; wherein the received instructions may be voice instructions received from the user, or may be user action (operation) instructions detected on an application interface displayed by a display component of the electronic device.

[0104] S102, monitoring equipment;

[0105] In the embodiments of the present disclosure, an operation simulation is performed on an application based on voice instructions and / or operation instructions, and during the operation simulation, the status of the application currently being tested is monitored.

[0106] S1021, log analysis;

[0107] In an embodiment of the present disclosure, in response to an abnormal state of an application currently being tested during an operation simulation, a process log of the application is captured and the log analysis is performed.

[0108] S1022, timed screenshots;

[0109] In the embodiment of the present disclosure, in response to the state of the application currently being tested being abnormal during the operation simulation process, an image including the simulation operation is captured at a fixed time.

[0110] S103: Generate a report.

[0111] In the disclosed embodiment, if the simulated operation of the application does not have an abnormality, a test result report indicating that the test is normal is generated. If the simulated operation of the application has an abnormality, a test failure result report including the log analysis result and the captured image is generated.

[0112] As can be seen from the foregoing, testing methods including the Monkey test are a pseudo-random testing method with a certain degree of randomness, which is quite different from the user's actual operation scenarios and is difficult to cover complex user operation scenarios.

[0113] In this regard, an embodiment of the present disclosure provides an application testing method, the execution subject of which can be an application testing device, for example, the application testing method can be executed by a terminal device or a server or other electronic device, wherein the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. If the execution subject is a server or other electronic device, after determining the target operation object to be operated based on the probability of each operation object responding to the operation, a control instruction indicating the target object to be operated is sent to the terminal device, so that the terminal device performs an operation simulation on the target operation object, and feeds back the test result to the server or other electronic device. In some possible implementations, the application testing method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0114] Figure 2 This is an example of an application testing method flow provided by the embodiment of the present disclosure Figure 1 ,Depend on Figure 2 It can be seen that the following steps are included:

[0115] S201, obtaining an interface image associated with the application to be tested; wherein the interface image includes at least one operation object;

[0116] S202, determining the probability of each operation object in the interface image responding to the operation;

[0117] S203, determining a target operation object to be operated based on the probability of each operation object responding to the operation;

[0118] S204: Obtain test results of the operation simulation performed on the target operation object.

[0119] In step S201, the test device obtains an interface image associated with the application to be tested; wherein the interface image includes at least one operation object; for example, the interface image is such as the home page image of a mobile phone, and the operation object may be an application icon in the interface image; the interface image is such as an interface image during the running of the application, and the operation object may be a control icon in the interface image, etc. It should be noted that the application icon is used to indicate the application, and the control icon is used to indicate the control. The test device may also indicate the application and / or control based on other identifiers, which is not limited in the present disclosure.

[0120] In step S202, the testing device determines the probability of each operation object in the interface image responding to the operation based on the interface image of the application to be tested; in some embodiments, the testing device determines the position information of each operation object in the interface image based on the marked operation objects in the interface image, and then based on the corresponding relationship between the preset position information and the probability, determines the probability corresponding to the position information of the operation object as the probability of the operation object responding to the operation.

[0121] It can be understood that since the position of the operation object in the interface image is correlated with the user's actual preference for interacting with the operation object, for example, it is easier for users to operate controls located at the lower part of the interface. Therefore, determining the probability of the operation object responding to the operation based on the position information of the operation object can improve the match between the probability of the operation object responding to the operation and the user's actual operation scenario.

[0122] In some embodiments, the testing device determines the probability of each operation object in the interface image responding to the operation based on the type of each operation object in the interface image and the correspondence between the preset type and the probability; wherein the type of the operation object can be divided based on the display style of the operation object, the function of the operation object, etc.

[0123] It can be understood that since the type of operation object is correlated with the user's actual preference for interacting with the operation object, for example, users prefer applications with instant messaging functions, and less frequently use settings and other applications; users prefer controls with animated display styles, and less frequently use static controls, etc. Therefore, determining the probability of the operation object responding to the operation based on the type of the operation object can improve the match between the probability of the operation object responding to the operation and the user's actual operation scenario.

[0124] In step S203, the test device determines the target operation object to be operated based on the probability of each operation object responding to the operation. For example, the test device may determine the operation object corresponding to the highest probability of responding to the operation as the target operation object to be operated; or determine multiple operation objects whose probability of responding to the operation is greater than a preset probability threshold as the target operation object.

[0125] In step S204, the testing device obtains the test result of the operation simulation on the target operation object; if the application testing method of the embodiment of the present disclosure is executed on the terminal device (considered to be the APK end of the installation package of the application to be tested), after determining the target operation object, the terminal device simulates the operation on the target operation object, thereby obtaining the test result of the operation simulation on the target operation object.

[0126] If steps S201 to S203 of the embodiment of the present disclosure are all executed in the aforementioned server or other electronic device (regarded as the platform end), the platform end is connected to the APK end; after the platform end determines the target operation object, the APK end simulates the operation on the target operation object and feeds back the test results to the platform end. The simulation operation on the target operation object can be performed based on tools such as UIAutomator, Robotium, MobileRunner, etc., and the present disclosure does not limit this.

[0127] It should be noted that if there is one target operation object, the testing device can perform a simulated operation on the target operation object as the object of the next simulated operation; if there are multiple target operation objects, the testing device can randomly generate a simulated operation sequence for the multiple target operation objects, or generate a simulated operation sequence based on factors such as the distance relationship of the positions of each target operation object, thereby performing a simulated operation on each target operation object based on the simulated operation sequence.

[0128] It can be understood that the test device of the embodiment of the present disclosure determines the probability of each operation object in the interface image of the application to be tested responding to the operation, and determines the target operation object based on the probability of each operation object responding to the operation, thereby simulating the operation of the target operation object to obtain the test result. Compared with the test based on the preset fixed operation sequence in the related art, the method of determining the target operation object in the embodiment of the present disclosure is more flexible, can cover more operation scenarios, and does not require manual maintenance of the operation sequence, thereby greatly reducing the labor cost while improving the intelligence of the test. On the other hand, compared with the test based on the pseudo-random operation sequence in the related art, the probability of the operation object responding to the operation in the embodiment of the present disclosure is more correlated with the user's operation preference, so that the target operation object to be operated based on the current interface image is more matched with the user's operation preference, and on the basis of improving the accuracy of the test results of the simulated operation based on the target operation object, the waste of resources caused by testing the scene that deviates from the user's operation preference is reduced.

[0129] In some embodiments, the at least one operation object in the interface image includes: at least one application icon of an application to be tested;

[0130] The step of determining the target operation object to be operated based on the probability of each operation object responding to the operation includes:

[0131] Based on the probability of the application icons of each application responding to the operation, the application icons that meet the preset first probability value are determined as the target operation objects to be operated.

[0132] In an embodiment of the present disclosure, at least one operation object in the interface image includes at least one application icon of an application to be tested. For example, the interface image includes a home page image of a mobile phone, and the operation objects are application icons of each application in the home page.

[0133] The test device of the disclosed embodiment determines the application icon that meets the preset first probability value as the target operation object to be operated based on the probability of the application icon of each application responding to the operation; wherein, if the target operation object includes one application icon, the simulation operation of the application indicated by the launch application icon is performed. If the target operation object includes multiple application icons, the simulation operation of each application icon is performed in sequence, so as to realize the needs of cross-testing between different applications, such as launching a phone application to answer a call when browsing a social application, returning to a video application to watch a video after replying a message in an instant chat application, etc.

[0134] It can be understood that the embodiment of the present disclosure determines the application icons that meet the preset first probability value as the target operation objects to be operated based on the probability of the application icons of each application responding to the operation, which can cover the scenario requirements such as initiating simulation tests on applications and cross-testing between multiple applications, thereby improving the comprehensiveness and reliability of the test results.

[0135] In some embodiments, the at least one operation object in the interface image includes: a control icon of at least one operation control included in the interface image of the application to be tested;

[0136] The step of determining the target operation object to be operated based on the probability of each operation object responding to the operation includes:

[0137] Based on the probability of the control icon of each operation control responding to the operation, the control icon that meets the preset second probability value is determined as the target operation object to be operated.

[0138] In an embodiment of the present disclosure, at least one operation object in the interface image includes a control icon of at least one operation control in the interface image of the application to be tested, for example, control icons of various controls in a certain interface during the running of the application.

[0139] The test device of the disclosed embodiment determines the control icon that meets the preset second probability value as the target operation object to be operated based on the probability of the control icon of each operation control responding to the operation; wherein, if the target operation object includes a control icon, a simulated operation of the function corresponding to the control indicated by the control icon is performed. If the target operation object includes multiple control icons, the simulated operation of each control icon is performed in sequence, so as to realize the cross-test of the reply message control after viewing the chat object information control in the instant chat application interface.

[0140] It can be understood that the embodiment of the present disclosure determines the control icon that meets the preset second probability value as the target operation object to be operated based on the probability of the control icon of each control in the application to be tested responding to the operation, which can cover the needs of simulation testing of the function corresponding to the running control icon and cross-testing between the functions corresponding to multiple controls, thereby improving the comprehensiveness and reliability of the test results.

[0141] In some embodiments, determining the target operation object to be operated based on the probability of each operation object responding to the operation includes:

[0142] In the case where the interface image includes multiple operation objects, based on the probability of each operation object responding to the operation, multiple target operation objects to be operated that meet a preset probability range are determined, and according to the probability ranking results of the multiple target operation objects to be operated responding to the operation, the operation order of the multiple target operation objects to be operated is determined.

[0143] In an embodiment of the present disclosure, when the interface image includes multiple operation objects, the testing device determines multiple target operation objects to be operated that meet a preset probability range based on the probability of each operation object responding to the operation; since the probability of each target operation object responding to the operation is positively correlated with the degree of user preference for interaction with the target operation object, the embodiment of the present disclosure determines the operation order of multiple target operation objects to be operated based on the probability ranking results of the multiple target operation objects to be operated responding to the operation, which can improve the matching degree between the operation order of each target operation object and the user's preferred operation order, thereby improving the purposefulness of performing simulation operations based on the operation order and the reliability of obtaining test results based on the simulation operations.

[0144] In some embodiments, determining the probability of each operation object in the interface image responding to the operation includes:

[0145] Performing target detection on the interface image to obtain coordinates of each operation object in the interface image;

[0146] For each operation object, determining an object identifier of the operation object based on the coordinates of the operation object in the interface image;

[0147] Based on the object identification and coordinates of each operation object, the probability of each operation object responding to the operation is determined.

[0148] In the disclosed embodiment, the testing device performs target detection on the interface image using a preset target detection algorithm to obtain the coordinates of each operation object in the interface image; wherein the preset target detection algorithm includes an image segmentation algorithm, a foreground algorithm, a feature extraction algorithm, and the like.

[0149] After determining the coordinates of each operation object, the testing device determines the object identifier of the operation object for each operation object based on the coordinates of the operation object in the interface image; for example, based on the correspondence between the preset coordinates and the identifier, the testing device determines the identifier corresponding to the coordinates of the operation object as the object identifier of the operation object; or, based on the correspondence between the preset coordinate range and the identifier, the testing device determines the identifier corresponding to the range within which the coordinates of the operation object fall as the object identifier of the operation object.

[0150] The test device of the disclosed embodiment determines the probability of each operation object responding to the operation based on the object identification and coordinates of each operation object. In some embodiments, the probability corresponding to the coordinates is determined based on the preset correspondence between the coordinates and the probability; the probability corresponding to the identification is determined based on the preset correspondence between the identification and the probability; the probability corresponding to the coordinates and the probability corresponding to the identification are weighted to obtain the probability of the operation object responding to the operation.

[0151] In some embodiments, based on a preset correspondence between coordinates, identifiers, and probabilities, the probability corresponding to the identifier of the operation object and the coordinates of the operation object is determined as the probability that the operation object responds to the operation.

[0152] It can be understood that since the position of the operation object in the interface image is associated with the user's actual preference for interacting with the operation object, and the type of the operation object is associated with the user's actual preference for interacting with the operation object, the test device of the embodiment of the present disclosure jointly determines the probability of each operation object responding to the operation based on the object identification and coordinates of each operation object, which can improve the matching degree between the probability and the user's actual operation scenario, thereby improving the reliability of the test results.

[0153] In some embodiments, performing target detection on the interface image to obtain coordinates of each operation object in the interface image includes:

[0154] The interface image is subjected to target detection using a preset target detection model to obtain coordinates of each operation object in the interface image; wherein the target detection model is obtained based on training of a first neural network model.

[0155] In the embodiment of the present disclosure, the testing device uses a preset target detection model to perform target detection on the interface image to obtain the coordinates of each operation object in the interface image; since the target detection model is obtained after multiple training convergence of the first neural network model, the accuracy of determining the coordinates of each operation object in the interface image based on the target detection model is relatively high.

[0156] In some embodiments, the method further comprises:

[0157] Acquire a sample interface image set; wherein the sample interface images in the sample interface image set include at least one sample operation object and a coordinate label value of the sample operation object;

[0158] Using the feature extraction layer of the first neural network model to extract features from each sample interface image in the sample interface image set, to obtain a first feature of the sample interface image;

[0159] Using the feature optimization layer of the first neural network model to perform feature optimization on the first feature of the sample interface image to obtain an optimized second feature;

[0160] Performing target detection based on the second feature of the sample interface image to determine the coordinate prediction value of each sample operation object in the sample interface image;

[0161] According to the coordinate label values ​​of each sample operation object included in each sample interface image, the coordinate prediction value of each sample operation object and the loss function, the parameters of the first neural network model are adjusted to obtain the target detection model.

[0162] In an embodiment of the present disclosure, a testing device acquires a sample image set, wherein a sample interface image in the sample image set includes at least one sample operation object annotated with a coordinate label value.

[0163] It should be noted that if the operation object has good symmetry, the coordinate value label value of the operation object can be the coordinate of the center of the operation object; if the operation object does not have good symmetry, the coordinate value label value of the operation object can be the coordinate of the center of mass of the operation object.

[0164] In the process of training the first neural network model to obtain the target detection model in the embodiment of the present disclosure, all sample interface images in the sample image set can be used as model training samples; or part of the sample interface images can be used as model training samples, such as 70% of the sample interface images as the training set, 20% of the sample interface images as the verification set, and 10% of the sample interface images as the test set.

[0165] The target detection model of the embodiment of the present disclosure (i.e., the first neural network model after training convergence) can be obtained by training models based on network architectures such as Convolutional Neural Networks (CNN), Deep Neural Networks (DNN), Yolov4, Yolov5, Mask RCNN, etc., and the present disclosure does not limit this.

[0166] The target detection model of the disclosed embodiment includes a feature extraction layer, a feature optimization layer, a prediction layer, etc.; the feature extraction layer (backbone network) is used to extract features from each sample interface image in the sample interface image set to obtain the first feature of the sample interface image; for example, the feature extraction layer can be a Focus, CSP, C3 or other structure or a combination of the above structures, and can include at least one of the following: Conv module, C3 module, SPP module. Among them, the Conv module is composed of a convolutional layer, a BN layer and an activation function to perform feature extraction; the C3 module is composed of multiple Conv modules to increase the depth of the network and improve the ability of feature extraction; the SPP module realizes the spatial invariance and position invariance of the input data, and is used to improve the accuracy of feature extraction of the neural network.

[0167] The feature optimization layer (Neck), such as FPN, PAN and other structures or a combination of the above structures, is used to optimize the first feature of the sample interface image to obtain an optimized second feature.

[0168] The prediction layer (Prediction) is used to perform target detection based on the second feature of the sample interface image, determine the coordinate prediction value of each sample operation object in the sample interface image; and adjust the parameters of the first neural network model according to the coordinate label value of each sample operation object included in each sample interface image, the coordinate prediction value of each sample operation object and the loss function to obtain the target detection model. Among them, the loss function can be a mean square error loss function (MSE), a quadratic loss function (QL), a mean absolute error loss function (MAE), a smoothed mean absolute error loss (Huber Loss), a cross entropy loss (CE), an IOU loss function, etc., which is not limited in this disclosure.

[0169] Figure 3 The figure shows an example of a target detection model provided by an embodiment of the present disclosure, which is composed of Figure 3It can be seen that the feature extraction layer adopts the Focus+C3 structure, the feature optimization layer adopts the FPN+PAN structure, and the prediction layer adopts the GIOU_Loss loss function. Among them, the Focus structure can use 32 convolution kernels to slice the interface image; the CSP structure includes CSP_1 and CSP_2, CSP_1 is applied to the feature extraction layer, and CSP_2 is applied to the feature optimization layer. The feature optimization layer can also adopt the CSP2 structure designed with reference to CSPnet to enhance the ability of network feature fusion. At the output end of the target detection model, CIOU_Loss is used as the loss function; under the same model parameters, DIOU_nms is used to improve the accuracy of detecting occluded / overlapped operation objects in the interface image. It can be understood that compared with ordinary CNN and RNN models, the target detection model based on the above structure in the embodiment of the present disclosure has better detection speed and detection accuracy.

[0170] Figure 4 An example diagram of an input end of a target detection model provided in an embodiment of the present disclosure, comprising Figure 4 It can be known that through the input end of the target detection model, multiple sample interface images can be subjected to data enhancement processing to obtain multiple sample interface images after data enhancement processing; wherein, the multiple sample interface images after data enhancement are used to constitute a sample interface image set; the data enhancement processing includes at least one of the following: cropping, splicing, and scaling. For example, using Mosaic data enhancement technology, the interface image is subjected to random scaling, random cropping, random arrangement, and other processing; using adaptive anchor frame technology, based on the anchor frame of the initial set length, a predicted frame including the operation object (corresponding to the predicted coordinate value of the area where the operation object is located) is output for subsequent object matching with the real frame (corresponding to the predicted coordinate value of the area where the operation object is located), and the parameters of the first neural network model are iterated by reverse updating; using adaptive image scaling technology, a black border is adaptively added to the original interface image, and the scaling ratio, the scaled size, and the black border filling value are calculated.

[0171] In some embodiments, the coordinate label value of the sample operation object includes a region label value of a region where the sample operation object is located; the coordinate prediction value of the sample operation object includes a region prediction value of a region where the sample operation object is located;

[0172] The adjusting the parameters of the first neural network model according to the coordinate prediction value of the sample operation object, the coordinate label value of the sample operation object and the loss function includes:

[0173] Determining, according to the region label value of the region where the sample operation object is located and the region prediction value, the degree of overlap between the region indicated by the region label value and the region indicated by the region prediction value;

[0174] According to the overlap and the loss function, the parameters of the first neural network model are adjusted.

[0175] In an embodiment of the present disclosure, the coordinate label value of the sample operation object includes the area label value of the area where the sample operation object is located; the coordinate prediction value of the sample operation object includes the area prediction value of the area where the sample operation object is located; wherein the label value of the area and the area and test value include coordinate values ​​that can indicate the shape of the area, for example, the area label value or coordinate value corresponding to the rectangular area is the center coordinate value of the four sides of the rectangular area, and the area label value or coordinate value corresponding to the circular area is the center coordinate value of the circular area and the radius size; wherein rectangle and circle are only examples, and the embodiment of the present disclosure does not limit the shape of the area where the operation object is located.

[0176] The test device of the disclosed embodiment determines the overlap between the area indicated by the area label value and the area indicated by the area prediction value according to the area label value of the area where the sample operation object is located, and adjusts the parameters of the first neural network model according to the overlap and the loss function, such as the aforementioned GIOU_Loss; wherein the overlap between areas includes the size of the overlapping area, the proportion of the overlapping area in the union area of ​​the area indicated by the area label value and the area indicated by the area prediction value (intersection-over-union ratio), etc.; and the parameters of the first neural network model that are adjusted include the step size, weight, etc.

[0177] It can be understood that, compared with adjusting the parameters of the first neural network model only based on the coordinates of the center point of the operation object or the coordinate label value and predicted value of the center of mass point, the embodiment of the present disclosure adjusts the parameters of the first neural network model according to the coordinate label value and predicted value of the area where the operation object is located that contains more operation object shape information, which can improve the accuracy of the adjustment of the parameters of the first neural network model, and further improve the accuracy of determining the coordinates of the operation object in the interface image based on the target detection model after the adjustment parameters converge.

[0178] Figure 5 This is an example diagram of the Yolov5 architecture of a target detection model provided in an embodiment of the present disclosure. Figure 5 In the figure, M1 identifies the feature extraction layer, M2 identifies the feature optimization layer, M3 identifies the output end, and M4 identifies the input end; the concat module is used to realize feature fusion; the upsampling module is used to sample the data (pixel value information) of the interface image from top to bottom. Figure 6 An example diagram of a C3 module provided for an embodiment of the present disclosure, wherein the bottleneck module is a convolutional layer, including performing two feature extractions on an input interface image through a Conv module, and then adding the output to the original interface image. Figure 7An example diagram of a Conv module provided for an embodiment of the present disclosure, wherein a BatchNorm module is used to accelerate the convergence speed of model training. Figure 8 An example diagram of a Focus module provided in an embodiment of the present disclosure, wherein the downsampling module is used to perform data sampling of an interface image from bottom to top. Fig. 9 An example diagram of an SPP module provided in an embodiment of the present disclosure, wherein the MaxPooling module is used to take the maximum value in the pooling core as the output value. Each module can be further understood in conjunction with the above description, which will not be repeated here.

[0179] In some embodiments, determining the probability of each operation object responding to the operation based on the object identification and coordinates of each operation object includes:

[0180] The object identification and coordinates of each operation object in the interface image are processed using a preset probability prediction model to obtain the probability of each operation object responding to the operation; wherein the operation prediction model is obtained based on the training of the second neural network model.

[0181] In the embodiment of the present disclosure, the testing device uses a preset probability prediction model to process the object identification and coordinates of each operation object in the interface image to obtain the probability of each operation object responding to the operation; since the probability prediction model is obtained after multiple training convergences of the second neural network model, the accuracy of determining the probability of each operation object responding to the operation based on the probability prediction model is relatively high.

[0182] In some embodiments, the method further comprises:

[0183] Acquire a set of sample operation objects; wherein the set of sample operation objects includes the coordinates of the sample operation objects in each sample interface image and the identifier of the sample operation object; each sample operation object is associated with an operation probability label value;

[0184] Using a second neural network model, the coordinates of each sample operation object in the sample operation object set and the identifier of the sample operation object are processed to determine an operation probability prediction value of the sample operation object;

[0185] According to the operation probability prediction value of each sample operation object and the operation probability label value of the sample operation object, the parameters of the second neural network model are adjusted to obtain the probability prediction model.

[0186] In an embodiment of the present disclosure, the test device obtains a sample operation object set, wherein the sample operation object set includes the coordinates of the sample operation object in each sample interface image and the identifier of the sample operation object; each sample operation object is associated with an operation probability label value. It should be noted that the sample interface image in the sample operation object set can be a sample interface image in the sample interface image set, or can be other sample interface images, and the present disclosure does not limit this.

[0187] The disclosed embodiment can directly use the coordinates and identifiers of each operation object in the sample operation object set as input data of the probability prediction model, or can vectorize the coordinates and identifiers of each operation object in the sample operation object set and use the vectorized coordinates and identifiers as input data of the probability prediction model.

[0188] Among them, the operation probability label value can be regarded as a reward or punishment value, which is used to mark the probability of the user interacting with the operation object. When setting the reward or punishment value, the reward or punishment value can be set based on the coordinates, identification and user operation preference of the operation object in the interface image; for example, in entering the instant chat interface, there is a first operation operation, a second operation object and a third operation object. Based on the position and identification (corresponding function) of the above three operation objects, the user's operation preference degree is determined, and the operation probability label value of the operation object that the user prefers to interact with is set higher, such as the probability of the user interacting with the first operation object is higher than the second operation object and the third operation object, so that the operation probability label value of the first operation object can be set to be greater than the operation probability label value of the second operation object, and the operation probability label value of the second operation object is equal to the operation probability label value of the third operation object. In the above manner, the operation probability label value of each operation object can be made more in line with the user's operation habits.

[0189] The embodiment of the present disclosure can set the number of iterative training times, and iteratively train the second neural network model based on the above method. The model after training convergence is the probability prediction model of the embodiment of the present disclosure, wherein the probability prediction model iterative training log example is as shown in FIG. Fig.10 As shown in the figure, Epoch represents the number of iterative training, loss represents the loss value in the gradient calculation, and mean average precision mAP (Mean Average Precision) is the average of the average precision AP (Average Precision) of each category in the statistical training process. The higher the mAP value, the higher the accuracy of the model prediction.

[0190] The probability prediction model of the disclosed embodiment (i.e., the second neural network model after training convergence) can be a model of a network architecture such as Yolov4, Yolov5, RCNN, Mask RCNN, CNN, etc. In the process of training the second neural network model to obtain the probability prediction model, the coordinates of each sample operation object in the sample operation object set and the identification of the sample operation object are processed by the second neural network model to determine the operation probability prediction value of the sample operation object; and according to the operation probability prediction value of each sample operation object and the operation probability label value of the sample operation object, the parameters of the second neural network model are adjusted to obtain the probability prediction model.

[0191] In some embodiments, the application testing method is applied to a first device, and an application to be tested is installed in a second device;

[0192] The method further comprises:

[0193] sending a control instruction indicating a target object to be operated to the second device;

[0194] The obtaining of the test result of the operation simulation on the target operation object includes:

[0195] A test result of the second device performing an operation simulation on the target operation object based on the control instruction is obtained.

[0196] In the embodiment of the present disclosure, the aforementioned application testing method is applied to a first device, such as the aforementioned server or other electronic device; the second device of the embodiment of the present disclosure is such as the aforementioned terminal device, and the application to be tested is installed in the second device.

[0197] The disclosed embodiment obtains an interface image of an associated application to be tested through a first device, determines the probability of each operation object in the interface image responding to an operation, determines a target operation object to be operated based on the probability of each operation object responding to the operation, and sends a control instruction indicating the target object to be operated to a second device.

[0198] After the second device receives the control instruction sent by the first device indicating the target object to be operated, it simulates the operation of the target operation object based on the control instruction and obtains the test result. The second device can also send the test result to the first device.

[0199] Exemplarily, the second device is a mobile phone, the first device is a computer, and the mobile phone is connected to the computer; the interface image of the application to be tested in the mobile phone is obtained through the computer, the probability of each operation object in the interface image responding to the operation is determined, and based on the probability of each operation object responding to the operation, the target operation object to be operated is determined, and a control instruction indicating the target object to be operated is sent to the mobile phone; after the mobile phone receives the control instruction indicating the target object to be operated from the computer, the mobile phone simulates the operation of the target operation object in the interface of the application based on the control instruction, obtains the test result, and feeds back the test result to the computer, so that the technician can debug the application in the mobile phone based on the test result displayed on the computer, thereby reducing the probability of abnormalities when the mobile phone runs the application.

[0200] It can be understood that compared with all the steps in the application testing method being executed in one device, the embodiment of the present disclosure decouples the determination of the target operation object and the execution of the simulation operation on the target operation object, which can lighten the second device (such as a mobile phone) and reduce the power consumption of the second device.

[0201] In some embodiments, obtaining a test result of the second device performing an operation simulation on the target operation object based on the control instruction includes:

[0202] In response to the second device detecting an abnormality during the operation simulation process on the target operation object, a test result including log data and operation video during the operation simulation process on the target operation object is obtained.

[0203] In the disclosed embodiment, during the process of the second device simulating the operation of the target operation object, if an abnormality is detected, such as no response to the operation, application crash, etc., the test results including the log data and the operation video during the operation simulation process of the target operation object are obtained. In this solution, a series of operation scenarios are completed through the simulation test of the second electronic device without human supervision, and the technician can reproduce the abnormal scene of the test one-to-one based on the test results including the log data and the operation video during the operation simulation process of the target operation object, which greatly reduces the difficulty of locating the abnormality and improves the efficiency and accuracy of abnormality locating.

[0204] In addition, after obtaining the test results, you can also automatically generate tasks including the test results based on the connected task management system, such as the Jira system, pingcode system, etc., so that technicians can receive tasks in a timely manner, which helps technicians solve test anomalies that occur in the test application based on the task and improve the stability of the application to be tested.

[0205] Fig.11 An example diagram of an application test provided by an embodiment of the present disclosure, comprising Fig.11It can be seen that the second device (mobile phone, corresponding to the APK side) obtains the control instruction indicating the target object to be operated sent by the first electronic device (platform side), and starts the operation simulation of the target operation object; the second device calls the script corresponding to the behavior simulation of the target operation object, and performs the operation simulation on the target operation object in the test application; in response to an exception in the application during the operation simulation, a message representing the exception is sent to the third-party exception analysis platform connected to the second device through the message integration component, so that the APK side notifies the platform side of the exception; the platform side obtains the log data of the application during the operation simulation of the target operation object and the test results of the operation video; after obtaining the test results on the platform side, the test results are sent to the Jira platform connected to the second device. The Jira platform automatically generates tasks based on the test results, so that technical personnel can receive the tasks in time and repair the exceptions in the application.

[0206] Fig.12 An example of a test method flow provided for an embodiment of the present disclosure Figure 2 ,Depend on Fig.12 It can be seen that the following steps are included:

[0207] S1201, obtaining an interface image;

[0208] In the embodiment of the present disclosure, the interface is obtained Figure 1 ,interface Figure 2 , interface image 3, ..., interface image n, may be multiple interface images of different applications, different interface images of the same application, or multiple image frames of a video stream within a preset time length, etc. Each interface image includes at least one operation object.

[0209] S1202, target detection;

[0210] The testing device of the embodiment of the present disclosure obtains the interface image associated with the application to be tested, and determines the coordinates of each operation object in each interface image through the target detection module.

[0211] S12003, coordinates and identification;

[0212] After obtaining the coordinates of each operation object, for each operation object, based on the coordinates of the operation object, obtain an identification of the operation object;

[0213] S1204, probability prediction;

[0214] Through the probability prediction model, based on the coordinates and identification of each operation object, the probability of the operation object's response operation is predicted, and based on the probability of each operation object's response operation, the target operation object to be operated is determined, thereby driving the operation simulation test of the target operation object to obtain the test result.

[0215] Fig.13is a diagram of an application test device shown in an embodiment of the present disclosure, comprising Fig.13 It is known that, including:

[0216] The first acquisition module 1301 is used to acquire an interface image associated with the application to be tested; wherein the interface image includes at least one operation object;

[0217] A first determination module 1302 is used to determine the probability of each operation object in the interface image responding to the operation;

[0218] The second determination module 1303 is used to determine the target operation object to be operated based on the probability of each operation object responding to the operation;

[0219] The second acquisition module 1304 is used to obtain the test result of the operation simulation performed on the target operation object.

[0220] In some embodiments, the at least one operation object in the interface image includes: at least one application icon of an application to be tested;

[0221] The second determining module 1303 is configured to determine, based on the probability of the application icons of each application responding to the operation, an application icon that meets a preset first probability value as the target operation object to be operated.

[0222] In some embodiments, the at least one operation object in the interface image includes: a control icon of at least one operation control included in the interface image of the application to be tested;

[0223] The second determination module 1303 is used to determine the control icon that meets the preset second probability value as the target operation object to be operated based on the probability of the control icon of each operation control responding to the operation.

[0224] In some embodiments, the second determination module 1303 is used to determine, when the interface image includes multiple operation objects, multiple target operation objects to be operated that meet a preset probability range based on the probability of each operation object responding to the operation, and determine the operation order of the multiple target operation objects to be operated according to the probability ranking results of the multiple target operation objects to be operated responding to the operation.

[0225] In some embodiments, the first determination module 1302 is used to perform target detection on the interface image to obtain the coordinates of each operation object in the interface image; for each operation object, determine the object identification of the operation object based on the coordinates of the operation object in the interface image; based on the object identification and coordinates of each operation object, determine the probability of each operation object responding to the operation.

[0226] In some embodiments, the first determination module 1302 is used to perform target detection on the interface image using a preset target detection model to obtain the coordinates of each operation object in the interface image; wherein the target detection model is obtained based on training of a first neural network model.

[0227] In some embodiments, the apparatus further comprises:

[0228] The third acquisition module 1305 is used to acquire a sample interface image set; wherein the sample interface image in the sample interface image set includes at least one sample operation object and a coordinate label value of the sample operation object;

[0229] A feature extraction module 1306 is used to extract features from each sample interface image in the sample interface image set using the feature extraction layer of the first neural network model to obtain a first feature of the sample interface image;

[0230] A feature optimization module 1307 is used to optimize the first feature of the sample interface image using the feature optimization layer of the first neural network model to obtain an optimized second feature;

[0231] A first prediction module 1308, configured to perform target detection based on the second feature of the sample interface image, and determine a coordinate prediction value of each sample operation object in the sample interface image;

[0232] The first adjustment module 1309 is used to adjust the parameters of the first neural network model according to the coordinate label values ​​of each sample operation object included in each sample interface image, the coordinate prediction value of each sample operation object and the loss function to obtain the target detection model.

[0233] In some embodiments, the coordinate label value of the sample operation object includes a region label value of a region where the sample operation object is located; the coordinate prediction value of the sample operation object includes a region prediction value of a region where the sample operation object is located;

[0234] The first adjustment module 1309 is used to determine the overlap between the area indicated by the area label value and the area indicated by the area prediction value according to the area label value of the area where the sample operation object is located and the area prediction value; and adjust the parameters of the first neural network model according to the overlap and the loss function.

[0235] In some embodiments, the first determination module 1302 is used to process the object identification and coordinates of each operation object in the interface image using a preset probability prediction model to obtain the probability of each operation object responding to the operation; wherein the operation prediction model is obtained based on training of the second neural network model.

[0236] In some embodiments, the apparatus further comprises:

[0237] The fourth acquisition module 1310 is used to acquire a sample operation object set; wherein the sample operation object set includes the coordinates of the sample operation object in each sample interface image and the identifier of the sample operation object; each sample operation object is associated with an operation probability label value;

[0238] The second prediction module 1311 is used to process the coordinates of each sample operation object in the sample operation object set and the identifier of the sample operation object by using a second neural network model to determine an operation probability prediction value of the sample operation object;

[0239] The second adjustment module 1312 is used to adjust the parameters of the second neural network model according to the operation probability prediction value of each sample operation object and the operation probability label value of the sample operation object to obtain the probability prediction model.

[0240] In some embodiments, the application testing method is applied to a first device, and an application to be tested is installed in a second device;

[0241] The device also includes:

[0242] A sending module 1313, configured to send a control instruction indicating a target object to be operated to the second device;

[0243] The second acquisition module 1304 is used to obtain a test result of the second device performing an operation simulation on the target operation object based on the control instruction.

[0244] In some embodiments, the second acquisition module is used to acquire test results including log data and operation video during the operation simulation process on the target operation object in response to the second device detecting an abnormality during the operation simulation process on the target operation object.

[0245] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0246] Fig.14 8 is a block diagram of an electronic device 800 shown in an embodiment of the present disclosure. For example, the electronic device 800 may be a mobile phone, a mobile computer, etc.

[0247] Reference Figure 4, the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0248] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0249] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0250] The power component 806 provides power to the various components of the electronic device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0251] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0252] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0253] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0254] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0255] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as Wi-Fi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0256] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0257] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of an electronic device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0258] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute the aforementioned application testing method.

[0259] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims above.

[0260] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An application testing method, characterized in that: The method comprises: Acquire an interface image associated with the application to be tested; wherein the interface image includes at least one operation object; Determining the probability of each operation object in the interface image responding to the operation; Determine the target operation object to be operated based on the probability of each operation object responding to the operation; A test result of performing an operation simulation on the target operation object is obtained.

2. The method according to claim 1, characterized in that The at least one operation object in the interface image includes: at least one application icon of an application to be tested; The step of determining the target operation object to be operated based on the probability of each operation object responding to the operation includes: Based on the probability of the application icons of each application responding to the operation, the application icons that meet the preset first probability value are determined as the target operation objects to be operated.

3. The method according to claim 1, characterized in that The at least one operation object in the interface image includes: a control icon of at least one operation control included in the interface image of the application to be tested; The step of determining the target operation object to be operated based on the probability of each operation object responding to the operation includes: Based on the probability of the control icon of each operation control responding to the operation, the control icon that meets the preset second probability value is determined as the target operation object to be operated.

4. The method according to claim 2 or 3, characterized in that: The step of determining the target operation object to be operated based on the probability of each operation object responding to the operation includes: In the case where the interface image includes multiple operation objects, based on the probability of each operation object responding to the operation, multiple target operation objects to be operated that meet a preset probability range are determined, and according to the probability ranking results of the multiple target operation objects to be operated responding to the operation, the operation order of the multiple target operation objects to be operated is determined.

5. The method according to claim 1, characterized in that The determining the probability of each operation object in the interface image responding to the operation includes: Performing target detection on the interface image to obtain coordinates of each operation object in the interface image; For each operation object, determining an object identifier of the operation object based on the coordinates of the operation object in the interface image; Based on the object identification and coordinates of each operation object, the probability of each operation object responding to the operation is determined.

6. The method according to claim 5, characterized in that The performing target detection on the interface image to obtain the coordinates of each operation object in the interface image includes: The interface image is subjected to target detection using a preset target detection model to obtain coordinates of each operation object in the interface image; wherein the target detection model is obtained based on training of a first neural network model.

7. The method according to claim 6, characterized in that The method further comprises: Acquire a sample interface image set; wherein the sample interface images in the sample interface image set include at least one sample operation object and a coordinate label value of the sample operation object; Using the feature extraction layer of the first neural network model to extract features from each sample interface image in the sample interface image set, to obtain a first feature of the sample interface image; Using the feature optimization layer of the first neural network model to perform feature optimization on the first feature of the sample interface image to obtain an optimized second feature; Performing target detection based on the second feature of the sample interface image to determine the coordinate prediction value of each sample operation object in the sample interface image; According to the coordinate label values ​​of each sample operation object included in each sample interface image, the coordinate prediction value of each sample operation object and the loss function, the parameters of the first neural network model are adjusted to obtain the target detection model.

8. The method according to claim 7, characterized in that The coordinate label value of the sample operation object includes the region label value of the region where the sample operation object is located; the coordinate prediction value of the sample operation object includes the region prediction value of the region where the sample operation object is located; The adjusting the parameters of the first neural network model according to the coordinate prediction value of the sample operation object, the coordinate label value of the sample operation object and the loss function includes: Determining, according to the region label value of the region where the sample operation object is located and the region prediction value, the degree of overlap between the region indicated by the region label value and the region indicated by the region prediction value; According to the overlap and the loss function, the parameters of the first neural network model are adjusted.

9. The method according to claim 5, characterized in that The determining the probability of each operation object responding to the operation based on the object identifier and coordinates of each operation object includes: The object identification and coordinates of each operation object in the interface image are processed using a preset probability prediction model to obtain the probability of each operation object responding to the operation; wherein the operation prediction model is obtained based on the training of the second neural network model.

10. The method according to claim 9, characterized in that The method further comprises: Acquire a set of sample operation objects; wherein the set of sample operation objects includes the coordinates of the sample operation objects in each sample interface image and the identifier of the sample operation object; each sample operation object is associated with an operation probability label value; Using a second neural network model, processing the coordinates of each sample operation object in the sample operation object set and the identifier of the sample operation object to determine an operation probability prediction value of the sample operation object; According to the operation probability prediction value of each sample operation object and the operation probability label value of the sample operation object, the parameters of the second neural network model are adjusted to obtain the probability prediction model.

11. The method according to claim 1, characterized in that The application testing method is applied in a first device, and an application to be tested is installed in a second device; The method further comprises: sending a control instruction indicating a target object to be operated to the second device; The obtaining of the test result of the operation simulation on the target operation object includes: A test result of the second device performing an operation simulation on the target operation object based on the control instruction is obtained.

12. The method according to claim 11, characterized in that The obtaining a test result of the second device performing an operation simulation on the target operation object based on the control instruction includes: In response to the second device detecting an abnormality during the operation simulation process on the target operation object, a test result including log data and operation video during the operation simulation process on the target operation object is obtained.

13. An application testing device, characterized in that: The device comprises: A first acquisition module is used to acquire an interface image associated with the application to be tested; wherein the interface image includes at least one operation object; A first determination module, used to determine the probability of each operation object in the interface image responding to the operation; A second determination module is used to determine a target operation object to be operated based on the probability of each operation object responding to the operation; The second acquisition module is used to obtain the test result of the operation simulation performed on the target operation object.

14. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the method as claimed in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 12.

Images