Method and device for detecting interface abnormity

Abstract

One or more embodiments of the invention provide an interface abnormity detection method and device. The interface abnormity detection method comprises the steps that for an interaction behavior implemented by a user in a graphical user interface (GUI), a behavior log of the interaction behavior and an interaction video corresponding to the interaction behavior are generated, and the interaction video is used for recording the process that the GUI updates interface content in response to the interaction behavior; and performing data fusion processing on the behavior log and the interactive video to generate behavior sequence data of the user in the GUI, and detecting interface abnormity of the GUI based on the behavior sequence data.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of terminal technology, and more particularly, to a method and device for detecting interface abnormalities. Background Art

[0002] As the entry point for users to interact with front-end applications, the normal display of interface content by GUI (Graphics User Interface) is crucial for front-end applications to achieve the expected results. However, due to reasons such as interface design, code implementation, software compatibility, and even network, the GUI may be abnormal and unable to display normally, have functional defects, or even display fraudulent content, which is called interface abnormality.

[0003] In order to improve the display quality of the front-end application, it is necessary to identify whether its GUI display is abnormal. Given that the user's interactive behaviors in normal and abnormal interfaces may be different, user behavior analysis can be performed based on user flow data to determine the abnormality of the GUI interface.

[0004] In related technologies, tracking is often performed in the code of the front-end application to obtain the user's behavior log, and by analyzing the behavior log, it is determined whether the GUI has interface abnormalities and the degree of abnormality, etc. However, this method can only obtain the back-end system events generated by the GUI in response to user interaction. Not only is the evaluation dimension single, which makes it impossible to accurately and comprehensively evaluate the abnormality of the GUI, but it is also difficult to intuitively locate the cause of the abnormality when the interface is determined to be abnormal, which needs to be improved. Summary of the invention

[0005] In view of this, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a method for detecting an interface abnormality is provided, the method comprising:

[0007] For the interactive behavior implemented by the user in the graphical user interface GUI, a behavior log of the interactive behavior and an interactive video corresponding to the interactive behavior are generated, wherein the interactive video is used to record the process of the GUI updating the interface content in response to the interactive behavior;

[0008] The behavior log and the interaction video are subjected to data fusion processing to generate behavior sequence data of the user in the GUI, and interface abnormalities of the GUI are detected based on the behavior sequence data.

[0009] According to a second aspect of one or more embodiments of this specification, a device for detecting an interface abnormality is provided, the device comprising:

[0010] A log and video generating unit, for generating a behavior log of an interactive behavior and an interactive video corresponding to the interactive behavior for an interactive behavior performed by a user in a graphical user interface GUI, wherein the interactive video is used to record a process in which the GUI updates interface content in response to the interactive behavior;

[0011] An anomaly detection unit is used to perform data fusion processing on the behavior log and the interactive video to generate behavior sequence data of the user in the GUI, and detect interface anomalies of the GUI based on the behavior sequence data.

[0012] According to a third aspect of one or more embodiments of this specification, an electronic device is proposed, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the method described in the first aspect by running the executable instructions.

[0013] According to a fourth aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0014] According to a fifth aspect of one or more embodiments of the present specification, a computer program product is proposed, comprising a computer program / instruction, which implements the steps of the method described in the first aspect when executed by a processor.

[0015] It can be seen from the above embodiments that after obtaining the behavior log of the interactive behavior implemented by the user in the GUI and the corresponding interactive video, the embodiment of this specification performs data fusion processing on the behavior log and the interactive video to obtain the user's behavior sequence data, and then detects the interface abnormality of the GUI based on the data (that is, detects whether the interface is abnormal and the degree of abnormality). It can be seen that in the process of detecting the interface abnormality of the GUI, in addition to obtaining the behavior log of the interactive behavior (used to reflect the back-end system events), the interactive video corresponding to the behavior is also obtained (used to reflect the real display effect of the front-end of the GUI), and then the log and video are fused into behavior sequence data and the interface abnormality of the GUI is detected accordingly. It can be understood that the scheme expands the evaluation dimension of the GUI by introducing the interactive video, realizes the effective synchronization of user operation and system response, and can provide a comprehensive view of user interaction, so that the interface abnormality that may exist in the GUI can be relatively comprehensive and comprehensive from the two dimensions of user operation and system response; and in the case of interface abnormality, the cause of the abnormality can be accurately and intuitively located based on the above-mentioned interactive video, and the abnormality detection effect is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1It is a schematic diagram of the topological structure of an interface anomaly detection system provided by an exemplary embodiment.

[0017] Figure 2 It is a schematic diagram of the topological structure of another interface anomaly detection system provided by an exemplary embodiment.

[0018] Figure 3 It is a flowchart of a method for detecting interface abnormality provided by an exemplary embodiment.

[0019] Figure 4 It is a schematic diagram of an interactive video provided by an exemplary embodiment.

[0020] Figure 5 An exemplary embodiment provides an interface content recognition result of an interactive video.

[0021] Figure 6 It is a flowchart of another method for detecting interface abnormality provided by an exemplary embodiment.

[0022] Figure 7 It is a structural schematic diagram of a device provided by an exemplary embodiment.

[0023] Figure 8 It is a block diagram of a device for detecting an interface abnormality provided by an exemplary embodiment. DETAILED DESCRIPTION

[0024] Figure 1 FIG. 1 is a schematic diagram of a structure of an interface anomaly detection system provided by an exemplary embodiment, and the system is used to implement the interface anomaly detection solution described in this specification. Figure 1 As shown, the system may include a network 10 and several electronic devices, such as a PC (Personal Computer) 12, a mobile phone 13, etc. In addition, the system may also include a server 11. The server 21, the server 22 and the electronic devices communicate with each other through the network 20.

[0025] The server 11 may be a physical server including an independent host, or a virtual server hosted by a host cluster. During operation, the server may run a server-side program of an application to implement the relevant functions of the application. For example, when the server 11 runs a program related to the interface anomaly detection service, it may be implemented as a server (hereinafter referred to as the server 11).

[0026] Similarly, any of the above electronic devices can run a program on the client side of a certain application to implement the relevant functions of the application. For example, when PC12 runs the relevant program of the interface anomaly detection service, it can be implemented as a client (or user front end), which can be used to display the GUI of the interface anomaly to be detected, so that the user can implement the corresponding interactive behavior in the GUI. Among them, the GUI described in this specification refers to a computer operation user interface displayed in a graphical manner, which is a dialogue interface between a computer and its user (i.e., the user), and is an important part of the computer system. It should be noted that PC and mobile phones are only some types of electronic devices that users can use. In fact, users can obviously also use electronic devices such as the following types: mobile phones, tablet devices, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smart watches, etc.), workstations (Work Station), etc. The electronic device only needs to be able to display the GUI, and one or more embodiments of this specification do not limit its specific form. The application of any of the above clients can be started and run on the electronic device. The program on the client side can be a native application (Application, referred to as APP) installed on the electronic device, or the program on the client side can be a small program, a quick application or other similar forms. Of course, when using web technologies such as HTML5 or similar, relevant functions can be implemented through pages displayed by browsers. The browser here can be an independent browser application or a browser module embedded in certain applications.

[0027] It can be seen that from the hardware perspective, the interface anomaly detection system may include only any electronic device (such as only PC 12 or only mobile phone 13), or it may include a server 11 and any electronic device; correspondingly, from the software perspective, the interface anomaly detection system may include only any client (such as only client 12 running in PC 12 or only client 13 running in mobile phone 13), or it may include the aforementioned server 11 and any of the aforementioned clients.

[0028] In addition, for the network 10 for interaction between the aforementioned devices, a wired or wireless network can be selected to achieve communication based on the communication mode supported by the corresponding device, and this specification does not limit this. For example, PC 12 can support both wired and wireless communication, so a wired or wireless network can be used to achieve communication as needed; mobile phone 13 can only support wireless communication, so a wireless network can be used to achieve communication.

[0029] The foregoing Figure 1 The interface anomaly detection system is described from the hardware level. Figure 2The system is described at a logical level. Figure 2 FIG. 1 is a topological diagram of another interface anomaly detection system provided by an exemplary embodiment. The system can detect anomalies on the interface based on the interactive behavior performed by the user in the GUI. Figure 2 As shown, the interface anomaly detection system may include multiple functional modules at the logical level, such as a video capture module 21 (used to generate an interactive video corresponding to the interactive behavior), a log generation module 22 (used to generate a behavior log of the interactive behavior), a data fusion module 23 (used to perform data fusion processing on the behavior log and the interactive video to generate behavior sequence data), and anomaly detection module 24 (used to detect GUI interface anomalies based on the behavior sequence data). In addition, the interface anomaly detection system may also include an anomaly reporting module 25 (used to initiate an anomaly notification message for the GUI to a preset associated party).

[0030] Among them, the above-mentioned various functional modules can all belong to the client, such as being deployed in the aforementioned PC12 (or being deployed in the mobile phone 13). In this case, the interface anomaly detection system does not include the aforementioned server 11 (or the aforementioned server 11). Alternatively, the above-mentioned video capture module 21 and log generation module 22 can all belong to the client, such as being deployed in the aforementioned PC12; while the data fusion module 23, anomaly detection module 24 and anomaly reporting module 25 belong to the server, such as being deployed in the aforementioned server 11. In this case, PC12 needs to submit the interactive video generated by the video capture module 21 and the behavior log generated by the log generation module 22 to the server 11. Alternatively, the above-mentioned video capture module 21, log generation module 22 and data fusion module 23 can all belong to the client, such as being deployed in the aforementioned PC12; while the anomaly detection module 24 and anomaly reporting module 25 belong to the server, such as being deployed in the aforementioned server 11. In this case, PC12 needs to submit the behavior sequence data generated by the data fusion module 25 to the server 11. No further description will be given.

[0031] Figure 3 This is a flowchart of a method for detecting an interface anomaly provided by an exemplary embodiment. As mentioned above, the method is applied to an interface anomaly detection system. The system may only include a client, in which case the method is applied to the client; or the system may also include a client and a server, in which case the method is applied to the client and the server, that is, the two are coordinated to achieve the same. Figure 3 As shown, the method includes steps 302-304.

[0032] Step 302, for the interactive behavior implemented by the user in the graphical user interface GUI, generate a behavior log of the interactive behavior and an interactive video corresponding to the interactive behavior, wherein the interactive video is used to record the process of the GUI updating the interface content in response to the interactive behavior.

[0033] In the process of the user using the client, the client can display a GUI for the user. The GUI can display various types of interface content, and the interface content can generally include two categories: components and data. Exemplarily, the components (or controls) can include buttons, sliders, single / multiple selection boxes, text input boxes, drop-down selection boxes and / or list boxes, etc.; the data can include text, pictures, audio and / or video, etc. For the above interface content displayed in the GUI, the user can implement interactive behaviors in the interface (such as clicking a link, clicking an "OK" button, clicking to view list details, clicking to jump to a new page, sliding up and down to view text, clicking to play a video, etc.) to trigger the GUI to implement the corresponding preset function. It can be understood that the interactive behavior implemented by the user for any GUI may include at least one operation. In response to the above interactive behavior implemented by the user, the GUI (which can be understood as the client) will update the interface content displayed accordingly, for example, the user can jump to a new page after clicking a link, the user can trigger the display of the content associated with the button after clicking the "OK" button, and the user can display the new page after jumping after clicking the jump button, etc., which will not be repeated.

[0034] The above-mentioned interactive behavior can reflect the user's operating rules, thereby reflecting whether there are interface abnormalities in the GUI. The interactive behavior can be detected by means of embedding points, etc. For details, please refer to the records in the relevant technology, which will not be repeated here. This solution uses the user's behavior sequence data in the GUI to detect interface abnormalities of the interface. Before generating the data, it is necessary to obtain the behavior log of the interactive behavior first. In addition, it is necessary to obtain the interactive video in order to intuitively record the interactive behavior.

[0035] Regarding the timing of generating the behavior log and the interaction video. In one embodiment, the behavior log and the interaction video may be generated when the GUI is displayed. In another embodiment, the behavior log and the interaction video may be generated when the interaction behavior is first detected during the display of the GUI.

[0036] Normally, users may not immediately implement interactive behaviors after any GUI is displayed, but decide what kind of interactive behaviors to implement after checking and determining the meaning of the interface content; and the interactive behaviors implemented for the triggerable controls in the interface content can relatively better reflect the user's operating rules in the GUI. In this regard, in another embodiment, the client can generate a behavior log of the interactive behavior and an interactive video corresponding to the interactive behavior in response to the triggerable control in the GUI being triggered by the interactive behavior; in other words, the client can start generating the behavior log and interactive video when it detects that any triggerable control in the GUI is triggered by an interactive behavior implemented by the user (such as a button being clicked, a slider being slid, a text input box being activated and text being entered, etc.). In this way, the behavior log and interactive video can be generated from the moment the triggerable control is triggered, thereby effectively reducing the amount of data in the behavior log and interactive video, which helps to improve the efficiency of subsequent generation of interface content and anomaly detection.

[0037] In one embodiment, the client (specifically, the aforementioned log generation module 22) can generate the behavior log using the interface events generated by the GUI. For example, given that the GUI will generate at least one interface event (or user interface event, i.e., UI Events) during its display time period, the interaction events that are associated with the interaction behavior can be first screened out from these interface events—any of the screened out interaction events is triggered by the corresponding operation of the interaction behavior. Among them, any screened out event can record information such as the action type, operation location, and operation time of the corresponding operation, and of course can also include other information, which will not be repeated. Furthermore, the screened out interaction events can be arranged in chronological order of occurrence to form a behavior log of the interaction behavior. In this way, the behavior log of the interaction behavior can be accurately generated to prepare data for subsequent data processing and anomaly detection.

[0038] In one embodiment, the client (specifically, the aforementioned video capture module 21) can obtain the interface data of the GUI, such as HTML (HyperText Markup Language) documents or DOM (Document Object Model) trees used for interface rendering, and detect the interactive behavior implemented by the user on the other hand, and then render the obtained interface data according to the detected interactive behavior to generate an interactive video corresponding to the behavior. This method can obtain the page data in advance and detect the interactive behavior during the GUI display process, which reduces the amount of calculation and resource usage in the process of generating the interactive video, and helps to avoid user operation jams.

[0039] In another embodiment, the client can also generate the interactive video by recording the screen during the display of the GUI. The specific screen recording implementation process can be found in the relevant technology and will not be repeated here. Because the screen recording can completely record all the user's operations and interface response processes in the GUI in the form of a video, this method can not only effectively avoid the omission of front-end information, but also ensure the recording of the real front-end display effect.

[0040] like Figure 4 FIG. 4 shows an exemplary interactive video generated by the aforementioned method. Figure 4 As shown, a play button 401 and a play progress bar 402 are displayed below the video (the play progress bar 402 shows that the total duration of the video is 2 seconds). In addition, a control 403 is displayed in the center of the interface, which can be triggered as a "reload" button.

[0041] In one embodiment, because the user may log into a personal account or enter personal sensitive information (such as ID information, personal photos and other identity information, account passwords, fund balances, etc.) when using the client, the interface content of the GUI may include sensitive information that is not suitable for public disclosure or is not suitable for being known by other relevant parties other than the user. In this regard, in the case where sensitive information is contained in the GUI, the sensitive information can be desensitized to generate an interactive video that does not contain the sensitive information, thereby ensuring that the user's sensitive information is not disclosed or leaked. Among them, the above-mentioned desensitization can be achieved by using methods such as coding (i.e., adding mosaics to blur the sensitive information) and / or replacing / overwriting non-sensitive information (i.e., replacing / overwriting the sensitive information with standardized non-sensitive information). Specifically, the desensitization process can be performed in the process of generating the interactive video, thereby directly generating an interactive video that does not contain sensitive information; alternatively, the desensitization process can also be performed on the original video that is initially generated, thereby processing the original video into an interactive video that does not contain sensitive information. Figure 5 A and Figure 5 FIG. 2 shows a video frame in the interactive video. The account name, last digit and bank of the receiving account in the interface are often sensitive information. Therefore, the above information in each video frame of the interactive video can be desensitized (e.g. Figure 5 Of course, as shown in the figure, the loan amount and the like may also be desensitized, which will not be described in detail.

[0042] In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0043] In one embodiment, the client may also mark the instruction information of the interactive behavior in the generated interactive video, and the information is at least used to indicate the operation position of the interactive behavior in the interface. Figure 4 As shown, the interactive video is marked with a location mark 404 of the interactive behavior, i.e., a dot in the figure used to indicate the location where the user clicked. In this way, if the GUI is subsequently detected to have an interface abnormality, the technician can quickly locate the abnormal location, the abnormal cause, etc. by viewing the interactive video and the indication information marked therein.

[0044] Step 304: perform data fusion processing on the behavior log and the interaction video to generate behavior sequence data of the user in the GUI, and detect interface abnormalities of the GUI based on the behavior sequence data.

[0045] After generating the behavior log and the interaction video as described above, the two can be further fused to generate the user's behavior sequence data in the GUI. It should be noted that the behavior sequence data described in the embodiments of this specification can be regarded as user movement line data or user behavior movement line, which refers to the behavior trajectory of the user in the process of interacting with the GUI. Since whether there are interface abnormalities in the GUI and the degree of verification of the interface abnormality will affect the user's behavior trajectory in the interface, that is, cause the behavior sequence data to be different, the interface abnormality of the GUI can be detected based on the data. Among them, "detecting interface abnormalities of the GUI" described in this specification may only include detecting whether there are abnormalities in the GUI; or, it may also include detecting whether there are abnormalities in the GUI and detecting the degree of abnormality (i.e., the severity of the abnormality) when there are abnormalities.

[0046] It can be seen from the above embodiments that after obtaining the behavior log of the interactive behavior implemented by the user in the GUI and the corresponding interactive video, the embodiment of this specification performs data fusion processing on the behavior log and the interactive video to obtain the user's behavior sequence data, and then detects the interface abnormality of the GUI based on the data (that is, detects whether the interface is abnormal and the degree of abnormality). It can be seen that in the process of detecting the interface abnormality of the GUI, in addition to obtaining the behavior log of the interactive behavior (used to reflect the back-end system events), the interactive video corresponding to the behavior is also obtained (used to reflect the real display effect of the front-end of the GUI), and then the log and video are fused into behavior sequence data and the interface abnormality of the GUI is detected accordingly. It can be understood that the scheme expands the evaluation dimension of the GUI by introducing the interactive video, realizes the effective synchronization of user operation and system response, and can provide a comprehensive view of user interaction, so that the interface abnormality that may exist in the GUI can be relatively comprehensive and comprehensive from the two dimensions of user operation and system response; and in the case of interface abnormality, the cause of the abnormality can be accurately and intuitively located based on the above-mentioned interactive video, and the abnormality detection effect is significantly improved.

[0047] In one embodiment, the interactive behavior may include at least one operation. In this regard, the data fusion module 23 may first align the behavior log and the interactive video to the same time axis during the data fusion process to ensure that the behavior log corresponding to each operation matches the corresponding video frame in the interactive video on the time axis. Further, the operation description information of each operation may be extracted from the behavior log, and the information is at least used to characterize the operation time and operation position of each operation (and may also include the behavior type, etc.); wherein the operation time of any operation may include the start time of the operation on the time axis and the duration of the operation, and the operation position of any operation is the coordinate of the contact point of the operation (such as the click position when the user clicks) in the interface, etc. Further, for each operation, the interface content of the GUI at the operation time and operation position of the operation may be identified from the interactive video, and the interface content and the operation description information of the operation constitute the behavior data of the operation; finally, according to the sequence of each operation, the behavior sequence data of the user in the GUI is constructed based on the behavior data of each operation, such as arranging the behavior data of each operation in sequence according to the sequence to obtain the behavior sequence data. In this way, the interactive video representing the GUI front-end display effect and the behavior log representing the GUI back-end response can be accurately fused, thereby ensuring that the generated behavior sequence data can fully reflect the display effect of the GUI under the interactive behavior from both the front-end and back-end aspects, and prepare data for subsequent accurate detection of interface anomalies.

[0048] The interface content of any target video frame may include a component frame and text, and the component frame of any component in the interface is used to identify the outer boundary of the component, thereby identifying the location of the component in the interface. Figure 5 The figure shows the process of identifying the interface content in any target video frame. Figure 5 a in the figure is any target video frame, and the interface content therein includes text 501 (such as "How long to borrow") and component 502 (such as "Agree to the agreement and use" button). Figure 5 The b in the Figure 5 The component boxes of each component identified in the target video frame shown in a in FIG. Figure 5 The c in the above is from Figure 5 The text recognized in the target video frame shown in a in FIG. 5 , such as the text 505 recognized from the component frame 504, is “12 months, payable at any time, how long?”, and other texts are not described in detail.

[0049] When the data fusion module 23 identifies the interface content of the GUI at the operation time and operation position of the operation from the interactive video, it can first extract the target video frame corresponding to the operation from the interactive video, and identify the interface content of the GUI at the operation time and operation position of the operation from the target video frame. The target video frame can be extracted from each video frame included in the interactive video, such as for each video frame included in the interactive video (each video frame has its own timestamp - each video frame is distributed at a different position in the aforementioned time axis), the video frame whose timestamp is closest to the time when the interactive behavior occurs can be determined as the target video frame corresponding to the interactive behavior; or the target video frame can also be extracted from the key frames included in the interactive video, such as for each key frame included in the interactive video (as a special video frame, each key frame also has its own timestamp - each key frame is also distributed at a different position in the aforementioned time axis), the key frame whose timestamp is closest to the time when the interactive behavior occurs can be determined as the target video frame corresponding to the interactive behavior.

[0050] For example, after opening the client, the user opens three GUIs in succession by performing interactive behaviors, and the behavior sequence data corresponding to each GUI can be obtained through the above method. For example, the behavior sequence data corresponding to the three GUIs are "Open-->Home-->Click Exit-->Click My-->Click Repay-->Stay on the page for 5.843 seconds", "Jump-->Post-loan Home-->Click Repay Anytime in 12 Months-->Stay on the page for 9.369 seconds", and "Jump-->Borrowing and Repayment Record List Page-->Click Repay-->Slide Multiple Times-->Stay on the page for 15.382 seconds".

[0051] In one embodiment, the anomaly detection module 24 can use a variety of detection methods when detecting the interface anomaly of the GUI based on the behavior sequence data. For example, the interface anomaly detection rule pre-generated for the GUI can be obtained, and then the matching between the behavior sequence data and the rule is determined, and the interface anomaly of the GUI is determined according to the matching. Among them, the interface anomaly detection rule can be manually generated by a technician after collecting common anomaly types, abnormal performances and other information of the GUI, or it can also be automatically collected by a rule generation module (not shown in the figure) The relevant information of the abnormal GUI is extracted, and the embodiment of this specification is not limited to this. In addition, the interface anomaly detection rule can include at least one, and in the case of including multiple rules, these rules can be attributed to at least one dimension such as interface type (such as APP interface, applet interface, web page interface, etc.), interface content, operation type (such as single / double click, up / down sliding, long press, etc.). It can be understood that the more detailed the dimensional division of the above rules is, the higher the detection accuracy is often, but the amount of calculation in the detection process may also be greater. Therefore, the rules can be reasonably generated or the dimensions of the rules can be divided according to the requirements of anomaly detection accuracy and duration requirements, and no further description is given.

[0052] For another example, a pre-generated interface anomaly detection model for a GUI may be obtained, and then the behavior sequence data may be input into the model, and the interface anomaly of the GUI may be determined based on the output of the model. In this way, a mature model trained based on machine learning technology and a large amount of sample data may be used to detect the interface anomaly of the GUI, facilitating efficient batch anomaly detection of a large number of GUIs.

[0053] Among them, when training the interface anomaly detection model, you can first obtain the sample behavior sequence data of the sample user in the sample GUI. The sample behavior sequence data is generated by data fusion processing of the sample behavior log of the sample interaction behavior performed by the sample user in the sample GUI and the sample interaction video corresponding to the sample interaction behavior; then use the sample behavior sequence data to train the basic model to obtain the interface anomaly detection model.

[0054] In addition, when the sample behavior sequence data is used to train the basic model, a variety of different training methods can be used. For example, the sample behavior sequence data can be directly used to perform unsupervised training on the basic model, such as using iForest (i.e., Isolation Forest) or cluster analysis (such as K-means, system clustering, and second-order clustering) and other machine learning algorithms to identify abnormal points in user interaction behaviors. For another example, the sample behavior sequence data can also be divided into positive sample data corresponding to sample GUIs with interface abnormalities and negative sample data corresponding to sample GUIs without interface abnormalities according to the labels of the sample behavior sequence data, and the positive sample data and the negative sample data can be used to perform supervised training on the basic model - the interface abnormality detection model trained in this way can be used to identify whether the GUI has abnormalities. For example, after the behavior sequence data is input into the model, if the abnormality score output by the model is 0, it means that the GUI does not have abnormalities, and if the abnormality score output is 1, it means that the GUI has abnormalities. For another example, the sample scores of the sample behavior sequence data can also be determined, wherein the abnormal score of any sample behavior sequence data is used to characterize the abnormality of the sample GUI corresponding to the data; and then the sample behavior sequence data and its sample score are used to conduct supervised training on the basic model - the interface abnormality detection model trained in this way can be used to identify the abnormality of the GUI. For example, after the behavior sequence data is input into the model, the higher the abnormality score output by the model, the more serious the abnormality of the GUI. If the abnormality score is zero, it indicates that there is no abnormality in the GUI. In addition, during the training process, features such as operation sequence, operation sequence duration, operation frequency, and operation interval duration can be extracted from the sample behavior sequence data, and these features can be used to construct the corresponding feature vector input model, which will not be repeated.

[0055] In one embodiment, when detecting the interface anomaly of the GUI based on the behavior sequence data, the anomaly of the GUI itself can be detected for a single GUI, or the interface anomaly of multiple consecutive GUIs can be detected uniformly. For example, the interface anomaly of the GUI can be detected based on the behavior sequence data of the user in any GUI, such as detecting whether the interface content of the GUI is displayed completely, whether the relative positions of the various interface contents are disordered, etc.; and / or, the interface anomaly of the multiple consecutive GUIs can be detected based on the behavior sequence data of the user in multiple consecutive GUIs, such as whether there is an abnormal jump between adjacent GUIs, etc. It can be understood that the specific detection method can be determined by the specific content of the aforementioned interface anomaly detection model or interface anomaly detection rules, and the embodiments of this specification do not limit this.

[0056] In one embodiment, when the detection result indicates that the GUI is abnormal or the degree of abnormality of the GUI is greater than a threshold value (such as the abnormality score output by the aforementioned interface abnormality detection model is higher than the preset score threshold), an abnormality notification message for the GUI can be initiated to the preset associated party. Among them, the preset associated party may include the GUI manager (usually the manager of the client). After the manager receives the abnormality notification message for a certain GUI uploaded by one or more clients, it can trigger the abnormality handling (such as interface update or software upgrade, etc.) process for the page after statistics / aggregation. Of course, the above-mentioned associated party can also be the user, so that the user is notified of the abnormality of the GUI in real time during the user's use of the client, so as to avoid the user from repeatedly operating because he is unaware of the abnormality of the GUI. In addition, the abnormality notification message may include abnormality cause description information, abnormality severity description information, abnormality type description information, etc., and may also include the interactive video so that the relevant personnel can intuitively trace back the interface abnormality by viewing the video.

[0057] Combine the following Figure 6 The complete process of this solution is introduced as an example. Figure 6 As shown, the client generates the behavior log of the interactive behavior and its corresponding interactive video respectively, and then aligns the time axis of the two. Next, for the interactive video, multiple target video frames are determined by extracting frames and the interface content is identified respectively, and the component boxes and texts contained in each target video frame are identified (of course, the positions of each component box and text in the interface must also be determined); for the behavior log, the operation information such as the action type, operation position and operation time are extracted for each operation in the behavior log. Next, according to the positions of the timestamps of the target video frame and the timestamps of each operation in the time axis, the interface content in the target video frame and the operation information of each operation are fused and processed to obtain the user's behavior sequence data in each GUI, such as "Open-->Home-->Click to Exit-->Click My-->Click to Repay-->Stay on the page for 5.843s", "Jump-->Post-loan Home-->Click to Repay at Any Time in 12 Months-->Stay on the page for 9.369s" and "Jump-->Borrowing and Repayment Record List Page-->Click to Repay-->Slide Multiple Times-->Stay on the page for 15.382s", etc., which will not be repeated. Finally, based on the above behavior sequence data, it is possible to detect whether the GUI has an abnormality, what kind of abnormality exists, the degree of abnormality, etc. In addition, when the GUI has an abnormality or the degree of abnormality is greater than a threshold, an abnormality notification message (i.e., abnormality reporting) for the GUI can be initiated to a preset associated party.

[0058] 7 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 7At the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, a memory 708, and a non-volatile memory 710, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 702 reading the corresponding computer program from the non-volatile memory 710 into the memory 708 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0059] Please refer to Figure 8 The interface abnormality detection device can be applied to Figure 7 The device shown in the figure is used to implement the technical solution of this specification. Among them, the interface abnormality detection device may include:

[0060] The log and video generating unit 801 is used to generate a behavior log of the interaction behavior and an interaction video corresponding to the interaction behavior for the interaction behavior performed by the user in the graphical user interface GUI, wherein the interaction video is used to record the process of the GUI updating the interface content in response to the interaction behavior;

[0061] The anomaly detection unit 802 is used to perform data fusion processing on the behavior log and the interaction video to generate behavior sequence data of the user in the GUI, and detect interface anomalies of the GUI based on the behavior sequence data.

[0062] Optionally, the log and video generating unit 801 is specifically used for:

[0063] In response to a triggerable control in the GUI being triggered by the interactive behavior, a behavior log of the interactive behavior and an interactive video corresponding to the interactive behavior are generated.

[0064] Optionally, the log and video generating unit 801 is specifically used for:

[0065] Interaction events associated with the interaction behavior are screened out from interface events generated by the GUI during its display time period, and the interaction events are arranged in chronological order of occurrence to form a behavior log of the interaction behavior.

[0066] Optionally, the log and video generating unit 801 is specifically used for:

[0067] The interactive video is generated by recording the screen during the display of the GUI.

[0068] Optionally, the log and video generating unit 801 is specifically used for:

[0069] In the case where the GUI contains sensitive information, the sensitive information is desensitized to generate an interactive video that does not contain the sensitive information.

[0070] Optionally, also include:

[0071] The indication marking unit 803 is used to mark indication information of the interactive behavior in the interactive video, and the indication information is at least used to indicate the operation position of the interactive behavior.

[0072] Optionally, the interactive behavior includes at least one operation, and the abnormality detection unit 802 is specifically configured to:

[0073] aligning the behavior log and the interaction video to the same timeline;

[0074] Extracting operation description information of each operation from the behavior log, wherein the operation description information is used to at least characterize the operation time and operation location of each operation;

[0075] For each operation, identifying the interface content of the GUI at the operation time and operation position of the operation from the interactive video, wherein the interface content and the operation description information of the operation constitute the behavior data of the operation;

[0076] According to the sequence of each operation, the behavior sequence data of the user in the GUI is constructed based on the behavior data of each operation.

[0077] Optionally, the anomaly detection unit 802 is specifically configured to:

[0078] A target video frame corresponding to the operation is extracted from the interactive video, and interface content of the GUI at the operation time and operation position of the operation is identified from the target video frame.

[0079] Optionally, the anomaly detection unit 802 is specifically configured to:

[0080] Inputting the behavior sequence data into an interface anomaly detection model, and determining the interface anomaly of the GUI according to the output result of the model; or,

[0081] According to the matching between the behavior sequence data and the interface anomaly detection rule, the interface anomaly of the GUI is determined.

[0082] Optionally, a model training unit 804 is further included, which is used to train the interface anomaly detection model, specifically for:

[0083] Acquire sample behavior sequence data of a sample user in a sample GUI, wherein the sample behavior sequence data is generated by data fusion processing of a sample behavior log of a sample interaction behavior performed by the sample user in the sample GUI and a sample interaction video corresponding to the sample interaction behavior;

[0084] The basic model is trained using the sample behavior sequence data to obtain the interface anomaly detection model.

[0085] Optionally, the model training unit 804 is specifically used for one of the following:

[0086] Using the sample behavior sequence data to perform unsupervised training on the basic model;

[0087] Dividing the sample behavior sequence data into positive sample data corresponding to sample GUIs with interface abnormalities and negative sample data corresponding to sample GUIs without interface abnormalities according to the labels of the sample behavior sequence data, and using the positive sample data and the negative sample data to perform supervised training on the basic model;

[0088] Determine the sample score of the sample behavior sequence data, wherein the abnormal score of any sample behavior sequence data is used to characterize the abnormality degree of the sample GUI corresponding to the data; and use the sample behavior sequence data and its sample score to perform supervised training on the basic model.

[0089] Optionally, the anomaly detection unit 802 is specifically configured to:

[0090] Detecting interface abnormality of any GUI based on the user's behavior sequence data in the GUI; and / or,

[0091] Based on the behavior sequence data of the user in the multiple consecutive GUIs, interface abnormalities of the multiple consecutive GUIs are detected.

[0092] Optionally, also include:

[0093] The abnormality reporting unit 805 is used to initiate an abnormality notification message for the GUI to a preset associated party when the detection result shows that the GUI is abnormal or the abnormality degree of the GUI is greater than a threshold.

[0094] Based on the same concept as the above method, this specification also provides an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the method described in any of the above embodiments by running the executable instructions.

[0095] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0096] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

Claims

1. A method for detecting an interface abnormality, the method comprising: For the interactive behavior implemented by the user in the graphical user interface GUI, a behavior log of the interactive behavior and an interactive video corresponding to the interactive behavior are generated, wherein the interactive video is used to record the process of the GUI updating the interface content in response to the interactive behavior; The behavior log and the interaction video are subjected to data fusion processing to generate behavior sequence data of the user in the GUI, and interface abnormalities of the GUI are detected based on the behavior sequence data.

2. According to the method of claim 1, generating a behavior log of the interactive behavior and an interactive video corresponding to the interactive behavior comprises: In response to a triggerable control in the GUI being triggered by the interactive behavior, a behavior log of the interactive behavior and an interactive video corresponding to the interactive behavior are generated.

3. According to the method of claim 1, the step of generating a behavior log of the interactive behavior comprises: Interaction events associated with the interaction behavior are screened out from interface events generated by the GUI during its display time period, and the interaction events are arranged in chronological order of occurrence to form a behavior log of the interaction behavior.

4. According to the method of claim 1, generating an interactive video corresponding to the interactive behavior comprises: The interactive video is generated by recording the screen during the display of the GUI.

5. According to the method of claim 1, generating an interactive video corresponding to the interactive behavior comprises: In the case where the GUI contains sensitive information, the sensitive information is desensitized to generate an interactive video that does not contain the sensitive information.

6. The method according to claim 1, further comprising: Indication information of the interactive behavior is marked in the interactive video, where the indication information is at least used to indicate an operation position of the interactive behavior.

7. The method according to claim 1, wherein the interactive behavior includes at least one operation, and the step of performing data fusion processing on the behavior log and the interactive video to generate the behavior sequence data of the user in the GUI comprises: Aligning the behavior log and the interaction video to the same timeline; Extracting operation description information of each operation from the behavior log, wherein the operation description information is used to characterize at least the operation time and operation location of each operation; For each operation, identifying the interface content of the GUI at the operation time and operation position of the operation from the interactive video, wherein the interface content and the operation description information of the operation constitute the behavior data of the operation; According to the sequence of each operation, the behavior sequence data of the user in the GUI is constructed based on the behavior data of each operation.

8. The method according to claim 7, wherein identifying the interface content of the GUI at the operation time and operation position of the operation from the interactive video comprises: A target video frame corresponding to the operation is extracted from the interactive video, and interface content of the GUI at the operation time and operation position of the operation is identified from the target video frame.

9. The method according to claim 1, wherein detecting an interface abnormality of the GUI based on the behavior sequence data comprises: Inputting the behavior sequence data into an interface anomaly detection model, and determining the interface anomaly of the GUI according to the output result of the model; or, According to the matching between the behavior sequence data and the interface anomaly detection rule, the interface anomaly of the GUI is determined.

10. The method according to claim 9, training the interface anomaly detection model comprises: Acquire sample behavior sequence data of a sample user in a sample GUI, wherein the sample behavior sequence data is generated by data fusion processing of a sample behavior log of a sample interaction behavior performed by the sample user in the sample GUI and a sample interaction video corresponding to the sample interaction behavior; The basic model is trained using the sample behavior sequence data to obtain the interface anomaly detection model.

11. The method according to claim 10, wherein the step of training a basic model using the sample behavior sequence data comprises one of the following: Using the sample behavior sequence data to perform unsupervised training on the basic model; Dividing the sample behavior sequence data into positive sample data corresponding to sample GUIs with interface abnormalities and negative sample data corresponding to sample GUIs without interface abnormalities according to the labels of the sample behavior sequence data, and using the positive sample data and the negative sample data to perform supervised training on the basic model; Determine the sample score of the sample behavior sequence data, wherein the abnormal score of any sample behavior sequence data is used to characterize the abnormality degree of the sample GUI corresponding to the data; and use the sample behavior sequence data and its sample score to perform supervised training on the basic model.

12. The method according to claim 1, wherein detecting an interface abnormality of the GUI based on the behavior sequence data comprises: Detecting interface abnormality of any GUI based on the behavior sequence data of the user in the GUI; and / or, Based on the behavior sequence data of the user in the multiple consecutive GUIs, interface abnormalities of the multiple consecutive GUIs are detected.

13. The method according to claim 1, further comprising: When the detection result indicates that the GUI is abnormal or the degree of abnormality of the GUI is greater than a threshold, an abnormality notification message for the GUI is initiated to a preset associated party.

14. A device for detecting interface abnormality, the device comprising: A log and video generating unit, for generating a behavior log of an interactive behavior and an interactive video corresponding to the interactive behavior for an interactive behavior performed by a user in a graphical user interface GUI, wherein the interactive video is used to record a process in which the GUI updates interface content in response to the interactive behavior; An anomaly detection unit is used to perform data fusion processing on the behavior log and the interactive video to generate behavior sequence data of the user in the GUI, and detect interface anomalies of the GUI based on the behavior sequence data.

15. An electronic device, characterized in that: include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 13 by executing the executable instructions.

16. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

17. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13.

Images