Application defect detection method, device and equipment and readable storage medium

Abstract

The invention discloses an application defect detection method, device and equipment and a readable storage medium, and the method comprises the steps: collecting an application operation video according to a preset collection rule, and the application operation video comprises at least one complete user operation process; key frame behavior detection is carried out on the application operation video, a video behavior text is generated through conversion, and operation item combinations and corresponding operation results forming the user operation processes in the application operation video are recorded in the video behavior text; the video behavior text is compared and detected according to a built-in behavior result mapping text, the abnormal condition of the operation result is determined, a defect list is generated, and the behavior result mapping text records each operation item combination and each correspondingly matched operation result. The method is higher in efficiency, accuracy and flexibility, reduces the maintenance cost, and can meet the requirements of rapidly changing iterative development and testing.

Description

Technical Field

[0001] The present application relates to the field of computer applications, and more specifically, to an application defect detection method, apparatus, device and readable storage medium. Background Art

[0002] Application software defect detection is crucial for modern software application development and maintenance, as it ensures the quality and stability of software. In the defect testing phase of existing software development and optimization and upgrade, testers usually rely on repeatedly using applications for testing purposes and performing a variety of test operations. During this process, testers must manually record each step of the operation and the corresponding expected results in detail. However, this testing method that relies heavily on manual operations has significant limitations.

[0003] First, it greatly consumes the time and energy of testers. The testing process is cumbersome and inefficient. Especially when faced with complex and changeable operating scenarios, testers may find it difficult to fully cover all test paths and boundary conditions, which undoubtedly increases the risk of test omissions. Secondly, the process of manually recording and comparing operating steps and expected results is very likely to introduce human errors, causing the accuracy of test results to be questioned. Furthermore, this traditional testing method lacks intelligent analysis capabilities and is difficult to automatically process and analyze large amounts of test data, making it difficult for the testing process to adapt to the rapid iteration of development rhythm and growing testing needs. In addition, manual testing also means an increase in testing costs, including labor costs and time costs, which is undoubtedly a heavy burden for modern software development processes that pursue efficient delivery and continuous integration.

[0004] Based on this, the present application provides an application defect detection solution, which not only improves the detection efficiency, but also significantly enhances the accuracy and flexibility of detection. Summary of the invention

[0005] In view of this, the present application provides an application defect detection method, device, equipment and readable storage medium, which have higher efficiency, accuracy and flexibility, reduce maintenance costs, and can cope with rapidly changing iterative development and testing needs.

[0006] An application defect detection method, comprising:

[0007] Collecting application operation videos according to preset collection rules, wherein the application operation videos include at least one complete user operation process;

[0008] Performing key frame behavior detection on the application operation video and converting it into a video behavior text, wherein the video behavior text records each combination of operation items and corresponding operation results constituting each user operation process in the application operation video;

[0009] The video behavior text is detected according to the built-in behavior result mapping text comparison, abnormal operation results are determined, and a defect ticket is generated. The behavior result mapping text records each operation item combination and the corresponding matching operation results.

[0010] Optionally, key frame behavior detection is performed on the application operation video, and the video behavior text is converted and generated, including:

[0011] Extracting each video key frame in the application operation video, and performing behavior detection on each video key frame to identify and generate operation behavior information;

[0012] Using OCR technology to perform text recognition on each video key frame to generate operation text information;

[0013] Performing time series analysis on the operation behavior information and the operation text information, and sequentially arranging the analysis results to obtain execution process data;

[0014] The execution process data is converted into text using NLP technology to generate video behavior text.

[0015] Optionally, the video behavior text is detected according to the built-in behavior result mapping text comparison, abnormal operation results are determined, and a defect ticket is generated, including:

[0016] Identify target operation item combinations in the built-in behavior result mapping text that are respectively consistent with the operation item combinations constituting the user operation processes in the application operation video, and determine target operation results that match the target operation item combinations;

[0017] Generate a first sentence embedding vector group based on the video behavior text;

[0018] Generate a second sentence embedding vector group based on the target operation item combinations and the corresponding matched target operation results;

[0019] Calculating a similarity score between the first sentence embedding vector group and the second sentence embedding vector group;

[0020] Based on the similarity score and a preset threshold, the abnormality of the operation result is determined and a defect ticket is generated.

[0021] Optionally, after collecting the application operation video according to the preset collection rules, the following is further included:

[0022] Performing abnormal behavior detection on the application operation video;

[0023] If abnormal behavior is detected, an abnormal behavior report is generated and the current application defect detection is stopped;

[0024] If no abnormal behavior is detected, the process of performing key frame behavior detection on the application operation video continues.

[0025] Optionally, abnormal behavior detection is performed on the application operation video, including:

[0026] Perform abnormal behavior detection of unexpected operations on the application operation video according to the built-in behavior result mapping text;

[0027] Perform abnormal behavior detection on the application operation video when there is no operation for a long time according to a preset operation stagnation time;

[0028] Abnormal behavior detection of invalid clicks is performed on the application operation video according to the validity of the response element.

[0029] Optionally, also include:

[0030] The defect nature of each defect recorded in the defect ticket is detected, and corresponding defect classification labels and processing priorities are set.

[0031] Optionally, also include:

[0032] A timestamp with the generation time is stamped on the defect sheet, and a corresponding defect handling cycle is set.

[0033] An application defect detection device, comprising:

[0034] A video acquisition unit, configured to acquire application operation videos according to preset acquisition rules, wherein the application operation videos include at least one complete user operation process;

[0035] A behavior detection unit, used to perform key frame behavior detection on the application operation video, and convert and generate a video behavior text, wherein the video behavior text records each operation item combination and each corresponding operation result constituting each user operation process in the application operation video;

[0036] The defect detection unit is used to detect the video behavior text according to the built-in behavior result mapping text comparison, determine the abnormal operation result, and generate a defect ticket. The behavior result mapping text records each operation item combination and the corresponding matching operation results.

[0037] An application defect detection device includes a memory and a processor;

[0038] The memory is used to store programs;

[0039] The processor is used to execute the program to implement each step of the application defect detection method as described in any one of the above items.

[0040] A readable storage medium stores a computer program thereon, wherein when the computer program is executed by a processor, each step of the application defect detection method as described in any one of the above items is implemented.

[0041] It can be seen from the above technical solution that an application defect detection method, device, equipment and readable storage medium provided by the embodiment of the present application first collects application operation videos according to preset collection rules, wherein the application operation video contains at least one complete user operation process. Then, key frame behavior detection is performed on the application operation video, and the video behavior text is converted to generate a video behavior text, wherein the video behavior text records the combination of each operation item and the corresponding operation results that constitute each user operation process in the application operation video. Finally, the video behavior text is detected according to the built-in behavior result mapping text comparison, the abnormal operation result is determined, and a defect ticket is generated, wherein the behavior result mapping text records the combination of each operation item and the corresponding matching operation results.

[0042] This application avoids the tedious process of manually recording operation steps and operation results by automatically collecting and analyzing user operation videos, realizes intelligent and automated testing, reduces manual intervention, and improves test efficiency and accuracy. At the same time, since this application can automatically analyze user operation items and operation results, and compare them with the target operation results recorded in the built-in behavior result mapping text, it can identify application defects. This defect identification method does not rely on fixed scripts and operation steps, and has greater flexibility and intelligent analysis capabilities. In addition, for the application software of this application, application developers only need to regularly retrieve defect orders from recent periods to understand the application defects and perform upgrades and maintenance. There is no need to perform a large number of test operations in order to detect defects, which reduces the cost of testing and maintenance, improves work efficiency, and can cope with rapidly changing iterative development and testing needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0044] Figure 1 A flowchart of an application defect detection method disclosed in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of an application defect detection device disclosed in an embodiment of the present application;

[0046] Figure 3This is a hardware structure block diagram of an application defect detection device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0048] The present application can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or equipment, etc.

[0049] An embodiment of the present application provides an application defect detection method, which can be applied to a platform or system involving application optimization development or defect detection, and can also be applied to various computer terminals or smart terminals. The execution subject can be a processor or server of the computer terminal or smart terminal.

[0050] Next, the present application scheme is introduced. The present application proposes the following technical scheme, please see below for details.

[0051] Figure 1 A flowchart of an application defect detection method disclosed in an embodiment of the present application.

[0052] like Figure 1 As shown, the method may include:

[0053] Step S1: Collect application operation videos according to preset collection rules.

[0054] Specifically, when executing step S1, the system will strictly collect application operation videos according to the preset collection rules, and the collected application operation videos contain at least one complete user operation process. The specific operation examples are as follows: First, the system starts the screen recording function, and the activation of this function marks the beginning of video collection. With the activation of the screen recording function, the system will capture all operations of the user on the application interface in real time, including clicks, slides, inputs, etc., and will also record the system's immediate feedback on user operations, such as page jumps, data changes, etc. In this way, the user's operations and system feedback are fully recorded to form an application operation video. When the user completes all steps according to the preset behavioral operation requirements, the system will automatically identify and stop video recording to ensure that the collected video content is both complete and accurate. Finally, this video file that records the user's complete operation process will be saved for use in subsequent analysis.

[0055] It should be noted that this application strictly follows the preset collection rules to collect user application operation videos to ensure that all collections are legal and compliant. Before the user uses this application for the first time, the relevant video collection behaviors have been explained and marked to the user in advance through clear terms and policies. The user uses this application on the premise of knowing and agreeing.

[0056] This application has the ability to recognize complex scenes, which is mainly achieved by increasing the amount of training data and model complexity of the AI ​​model. This enables the system to better adapt to various complex scenes and provide users with more accurate and comprehensive video recording and analysis services.

[0057] For example, multi-step combined operations: The system can identify a series of continuous operations performed by the user, that is, multi-step combined operations. Through AI algorithms, the system can accurately capture the user's complex operation path, whether it is a simple click or a complex combined operation, it can be clearly recorded by the system. This ensures that even in the face of complex user operations, the system can provide accurate and complete video records.

[0058] Rapid continuous clicks: The system can handle situations where users quickly click on different elements, i.e. rapid continuous clicks. This requires the system to have a high degree of response speed and accuracy to correctly identify the user's operation intention. Through the recognition algorithm of the AI ​​model, the system can accurately capture each click of the user and determine the operation intention behind it, thereby providing users with more accurate feedback.

[0059] Drag operation: The system can recognize the user's behavior of dragging elements on the interface. Drag operation is a relatively complex user interaction method, requiring the system to track the user's dragging path in real time and accurately record the starting and ending points of the drag. By combining AI models for tracking, the system can generate corresponding text descriptions to help users or developers better understand the specific content and impact of drag operations.

[0060] Dynamic content loading and pop-up windows: The system has the ability to identify dynamic content loading and pop-up window prompts. In actual applications, dynamically loaded content and pop-up window prompts are common user interaction elements, which may have a significant impact on user operations. Through the intelligent recognition of AI models, the system can capture and record such changes, thereby providing users with more comprehensive and accurate video recording and analysis results.

[0061] Step S2: perform key frame behavior detection on the application operation video and convert it into video behavior text.

[0062] Specifically, the video behavior text records the combination of each operation item and the corresponding operation results that constitute the user operation process in the application operation video. In step S2, the system will perform key frame behavior detection on the collected application operation video, and generate a video behavior text based on these key frame information conversions. The core of this step is to accurately identify and record each key step of the user operation in the video and its corresponding results, thereby generating a video behavior text.

[0063] Specifically, the system will first analyze the application operation video, and automatically detect and extract the key frames in the video through advanced algorithm technologies such as image recognition and motion detection. These key frames represent important nodes in the user operation process, such as the start and end time of clicks, drags, slides, etc.

[0064] Next, the system will further analyze and interpret these keyframes to identify the specific operation items represented by each keyframe, such as clicking a button, dragging an element, etc. At the same time, the system will also record the corresponding operation results caused by these operation items, such as page jumps, data updates, etc.

[0065] Finally, the system will generate a detailed video behavior text based on these key frame information. This text will clearly list the various operation item combinations that constitute the user operation process in the application operation video, as well as the various operation results corresponding to each operation item combination. For example, application operation video 1 contains an operation item combination, the operation item combination is "click button A → drag element B → click button C", and the operation result corresponding to this operation item combination is "display page D"; application operation video 2 contains two operation item combinations, operation item combination 1 is "click button E", and the operation result corresponding to operation item combination 1 is "display page F", and operation item combination 2 is "enter text in element G → click button H", and the operation result corresponding to operation item combination 2 is "pop-up link I".

[0066] Step S3: Detect the video behavior text according to the built-in behavior result mapping text comparison, determine the abnormality of the operation result, and generate a defect ticket.

[0067] Specifically, the system relies on the built-in behavior result mapping text of the system to conduct an in-depth comparison and detection of the video behavior text to determine whether there are abnormal operation results and generate a defect ticket accordingly. The behavior result mapping text records each operation item combination and the corresponding matching operation results. The corresponding matching operation results should be the operation results that should be correctly displayed after executing various operation item combinations.

[0068] Specifically, the system first reads the combination of operation items and their operation results recorded in the video behavior text, and then compares them with the records in the behavior result mapping text to identify whether the operation results in the video behavior text are consistent with the correct results recorded in the behavior result mapping text.

[0069] During the comparison process, if the operation result in the video behavior text is found to be inconsistent with the record in the behavior result mapping text, the system will determine that the operation result is abnormal. At this time, the system will automatically record the abnormal information and generate a defect ticket. The defect ticket records in detail the abnormal operation item combination, the wrong operation result, and the expected correct result, providing clear and accurate information for subsequent defect repair.

[0070] Through the comparison test in step S3, the system can promptly detect and report abnormal operation results, thereby ensuring the quality of the application and user experience. At the same time, the generated defect tickets also provide valuable feedback to developers, helping application developers to quickly locate and fix problems, and further improve the stability and reliability of the application.

[0071] In addition, after the defect ticket is generated, this application also designs a series of subsequent processing processes to ensure that the defect can be managed and resolved in a timely and effective manner. This application can also perform the following processing on the defect ticket, including:

[0072] ① Put a timestamp with the generation time on the defect ticket and set the corresponding defect processing cycle.

[0073] Specifically, for each defect ticket generated, the system will automatically stamp it with a timestamp with the generation time and set the corresponding defect handling cycle. This helps the development team clearly understand the discovery time of each defect and the expected repair time, so as to better plan and manage the repair work.

[0074] When the system detects a defect and generates a defect ticket, it will automatically send a notification to the relevant developers so that they can understand and handle the defect in a timely manner. Once the developer completes the repair work, the system will automatically prompt the creator of the defect ticket to verify to ensure the effectiveness of the repair. In addition, if a defect is not repaired within the corresponding defect handling cycle, the system will automatically send a reminder email to remind you to pay attention to the repair progress of the defect, so as to ensure that all defects can be handled in a timely manner.

[0075] ② Detect the defect nature of each defect recorded in the defect ticket, and set corresponding defect classification labels and processing priorities.

[0076] Specifically, the system can also intelligently classify and prioritize each defect recorded in the defect ticket. Based on the nature of the defect (such as severity, frequency, scope of impact, etc.) and the contextual information of the project, each defect can be automatically assigned a classification label and set a processing priority. For example, the system can identify defects that affect core functional modules and automatically set them to "high priority" so that the development team can prioritize these issues. For some UI issues with less impact, the system will set them to "low priority", thereby gradually optimizing the user experience while ensuring the stability of core functions.

[0077] It can be seen from the above technical solution that an application defect detection method, device, equipment and readable storage medium provided by the embodiment of the present application first collects application operation videos according to preset collection rules, wherein the application operation video contains at least one complete user operation process. Then, key frame behavior detection is performed on the application operation video, and the video behavior text is converted to generate a video behavior text, wherein the video behavior text records the combination of each operation item and the corresponding operation results that constitute each user operation process in the application operation video. Finally, the video behavior text is detected according to the built-in behavior result mapping text comparison, the abnormal operation result is determined, and a defect ticket is generated, wherein the behavior result mapping text records the combination of each operation item and the corresponding matching operation results.

[0078] This application avoids the tedious process of manually recording operation steps and operation results by automatically collecting and analyzing user operation videos, realizes intelligent and automated testing, reduces manual intervention, and improves test efficiency and accuracy. At the same time, since this application can automatically analyze user operation items and operation results, and compare them with the target operation results recorded in the built-in behavior result mapping text, it can identify application defects. This defect identification method does not rely on fixed scripts and operation steps, and has greater flexibility and intelligent analysis capabilities. In addition, for the application software of this application, application developers only need to regularly retrieve defect orders from recent periods to understand the application defects and perform upgrades and maintenance. There is no need to perform a large number of test operations in order to detect defects, which reduces the cost of testing and maintenance, improves work efficiency, and can cope with rapidly changing iterative development and testing needs.

[0079] In some embodiments of the present application, step S2, performing key frame behavior detection on the application operation video and converting and generating video behavior text for introduction, may specifically include:

[0080] Step S21: extract each video key frame in the application operation video, perform behavior detection on each video key frame, and identify and generate operation behavior information.

[0081] Specifically, the system first extracts key frames from the application operation video. These key frames usually represent important nodes or key actions in the user's operation process. Then, the system performs behavior detection on these key frames, using advanced image processing and machine learning algorithms to identify the user's operation behavior in the key frames and generate corresponding operation behavior information. This information records in detail the specific operations performed by the user in each key frame.

[0082] Step S22: Use OCR technology to perform text recognition on each video key frame to generate operation text information.

[0083] Specifically, after extracting the keyframes and identifying the operation behavior, the system will also use OCR (Optical Character Recognition) technology to identify the text information in the keyframes. The purpose of this step is to obtain any text content displayed in the keyframes, such as button labels, prompt information, etc. These text contents are crucial to understanding the user operation context and results. Through OCR technology, the system can convert these text contents into readable text information, providing a basis for subsequent analysis and conversion.

[0084] Step S23: perform time series analysis on the operation behavior information and the operation text information, and sequentially organize the analysis results to obtain execution process data.

[0085] Specifically, the operation behavior information and operation text information generated in steps S21 and S22 are subjected to time series analysis. The purpose of this step is to determine the sequence and logical relationship of this information, so as to construct a complete execution process of the user's operation. Through time series analysis, the system can accurately identify the dependency and sequence between each operation behavior, and then sequentially organize the execution process data based on these analysis results. These data record in detail each step of the user's operation from start to finish and its corresponding text information.

[0086] Step S24: Use NLP technology to perform text conversion on the execution process data to generate video behavior text.

[0087] Specifically, the last step is to use NLP (natural language processing) technology to convert the execution process data into text and generate video behavior text. In this step, the system will use the natural language generation algorithm in NLP technology to convert the operation behavior and text information in the execution process data into a coherent and readable text description. These text descriptions not only record each step and result of the user's operation, but also present the complete process and context information of the user's operation in the form of natural language. The generated video behavior text can be used as a basis for subsequent analysis, reporting or feedback.

[0088] In some embodiments of the present application, step S3, detecting the video behavior text according to the built-in behavior result mapping text comparison, determining the abnormality of the operation result, and generating a defect ticket is introduced, which may specifically include:

[0089] Step S31, identifying target operation item combinations in the built-in behavior result mapping text that are respectively consistent with the operation item combinations constituting the user operation processes in the application operation video, and determining target operation results corresponding to the target operation item combinations.

[0090] Specifically, the system first reads the built-in behavior result mapping text, which records in detail various operation item combinations and their corresponding correct operation results. Then, the system identifies the target operation item combinations that are consistent with the operation item combinations that constitute the user operation process in the application operation video, and determines the target operation results that these target operation item combinations correspond to in the behavior result mapping text. This step provides benchmark data for subsequent comparison detection.

[0091] It is understandable that the built-in behavior result mapping text should include all possible combinations of operation items that the user may perform and the corresponding correct operation results. In theory, if the user's operation process is correct, the corresponding target operation item combination and its corresponding target operation result should be found in the behavior result mapping text.

[0092] Step S32: Generate a first sentence embedding vector group based on the video behavior text.

[0093] Specifically, in order to perform efficient text comparison, the system converts the video behavior text into numerical form. The system performs preprocessing operations such as word segmentation and stop word removal on each sentence in the video behavior text, and then uses deep learning models (such as BERT, GPT, etc.) to convert the processed sentences into sentence embedding vectors. These sentence embedding vectors can capture the semantic information of the sentence, which facilitates subsequent comparison and analysis. The system combines all these sentence embedding vectors to form a first sentence embedding vector group. In general, one operation item combination is converted into a sentence embedding vector, but because the application operation video may include more than one operation item combination, a first sentence embedding vector group consisting of multiple sentence embedding vectors will be obtained.

[0094] Step S33: Generate a second sentence embedding vector group based on the target operation item combinations and the corresponding matching target operation results.

[0095] Specifically, the system will also pre-process the target operation results corresponding to the target operation item combination in the behavior result mapping text and convert them into sentence embedding vectors. These sentence embedding vectors can also capture the semantic information of the target operation results. The system combines all these sentence embedding vectors to form a second sentence embedding vector group.

[0096] Step S34: Calculate the similarity score between the first sentence embedding vector group and the second sentence embedding vector group.

[0097] Specifically, after obtaining the first sentence embedding vector group and the second sentence embedding vector group, the system will calculate the similarity score between them. This is usually achieved by calculating the cosine similarity, Euclidean distance and other metrics between the two sets of vectors. The higher the similarity score, the closer the operation result in the video behavior text is to the target operation result in the behavior result mapping text; the lower the similarity score, the more abnormal the operation result may be.

[0098] Step S35: Determine the abnormality of the operation result according to the similarity score and the preset threshold, and generate a defect ticket.

[0099] Specifically, the similarity score and the preset threshold are used to determine whether the operation result is abnormal, and a defect ticket is generated accordingly. The system will compare the calculated similarity score with the preset threshold. If the similarity score is lower than the threshold, the system considers that the operation result is abnormal and automatically generates a defect ticket. The defect ticket will record in detail the abnormal operation item combination, the wrong operation result, the expected correct result, the similarity score and other information, providing a clear and accurate basis for subsequent defect repair.

[0100] In some embodiments of the present application, in order to ensure the accuracy and efficiency of application defect detection, an abnormal behavior detection link may be added.

[0101] After step S1, collecting the application operation video according to the preset collection rule, the following may also be included:

[0102] Performing abnormal behavior detection on the application operation video;

[0103] If abnormal behavior is detected, an abnormal behavior report is generated and the current application defect detection is stopped;

[0104] If no abnormal behavior is detected, the process of performing key frame behavior detection on the application operation video continues.

[0105] Specifically, after collecting the application operation video, the system will first use advanced image processing and machine learning algorithms to conduct a comprehensive abnormal behavior detection on the video content. Abnormal behavior may include but is not limited to: user operations do not meet preset specifications, unexpected elements appear in the interface, application crashes or freezes, etc. If the system detects abnormal behavior in the video, it will immediately generate an abnormal behavior report. The abnormal behavior report will record in detail the specific manifestations, occurrence time, possible causes and other information of the abnormal behavior, providing developers with a clear basis for troubleshooting. At the same time, the system will stop the current application defect detection process to avoid unnecessary detection in the presence of obvious abnormal behavior, thereby improving detection efficiency. If the system does not detect abnormal behavior, it is considered that the video content meets the detection requirements and the subsequent key frame behavior detection process can continue.

[0106] By adding an abnormal behavior detection link, this application can identify abnormal situations that may cause detection failure or inaccurate results at an early stage, and promptly feedback to developers for processing. This can not only improve the accuracy and efficiency of application defect detection, but also help developers to promptly discover and fix potential problems in the application, thereby improving the overall quality of the application and user experience.

[0107] It should be noted that the specific implementation of abnormal behavior detection may vary depending on the application scenario and requirements. In actual applications, developers can select appropriate algorithms and models to perform abnormal behavior detection according to specific circumstances, and take corresponding processing measures based on the detection results. This application should include but is not limited to the abnormal behavior detection methods mentioned in the application.

[0108] Furthermore, in this application, the following three forms are provided for detecting abnormal behavior of the application operation video, specifically:

[0109] The first type is abnormal behavior detection of unexpected operations:

[0110] The application operation video is subjected to abnormal behavior detection of unexpected operations according to the built-in behavior result mapping text.

[0111] Specifically, based on the built-in behavior result mapping text, the system can identify whether each step of the user's operation in the application operation video is in line with expectations. If the user's operation deviates from the preset behavior path, that is, an unexpected operation occurs, the system will determine it as abnormal behavior.

[0112] For example, the default behavior is "click button A to switch to page B". However, in actual operation, the user accidentally clicks button C, resulting in failure to complete the expected operation path. At this time, the system will detect this abnormal behavior and generate a corresponding abnormal behavior report: "The user clicked the unexpected button C and failed to complete the expected operation path."

[0113] Second, abnormal behavior detection with long-term no operation:

[0114] The abnormal behavior of long-term non-operation is detected on the application operation video according to the preset operation stagnation time.

[0115] Specifically, the system presets a reasonable stagnation time between each operation link. If the user stays on a page or operation link for longer than this preset time in the application operation video, the system will determine it as abnormal behavior of long-term inactivity.

[0116] For example, after the user completes the first few steps, the system continues to monitor the user's operation behavior. If the system finds that the user stays on a certain page (such as page X) for more than the preset stagnation time (such as 5 minutes), the system will generate a warning prompt: "The user has stayed on page X for more than 5 minutes without any operation." This prompt helps to promptly detect and handle abnormal user behavior, avoiding system resource waste or application failures that may be caused by long-term inactivity.

[0117] The third type: abnormal behavior detection of invalid clicks:

[0118] Abnormal behavior detection of invalid clicks is performed on the application operation video according to the validity of the response element.

[0119] Specifically, the system will verify the validity of each click operation in the application operation video. If the user's click operation fails to trigger the expected response element (such as a button, link, etc.), or triggers an invalid response element (such as an invalid link, a disabled button, etc.), the system will determine it as an abnormal behavior of invalid click.

[0120] For example, a user clicks a button in an app, but the button fails to produce the expected response for some reason (such as insufficient permissions, the button is invalid, etc.). At this time, the system will detect this invalid click behavior and generate a corresponding abnormal behavior report to help developers locate and fix the problem.

[0121] An application defect detection device provided in an embodiment of the present application is described below. The application defect detection device described below and the application defect detection method described above can refer to each other.

[0122] See also Figure 2 , Figure 2 A schematic diagram of an application defect detection device disclosed in an embodiment of the present application.

[0123] like Figure 2 As shown, the application defect detection device may include:

[0124] The video acquisition unit 110 is used to acquire application operation videos according to preset acquisition rules, wherein the application operation videos include at least one complete user operation process;

[0125] The behavior detection unit 120 is used to perform key frame behavior detection on the application operation video and convert it into a video behavior text, wherein the video behavior text records each combination of operation items and corresponding operation results constituting each user operation process in the application operation video;

[0126] The defect detection unit 130 is used to detect the video behavior text according to the built-in behavior result mapping text comparison, determine the abnormal operation result, and generate a defect ticket. The behavior result mapping text records each operation item combination and the corresponding matching operation results.

[0127] It can be seen from the above technical solution that an application defect detection method, device, equipment and readable storage medium provided by the embodiment of the present application first collects application operation videos according to preset collection rules, wherein the application operation video contains at least one complete user operation process. Then, key frame behavior detection is performed on the application operation video, and the video behavior text is converted to generate a video behavior text, wherein the video behavior text records the combination of each operation item and the corresponding operation results that constitute each user operation process in the application operation video. Finally, the video behavior text is detected according to the built-in behavior result mapping text comparison, the abnormal operation result is determined, and a defect ticket is generated, wherein the behavior result mapping text records the combination of each operation item and the corresponding matching operation results.

[0128] This application avoids the tedious process of manually recording operation steps and operation results by automatically collecting and analyzing user operation videos, realizes intelligent and automated testing, reduces manual intervention, and improves test efficiency and accuracy. At the same time, since this application can automatically analyze user operation items and operation results, and compare them with the target operation results recorded in the built-in behavior result mapping text, it can identify application defects. This defect identification method does not rely on fixed scripts and operation steps, and has greater flexibility and intelligent analysis capabilities. In addition, for the application software of this application, application developers only need to regularly retrieve defect orders from recent periods to understand the application defects and perform upgrades and maintenance. There is no need to perform a large number of test operations in order to detect defects, which reduces the cost of testing and maintenance, improves work efficiency, and can cope with rapidly changing iterative development and testing needs.

[0129] Optionally, the video acquisition unit may include:

[0130] A behavior information unit, used to extract each video key frame in the application operation video, and perform behavior detection on each video key frame to identify and generate operation behavior information;

[0131] A text information unit, used to perform text recognition on each of the video key frames using OCR technology to generate operation text information;

[0132] An analysis and sorting unit, used to perform time series analysis on the operation behavior information and the operation text information, and to sort the analysis results in order to obtain execution process data;

[0133] The text conversion unit is used to perform text conversion on the execution process data by using NLP technology to generate video behavior text.

[0134] Optionally, the defect detection unit may include:

[0135] A combination matching unit, used to identify target operation item combinations in the built-in behavior result mapping text that are respectively consistent with the operation item combinations constituting the user operation processes in the application operation video, and determine the target operation results that are matched to the target operation item combinations;

[0136] A first vector unit, configured to generate a first sentence embedding vector group based on the video behavior text;

[0137] A second vector unit, configured to generate a second sentence embedding vector group based on the target operation item combinations and the corresponding matched target operation results;

[0138] A similarity calculation unit, configured to calculate a similarity score between the first sentence embedding vector group and the second sentence embedding vector group;

[0139] The defect determination unit is used to determine the abnormality of the operation result according to the similarity score and a preset threshold value, and generate a defect ticket.

[0140] Optionally, the application defect detection device may further include:

[0141] The anomaly detection unit is used to perform abnormal behavior detection on the application operation video after collecting the application operation video according to the preset collection rules. If abnormal behavior is detected, an abnormal behavior report is generated and the application defect detection is stopped. If no abnormal behavior is detected, the process of performing key frame behavior detection on the application operation video continues.

[0142] Optionally, the abnormality detection unit may include:

[0143] A first detection unit, configured to perform abnormal behavior detection of unexpected operations on the application operation video according to a built-in behavior result mapping text;

[0144] A second detection unit is used to detect abnormal behavior of long-term non-operation of the application operation video according to a preset operation stagnation time;

[0145] The third detection unit is used to detect abnormal behavior of invalid clicks on the application operation video according to the validity of the response element.

[0146] Optionally, the application defect detection device may further include:

[0147] The defect sheet annotation unit is used to detect the defect nature of each defect recorded in the defect sheet and set corresponding defect classification labels and processing priorities.

[0148] Optionally, the application defect detection device may further include:

[0149] The processing cycle marking unit is used to stamp a timestamp with a generation time on the defect sheet and set a corresponding defect processing cycle.

[0150] The application defect detection apparatus provided in the embodiment of the present application can be applied to application defect detection equipment. Figure 3 The hardware structure diagram of the application defect detection device is shown in FIG. Figure 3 ,The hardware structure of the application defect detection device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0151] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0152] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0153] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), etc., such as at least one disk memory;

[0154] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0155] Collecting application operation videos according to preset collection rules, wherein the application operation videos include at least one complete user operation process;

[0156] Performing key frame behavior detection on the application operation video and converting it into a video behavior text, wherein the video behavior text records each combination of operation items and corresponding operation results constituting each user operation process in the application operation video;

[0157] The video behavior text is detected according to the built-in behavior result mapping text comparison, abnormal operation results are determined, and a defect ticket is generated. The behavior result mapping text records each operation item combination and the corresponding matching operation results.

[0158] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0159] The embodiment of the present application further provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0160] Collecting application operation videos according to preset collection rules, wherein the application operation videos include at least one complete user operation process;

[0161] Performing key frame behavior detection on the application operation video and converting it into a video behavior text, wherein the video behavior text records each combination of operation items and corresponding operation results constituting each user operation process in the application operation video;

[0162] The video behavior text is detected according to the built-in behavior result mapping text comparison, abnormal operation results are determined, and a defect ticket is generated. The behavior result mapping text records each operation item combination and the corresponding matching operation results.

[0163] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0164] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0165] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0166] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An application defect detection method, characterized in that: include: Collecting application operation videos according to preset collection rules, wherein the application operation videos include at least one complete user operation process; Performing key frame behavior detection on the application operation video and converting it into a video behavior text, wherein the video behavior text records each combination of operation items constituting each user operation process in the application operation video and each corresponding operation result; The video behavior text is detected according to the built-in behavior result mapping text comparison, abnormal operation results are determined, and a defect ticket is generated. The behavior result mapping text records each operation item combination and the corresponding matching operation results.

2. The method according to claim 1, characterized in that Perform key frame behavior detection on the application operation video and convert it into video behavior text, including: Extracting each video key frame in the application operation video, and performing behavior detection on each video key frame to identify and generate operation behavior information; Using OCR technology to perform text recognition on each video key frame to generate operation text information; Performing time series analysis on the operation behavior information and the operation text information, and sequentially arranging the analysis results to obtain execution process data; The execution process data is converted into text using NLP technology to generate video behavior text.

3. The method according to claim 1, characterized in that The video behavior text is detected based on the built-in behavior result mapping text comparison, abnormal operation results are determined, and a defect ticket is generated, including: Identify target operation item combinations in the built-in behavior result mapping text that are respectively consistent with the operation item combinations constituting the user operation processes in the application operation video, and determine target operation results that match the target operation item combinations; Generate a first sentence embedding vector group based on the video behavior text; Generate a second sentence embedding vector group based on the target operation item combinations and the corresponding matched target operation results; Calculating a similarity score between the first sentence embedding vector group and the second sentence embedding vector group; Based on the similarity score and a preset threshold, the abnormality of the operation result is determined and a defect ticket is generated.

4. The method according to claim 1, characterized in that: After collecting the application operation video according to the preset collection rules, it also includes: Performing abnormal behavior detection on the application operation video; If abnormal behavior is detected, an abnormal behavior report is generated and the current application defect detection is stopped; If no abnormal behavior is detected, the process of performing key frame behavior detection on the application operation video continues.

5. The method according to claim 4, characterized in that Perform abnormal behavior detection on the application operation video, including: Perform abnormal behavior detection of unexpected operations on the application operation video according to the built-in behavior result mapping text; Perform abnormal behavior detection on the application operation video when there is no operation for a long time according to a preset operation stagnation time; Abnormal behavior detection of invalid clicks is performed on the application operation video according to the validity of the response element.

6. The method according to claim 1, characterized in that Also includes: The defect nature of each defect recorded in the defect ticket is detected, and corresponding defect classification labels and processing priorities are set.

7. The method according to claim 1, characterized in that Also includes: A timestamp with the generation time is stamped on the defect sheet, and a corresponding defect handling cycle is set.

8. An application defect detection device, characterized in that: include: A video acquisition unit, configured to acquire application operation videos according to preset acquisition rules, wherein the application operation videos include at least one complete user operation process; A behavior detection unit, used to perform key frame behavior detection on the application operation video, and convert and generate a video behavior text, wherein the video behavior text records each operation item combination and each corresponding operation result constituting each user operation process in the application operation video; The defect detection unit is used to detect the video behavior text according to the built-in behavior result mapping text comparison, determine the abnormal operation result, and generate a defect ticket. The behavior result mapping text records each operation item combination and the corresponding matching operation results.

9. An application defect detection device, characterized in that: including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the application defect detection method as described in any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the application defect detection method according to any one of claims 1 to 7 is implemented.

Images