Software testing method based on multi-modal analysis and terminal

Abstract

The invention discloses a software testing method based on multi-modal analysis and a terminal. The method comprises the following steps: monitoring a picture of equipment running to-be-tested software in real time, obtaining a frame-dropping picture in the picture and context data thereof, and analyzing the frame-dropping picture by using a multi-modal picture analysis model based on the context data to obtain a picture analysis text; extracting keywords in the picture analysis text, performing clustering operation on the extracted keywords, generating a question label corresponding to each cluster, and determining all check points according to the question labels; and checking the frame drop picture according to all the check points, marking the check points which do not pass the check result as problem data, and generating a test report, so that picture-level BUG detection can be realized, and the software test quality and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and in particular to a software testing method and terminal based on multimodal analysis. Background Art

[0002] With the rapid development of information technology, software and application development and testing have become critical for ensuring product quality. However, in the current technological landscape, despite the availability of a variety of testing tools and methods, their approaches to identifying and addressing screen-level bugs remain limited. Specifically, traditional testing tools such as Testin Cloud Testing and XDebug can help developers identify software issues to a certain extent, but they rely on manual intervention and cannot automatically identify screen-level bugs. Cross-platform testing methods primarily focus on performance data comparison, assessing software compatibility and performance by collecting and comparing performance metrics across different platforms. These methods lack dynamic analysis of screen anomalies and are unable to promptly and accurately detect issues such as display errors and confusing interface layouts that may exist on different platforms. AI monitoring technology is often used in operational optimization or security applications. By leveraging AI algorithms to analyze and predict user behavior or identify and alert anomalies in monitoring images, these technologies' algorithmic models and data formats differ significantly from the requirements of software testing scenarios, making them inappropriate for direct application and difficult to meet the specific requirements of software testing for screen-level bug detection and functional integrity verification. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a software testing method based on multimodal analysis, which can realize screen-level BUG detection and improve the quality and efficiency of software testing.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: A software testing method based on multimodal analysis comprises the following steps: S1. Monitoring a screen of a device running the software to be tested in real time, obtaining a dropped frame screen and its context data in the screen, and analyzing the dropped frame screen using a multimodal screen analysis model based on the context data to obtain a screen analysis text; S2. Extract keywords from the image analysis text, perform clustering on the extracted keywords, generate a question label corresponding to each cluster, and determine all checkpoints based on the question label; S3. Check the frame drop images according to all the checkpoints, mark the checkpoints that fail the check as problem data, and generate a test report.

[0005] In order to solve the above technical problems, another technical solution adopted by the present invention is: A software testing terminal based on multimodal analysis includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor implements the various steps of the above-mentioned software testing method based on multimodal analysis.

[0006] The beneficial effects of the present invention are as follows: a software testing method and terminal based on multimodal analysis provided by the present invention, which monitors the screen of the device running the software to be tested in real time, captures the dropped frame screen and its context data, reduces manual intervention, improves monitoring efficiency, uses a multimodal screen analysis model to analyze the dropped frame screen to obtain screen analysis text, and improves the accuracy of recognition and analysis; processes the screen analysis text through keyword extraction and clustering technology, generates problem tags, determines checkpoints based on the problem tags, optimizes the problem diagnosis process, improves the problem location efficiency, and checks all checkpoints to ensure the comprehensiveness of the test, realizes screen-level BUG detection, marks the checkpoints that fail the inspection results as problem data, generates a test report, and provides strong support for subsequent problem analysis and resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 Flowchart of a software testing method based on multimodal analysis according to an embodiment of the present invention; Figure 2 This is an architectural diagram of a software testing method based on multimodal analysis according to an embodiment of the present invention; Figure 3 is another flow chart of a software testing method based on multimodal analysis according to an embodiment of the present invention; Figure 4 Schematic diagram of a software testing terminal based on multimodal analysis according to an embodiment of the present invention; Description of labels: 1. A software testing terminal based on multimodal analysis; 2. Memory; 3. Processor. DETAILED DESCRIPTION

[0008] To illustrate the technical content, achieved objectives and effects of the present invention in detail, the following description is given in conjunction with the embodiments and accompanying drawings.

[0009] Please refer to Figure 1 , an embodiment of the present invention provides a software testing method based on multimodal analysis, comprising the steps of: S1. Monitoring a screen of a device running the software to be tested in real time, obtaining a dropped frame screen and its context data in the screen, and analyzing the dropped frame screen using a multimodal screen analysis model based on the context data to obtain a screen analysis text; S2. Extract keywords from the image analysis text, perform clustering on the extracted keywords, generate a question label corresponding to each cluster, and determine all checkpoints based on the question label; S3. Check the frame drop images according to all the checkpoints, mark the checkpoints that fail the check as problem data, and generate a test report.

[0010] From the above description, it can be seen that the beneficial effects of the present invention are: by real-time monitoring of the screen of the device running the software to be tested, capturing the dropped frame screen and its context data, reducing manual intervention, improving monitoring efficiency, using a multimodal screen analysis model to analyze the dropped frame screen to obtain screen analysis text, and improving the accuracy of recognition and analysis; processing the screen analysis text through keyword extraction and clustering technology, generating problem labels, determining checkpoints based on the problem labels, optimizing the problem diagnosis process, improving the efficiency of problem location, and checking all checkpoints to ensure the comprehensiveness of the test, realizing screen-level BUG detection, marking the checkpoints that fail the inspection results as problem data, generating a test report, and providing strong support for subsequent problem analysis and resolution.

[0011] Furthermore, real-time monitoring of the screen of the device running the software to be tested and obtaining frame drop images and their context data in the screen include: The frame rate of the screen of the device running the software to be tested is calculated according to the real-time monitoring situation. If the frame rate is lower than a preset frame rate threshold, it is determined to be a frame drop, and the corresponding frame drop screen and its context data are obtained.

[0012] As can be seen from the above description, by monitoring the frame rate in real time, accurately determining the moment of frame drops, ensuring that no frame drops are missed, and obtaining the corresponding frame drops and their context data, providing data support for subsequent analysis, so as to improve the picture smoothness and stability of the software under test running on the device.

[0013] Furthermore, the multimodal image analysis model includes a basic model, a target detection model and an image segmentation model; Analyzing the dropped frame image using a multimodal image analysis model based on the context data to obtain an image analysis text, including: Generate a text instruction according to the context data, input the dropped frame image and the text instruction into the basic model to generate a picture description text; Using the object detection model to detect the coordinates of each element in the dropped frame picture through an anchor frame mechanism, determining whether there is a picture adaptation problem based on the coordinates of each element, and obtaining an analysis result of the object detection model; Using the image segmentation model to identify and quantify the missing texture area in the dropped frame image, obtaining an area ratio of the missing texture area, determining whether there is a screen rendering problem based on the area ratio of the missing texture area, and obtaining an analysis result of the image segmentation model; The picture description text, the analysis result of the target detection model and the analysis result of the image segmentation model are combined to generate a picture analysis text.

[0014] From the above description, it can be seen that the screen description text generated by the basic model can provide a relatively comprehensive text description of the frame drop screen, providing basic information for subsequent analysis. The target detection model is used to accurately detect the coordinates of screen elements through the anchor frame mechanism, effectively judge the screen adaptation problems, and can discover abnormalities in screen layout, element placement, etc. The image segmentation model is used to identify and quantify the missing map area, accurately judge the screen rendering problems, and effectively identify screen-level BUGs. Combining the results of the three to generate screen analysis text, ensure the comprehensiveness of the frame drop screen analysis, help to quickly locate and solve the problems that cause frame drops, thereby improving the software screen quality and operation stability.

[0015] Furthermore, the frame drop image is analyzed using a multimodal image analysis model based on the context data to obtain an image analysis text, further comprising: Setting the output probability of the basic model, performing consistency scoring on the analysis results corresponding to the object detection model and the image segmentation model and the picture description text, and obtaining a multimodal consistency value; The area ratio of the missing region in the image segmentation model is weighted and scored according to a preset image difference calculation formula to obtain the abnormal feature strength; Corresponding weights are configured for the output probability, the multimodal consistency value and the abnormal feature strength. The confidence of the picture analysis text is calculated by combining the output probability and its weight, the multimodal consistency value and its weight, and the abnormal feature strength and its weight. If the confidence is less than the preset confidence threshold, the frame drop review process is triggered.

[0016] From the above description, it can be seen that by setting the output probability of the basic model and combining the analysis results of the target detection model and the image segmentation model for consistency scoring to obtain a multimodal consistency value, the degree of fit between the analysis results of different models can be measured as a whole, avoiding the deviation that may occur in a single model. At the same time, the picture analysis text is evaluated in combination with the strength of abnormal features to effectively ensure the reliability of the picture analysis text. When the confidence level is less than the preset confidence threshold, the frame drop review process is triggered to avoid ignoring the frame drop problem due to erroneous or inaccurate analysis, which helps to improve the accuracy of the frame drop problem analysis.

[0017] Furthermore, the step S2 then includes: Adjusting parameters of the multimodal image analysis model according to the test report, and adjusting output results of the multimodal image analysis model using the received natural language instructions; The adjusted multimodal image analysis model is pruned and quantized to deploy the multimodal image analysis model in a lightweight manner.

[0018] From the above description, it can be seen that adjusting the parameters of the multimodal image analysis model according to the test report can optimize the model performance in a targeted manner, improve the accuracy and effectiveness of the analysis, adjust the output results through the received natural language instructions, increase the flexibility of the model, meet user needs, and prune and quantize the adjusted multimodal image analysis model to achieve lightweight deployment, reduce the model's storage space and computing resource usage, and improve the model's practicality and popularity.

[0019] Please refer to Figure 4 Another embodiment of the present invention provides a software testing terminal based on multimodal analysis, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor implements the various steps of the above-mentioned software testing method based on multimodal analysis.

[0020] The software testing method and terminal based on multimodal analysis described above are applicable to software testing scenarios, can implement screen-level BUG detection, and improve software testing quality and efficiency. The following is an explanation of the specific implementation methods: Please refer to Figures 1 to 3 , embodiment 1 of the present invention is: A software testing method based on multimodal analysis comprises the following steps: S1. Monitor the screen of the device running the software to be tested in real time, obtain the frame-dropping screen and its context data in the screen, analyze the frame-dropping screen using a multimodal screen analysis model based on the context data, and obtain a screen analysis text, specifically including: S1.1-S1.3.

[0021] S1.1. Calculate the frame rate of the screen of the device running the software to be tested according to the real-time monitoring situation. If the frame rate is lower than a preset frame rate threshold, it is determined to be a frame drop, and obtain the corresponding frame drop screen and its context data.

[0022] In this embodiment, the screen of the device running the software under test is monitored in real time through a camera, SDK, or ADB automation tool. When the frame drop rate exceeds a threshold, the screen is automatically captured and the context (such as device status and operation log) is recorded. The implementation method of monitoring the frame rate includes: 1. Single external camera monitoring A high-speed camera (≥60FPS) is aimed at the device screen. Image masking technology is used to lock the game screen area to eliminate ambient light interference (such as reflections from curtains and occlusion by the operator's hands). The pixel difference ratio between two consecutive frames is calculated (pixel difference ratio = number of difference pixels / total pixels × 100%). If the pixel difference ratio is ≥5%, it is determined to be a screen update. The actual frame rate is calculated based on the timestamp (actual frame rate = number of detected screen updates / monitoring time period (seconds)).

[0023] 2. Hybrid monitoring Use the camera to monitor the device in conjunction with ADB / SDK. Use ADB to directly obtain the device's internal frame rate data (for example, adbshell dumpsys gfxinfo). The camera is only used for synchronous screenshots to ensure the accuracy of frame rate detection.

[0024] In addition, in this embodiment, the preset frame rate threshold can be dynamically adjusted according to the device performance, such as the frame rate threshold of a high-end mobile phone is set to 60FPS, and that of a low-end device is set to 30FPS.

[0025] S1.2. Analyze the dropped frame image using a multimodal image analysis model based on the context data to obtain an image analysis text, specifically including S1.2.1-S1.2.4.

[0026] S1.2.1. Generate text instructions based on the context data, input the dropped frame image and the text instructions into the basic model, and generate image description text.

[0027] In this embodiment, the multimodal image analysis model includes a base model, which can be CLIP or GPT-4V. Dropped frames are fed into the base model to generate a textual description of the frame. For example, if a frame experiences delayed texture loading, CLIP might output "ground texture blur," while GPT-4V might further describe it as "ground texture resolution is below standard at the 3rd second." Contextual data, including device status (GPU load, memory usage, etc.) and operation logs, is fed into the base model via a conditional embedding layer, adjusting its attention weight. For example, in low-end device scenarios, the model prioritizes texture loading delays over HDR overexposure. Operation logs are also aligned with the timestamps of dropped frame images to help the model understand dynamic scene context.

[0028] S1.2.2. Use the target detection model to detect the coordinates of each element in the dropped frame image through the anchor frame mechanism, determine whether there is a picture adaptation problem based on the coordinates of each element, and obtain the analysis result of the target detection model.

[0029] In this embodiment, the multimodal screen analysis model includes a target detection model, which can be YOLOv8. The target detection model accurately detects dynamic UI positions (such as health bar offset pixel values, button click area bounding boxes) through an anchor box mechanism. The output format is (x_min, y_min, x_max, y_max, category, confidence). For example, when it is detected that the "skill button is not aligned with the screen edge" in the screen, it is marked as a UI adaptation problem.

[0030] S1.2.3. Use the image segmentation model to identify and quantify the texture missing areas in the frame drop screen, obtain the area ratio of the texture missing areas, determine whether there is a screen rendering problem based on the area ratio of the texture missing areas, and obtain the analysis result of the image segmentation model.

[0031] In this embodiment, the multimodal image analysis model includes an image segmentation model, which can be a Mask R-CNN. The pixel-level mask (Mask) is generated through instance segmentation to identify areas with missing maps (such as ground breaks and jagged edges of character models) and quantify the area ratio of abnormal areas. For example, the missing part of the water surface reflection map is segmented and its proportion of the total area of ​​the picture is calculated (such as 15%).

[0032] S1.2.4. Generate a picture analysis text by combining the picture description text, the analysis result of the target detection model, and the analysis result of the image segmentation model.

[0033] In this embodiment, three models are used to analyze dropped frames and generate analysis text. This ensures comprehensive analysis of dropped frames, helps quickly locate and resolve issues that cause them, and thus improves software image quality and operational stability. Furthermore, in this embodiment, different devices use prompt word engineering to limit the model output range to avoid generating UI problem descriptions that are inappropriate for the current device.

[0034] S1.3. Conduct confidence assessment on the image analysis text to ensure the accuracy of the image analysis results, including: Setting the output probability of the basic model, performing consistency scoring on the analysis results corresponding to the object detection model and the image segmentation model and the picture description text, and obtaining a multimodal consistency value; The area ratio of the missing region in the image segmentation model is weighted and scored according to a preset image difference calculation formula to obtain the abnormal feature strength; Corresponding weights are configured for the output probability, the multimodal consistency value and the abnormal feature strength. The confidence of the picture analysis text is calculated by combining the output probability and its weight, the multimodal consistency value and its weight, and the abnormal feature strength and its weight. If the confidence is less than the preset confidence threshold, the frame drop review process is triggered.

[0035] In this embodiment, the probability value is output by the classification head of the base model, and a multimodal consistency score is performed in combination with the analysis results of the target detection model and the image segmentation model to obtain a multimodal consistency value. If the analysis result of the target detection model is "button offset" and the screen description text generated by the base model includes "UI error", it indicates multimodal consistency. The area ratio of the missing map region in the image segmentation model is weighted and scored according to a preset image difference calculation formula to obtain the abnormal feature strength. Thus, the confidence of the screen analysis text is calculated by combining the output probability and its weight, the multimodal consistency value and its weight, and the abnormal feature strength and its weight. If the confidence is less than a preset confidence threshold, the frame drop review process is triggered to avoid ignoring the frame drop problem due to erroneous or inaccurate analysis, which helps to improve the accuracy of the frame drop problem analysis. Among them, the confidence threshold can be dynamically adjusted according to the performance of the device. For example, when the frame rate fluctuation is detected, the confidence threshold is lowered to reduce false alarms.

[0036] S2. Extract keywords from the image analysis text, perform clustering on the extracted keywords, generate question labels corresponding to each cluster, and determine all checkpoints based on the question labels.

[0037] In this embodiment, keywords (such as "piercing the model" and "missing textures") are extracted from the text of the picture analysis, and the cosine similarity algorithm is used to cluster similar descriptions into problem labels of the same category (such as "physical collision bug" and "rendering abnormality"). Semantic-driven dynamic matching is adopted to match the checkpoints corresponding to the problem labels (such as the "rendering parameter error" label only triggers checkpoints such as "shadow quality" and "HDR brightness"). If the problem label has no preset checkpoints, rules are dynamically generated (such as "water surface reflection ripple ≤5Hz" is generated after analyzing the picture through GPT-4V). This can reduce the amount of comparison calculations and improve the coverage of long-tail problems.

[0038] S3. Check the frame drop images according to all the checkpoints, mark the checkpoints that fail the check as problem data, and generate a test report.

[0039] In this embodiment, checkpoints include performance checkpoints, functional checkpoints, and compatibility checkpoints. When the frame drop image does not meet the preset rules corresponding to the checkpoint, the corresponding BUG mark is performed to achieve image-level BUG detection and generate a test report to provide strong support for subsequent problem analysis and resolution.

[0040] S4. Adjusting parameters of the multimodal image analysis model according to the test report, and adjusting output results of the multimodal image analysis model using the received natural language instructions; The adjusted multimodal image analysis model is pruned and quantized to deploy the multimodal image analysis model in a lightweight manner.

[0041] In this embodiment, whenever the amount of marked test data reaches a certain level, model fine-tuning is triggered. Only the bug data marked during testing is used to adjust the parameters of the multimodal image analysis model. This optimizes the model's sensitivity to specific bugs, reducing the false alarm rate in specific scenarios. The model also uses received natural language instructions (such as "analyze the model penetration phenomenon in the image") to adjust the model output, increasing the model's flexibility and meeting user needs. The adjusted multimodal image analysis model is pruned and quantized to achieve lightweight deployment, reducing the model's storage space and computing resource usage, and improving the model's practicality and ubiquity.

[0042] The present invention also provides the following specific application scenarios: 1. Cloud gaming test: Detects the synchronization delay between the cloud-rendered screen and local operation commands. If it is detected that the screen is not updated within 200ms after the operation command is issued, it is marked as a "network synchronization bug."

[0043] 2. AR / VR application testing: Identify the offset between the helmet display image and the gyroscope data. If the head turns 10° but the image is delayed by 5°, it will be marked as "posture tracking abnormality."

[0044] 3. PC Game Testing: If screen tearing or rendering asynchrony issues are detected in multi-GPU CrossFire (SLI / CrossFire) mode, it will be marked as "Multi-GPU Compatibility Bug". If user experience issues caused by HUD elements being too small at high resolutions (4K / 8K) are identified, it will be marked as "High DPI Adaptation Anomaly".

[0045] 4. Mobile Game Testing: If a delay in model LOD (Level of Detail) switching is detected due to frequency reduction on low-end devices (such as Snapdragon 6 series chips), it will be marked as a "Performance Degradation Bug." If a conflict between full-screen gestures and game buttons is detected (such as swiping up to return causing the character to unexpectedly jump), it will be marked as an "Interaction Conflict Anomaly."

[0046] 5. Console Game Testing: If the PS5 / Xbox Series X's HDR mode is turned on and the color gamut of the cutscene is abnormal, it will be marked as "HDR metadata configuration error." If the controller vibration feedback is detected to be out of sync with the screen events (such as an explosion scene triggering vibration with a 0.5-second delay), it will be marked as "Cross-modal synchronization failure."

[0047] Please refer to Figure 4 , embodiment 2 of the present invention is: a software testing terminal 1 based on multimodal analysis, comprising a memory 2, a processor 3, and a computer program stored in the memory 2 and executable on the processor 3, wherein the processor 3 implements each step of a software testing method based on multimodal analysis of embodiment 1 when executing the computer program.

[0048] In summary, the present invention provides a software testing method and terminal based on multimodal analysis, which monitors the frame rate of the device running the software to be tested in real time, captures the dropped frame screen and its context data, reduces manual intervention, and improves monitoring efficiency; generates screen description text through the basic model, uses the target detection model to detect the coordinates of the screen elements through the anchor frame mechanism, uses the image segmentation model to identify and quantify the missing map area, and combines the output results of the three models to generate screen analysis text to ensure the comprehensiveness of the dropped frame screen analysis, and performs confidence evaluation on the screen analysis text. When the confidence is less than the preset confidence threshold, the dropped frame screen review process is triggered to improve the accuracy of recognition and analysis; processes the screen analysis text through keyword extraction and clustering technology to generate problem labels, determines checkpoints according to the problem labels, optimizes the problem diagnosis process, improves the problem location efficiency, and checks all checkpoints to ensure the comprehensiveness of the test, realizes screen-level BUG detection, marks the checkpoints that fail the inspection as problem data, generates a test report, and provides strong support for subsequent problem analysis and resolution. In addition, the parameters of the multimodal image analysis model are adjusted according to the test report, which can optimize the model performance in a targeted manner, improve the accuracy and effectiveness of the analysis, adjust the output results through the received natural language instructions, increase the flexibility of the model, meet user needs, and prune and quantize the adjusted multimodal image analysis model to achieve lightweight deployment, reduce the model's storage space and computing resource usage, and improve the model's practicality and popularity.

[0049] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's description and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A software testing method based on multimodal analysis, characterized in that: Including steps: S1. Monitoring a screen of a device running the software to be tested in real time, obtaining a dropped frame screen and its context data in the screen, and analyzing the dropped frame screen using a multimodal screen analysis model based on the context data to obtain a screen analysis text; S2. Extract keywords from the image analysis text, perform clustering on the extracted keywords, generate a question label corresponding to each cluster, and determine all checkpoints based on the question label; S3. Check the frame drop images according to all the checkpoints, mark the checkpoints that fail the check as problem data, and generate a test report.

2. A software testing method based on multimodal analysis according to claim 1, characterized in that: Monitor the screen of the device running the software under test in real time, and obtain the frame drop images and their context data in the screen, including: The frame rate of the screen of the device running the software to be tested is calculated according to the real-time monitoring situation. If the frame rate is lower than a preset frame rate threshold, it is determined to be a frame drop, and the corresponding frame drop screen and its context data are obtained.

3. A software testing method based on multimodal analysis according to claim 1, characterized in that: The multimodal image analysis model includes a basic model, a target detection model and an image segmentation model; Analyzing the dropped frame image using a multimodal image analysis model based on the context data to obtain an image analysis text, including: Generate a text instruction according to the context data, input the dropped frame image and the text instruction into the basic model to generate a picture description text; Using the object detection model to detect the coordinates of each element in the dropped frame picture through an anchor frame mechanism, determining whether there is a picture adaptation problem based on the coordinates of each element, and obtaining an analysis result of the object detection model; Using the image segmentation model to identify and quantify the missing texture area in the dropped frame image, obtaining an area ratio of the missing texture area, determining whether there is a screen rendering problem based on the area ratio of the missing texture area, and obtaining an analysis result of the image segmentation model; The picture description text, the analysis result of the target detection model and the analysis result of the image segmentation model are combined to generate a picture analysis text.

4. A software testing method based on multimodal analysis according to claim 3, characterized in that: Analyzing the dropped frame image using a multimodal image analysis model based on the context data to obtain an image analysis text further includes: Setting the output probability of the basic model, performing consistency scoring on the analysis results corresponding to the object detection model and the image segmentation model and the picture description text, and obtaining a multimodal consistency value; The area ratio of the missing region in the image segmentation model is weighted and scored according to a preset image difference calculation formula to obtain the abnormal feature strength; Corresponding weights are configured for the output probability, the multimodal consistency value and the abnormal feature strength. The confidence of the picture analysis text is calculated by combining the output probability and its weight, the multimodal consistency value and its weight, and the abnormal feature strength and its weight. If the confidence is less than the preset confidence threshold, the frame drop review process is triggered.

5. The software testing method based on multimodal analysis according to claim 1, characterized in that: The step S2 then includes: Adjusting parameters of the multimodal image analysis model according to the test report, and adjusting output results of the multimodal image analysis model using the received natural language instructions; The adjusted multimodal image analysis model is pruned and quantized to deploy the multimodal image analysis model in a lightweight manner.

6. A software testing terminal based on multimodal analysis, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: S1. Monitoring a screen of a device running the software to be tested in real time, obtaining a dropped frame screen and its context data in the screen, and analyzing the dropped frame screen using a multimodal screen analysis model based on the context data to obtain a screen analysis text; S2. Extract keywords from the image analysis text, perform clustering on the extracted keywords, generate a question label corresponding to each cluster, and determine all checkpoints based on the question label; S3. Check the frame drop images according to all the checkpoints, mark the checkpoints that fail the check as problem data, and generate a test report.

7. A software testing terminal based on multimodal analysis according to claim 6, characterized in that: Monitor the screen of the device running the software under test in real time, and obtain the frame drop images and their context data in the screen, including: The frame rate of the screen of the device running the software to be tested is calculated according to the real-time monitoring situation. If the frame rate is lower than a preset frame rate threshold, it is determined to be a frame drop, and the corresponding frame drop screen and its context data are obtained.

8. The software testing terminal based on multimodal analysis according to claim 6, characterized in that: The multimodal image analysis model includes a basic model, a target detection model and an image segmentation model; Analyzing the dropped frame image using a multimodal image analysis model based on the context data to obtain an image analysis text, including: Generate a text instruction according to the context data, input the dropped frame image and the text instruction into the basic model to generate a picture description text; Using the object detection model to detect the coordinates of each element in the dropped frame picture through an anchor frame mechanism, determining whether there is a picture adaptation problem based on the coordinates of each element, and obtaining an analysis result of the object detection model; Using the image segmentation model to identify and quantify the missing texture area in the dropped frame image, obtaining an area ratio of the missing texture area, determining whether there is a screen rendering problem based on the area ratio of the missing texture area, and obtaining an analysis result of the image segmentation model; The picture description text, the analysis result of the target detection model and the analysis result of the image segmentation model are combined to generate a picture analysis text.

9. The software testing terminal based on multimodal analysis according to claim 8, characterized in that: Analyzing the dropped frame image using a multimodal image analysis model based on the context data to obtain an image analysis text further includes: Setting the output probability of the basic model, performing consistency scoring on the analysis results corresponding to the object detection model and the image segmentation model and the picture description text, and obtaining a multimodal consistency value; The area ratio of the missing region in the image segmentation model is weighted and scored according to a preset image difference calculation formula to obtain the abnormal feature strength; Corresponding weights are configured for the output probability, the multimodal consistency value and the abnormal feature strength. The confidence of the picture analysis text is calculated by combining the output probability and its weight, the multimodal consistency value and its weight, and the abnormal feature strength and its weight. If the confidence is less than the preset confidence threshold, the frame drop review process is triggered.

10. The software testing terminal based on multimodal analysis according to claim 6, characterized in that: The step S2 then includes: Adjusting parameters of the multimodal image analysis model according to the test report, and adjusting output results of the multimodal image analysis model using the received natural language instructions; The adjusted multimodal image analysis model is pruned and quantized to deploy the multimodal image analysis model in a lightweight manner.

Images