Android all-in-one machine automatic test method and system based on multi-modal information fusion

Abstract

This application relates to the field of all-in-one machine testing, providing an automated testing method and system for Android all-in-one machines based on multimodal information fusion. The method involves configuring the testing end based on the test scenario and adjusting the acquisition posture of a robotic arm or mobile platform to automate the process of the Android all-in-one machine. Then, it performs consistency analysis and fusion judgment on multi-source data such as logs, video, and audio after aligning them according to synchronization information, outputting test conclusions and results. This application achieves higher accuracy and stability of test results by adjusting the acquisition posture of the robotic arm or mobile platform and performing time alignment and fusion judgment on multimodal information such as logs, video, and audio based on process synchronization. It solves the technical problem of insufficient accuracy and stability of test results due to the difficulty in synchronously acquiring, accurately aligning, and reliably comprehensively judging multi-source information such as system logs, screen displays, and audio output under different acquisition angles and complex test scenarios.

Description

Technical Field

[0001] This application relates to the field of all-in-one machine testing technology, and in particular to an automated testing method and system for Android all-in-one machines based on multimodal information fusion. Background Technology

[0002] Currently, in the R&D, production testing, and delivery verification of Android all-in-one devices, testers generally rely on visually observing the screen and listening to the sound, using testing tools to perform preset operations on the device under test, and judging the test results through return codes, log output, or screenshots. While this approach can perform tests, in the actual application scenarios of Android all-in-one devices, test tasks often involve multiple output forms such as screen display and speaker playback simultaneously. Moreover, the testing process has timing characteristics such as interface switching, pop-up window obstruction, and instantaneous triggering of audio prompts. Relying solely on system logs or single-modal acquisition methods is insufficient to make a stable judgment on whether the display and audio output meet expectations.

[0003] Furthermore, the relative position and angle of the acquisition device and the device under test in the engineering site or production line environment may change with the layout of the workstation or the placement of the equipment, resulting in insufficient clarity, signal-to-noise ratio and repeatability of video and audio acquisition. At the same time, the acquisition of multi-source data such as logs, videos and audios often lacks a unified process synchronization mechanism, which is prone to time alignment deviations. This makes it difficult to accurately correlate evidence fragments corresponding to the same target operation behavior, resulting in misjudgment, omission, or difficulty in anomaly location, affecting the accuracy and stability of automated test conclusions.

[0004] Therefore, existing technologies urgently need an automated testing solution that can perform process synchronization, time alignment, and consistency verification of multimodal information such as system logs, screen display video, and speaker audio under variable acquisition angle conditions, in order to improve the reliability and reproducibility of automated testing results for Android all-in-one devices. Summary of the Invention

[0005] In view of this, embodiments of this application provide an automated testing method and system for Android all-in-one machines based on multimodal information fusion, in order to solve the technical problem that when conducting automated testing of Android all-in-one machines, the existing technology is unable to synchronously collect, accurately align and reliably comprehensively judge multi-source information such as system logs, screen display and audio output under different collection angles and complex test scenarios, resulting in insufficient accuracy and stability of test results.

[0006] In a first aspect, embodiments of this application provide an automated testing method for an Android all-in-one device based on multimodal information fusion. The method is executed by a testing terminal on the Android all-in-one device to be tested, and includes: Based on the test task of the Android all-in-one device under test, the target operation behavior and the corresponding expected output state are associated and configured to obtain the test scenario configuration. The test scenario configuration includes at least the expected display screen benchmark and / or expected audio playback benchmark corresponding to the target operation behavior. According to the test scenario configuration, the robotic arm or mobile platform is controlled to adjust the acquisition posture of the acquisition device used to acquire video stream data and / or audio stream data to obtain target acquisition posture parameters, wherein the target acquisition posture parameters include at least the relative position and / or angle of the acquisition device relative to the Android all-in-one device under test. Based on the test scenario configuration and the target acquisition posture parameters, an automated test driver is executed on the Android all-in-one device under test and the process is synchronized to obtain process synchronization information. Based on the process synchronization information, system log data, video stream data, and / or audio stream data are collected and time-aligned, and consistency analysis is performed with the expected output state to obtain first evaluation information, second evaluation information, and third evaluation information. Multimodal evaluation information is then formed based on the first evaluation information, the second evaluation information, and the third evaluation information. Based on the multimodal evaluation information, a fusion judgment is performed to obtain the test conclusion and output the test result information.

[0007] Preferably, the step of executing automated test drive and synchronizing the process on the Android all-in-one device under test according to the test scenario configuration and the target acquisition posture parameters to obtain process synchronization information includes: During the execution of the target operation, the execution sequence of the target operation is marked to obtain execution sequence marking information; Based on the execution timing marker information, generate alignment index information corresponding to the system log data, the video stream data, and the audio stream data; The system log data, video stream data, and audio stream data are collected and time-aligned based on the alignment index information.

[0008] Preferably, the consistency analysis includes: Based on the alignment index information, the log alignment segment corresponding to the target operation behavior is determined, and the preset log events in the log alignment segment are matched to obtain the log consistency matching result. Based on the log consistency matching results, a first evaluation message representing an abnormal running status is generated.

[0009] Preferably, the consistency analysis further includes: Based on the alignment index information, extract the set of screen alignment frames corresponding to the target operation behavior from the video stream data; Based on the expected display frame reference, a template similarity calculation is performed on the set of aligned frames to obtain the template similarity result; When the template similarity result does not meet the preset similarity conditions, a second evaluation message representing abnormal displayed content is generated.

[0010] Preferably, the visual consistency analysis further includes: During the template similarity calculation of the set of aligned frames, display defect detection is performed on the set of aligned frames to obtain display defect detection results; The display defect detection includes at least color anomaly detection based on color histogram, screen flickering or ghosting detection based on edge features or texture features, and black screen or white screen detection based on brightness features. The defect detection results are incorporated into the second evaluation information.

[0011] Preferably, the consistency analysis further includes: Based on the alignment index information, determine the audio alignment segment corresponding to the target operation behavior; The average energy characteristics of the audio alignment segment are calculated to obtain the energy determination result; When the energy determination result meets the preset silent abnormality condition, a third evaluation information characterizing the silent abnormality is generated.

[0012] Preferably, the acoustic consistency analysis further includes: The spectral characteristics of the audio alignment segment are calculated to obtain the spectral determination result; When the spectrum determination result meets the preset spectrum anomaly conditions, third evaluation information characterizing distortion or noise anomalies is generated; and, Based on the expected playback audio benchmark, a content consistency comparison is performed on the audio alignment segment to obtain the content comparison result; The content consistency comparison is implemented based on at least one of speech recognition or voiceprint matching, and a third evaluation information representing the inconsistency of the played content is generated when the content comparison result does not meet the preset consistency conditions.

[0013] Preferably, the process of forming multimodal evaluation information further includes: Generate confidence parameters for the first evaluation information, the second evaluation information, and the third evaluation information, respectively; Based on the alignment index information, an alignment time window corresponding to the same target operation behavior is determined, and within the alignment time window, the first evaluation information, the second evaluation information, and the third evaluation information are constrained based on the confidence parameter to obtain multimodal evaluation information for fusion judgment. The test conclusion is output based on the multimodal evaluation information.

[0014] Preferably, the fusion determination further includes generating evidence association information while outputting test result information. The evidence association information is used to associate the log fragments, video frame images, and audio fragments corresponding to the anomaly with the alignment index information; wherein, generating the evidence association information includes: The alignment index information is used to determine the target operation behavior identifier corresponding to the anomaly and the alignment time window corresponding to the target operation behavior identifier. According to the alignment time window, target log segments are extracted from the system log data, target video frame sets are extracted from the video stream data, and target audio segments are extracted from the audio stream data. Based on the confidence parameter, the target log segment, the target video frame set, and the target audio segment are labeled with evidence credibility to obtain evidence credibility results. The evidence credibility label is used at least to characterize the reliability of the association between the target log segment, the target video frame set, and the target audio segment and the anomaly. The target operation behavior identifier, the alignment time window, the target log segment, the target video frame set, the target audio segment, and the evidence credibility result are associated and encapsulated to obtain the evidence association information, and the evidence association information is bound to the test result information for output.

[0015] Secondly, embodiments of this application provide an automated testing system for an Android all-in-one machine based on multimodal information fusion, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method described in any of the above.

[0016] Beneficial effects: Compared with existing technologies, the automated testing method for Android all-in-one devices based on multimodal information fusion provided in this application embodiment is executed by the testing terminal on the Android all-in-one device under test. The method includes: configuring a test scenario configuration by associating the target operation behavior with the corresponding expected output state according to the test task of the Android all-in-one device under test; and adjusting the acquisition posture of the acquisition device used to acquire video stream data and / or audio stream data according to the test scenario configuration to obtain target acquisition posture parameters, wherein the target acquisition posture parameters include at least the expected display screen reference and / or expected audio playback reference corresponding to the target operation behavior. The method involves determining the relative position and / or angle of the acquisition device relative to the Android all-in-one device under test; executing automated test driving and process synchronization on the Android all-in-one device under test based on the test scenario configuration and the target acquisition posture parameters to obtain process synchronization information; acquiring and time-aligning system log data, video stream data, and / or audio stream data based on the process synchronization information, and performing consistency analysis with the expected output state to obtain first evaluation information, second evaluation information, and third evaluation information; forming multimodal evaluation information based on the first evaluation information, second evaluation information, and third evaluation information; performing fusion judgment based on the multimodal evaluation information to obtain test conclusions and output test result information. This method adjusts the acquisition posture through a robotic arm or mobile platform and performs time alignment and fusion judgment on multimodal information such as logs, videos, and audio based on process synchronization, achieving higher accuracy, stability, and repeatability of automated test results for Android all-in-one devices.

[0017] Compared with the prior art, the Android all-in-one device automated testing system based on multimodal information fusion provided in this application includes at least one processor, at least one memory, and computer program instructions stored in the memory. When executed by the processor, it implements the Android all-in-one device automated testing method based on multimodal information fusion as described above. It is understood that this Android all-in-one device automated testing system based on multimodal information fusion can possess all the technical features and beneficial effects of the aforementioned Android all-in-one device automated testing method based on multimodal information fusion, which will not be elaborated further here. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, and these are all within the protection scope of this application.

[0019] Figure 1 A schematic diagram of the overall process of an automated testing method for an Android all-in-one device based on multimodal information fusion provided in an embodiment of this application; Figure 2 A schematic diagram of the overall process of an automated testing method for an Android all-in-one device based on multimodal information fusion, provided in another embodiment of this application; Figure 3 A schematic diagram of the overall process of an automated testing method for an Android all-in-one device based on multimodal information fusion, provided in another embodiment of this application; Figure 4 A schematic diagram of the overall process of an automated testing method for an Android all-in-one device based on multimodal information fusion, provided in another embodiment of this application; Figure 6 A schematic diagram of the overall process of an automated testing method for an Android all-in-one device based on multimodal information fusion, provided in another embodiment of this application; Figure 7 A schematic diagram of the overall process of an automated testing method for an Android all-in-one device based on multimodal information fusion, provided in another embodiment of this application; Figure 8 A schematic diagram of the overall process of an automated testing method for an Android all-in-one device based on multimodal information fusion, provided in another embodiment of this application; Figure 9 A schematic diagram of the overall process of an automated testing method for an Android all-in-one device based on multimodal information fusion, provided in another embodiment of this application; Figure 10 This is a schematic diagram of the structure of an automated testing system for an Android all-in-one device based on multimodal information fusion, provided as an embodiment of this application. Detailed Implementation

[0020] The features and exemplary embodiments of various aspects of this application will now be described in detail. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only configured to explain this application and are not configured to limit this application. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples of this application.

[0021] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0022] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and regulations of the locality and with authorization from the owner of the relevant device.

[0023] Firstly, please see Figures 1 to 9 This application provides an automated testing method for Android all-in-one devices based on multimodal information fusion. The method is executed by the testing terminal on the Android all-in-one device to be tested, and includes: S1: Based on the test task of the Android all-in-one device under test, the target operation behavior and the corresponding expected output state are associated and configured to obtain the test scenario configuration. The test scenario configuration includes at least the expected display screen benchmark and / or expected audio playback benchmark corresponding to the target operation behavior. S2: According to the test scenario configuration, control the robotic arm or mobile platform to adjust the acquisition posture of the acquisition device used to acquire video stream data and / or audio stream data to obtain target acquisition posture parameters, wherein the target acquisition posture parameters include at least the relative position and / or angle of the acquisition device relative to the Android all-in-one device under test; S3: Based on the test scenario configuration and the target acquisition posture parameters, execute automated test driver on the Android all-in-one device under test and perform process synchronization to obtain process synchronization information; S4: Based on the process synchronization information, collect and time-align the system log data, the video stream data, and / or the audio stream data, and perform consistency analysis with the expected output state to obtain first evaluation information, second evaluation information, and third evaluation information, and form multimodal evaluation information based on the first evaluation information, the second evaluation information, and the third evaluation information; S5: Perform fusion judgment based on the multimodal evaluation information, obtain test conclusions, and output test result information.

[0024] It should be noted that Android all-in-one devices refer to all-in-one devices with the Android operating system, such as smart advertising machines, smart conference tablets, and self-service kiosks, which are widely used in various scenarios. Before leaving the factory or after repair, Android all-in-one devices need to undergo rigorous testing of their screen display, sound playback system, and system functions.

[0025] In this embodiment, the test task is first decomposed into target operation behaviors and bound to the expected display screen benchmark and / or expected audio playback benchmark to form a reusable test scenario configuration. Then, the relative position and / or angle of the acquisition device is adjusted by a robotic arm or mobile platform to ensure that the video and audio acquisition posture is controllable under different work positions or placement conditions. Subsequently, process synchronization information is generated during the automated test drive process, and based on this, system logs, video streams, and audio streams are collected, time-aligned, and consistency analyzed. Three types of evaluation information are output and multimodal evaluation information is formed. Finally, the test conclusion and result output are given through fusion judgment, which solves the problem of unstable conclusions caused by the difficulty in synchronizing, aligning, and comprehensively judging multi-source information.

[0026] In the above embodiments, automated testing tasks typically involve multiple operations and output formats, lacking a unified expected benchmark, which easily leads to inconsistent judgment criteria. In complex scenarios, reliance on human experience can result in misjudgments and omissions. As described in step S1, based on the test task of the Android all-in-one device under test, the target operation behavior is associated with the corresponding expected output state to obtain a test scenario configuration. This test scenario configuration at least fixes the expected display screen benchmark and / or the expected audio playback benchmark, allowing subsequent analysis to use this benchmark as a reference for consistency verification. Step S1 establishes test execution and result judgment on a reusable benchmark configuration, improving the consistency and reproducibility of test judgments and reducing the drift between different test cases.

[0027] Specifically, the testing end receives test task input, which can correspond to a set of test cases or test steps. The testing end parses the test task into a set of target operation behaviors, which can be represented as a sequence of operation behaviors, each corresponding to an executable action or group of actions. The testing end configures the expected output state for each target operation behavior, including at least the expected display screen baseline and / or the expected audio playback baseline. The testing end writes the correspondence between target operation behaviors and expected output states into the test scenario configuration, forming structured configuration data that can be called by subsequent modules.

[0028] In the above embodiments, different workstation layouts, equipment placements, and changes in viewing angles can lead to fluctuations in the quality of external video and audio acquisition, causing data instability for the same use case under different environments, further affecting the stability of the judgment. As described in step S1, based on the test scenario configuration, the robotic arm or mobile platform is controlled to adjust the acquisition posture of the acquisition device to obtain the target acquisition posture parameters. These target acquisition posture parameters include at least the relative position and / or angle of the acquisition device relative to the Android all-in-one device under test, enabling the acquisition conditions to adapt to the use case and the on-site conditions. Step S1 makes the acquisition input more stable and repeatable, reducing the problems of image loss or sound attenuation caused by occlusion, angle deviation, and distance changes, providing a more reliable data foundation for subsequent multimodal alignment and fusion judgment.

[0029] Specifically, the test involves using acquisition devices, such as an external camera, to capture the screen video stream of the all-in-one device, and an external audio acquisition device to capture the audio stream played by the speakers. The test end drives a robotic arm or mobile platform via a control interface. The robotic arm or mobile platform then adjusts the spatial position or angle of the external camera, which is used to acquire video and / or audio stream data. The test end reads the test case information or target operation behavior set from the test scenario configuration to determine the required acquisition viewpoint for the current test case or operation behavior to be executed. The test end issues a posture adjustment command to the robotic arm or mobile platform, which includes parameter constraints or a description of the target posture. The robotic arm or mobile platform executes motion control to adjust the relative position and / or angle of the external camera relative to the Android all-in-one device under test. The test end acquires and records the adjusted target acquisition posture parameters, which include at least the relative position and / or angle of the external camera relative to the Android all-in-one device under test, and uses these parameters as associated parameters for subsequent acquisition in the later stages of the process.

[0030] In the above embodiments, it should be noted that logs, videos, and audios typically come from different channels. Without a unified execution timing reference, it is difficult to accurately correspond to the same target operation behavior, easily leading to cross-modal mismatches and difficulty in localization. As described in step S3, based on the test scenario configuration and target acquisition posture parameters, the Android all-in-one device under test is executed with automated test drive and process synchronization is performed to obtain process synchronization information, ensuring that the execution process of the test drive has a traceable synchronization benchmark. Through step S3, a unified timing framework can be provided for subsequent multi-source data acquisition and alignment, enabling evidence from various modalities to be aggregated around the same operation behavior, reducing misalignment caused by sampling delays or buffering.

[0031] Specifically, the test client executes automated test drivers on the Android all-in-one device under test via a debug connection or control interface. Simultaneously, the test client generates process synchronization information to mark the execution sequence of operational behaviors. The test client loads the test scenario configuration and target acquisition posture parameters, entering the test case execution phase. The test client starts the automated test driver, issuing operation commands to the Android all-in-one device under test sequentially according to the target operation behavior sequence, driving the device to complete corresponding interface switching, function triggering, or playback actions. When each target operation behavior is triggered, the test client marks the operation behavior with a time sequence, forming process synchronization information. This process synchronization information can include the operation behavior identifier and corresponding execution time information, used as a reference for subsequent alignment. In implementations requiring stronger alignment capabilities, the test client further generates alignment index information based on the time sequence markings, associating the alignment index information with subsequently collected system log data, video stream data, and audio stream data, ensuring that all three types of data can be mapped to the same operation behavior time sequence framework.

[0032] In the above embodiments, single-dimensional detection is easily affected by noise, occlusion, or occasional anomalies, and if multi-source data is not aligned, it is difficult to form a reliable chain of evidence, leading to unstable comprehensive judgment. As described in step S4, system log data, video stream data, and / or audio stream data are collected and time-aligned based on process synchronization information, and consistency analysis is performed with the expected output state to obtain first evaluation information, second evaluation information, and third evaluation information. Then, multimodal evaluation information is formed based on the three types of evaluation information. Through step S4, the original multi-source data can be transformed into an aligned, comparable, and fusionable structured evaluation result, enabling logs, images, and sounds to cross-verify the same operation behavior, reducing the probability of single-modal misjudgment, and improving the accuracy and coverage of anomaly identification.

[0033] Specifically, the test server initiates the acquisition and analysis process for three data channels: system log channel, video channel, and audio channel, driving time alignment with process synchronization information or alignment index information. After alignment, consistency analysis is performed to generate first, second, and third evaluation information, which are then aggregated into multimodal evaluation information. During test case execution, the test server collects system log data, obtaining log streams related to the runtime status through log capture. The test server acquires video stream data via acquisition devices, obtaining continuous image data of the screen display status. The test server also acquires audio stream data via acquisition devices, obtaining continuous audio data of the speaker output status. The test server uses process synchronization information or alignment index information to establish a unified alignment benchmark for log, video, and audio data. For each target operation, the test server determines the alignment time window based on the alignment index information and locates the corresponding data segment or data frame set in the three types of data, thereby completing the time alignment of the three channels of data on the same operation dimension. The testing end extracts log segments corresponding to the target operation behavior from the aligned system log data, matches and verifies these log segments against the expected output state or preset rules, and outputs first evaluation information indicating whether the operating state is abnormal. The testing end extracts screen frames or sets of frames from the aligned video stream data, performs similarity calculations or template comparisons using the expected display frame as a reference, obtains display consistency results, and generates second evaluation information accordingly. The testing end extracts audio segments from the aligned audio stream data, calculates acoustic features such as energy or spectrum for these segments, performs consistency analysis with the expected playback audio frame or threshold rules, and outputs third evaluation information. The testing end aggregates the first, second, and third evaluation information according to the target operation behavior or aligned time window to form multimodal evaluation information, which serves as the direct input for fusion judgment.

[0034] In the above embodiments, it should be noted that existing automated testing often only provides a pass or fail result, lacking a comprehensive decision-making mechanism based on multi-source evidence, leading to inaccurate and unstable test results. As described in step S5, a fusion judgment is performed based on multimodal evaluation information to obtain test conclusions and output test result information, achieving a comprehensive decision-making output based on log, visual, and acoustic evaluation results. The test conclusions obtained through step S5 are supported by multiple pieces of evidence, have stronger anti-interference capabilities, can maintain higher accuracy and stability in complex scenarios, and improve the traceability and verifiability of test results.

[0035] Specifically, the testing end receives multimodal evaluation information, which includes first, second, and third evaluation information corresponding to each target operation behavior. The testing end performs a fusion judgment on the multimodal evaluation results for the same target operation behavior or within the same aligned time window, integrating the multi-source evaluation results to generate a single test conclusion. The fusion judgment can be reflected in the consistency verification and comprehensive decision-making of multiple evaluation information, ensuring that the conclusion does not depend on a single channel. The testing end generates test result information, which at least includes the test conclusion and may further include the operation behavior identifier or time window index corresponding to the anomaly for subsequent traceability and review.

[0036] Please see Figures 1 to 9 In one embodiment, the step of executing automated test driving and synchronizing the process on the Android all-in-one device under test according to the test scenario configuration and the target acquisition posture parameters to obtain process synchronization information includes: During the execution of the target operation sequence, the execution timing of the target operation sequence is marked to obtain execution timing marking information; Based on the execution timing marker information, generate alignment index information corresponding to the system log data, the video stream data, and the audio stream data; The system log data, video stream data, and audio stream data are collected and time-aligned based on the alignment index information.

[0037] In this embodiment, the control and communication module of the test terminal, together with the automated test script, marks the execution sequence and generates alignment index information during the execution of the target operation behavior sequence. The test terminal drives the Android all-in-one device under test to execute preset test cases through the debugging connection, and simultaneously captures the logcat system log stream.

[0038] Specifically, the test terminal reads the test scenario matrix or equivalent test scenario configuration to obtain the target operation behavior sequence and its order. The test terminal sequentially issues control commands to the Android all-in-one device under test to execute each target operation behavior, while simultaneously generating a timing marker record at the trigger time or duration of each target operation behavior. This record contains at least the target operation behavior identifier and its corresponding time information, ensuring that each operation behavior has a traceable temporal position during the test. Based on the aforementioned timing markers, the test terminal generates alignment index information. The alignment index information is essentially a mapping relationship used to associate the same target operation behavior with system log data, video stream data, and audio stream data in the time dimension, enabling subsequent location of the corresponding segments of the three types of data according to the target operation behavior. During script execution, the test terminal continuously and synchronously captures the logcat system log stream, forming a log data stream with time information. External image acquisition devices, such as external cameras, continuously capture screen video streams, forming a continuous video frame sequence. External audio acquisition devices continuously capture audio streams played from speakers, forming continuous audio data. The test server reads the alignment index information, determines the corresponding alignment time window or equivalent positioning condition based on the target operation behavior, and locates the log record set, video frame set, and audio segment falling within the time window in the three types of raw data. The test server then links the above three positioning results to the same target operation behavior identifier, forming an aligned three-element data structure. For example, a target operation behavior corresponds to a log segment, a set of video frames, and an audio segment. This achieves time alignment of system log data, video stream data, and audio stream data, enabling direct comparison and joint analysis of the three types of evidence along the same target operation behavior dimension. The aligned results can directly support the association of abnormal time point evidence mentioned in the material, that is, when an anomaly is subsequently determined, the corresponding log segment, video frame image, and audio segment can be output according to the same aligned time window for report presentation and review.

[0039] Please see Figures 1 to 9 In one embodiment, the consistency analysis includes: Based on the alignment index information, the log alignment segment corresponding to the target operation behavior is determined, and the preset log events in the log alignment segment are matched to obtain the log consistency matching result. Based on the log consistency matching results, a first evaluation message representing an abnormal running status is generated.

[0040] It should be noted that without alignment and localization, a large amount of system printouts and background information in the logs will be mixed with the target operation behavior, which may easily lead to situations where an anomaly occurs at a certain step but cannot be accurately attributed to that step, thus affecting the stability of fault localization and judgment.

[0041] In this embodiment, the control module, communication module, and log analysis module in the test terminal work together. The control module and communication module are responsible for establishing a connection with the Android all-in-one device under test and synchronously capturing the system log stream, while the log analysis module is responsible for subsequent location and analysis of the log stream. Specifically, the test terminal continuously captures the logcat system log stream during the execution of the automated test script, forming system log data with time sequence. Simultaneously, the test terminal marks the execution time sequence of the target operation behavior sequence and generates alignment index information to characterize the time correspondence between the target operation behavior and multimodal data. Here, the alignment index information serves as input. The log analysis module searches for the corresponding alignment time window or equivalent location range according to the target operation behavior identifier, and then extracts the set of log records falling within this time range from the system log data to obtain the log fragment corresponding to the target operation behavior. This log fragment serves as the analysis object for the next step of matching preset log events. Through the above steps, system log data can be segmented and archived according to the granularity of target operation behavior. Log analysis is transformed from full scanning to step-by-step focusing, reducing noise log interference, improving the pertinence and traceability of anomaly location, and providing alignable log evidence input for subsequent multimodal fusion.

[0042] It is further important to note that relying solely on visual or audio output for judgment cannot cover faults at the system's underlying and application layers. Some anomalies may not be immediately apparent at the output layer, but they will manifest as errors, crashes, exception stacks, or critical exception patterns in the system logs. Without consistent log analysis, system-level and application-level errors are easily missed, leading to insufficient test coverage.

[0043] In the above embodiments, the log analysis module performs real-time analysis on the system log stream or the aforementioned log fragments, identifies system-level and application-level errors based on preset keyword or anomaly pattern matching rules, and generates a first judgment result characterizing the anomaly, corresponding to the first evaluation information in the claims. Specifically, the log analysis module receives the log fragments output in step one and reads a preset log event set. The preset log events can be represented as a keyword set or anomaly pattern matching rule set, used to cover the log formats of common system errors and application errors. The log analysis module performs matching operations on each log fragment to obtain matching results, and generates first evaluation information based on the matching results. The first evaluation information is at least used to characterize the existence or absence of an abnormal operating state, and can further carry information such as the anomaly type or the time point of the anomaly occurrence, so as to associate the log fragments with the anomaly time point in subsequent reports. Through the above steps, the aligned log fragments are then subjected to keyword or anomaly pattern matching, which can more accurately identify the source of the error in the context corresponding to the target operation behavior, avoid false alarms caused by global scanning, and improve the ability to detect underlying system stability problems.

[0044] Please see Figures 1 to 9 In one embodiment, the consistency analysis further includes: Based on the alignment index information, extract the set of screen alignment frames corresponding to the target operation behavior from the video stream data; Based on the expected display frame reference, a template similarity calculation is performed on the set of aligned frames to obtain the template similarity result; When the template similarity result does not meet the preset similarity conditions, a second evaluation message representing abnormal displayed content is generated.

[0045] It should be noted that if frames are not aligned with the process synchronization information, residual frames from the previous screen, transition animation frames, or frames from irrelevant time periods may be included in the analysis, leading to misjudgments or inaccurate anomaly localization. Furthermore, in existing complex test scenarios, the content of the screen changes frequently, making manual verification inefficient and highly subjective. Additionally, slight distortions or brightness variations exist in the image when captured from different angles, necessitating quantitative comparison methods to maintain consistent judgments.

[0046] In this embodiment, this can be achieved by a template matching unit within the image analysis module. The image analysis module receives screen frames or screen area images extracted by an external camera device, and simultaneously reads the expected display screen reference from the test scene configuration. The template matching unit performs a similarity comparison operation on each frame of screen area image and the standard template image, outputting the similarity value or similarity sequence for each frame as direct input for subsequent threshold determination. Through the above steps, the determination of the correctness of the display content can be transformed into a quantifiable and reproducible similarity index, ensuring that the image verification under different batches, different workstations, and different acquisition angles maintains the same judgment standard, improving objectivity and consistency.

[0047] In the above embodiments, the image analysis module can perform threshold determination and output a second determination result, which corresponds to second evaluation information. The similarity value obtained from template similarity calculation enters the threshold comparison logic. The image analysis module reads a preset threshold and compares the similarity with the threshold; when the similarity is lower than the threshold, it generates second evaluation information representing an anomaly in the displayed content. This second evaluation information can be bound to the target operation behavior identifier and the alignment time window as input for subsequent multimodal fusion and report association. Through the above steps, when the displayed content is inconsistent, an anomaly marker can be automatically triggered and the second evaluation information can be output, providing a strong evidence channel for fusion determination and reducing the probability of missed detection; at the same time, binding the anomaly with the alignment time window facilitates accurate location of the step and time of the anomaly in subsequent reports, improving the traceability of test results.

[0048] Please see Figures 1 to 9 In one embodiment, the visual consistency analysis further includes: During the template similarity calculation of the set of aligned frames, display defect detection is performed on the set of aligned frames to obtain display defect detection results; The display defect detection includes at least color anomaly detection based on color histogram, screen flickering or ghosting detection based on edge features or texture features, and black screen or white screen detection based on brightness features. The defect detection results are incorporated into the second evaluation information.

[0049] It should be noted that template similarity comparison primarily captures the correctness of content, but it does not adequately cover display defects at the physical or signal level. This can easily lead to situations where content similarity is acceptable, but issues such as color cast, screen distortion, ghosting, or black-and-white displays go undetected. Furthermore, defects at the structural and textural level, such as screen distortion and ghosting, are unstable and easily affected by the shooting angle, making them difficult to determine manually or with a single indicator. In particular, ghosting can overlap with content changes, making template comparison alone prone to misjudgment or omission.

[0050] In this embodiment, the initial input is the current screen area image or a screen frame. The defect detection unit calculates the color histogram features of the frame to obtain statistical results characterizing the color distribution of the frame. Then, this color distribution is compared with preset normal color distribution rules, using the degree of color distribution shift or abnormal concentration as criteria. If the criteria meet the abnormal conditions, a color anomaly detection result is output, marking the presence of color shift or color block defects, and a defect type label can be attached for subsequent inclusion in the second evaluation information. Through the above steps, anomalies at the display color level can be captured through the statistical characteristics of color distribution, enabling consistent identification of color defects even under different angles and brightness conditions, thus improving the visual anomaly coverage and detection rate.

[0051] In the above embodiments, the input is an aligned screen area image frame or set of frames. The defect detection unit extracts edge or texture features from each frame to obtain feature results reflecting the local structure and texture regularity of the image. Based on preset rules, it determines whether there are abnormal patterns such as abnormal edge density, abnormal texture repetition, or increased local random noise, and outputs the detection results of screen tearing or ghosting. When frame set processing is used, edge or texture features can be compared between adjacent frames. If persistent abnormalities or abnormal patterns persist over time, the reliability of screen tearing or ghosting detection is enhanced, and corresponding defect markers are formed. Through the above steps, defects such as screen tearing and ghosting can be transformed into calculable edge and texture abnormal patterns, enabling visual judgment to remain stable under non-contact, automated acquisition conditions, reducing reliance on manual visual inspection, and improving the timeliness and consistency of anomaly detection.

[0052] In the above embodiment, the input is a screen area image frame. The defect detection unit calculates the brightness features of this frame to obtain the overall brightness level or brightness distribution. The brightness features are compared with a preset threshold or rule. If the brightness is consistently below the threshold, it is marked as a black screen tendency; if the brightness is consistently above the threshold, it is marked as a white screen tendency. The black screen or white screen detection result is output and can be bound to an alignment time window to pinpoint the target operation during which it occurred. Through the above steps, black and white screen anomaly prompts can be quickly given in complex scenarios, serving as strong visual evidence to supplement the second evaluation information, thereby improving the stability and reliability of the comprehensive judgment.

[0053] In the above embodiments, the color anomaly, screen flickering, and black-and-white screen detection results output by the defect detection unit are combined with the template similarity comparison results and incorporated into the second evaluation information. The second evaluation information can be organized according to the target operation behavior or aligned with the time window, so that the subsequent fusion stage can directly include any visual anomaly in the comprehensive test report when it occurs, and can further associate it with the corresponding video frame image.

[0054] Please see Figures 1 to 9 In one embodiment, the consistency analysis further includes: Based on the alignment index information, determine the audio alignment segment corresponding to the target operation behavior; The average energy characteristics of the audio alignment segment are calculated to obtain the energy determination result; When the energy determination result meets the preset silent abnormality condition, a third evaluation information characterizing the silent abnormality is generated.

[0055] It should be noted that in the current technology, relying solely on the ADB (Android Debug Bridge) function for traversal and screenshot analysis is insufficient to detect audio problems and cannot provide non-contact audio testing.

[0056] In the above embodiments, the multimodal acquisition module includes an external audio acquisition device, such as a microphone, for capturing the audio stream. The audio analysis module processes and analyzes the audio stream, detects audio anomalies, and generates a third judgment result. Specifically, the external audio acquisition device continuously acquires the audio stream played by the speaker of the Android all-in-one device under test during the execution of the test script, forming audio stream data input. The audio analysis module receives process synchronization information or alignment index information to determine the alignment time window corresponding to the current target operation behavior, thereby extracting the target audio segment within the time window from the continuous audio stream to avoid irrelevant segments interfering with subsequent calculations. The audio analysis module performs average energy calculation on the target audio segment to obtain the average energy feature value, which serves as the quantitative basis for subsequent silence judgment. Through the above steps, silence judgment can be transformed from human auditory perception into a repeatable quantitative feature, enabling automated testing to stably output consistent detection results in long-term, batch testing scenarios, improving testing efficiency and consistency.

[0057] In the above embodiments, the audio analysis module can execute the threshold determination logic and output a third determination result, which corresponds to third evaluation information. The audio analysis module reads a preset silence threshold as a determination condition. This threshold can be preset at the test end and used as a rule parameter for acoustic consistency analysis. The audio analysis module compares the average energy characteristics with the silence threshold. That is, within the alignment time window, the silence anomaly output is triggered only when the average energy remains below the silence threshold for a continuous period of time, avoiding false alarms caused by instantaneous fluctuations. When the condition of being continuously below the threshold is met, third evaluation information characterizing the silence anomaly is generated. This third evaluation information can be bound to the target operation behavior identifier and the alignment time window for easy subsequent fusion determination and report traceability. Through the above steps, without relying on manual listening, anomalies such as speaker silence, audio link disconnection, or playback failure can be reliably detected, and these anomalies can be incorporated into multimodal fusion as third evaluation information, thereby improving the reliability and coverage of the comprehensive determination.

[0058] Please see Figures 1 to 9 In one embodiment, the acoustic consistency analysis further includes: The spectral characteristics of the audio alignment segment are calculated to obtain the spectral determination result; When the spectrum determination result meets the preset spectrum anomaly conditions, third evaluation information characterizing distortion or noise anomalies is generated; and, Based on the expected playback audio benchmark, a content consistency comparison is performed on the audio alignment segment to obtain the content comparison result; The content consistency comparison is implemented based on at least one of speech recognition or voiceprint matching, and a third evaluation information representing the inconsistency of the played content is generated when the content comparison result does not meet the preset consistency conditions.

[0059] It should be noted that in practical applications, average energy alone is insufficient to identify distortion or noise. Average energy is more suitable for determining silence, but distortion and noise often manifest as abnormal energy peaks or distributions in certain frequency bands. Relying solely on the average energy may still lead to a judgment that everything is normal, thus easily resulting in missed detections. Furthermore, distortion and noise may only appear in certain test cases or during a specific playback phase. Without continuous judgment and alignment time window constraints, false alarms or failure to attribute the cause to specific operational behaviors are likely to occur.

[0060] In the above embodiment, the test end captures the audio stream played by the speaker through an external audio acquisition device, obtaining audio stream data input. The audio analysis module receives the audio stream and, in conjunction with process synchronization information or alignment index information, first locates the alignment time window corresponding to a certain target operation behavior. It then extracts the target audio segment within this time window from the continuous audio stream, avoiding including irrelevant segments in the spectrum analysis. Subsequently, it performs spectrum analysis on the target audio segment to obtain spectrum feature results, which characterize the energy distribution at different frequencies. By transforming audio quality anomalies from subjective listening perception into quantifiable features in the frequency domain, the test system can reliably detect distortion and noise defects, improving the coverage and stability of automated audio testing. The audio analysis module receives the spectrum feature results and reads the preset frequency band and abnormal peak judgment rules. The audio analysis module detects the presence of abnormal peaks within the preset frequency band and judges their persistence. That is, abnormal peaks only trigger distortion or noise anomaly output if they continuously appear within the alignment time window, avoiding false alarms caused by instantaneous spikes. When the conditions are met, third evaluation information characterizing distortion or noise anomalies is generated. This third evaluation information can be bound to the target operation behavior identifier and the alignment time window, which facilitates subsequent fusion judgment and association of abnormal audio segments in the report.

[0061] Furthermore, in practical applications, simply detecting silence and noise may not guarantee the accuracy of the playback content. For example, the device may play an error message, play other program sources, or play in the wrong order. In these cases, the average energy and spectrum may appear normal, but the content itself may be incorrect. Without content comparison, these errors will be missed.

[0062] In the above embodiments, after the audio analysis module extracts the target audio segment within the alignment time window, it reads the corresponding expected playback audio benchmark and compares the two with the content consistency comparison algorithm to obtain the content comparison result. Specifically, the audio analysis module performs speech recognition on the target audio segment to obtain the recognized text, and then performs a consistency judgment with the expected text or content identifier corresponding to the expected playback audio benchmark; alternatively, the audio analysis module extracts the voiceprint features of the target audio segment and matches them with the voiceprint features corresponding to the expected playback audio benchmark to determine whether they are the same expected sound source or the same expected content. Through the above steps, the audio testing can be elevated from whether there is sound or whether the sound is clean to whether the playback content is correct, thereby expanding the testing coverage of the audio dimension.

[0063] In the above embodiments, the audio analysis module receives the content comparison results and reads the preset consistency conditions. When the content comparison results do not meet the consistency conditions, it generates third evaluation information indicating inconsistency in the played content. This third evaluation information can also be bound to the target operation behavior identifier and the alignment time window, and associated with the corresponding abnormal audio segments as evidence output when generating the comprehensive test report. This achieves automated verification at the audio content level, reduces false positives and false negatives, improves the interpretability and traceability of test results, and is integrated with logs and screen judgments for fusion judgment, thereby improving the reliability of the overall comprehensive test report.

[0064] Please see Figures 1 to 9 In one embodiment, the process of forming multimodal evaluation information further includes: Generate confidence parameters for the first evaluation information, the second evaluation information, and the third evaluation information, respectively; Based on the alignment index information, an alignment time window corresponding to the same target operation behavior is determined, and within the alignment time window, the first evaluation information, the second evaluation information, and the third evaluation information are constrained based on the confidence parameter to obtain multimodal evaluation information for fusion judgment. The test conclusion is output based on the multimodal evaluation information.

[0065] In this embodiment, each analysis module can output its quantifiable judgment values, which are then aggregated by the data fusion and report generation module to form confidence parameters. These confidence parameters include template similarity confidence, spectral anomaly confidence, and log consistency confidence. Specifically, the template similarity confidence comes from the template similarity comparison results of the image analysis module; the spectral anomaly confidence comes from the spectral analysis and abnormal peak determination process of the audio analysis module; and the log consistency confidence comes from the matching results of log keywords or abnormal patterns by the log analysis module. Through these steps, the quantified results can be encapsulated into multimodal evaluation information using unified fields, forming a set of confidence parameters corresponding to the first, second, and third evaluation information, for use in fusion and judgment.

[0066] Specifically, in the above embodiments, the control and communication module can drive the execution of the test script and capture the log stream, while an external acquisition device captures the video and audio streams. These three data streams are generated during the execution of the same test case, and the fusion module uses this data to establish a time range for the same target operation behavior. The test end generates a sequence of target operation behaviors based on the test case execution sequence, and the log stream, video stream, and audio stream are generated synchronously under this sequence. The fusion module maps the start and end times of the target operation behavior in the script or their corresponding markers to alignment index information, and determines the alignment time window accordingly. This window is used to extract a set of data segments corresponding to the same behavior from the log, video, and audio, ensuring that subsequent credibility constraints are performed at the same operation behavior scale.

[0067] Specifically, within the alignment time window, the fusion module reads the confidence parameters from the multimodal evaluation information and applies confidence filtering or weighted constraints to the evaluation results of each modality. For example, if the template similarity is close to a threshold and the screen area image is easily affected by viewing angle, the effectiveness of the second evaluation information in the fusion is reduced. If the abnormal peak in the spectrum is not persistent or only appears briefly, the abnormal audio distortion is marked as low confidence to avoid transient interference triggering failure. If the log keyword hit is weakly correlated or an occasional prompt, the first evaluation information is processed with low confidence to avoid irrelevant alarms leading to misjudgment. Through the above steps, boundary anomalies of a single modality can be distinguished from real faults, reducing misjudgments caused by transient audio spikes due to changes in viewing angle and noise during acquisition, making the fusion conclusion more reliable.

[0068] In the above embodiments, the data fusion and report generation module can output conclusions and generate a comprehensive test report. After completing the credibility constraints within the aligned time window, the fusion module checks whether the preset credibility conditions are met. For example, test case failure is triggered only when an anomaly judgment result occurs and its confidence level reaches a threshold, or when multimodal evidence corroborates each other within the same time window. After the conditions are met, the test conclusion is output, and the conclusion is bound to the multi-source evidence at the anomaly time point and included in the comprehensive test report.

[0069] Please see Figures 1 to 9 In one embodiment, the fusion determination further includes generating evidence association information while outputting test result information. The evidence association information is used to associate the log fragments, video frame images, and audio fragments corresponding to the anomaly with the alignment index information; wherein, generating the evidence association information includes: The alignment index information is used to determine the target operation behavior identifier corresponding to the anomaly and the alignment time window corresponding to the target operation behavior identifier. According to the alignment time window, target log segments are extracted from the system log data, target video frame sets are extracted from the video stream data, and target audio segments are extracted from the audio stream data. Based on the confidence parameter, the target log segment, the target video frame set, and the target audio segment are labeled with evidence credibility to obtain evidence credibility results. The evidence credibility label is used at least to characterize the reliability of the association between the target log segment, the target video frame set, and the target audio segment and the anomaly. The target operation behavior identifier, the alignment time window, the target log segment, the target video frame set, the target audio segment, and the evidence credibility result are associated and encapsulated to obtain the evidence association information, and the evidence association information is bound to the test result information for output.

[0070] In this embodiment, determining the target operation behavior identifier and alignment time window corresponding to the anomaly based on the alignment index information can be achieved by the alignment index management logic of the automated test script execution record and fusion module on the test end. When the automated test script executes each target operation behavior, it generates a corresponding execution timing mark or index information, which is then entered into the fusion module as part of the alignment index information. When the fusion module receives any one of the first judgment result, second judgment result, or third judgment result indicating an anomaly, it can trace back the anomaly to the corresponding target operation behavior.

[0071] In the above embodiments, the extraction of target log segments, target video frame sets, and target audio segments according to the aligned time window can be achieved collaboratively by the control and communication module, the multimodal acquisition module, and the fusion module. Specifically, the control module and the communication module continuously capture the system log stream, and the fusion module extracts target log segments from the system log data according to the aligned time window; the camera captures the screen video stream, and the image analysis module can first extract the screen area image frame by frame, and the fusion module extracts the target video frame set from the video stream according to the aligned time window or directly reuses the corresponding frame set output by the image analysis module; the microphone captures the speaker audio stream, and the fusion module extracts target audio segments from the audio stream according to the aligned time window, or reuses the audio segments analyzed by the audio analysis module within the corresponding time window.

[0072] In the above embodiments, the evidence credibility results are obtained by labeling the three types of evidence with confidence parameters. The confidence parameters can be generated by the fusion module when summarizing multimodal evaluation information, and then written back to the evidence level to form evidence credibility labels. For example, on the image side, similarity comparison results can be mapped to template similarity confidence criteria; on the audio side, continuous abnormal peaks in a specific frequency band can be mapped to spectral anomaly confidence criteria; and on the log side, keyword or abnormal pattern matching can be mapped to log consistency confidence criteria.

[0073] In the above embodiments, the associative encapsulation of evidence association information and its binding with test result information can be performed by the data fusion and report generation module. For example, the fusion module associatively encapsulates the target operation behavior identifier, alignment time window, target log segment, target video frame set, and target audio segment evidence credibility result fields into evidence association information objects or records. After encapsulation, the evidence association information is bound to the test result information and output to the comprehensive test report, forming a synchronous output of anomaly conclusions and evidence chains.

[0074] Secondly, this application also provides an automated testing system for an Android all-in-one machine based on multimodal information fusion, comprising: at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method described in any of the above.

[0075] It is understandable that the Android all-in-one device automated testing system based on multimodal information fusion can have all the technical features and beneficial effects of the aforementioned Android all-in-one device automated testing method based on multimodal information fusion, which will not be elaborated here.

[0076] In one embodiment, the Android all-in-one device automated testing system based on multimodal information fusion further includes an Android all-in-one device automated testing apparatus based on multimodal information fusion, the apparatus comprising: The scenario configuration module is used to associate the target operation behavior with the corresponding expected output state according to the test task of the Android all-in-one device under test, so as to obtain the test scenario configuration. The test scenario configuration includes at least the expected display screen benchmark and / or expected audio playback benchmark corresponding to the target operation behavior. The posture adjustment module is used to control the robotic arm or mobile platform to adjust the acquisition posture of the acquisition device used to acquire video stream data and / or audio stream data according to the test scenario configuration, so as to obtain the target acquisition posture parameters, wherein the target acquisition posture parameters include at least the relative position and / or angle of the acquisition device relative to the Android all-in-one device under test. The test driver and synchronization module is used to execute automated test driver and perform process synchronization on the Android all-in-one device under test according to the test scenario configuration and the target acquisition posture parameters, and obtain process synchronization information. The multi-source acquisition and consistency analysis module is used to acquire and time-align system log data, video stream data, and / or audio stream data according to the process synchronization information, and perform consistency analysis with the expected output state to obtain first evaluation information, second evaluation information, and third evaluation information, and form multimodal evaluation information based on the first evaluation information, the second evaluation information, and the third evaluation information. The fusion determination module is used to perform fusion determination based on the multimodal evaluation information, obtain test conclusions, and output test result information.

[0077] In one embodiment, the test-driven and synchronization module includes: The timing marking unit is used to mark the execution timing of the target operation sequence during the execution of the target operation sequence, and generate alignment index information corresponding to the system log data, the video stream data and the audio stream data based on the execution timing; Furthermore, the multi-source acquisition and consistency analysis module includes an alignment unit, used to align the system log data, the video stream data, and the audio stream data according to the alignment index information.

[0078] In one embodiment, the multi-source acquisition and consistency analysis module includes a log consistency analysis unit, which is used to determine the log segment corresponding to the target operation behavior according to the alignment index information, and to match the preset log events in the log segment to generate first evaluation information representing abnormal operation status.

[0079] In one embodiment, the multi-source acquisition and consistency analysis module includes a visual consistency analysis unit, which is used to extract screen frames aligned with the process synchronization information from the video stream data, perform template similarity calculation based on the expected display screen benchmark, and generate second evaluation information characterizing abnormal display content when the similarity is lower than a preset threshold.

[0080] In one embodiment, the visual consistency analysis unit further includes a display defect detection unit, which performs display defect detection on the screen frame. The display defect detection includes at least color anomaly detection based on color histogram, screen distortion or ghosting detection based on edge features or texture features, and black screen or white screen detection based on brightness features. The detection results are then incorporated into the second evaluation information.

[0081] In one embodiment, the multi-source acquisition and consistency analysis module includes an acoustic consistency analysis unit, which is used to calculate the average energy characteristics of the audio stream data and generate third evaluation information characterizing the silence abnormality when the average energy is continuously lower than a preset silence threshold.

[0082] In one embodiment, the acoustic consistency analysis unit further includes: The spectrum analysis and abnormal peak detection unit is used to calculate the spectrum characteristics of the audio stream data and generate third evaluation information characterizing distortion or noise abnormality when a continuous abnormal peak occurs in a preset frequency band. The content consistency comparison unit is used to perform a content consistency comparison between the audio stream data and the expected playback audio benchmark to obtain a content comparison result. The content consistency comparison is implemented based on at least one of speech recognition or voiceprint matching. When the content comparison result does not meet the preset consistency conditions, a third evaluation information representing the inconsistency of the playback content is generated.

[0083] In one embodiment, the multi-source acquisition and consistency analysis module is further configured to generate confidence parameters corresponding to the first evaluation information, the second evaluation information, and the third evaluation information, wherein the confidence parameters include at least one of template similarity confidence, spectrum anomaly confidence, and log consistency confidence. The fusion determination module includes a credibility constraint unit, which is used to determine the alignment time window corresponding to the same target operation behavior based on the alignment index information, and to perform credibility constraints on the first evaluation information, the second evaluation information and the third evaluation information according to the confidence parameter within the alignment time window, and to output the test conclusion when the preset credibility condition is met.

[0084] In one embodiment, the fusion determination module further includes an evidence association generation unit, used to generate evidence association information while outputting test result information. The evidence association information is used to associate log fragments, video frame images, and audio fragments corresponding to the anomaly with the alignment index information; wherein, the evidence association generation unit is used to: The alignment index information is used to determine the target operation behavior identifier corresponding to the anomaly and the alignment time window corresponding to the target operation behavior identifier. According to the alignment time window, target log segments are extracted from the system log data, target video frame sets are extracted from the video stream data, and target audio segments are extracted from the audio stream data. Based on the confidence parameter, the target log segment, the target video frame set, and the target audio segment are labeled with evidence credibility to obtain evidence credibility results. The evidence credibility label is used at least to characterize the reliability of the association between the target log segment, the target video frame set, and the target audio segment and the anomaly. The target operation behavior identifier, the alignment time window, the target log segment, the target video frame set, the target audio segment, and the evidence credibility result are associated and encapsulated to obtain the evidence association information, and the evidence association information is bound to the test result information for output.

[0085] In addition, combined Figure 1 The Android all-in-one machine automated testing method based on multimodal information fusion described in this application embodiment can be implemented by an Android all-in-one machine automated testing system based on multimodal information fusion. Figure 10 A schematic diagram of the hardware structure of the Android all-in-one automated testing system based on multimodal information fusion provided in this application embodiment is shown.

[0086] An automated testing system for Android all-in-one devices based on multimodal information fusion may include a processor and a memory storing computer program instructions.

[0087] Specifically, the processor may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0088] The memory may include a large-capacity storage device for data or instructions. For example, and not limitingly, the memory may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory may include removable or non-removable (or fixed) media. Where appropriate, the memory may be internal or external to a data processing device. In a particular embodiment, the memory is a non-volatile solid-state memory. In a particular embodiment, the memory includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0089] The processor reads and executes computer program instructions stored in the memory to implement any of the Android all-in-one device automated testing methods based on multimodal information fusion in the above embodiments.

[0090] In one example, an automated testing system for an Android all-in-one device based on multimodal information fusion may also include a communication interface and a bus. For example, Figure 10 As shown, the processor, memory, and communication interface are connected via a bus and communicate with each other.

[0091] The communication interface is mainly used to enable communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0092] A bus, including hardware, software, or both, couples together components of an Android all-in-one automated testing system based on multimodal information fusion. For example, and not limitingly, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, a bus may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0093] Furthermore, in conjunction with the Android all-in-one device automated testing method based on multimodal information fusion in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the Android all-in-one device automated testing methods based on multimodal information fusion in the above embodiments.

[0094] In summary, the automated testing method and system for Android all-in-one devices based on multimodal information fusion provided in this application embodiment, wherein the method is executed by the test terminal on the Android all-in-one device under test, includes: configuring the target operation behavior and the corresponding expected output state according to the test task of the Android all-in-one device under test to obtain a test scenario configuration, wherein the test scenario configuration includes at least the expected display screen reference and / or expected audio playback reference corresponding to the target operation behavior; and, according to the test scenario configuration, controlling a robotic arm or mobile platform to adjust the acquisition posture of the acquisition device used to acquire video stream data and / or audio stream data to obtain target acquisition posture parameters, wherein the target acquisition posture parameters include at least the expected display screen reference and / or expected audio playback reference corresponding to the target operation behavior. The method involves determining the relative position and / or angle of the acquisition device relative to the Android all-in-one device under test; executing automated test driving and synchronizing the process on the Android all-in-one device under test according to the test scenario configuration and the target acquisition posture parameters to obtain process synchronization information; acquiring and time-aligning system log data, video stream data, and / or audio stream data according to the process synchronization information, and performing consistency analysis with the expected output state to obtain first evaluation information, second evaluation information, and third evaluation information, and forming multimodal evaluation information based on the first evaluation information, second evaluation information, and third evaluation information; performing fusion judgment based on the multimodal evaluation information to obtain test conclusions and output test result information. This method adjusts the acquisition posture through a robotic arm or mobile platform and performs time alignment and fusion judgment on multimodal information such as logs, videos, and audios based on process synchronization, achieving higher accuracy, stability, and repeatability of automated test results for Android all-in-one devices. The Android all-in-one device automated testing system based on multimodal information fusion provided in this application includes at least one processor, at least one memory, and computer program instructions stored in the memory. When executed by the processor, the instructions implement the Android all-in-one device automated testing method based on multimodal information fusion as described above. It is understood that this Android all-in-one device automated testing system based on multimodal information fusion can possess all the technical features and beneficial effects of the aforementioned Android all-in-one device automated testing method based on multimodal information fusion.

[0095] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0096] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0097] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0098] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. An automated testing method for Android all-in-one devices based on multimodal information fusion, characterized in that, The method is executed by the test terminal on the Android all-in-one device to be tested, including: Based on the test task of the Android all-in-one device under test, the target operation behavior and the corresponding expected output state are associated and configured to obtain the test scenario configuration. The test scenario configuration includes at least the expected display screen benchmark and / or expected audio playback benchmark corresponding to the target operation behavior. According to the test scenario configuration, control the robotic arm or mobile platform to adjust the acquisition posture of the acquisition device used to acquire video stream data and / or audio stream data, and obtain the target acquisition posture parameters. Based on the test scenario configuration and the target acquisition posture parameters, an automated test driver is executed on the Android all-in-one device under test and the process is synchronized to obtain process synchronization information. Based on the process synchronization information, system log data, video stream data, and / or audio stream data are collected and time-aligned, and consistency analysis is performed with the expected output state to obtain first evaluation information, second evaluation information, and third evaluation information. Multimodal evaluation information is then formed based on the first evaluation information, the second evaluation information, and the third evaluation information. Based on the multimodal evaluation information, a fusion judgment is performed to obtain the test conclusion and output the test result information.

2. The method according to claim 1, characterized in that, The step of executing automated test drive and synchronizing the process of the Android all-in-one device under test according to the test scenario configuration and the target acquisition posture parameters to obtain process synchronization information includes: During the execution of the target operation, the execution sequence of the target operation is marked to obtain execution sequence marking information; Based on the execution timing marker information, generate alignment index information corresponding to the system log data, the video stream data, and the audio stream data; The system log data, video stream data, and audio stream data are collected and time-aligned based on the alignment index information.

3. The method according to claim 2, characterized in that, The consistency analysis includes: Based on the alignment index information, the log alignment segment corresponding to the target operation behavior is determined, and the preset log events in the log alignment segment are matched to obtain the log consistency matching result. Based on the log consistency matching results, a first evaluation message representing an abnormal running status is generated.

4. The method according to claim 2, characterized in that, The visual consistency analysis also includes: Based on the alignment index information, extract the set of screen alignment frames corresponding to the target operation behavior from the video stream data; Based on the expected display frame reference, a template similarity calculation is performed on the set of aligned frames to obtain the template similarity result; When the template similarity result does not meet the preset similarity conditions, a second evaluation message representing abnormal displayed content is generated.

5. The method according to claim 4, characterized in that, The visual consistency analysis also includes: During the template similarity calculation of the set of aligned frames, display defect detection is performed on the set of aligned frames to obtain display defect detection results; The display defect detection includes at least color anomaly detection based on color histogram, screen flickering or ghosting detection based on edge features or texture features, and black screen or white screen detection based on brightness features. The defect detection results are incorporated into the second evaluation information.

6. The method according to claim 1, characterized in that, The consistency analysis also includes: Based on the alignment index information, determine the audio alignment segment corresponding to the target operation behavior; The average energy characteristics of the audio alignment segment are calculated to obtain the energy determination result; When the energy determination result meets the preset silent abnormality condition, a third evaluation information characterizing the silent abnormality is generated.

7. The method according to claim 6, characterized in that, The acoustic consistency analysis also includes: The spectral characteristics of the audio alignment segment are calculated to obtain the spectral determination result; When the spectrum determination result meets the preset spectrum anomaly conditions, third evaluation information characterizing distortion or noise anomalies is generated; and, Based on the expected playback audio benchmark, a content consistency comparison is performed on the audio alignment segment to obtain the content comparison result; The content consistency comparison is implemented based on at least one of speech recognition or voiceprint matching, and a third evaluation information representing the inconsistency of the played content is generated when the content comparison result does not meet the preset consistency conditions.

8. The method according to claim 2, characterized in that, The formation of multimodal evaluation information also includes: Generate confidence parameters for the first evaluation information, the second evaluation information, and the third evaluation information, respectively; Based on the alignment index information, an alignment time window corresponding to the same target operation behavior is determined, and within the alignment time window, the first evaluation information, the second evaluation information, and the third evaluation information are constrained based on the confidence parameter to obtain multimodal evaluation information for fusion judgment. The test conclusion is output based on the multimodal evaluation information.

9. The method according to claim 8, characterized in that, The fusion determination also includes generating evidence association information while outputting test result information. This evidence association information is used to associate the log fragments, video frame images, and audio fragments corresponding to the anomaly with the alignment index information. The generation of evidence association information includes: The alignment index information is used to determine the target operation behavior identifier corresponding to the anomaly and the alignment time window corresponding to the target operation behavior identifier. According to the alignment time window, target log segments are extracted from the system log data, target video frame sets are extracted from the video stream data, and target audio segments are extracted from the audio stream data. Based on the confidence parameter, the target log segment, the target video frame set, and the target audio segment are labeled with evidence credibility to obtain evidence credibility results. The evidence credibility label is used at least to characterize the reliability of the association between the target log segment, the target video frame set, and the target audio segment and the anomaly. The target operation behavior identifier, the alignment time window, the target log segment, the target video frame set, the target audio segment, and the evidence credibility result are associated and encapsulated to obtain the evidence association information, and the evidence association information is bound to the test result information for output.

10. An automated testing system for an Android all-in-one device based on multimodal information fusion, characterized in that, The system includes: at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement the method as described in any one of claims 1-9.

Images