Test result determination method and apparatus, device, storage medium, and program product

Abstract

The application discloses a test result determination method and device, equipment, a storage medium and a program product, and relates to the technical field of testing. The method comprises the following steps: performing visual feature extraction on a video frame corresponding to each timestamp in a video stream of a device under test, and determining a video feature vector corresponding to each timestamp; in the case where a text box is contained in the video frame, determining a semantic feature vector corresponding to the text in the text box; performing audio feature extraction on an audio frame corresponding to each timestamp in an audio stream of the device under test, and determining an audio feature vector corresponding to each timestamp; and determining a test result according to the visual feature vector, the semantic feature vector and the audio feature vector corresponding to each timestamp. Thus, the accuracy and comprehensiveness of the test result are improved, the audio and video output quality and overall performance of the device under test can be more reliably evaluated, and more accurate test basis is provided for performance optimization and quality control of the device under test.

Description

Technical Field

[0001] This application relates to the field of testing technology, and in particular to methods, apparatus, equipment, storage media, and program products for determining test results. Background Technology

[0002] With the widespread adoption of smart devices (such as smart monitoring equipment, smart terminals, and in-vehicle displays) and the continuous expansion of their application scenarios, the audio and video output function of the device under test (DUT) has become a crucial component of its core performance. The quality, accuracy, and synchronization of multi-dimensional information in the audio and video output directly affect the user experience, functional completeness, and application reliability of the device. Therefore, testing the audio and video output performance of the DUT is of paramount importance. Summary of the Invention

[0003] The main objective of this application is to provide a method, apparatus, device, storage medium, and program product for determining test results, aiming to solve the technical problem of low accuracy in the test results of the audio and video output related performance of the device under test.

[0004] To achieve the above objectives, this application proposes a method for determining test results, the method comprising: Visual features are extracted from the video frames corresponding to each timestamp in the video stream of the device under test, and the video feature vector corresponding to each timestamp is determined. If the video frame contains a text box, determine the semantic feature vector corresponding to the text in the text box; Audio features are extracted from the audio frames corresponding to each timestamp of the audio stream of the device under test, and the audio feature vector corresponding to each timestamp is determined. The test result is determined based on the visual feature vector, semantic feature vector, and audio feature vector corresponding to each timestamp.

[0005] Furthermore, to achieve the above objectives, this application also proposes a device for determining test results, the device comprising: The first determining module is used to extract visual features from the video frame corresponding to each timestamp in the video stream of the device under test, and determine the video feature vector corresponding to each timestamp. The second determining module is used to determine the semantic feature vector corresponding to the text in the text box when the video frame contains a text box; The third determining module is used to extract audio features from the audio frames corresponding to each timestamp of the audio stream of the device under test, and determine the audio feature vector corresponding to each timestamp. The fourth determining module is used to determine the test result based on the visual feature vector, the semantic feature vector, and the audio feature vector corresponding to each timestamp.

[0006] In addition, to achieve the above objectives, this application also proposes a test result determination device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the test result determination method as described above.

[0007] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the method for determining test results as described above.

[0008] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the method for determining test results as described above. Attached Figure Description

[0009] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating the method for determining test results in this application (Example 1); Figure 2 A flowchart illustrating the second embodiment of the method for determining the test results of this application; Figure 3 This is a schematic diagram of the module structure of the device for determining test results in an embodiment of this application; Figure 4 This is a schematic diagram of the hardware operating environment involved in the method for determining test results in the embodiments of this application.

[0012] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0013] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0014] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0015] Currently, functional testing of electronic devices (such as commercial digital signage, televisions, monitors, etc.) usually relies on manual operation and observation: 1. Testers observe the output screen of the device through the monitor to determine whether the function is normal; 2. They operate the device through a mouse, keyboard, or remote control to trigger different functional scenarios.

[0016] This approach has the following problems: 1. Low testing efficiency: Manually executing test cases step by step is time-consuming; 2. High labor costs: Requires a large number of testers; 3. Subjective judgment is prone to errors: Different testers may have inconsistent understandings of the visuals and sounds, leading to unstable results; 4. Data is not reusable: The testing process lacks complete records, making it difficult to backtrack and analyze.

[0017] Therefore, this application provides a solution to extract visual features from video frames corresponding to each timestamp in the video stream of the device under test, and determine the video feature vector corresponding to each timestamp; if the video frame contains a text box, determine the semantic feature vector corresponding to the text in the text box; extract audio features from audio frames corresponding to each timestamp in the audio stream of the device under test, and determine the audio feature vector corresponding to each timestamp; and determine the test result based on the visual feature vector, the semantic feature vector, and the audio feature vector corresponding to each timestamp.

[0018] As can be seen from the above embodiments, by extracting visual features from video frames corresponding to each timestamp of the video stream of the device under test to obtain video feature vectors, extracting text semantic features when the video frame contains a text box to obtain semantic feature vectors, and simultaneously extracting audio features from audio frames corresponding to each timestamp of the audio stream to obtain audio feature vectors, and then combining the three feature vectors under the same timestamp to comprehensively determine the test results, multi-dimensional collaborative detection of the audio and video output of the device under test is achieved. This not only comprehensively captures video visual anomalies and audio output anomalies, but also accurately identifies text semantic errors in the video, text and visual / audio asynchrony issues, etc., significantly improving the accuracy and comprehensiveness of the test results. It can more reliably evaluate the audio and video output quality and overall performance of the device under test, and provide more accurate test basis for the performance optimization and quality control of the device under test.

[0019] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or test result determination device capable of performing the above functions. The following description uses a test result determination device as an example to illustrate this embodiment and the subsequent embodiments.

[0020] Based on this, embodiments of this application provide a method for determining test results, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the method for determining test results in this application. Figure 1 As shown, the method for determining the test results includes steps S10 to S40: Step S10: Visual features are extracted from the video frames corresponding to each timestamp in the video stream of the device under test, and the video feature vector corresponding to each timestamp is determined.

[0021] The device under test can be a commercial digital signage device, a television, a monitor, or other equipment that can be tested.

[0022] Among them, the video stream is a continuous dynamic image data sequence output by the device under test, which is a collection of video frames.

[0023] The timestamp is a time identifier added to each frame of data in the video stream.

[0024] In some embodiments, deep learning models such as the YOLO object detection model and Convolutional Neural Network (CNN) can be used to extract feature information that can characterize the content of the video frame, including the position information and confidence level of interface elements. Then, the extracted visual information is digitized and vectorized into a multidimensional array to obtain the video feature vector. The interface elements may include buttons, pop-ups, screen structures, visual actions, text boxes, etc.

[0025] In one possible implementation, the video stream can be acquired via a wired video interface or a wireless transmission protocol, depending on the type of video stream acquisition supported by the device under test. It should be noted that different types of devices under test (TVs, set-top boxes, game consoles) have different video output methods; supporting the selection of the video stream acquisition method based on the device's supported video stream acquisition type can improve compatibility.

[0026] Among them, the wired video interface is the hardware interface for enabling wired transmission of video streams. For example, it supports High-Definition Multimedia Interface (HDMI) and DisplayPort (DP).

[0027] Among them, wireless transmission protocols refer to communication protocols that enable wireless transmission of video streams. For example, they support Wi-Fi Display and Real-Time Streaming Protocol (RTSP) to solve the spatial limitations of wired acquisition.

[0028] The video stream acquisition type refers to the video output method of the device under test. It is divided into two types: wired output (supporting HDMI / DP) and wireless output (supporting Wi-Fi Display / RTSP).

[0029] In possible implementations, if the device under test (DUT) supports wired output but not wireless output, the video stream is obtained through a wired video interface. Alternatively, if the DUT supports wireless output but not wired output, the video stream is obtained through a wireless transmission protocol. Or, if the DUT supports both wireless and wired output, it can choose either a wired video interface or a wireless transmission protocol to obtain the video stream.

[0030] Step S20: If the video frame contains a text box, determine the semantic feature vector corresponding to the text in the text box.

[0031] Among them, text boxes are interface areas in video frames used to display text information, such as error message boxes, function menu boxes, status message boxes, etc.

[0032] Among them, the semantic feature vector is a fixed-dimensional vector that is transformed from the text content in the text box through natural language processing technology, and can represent the semantic information of the text.

[0033] In one possible implementation, deep learning models such as YOLO object detection model and Convolutional Neural Network (CNN) can be used to extract the position information and confidence of interface elements such as buttons, pop-ups, text boxes, screen structure, and visual actions from video frames, thereby determining whether the video frame contains a text box.

[0034] In one possible implementation, if the video frame contains a text box, the video frame is cropped based on the text box's position within the video frame to obtain a text box image. Then, Optical Character Recognition (OCR) can be used to recognize the text in the text box image to obtain the text content. The text content is then input into a Natural Language Processing (NLP) semantic understanding module to convert it into a fixed-dimensional semantic feature vector. For example, it can identify that "network connection timeout" and "ServerError" belong to the same type of abnormal semantic features.

[0035] Step S30: Extract audio features from the audio frames corresponding to each timestamp of the audio stream of the device under test, and determine the audio feature vector corresponding to each timestamp.

[0036] The audio stream is a continuous sequence of sound data output by the device under test, such as click effects, audio playback, and prompts.

[0037] The audio frame is a single sound data segment obtained after processing the audio stream into frames, and it corresponds one-to-one with the timestamp of the video frame, realizing synchronized audio-visual analysis. It should be noted that the audio acquisition module and the video acquisition module use a unified timestamp for synchronization, and the timestamps of the video stream and the audio stream are kept consistent.

[0038] In one possible implementation, the audio frames at each timestamp are converted into spectrograms. A CNN acoustic model is used to perform convolution operations on the spectrograms to extract the spectral features of the audio. The spectral features are then mapped into fixed-dimensional audio feature vectors through a fully connected layer.

[0039] In another possible implementation, Mel Frequency Cepstral Coefficient (MFCC) features of the audio frame are extracted, and the MFCC features are input into a Deep Neural Network (DNN) for feature learning and enhancement to generate a fixed-dimensional audio feature vector.

[0040] Step S40: Determine the test result based on the visual feature vector, semantic feature vector, and audio feature vector corresponding to each timestamp.

[0041] The test results are the determination of the current functional status of the device under test based on the fusion analysis of multimodal features. These results may include test states such as picture without sound, no response to operation, and audio-visual asynchrony.

[0042] In one possible implementation, visual, semantic, and audio feature vectors are mapped to the same fixed dimension (e.g., 512-dimensional) through a linear fully connected layer, ensuring that the vectors can be concatenated. Then, the three aligned feature vectors under each timestamp are concatenated dimensionally to generate a single timestamp fused feature vector. Subsequently, the fused feature vectors of the timestamps are concatenated again in chronological order to generate a temporal fused feature vector, preserving the state change information within a short period of time. Finally, the temporal fused feature vector is input into a lightweight classifier (e.g., logistic regression, shallow DNN, SVM). The lightweight classifier outputs the judgment results of each test state (e.g., picture without sound, audio and picture out of sync, no response to operation, etc.) through feature mapping, directly obtaining the test results.

[0043] In this embodiment, visual features are extracted from video frames corresponding to each timestamp of the video stream of the device under test to obtain video feature vectors. When a video frame contains a text box, text semantic features are extracted to obtain semantic feature vectors. At the same time, audio features are extracted from audio frames corresponding to each timestamp of the audio stream to obtain audio feature vectors. The test results are then determined by combining the feature vectors of the three at the same timestamp. This achieves multi-dimensional collaborative detection of the audio and video output of the device under test. It can not only comprehensively capture video visual anomalies and audio output anomalies, but also accurately identify text semantic errors in the video, text and visual / audio asynchrony issues, etc., which significantly improves the accuracy and comprehensiveness of the test results. It can more reliably evaluate the audio and video output quality and overall performance of the device under test, and provide more accurate test basis for the performance optimization and quality control of the device under test.

[0044] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 . Figure 2 This is a flowchart illustrating the second embodiment of the method for determining test results in this application, as shown below. Figure 2 As shown, step S40 further includes steps S41 to S45: Step S41: Align the visual feature vector, the semantic feature vector, and the audio feature vector along the feature dimension.

[0045] In one possible implementation, independent linear fully connected layers are configured for the three feature vectors, and each vector is projected to a preset fixed dimension (such as 512-dimensional or 1024-dimensional) to achieve dimension unification.

[0046] In another possible implementation, principal component analysis is used to reduce the dimensionality of high-dimensional feature vectors, while linear interpolation is used to increase the dimensionality of low-dimensional feature vectors, thus adjusting all three types of vectors to the same dimension. This approach can adapt to the computational needs of low-computing-power testing systems.

[0047] Step S42: Add corresponding modality type labels to the aligned visual feature vector, semantic feature vector, and audio feature vector, respectively.

[0048] The modality type label is a unique identifier added to the feature vectors of different modalities (e.g., visual feature vector is labeled V, semantic feature vector is labeled S, and audio feature vector is labeled A), which enables the multimodal feature fusion network to distinguish feature information of different modalities and realize cross-modal feature association calculation.

[0049] In one possible implementation, a fixed-dimensional modality type identifier vector is added to the head of each aligned feature vector to form a labeled feature vector.

[0050] Step S43: According to the time sequence of the timestamps, the aligned visual feature vector, semantic feature vector, audio feature vector and modality type label corresponding to each timestamp are concatenated to obtain the time sequence input sequence.

[0051] The temporal input sequence is a one-dimensional feature sequence formed by concatenating labeled feature vectors with different timestamps and modalities in chronological order. It is the core input data of the multimodal fusion network, and the sequence contains both modal and temporal information.

[0052] In one possible implementation, the visual, semantic, and audio labeled feature vectors corresponding to each timestamp are concatenated sequentially in order from early to late, forming a temporal input sequence of "timestamp 1-visual + timestamp 1-semantic + timestamp 1-audio + timestamp 2-visual + timestamp 2-semantic + timestamp 2-audio + ...".

[0053] Step S44: Input the time-series input sequence into the multimodal feature fusion network to obtain context aggregation features.

[0054] Among them, multimodal feature fusion networks can achieve cross-modal and cross-temporal feature fusion and information interaction. In one possible implementation, the multimodal feature fusion network can be a deep learning model such as Transformer.

[0055] Among them, context aggregation features are fused features output by multimodal feature fusion networks after feature fusion and information interaction of temporal input sequences. They can represent contextual information across time and modality (such as audio feedback corresponding to a visual action, and the correlation between image and sound changes in consecutive timestamps), and are the core feature data for subsequent test result determination.

[0056] In one possible implementation, the multimodal feature fusion network may include an encoding layer and a global pooling layer. The encoding layer includes a multi-head self-attention sublayer, a first residual connection sublayer, a first normalization sublayer, a feedforward network sublayer, a second residual connection sublayer, and a second normalization sublayer. Step S44 includes steps S441 to S445: Step S441: Input the time-series input sequence into the multi-head self-attention sub-layer to obtain a first fusion feature sequence, wherein the first fusion feature sequence contains the fusion feature corresponding to each feature vector.

[0057] Among them, the multi-head self-attention sub-layer can simultaneously calculate the correlation weights between all feature vectors in the temporal input sequence through multiple attention heads, thereby realizing cross-modal and cross-temporal feature information interaction and fusion. The first fused feature sequence is the output sequence of the multi-head self-attention sublayer after fusing features from the temporal input sequence. The fused feature corresponding to each feature vector incorporates information from other relevant feature vectors in the sequence.

[0058] It should be noted that the multi-head self-attention sublayer can dynamically calculate the correlation weights between features, assigning high weights to important feature correlations (such as click actions and corresponding sound effects) and low weights to irrelevant correlations, thereby achieving adaptive feature fusion.

[0059] Step S442: Input the first fused feature sequence into the first residual connection sub-layer and the first normalization sub-layer to obtain the second fused feature sequence. The first residual connection sub-layer is used to add the first fused feature sequence to the time-series input sequence by residual addition. The first normalization sub-layer is used to normalize the output of the first residual connection sub-layer.

[0060] The residual connection sublayer can solve the gradient vanishing problem in deep learning networks by adding the network's input and output, thereby improving the network's training and inference performance. The first residual connection sublayer can achieve the addition of the residuals between the temporal input sequence and the first fused feature sequence.

[0061] The normalization sublayer normalizes the numerical values ​​of the feature sequence, ensuring that the feature values ​​are distributed within a fixed range, thus improving the convergence speed and stability of the network. Specifically, the first normalization sublayer normalizes the result of summing the residuals.

[0062] The second fused feature sequence is the feature sequence output after processing by the first residual connection sub-layer and the first normalization sub-layer, which solves the gradient vanishing problem of the multi-head self-attention sub-layer and improves the stability of the feature sequence.

[0063] Step S443: Input the second fused feature sequence into the feedforward network sub-layer to obtain the third fused feature sequence. The feedforward network sub-layer is used to abstract and extract features from the second fused feature sequence and enhance its features.

[0064] Among them, the feedforward network sub-layer can be a sub-layer composed of fully connected layers and activation functions (such as ReLU), which can perform nonlinear transformation, abstract extraction and feature enhancement on the second fused feature sequence, and capture the deep nonlinear correlation of features.

[0065] The third fusion feature sequence is the output sequence of the feedforward network sublayer after processing the second fusion feature sequence. The representation ability of the third fusion feature sequence is further enhanced, and it contains more deep nonlinear context information.

[0066] In one possible implementation, the feedforward network sublayers may include two fully connected layers and a ReLU activation function. The first fully connected layer increases the dimensionality of the features, and the ReLU activation function performs a non-linear transformation. The second fully connected layer reduces the dimensionality of the features, resulting in a third fused feature sequence.

[0067] In another possible implementation, the feedforward network sublayer can include a single fully connected layer and a GELU activation function. Using a single fully connected layer in conjunction with the GELU activation function to perform a nonlinear transformation on the second fused feature sequence to obtain the third fused feature sequence can reduce computational load and adapt to low-computing-power scenarios.

[0068] It should be noted that multi-head self-attention sublayers mainly realize the correlation and fusion between features, which is a linear transformation and cannot capture the non-linear correlation between features (such as the non-linear temporal correlation between visual changes and sound effects). Feedforward network sublayers, through non-linear activation functions, can realize non-linear transformation of features, capture deep non-linear contextual information, and improve the representation ability of features.

[0069] Step S444: Input the third fused feature sequence into the second residual connection sub-layer and the second normalization sub-layer to obtain the fourth fused feature sequence. The second residual connection sub-layer is used to add the residuals of the third fused feature sequence and the first fused feature sequence. The second normalization sub-layer is used to normalize the output of the second residual connection sub-layer.

[0070] It should be noted that the nonlinear transformation of the feedforward network sublayer is prone to gradient vanishing and feature distribution shift. The second residual connection sublayer solves the gradient vanishing problem by adding the third fused feature sequence to the first fused feature sequence, while retaining the fusion information of the multi-head self-attention sublayer. The second normalization sublayer standardizes the output of the second residual connection sublayer, making the feature value distribution stable, avoiding the abnormal feature distribution caused by the nonlinear transformation of the feedforward network sublayer, and improving the processing effect of the subsequent global pooling layer.

[0071] Step S445: Input the fourth fused feature sequence into the global pooling layer to obtain the context aggregation feature.

[0072] The global pooling layer can pool high-dimensional fourth-dimensional fusion feature sequences, compressing the temporal-dimensional feature sequences into fixed-dimensional feature vectors, thus achieving global feature aggregation. Commonly used pooling methods include global average pooling and global max pooling.

[0073] For example, when the global pooling layer uses global average pooling, the mean of all feature vectors in the fourth fused feature sequence is calculated to generate a fixed-dimensional contextual aggregated feature. This method can preserve the overall feature information of the sequence. When the global pooling layer uses global max pooling, the maximum value is taken from all feature vectors in the fourth fused feature sequence to generate a fixed-dimensional contextual aggregated feature. This method can highlight the key feature information in the sequence.

[0074] It should be noted that the fourth fusion feature sequence is a high-dimensional sequence containing temporal information, which cannot be directly input into the classifier for test result determination. The global pooling layer can realize the global aggregation of features, condensing the contextual information across time and modality into a fixed-dimensional vector, preserving the core information of feature fusion, and reducing the overhead of subsequent computation.

[0075] In one possible implementation, the multimodal feature fusion network may include multiple encoding layers, with the output of the first encoding layer input into the second encoding layer, the output of the second encoding layer input into the third encoding layer, and the output of the last encoding layer input into the global pooling layer.

[0076] Therefore, relying on the multi-head self-attention mechanism, cross-modal and cross-temporal feature association calculations are performed on temporal feature sequences with modality labels to achieve preliminary context fusion between features. Then, residual connection and normalization techniques are used to solve the gradient vanishing problem in deep networks, stabilize the feature value distribution to ensure fusion effectiveness, and combine feedforward networks to complete the nonlinear abstract extraction and representation capability enhancement of features, making up for the deficiency of linear fusion in capturing deep nonlinear feature associations. Finally, global pooling is used to achieve dimensionality reduction and global information aggregation of high-dimensional temporal fusion features, ultimately generating aggregated features that have both global context association information and high representation capability. This retains the core feature associations of the functional state of the device under test and completes the standardization of feature format, providing a high-quality and highly adaptable feature foundation for the accurate judgment of subsequent test results, so that the fused features can comprehensively and accurately represent the actual functional state of the device under test.

[0077] Step S45: Input the context aggregation features into the classifier to obtain the test results.

[0078] Among them, the classifier can map contextual aggregate features to confidence levels for different test states, thereby enabling automatic determination of test results.

[0079] It should be noted that context aggregation features are fixed-dimensional fusion features and cannot directly reflect the test results. The classifier transforms the fusion features into intuitive test state confidence through feature mapping and activation functions, thereby achieving automated determination of test results.

[0080] The test results can be confidence levels corresponding to various possible test states, such as picture without sound, audio and video desynchronization, and no response to operation. Alternatively, the test results can also be test states with confidence levels greater than or equal to a confidence threshold. For example, if the confidence level for picture without sound is greater than the corresponding confidence threshold, the test results can include picture without sound; if the confidence level for audio and video desynchronization is less than the corresponding confidence threshold, the test results can exclude audio and video desynchronization.

[0081] In one possible implementation, the classifier includes a fully connected sublayer and an activation function, and step S45 includes steps S451 and S452: Step S451: Input the context aggregation features into the fully connected sub-layer to obtain the test state score vector.

[0082] The fully connected sublayer is the core component of the classifier, which can map the context aggregation features into a score vector consistent with the number of test states through linear transformation.

[0083] The test state score vector is a vector with the same number of dimensions as the number of test states. The value of each dimension represents the original score of the device under test in the corresponding test state, such as a score for picture without sound, a score for audio and picture out of sync, and a score for no response to operation.

[0084] Step S452: Normalize the test state score vector using an activation function to obtain the confidence level corresponding to each test state. The confidence level corresponding to each test state is the test result.

[0085] The activation function is a function that performs a non-linear transformation and normalizes the values. This step is used to normalize the test state score vector to a confidence level between 0 and 1. The activation function can be Softmax, etc.

[0086] The confidence level, ranging from 0 to 1, represents the probability that the device under test is in the corresponding test state; the closer the value is to 1, the higher the probability. The sum of the confidence levels for all test states is 1.

[0087] Therefore, by using linear transformation of the fully connected layer to achieve feature mapping from high-dimensional fusion aggregated features to test state dimensions, the abstract fusion features are transformed into raw test state scores. Then, the raw scores are normalized by the activation function, transforming the scores without a fixed range into probabilistic confidence scores in the 0-1 interval. This realizes the transformation from abstract fusion features to quantitative and probabilistic test results, providing a unified quantitative standard for judging the functional states of the device under test. This not only intuitively reflects the possibility of the device being in each test state, but also provides a quantifiable objective basis for the refined judgment of subsequent test results, improving the flexibility, scientificity, and interpretability of test result judgment.

[0088] In this embodiment, by unifying the dimensions and identifying the modalities of visual, semantic, and audio features, and combining the temporal dimension to complete the temporal splicing of features, and then relying on a multimodal fusion network to achieve deep feature association and contextual information aggregation across modalities and time, the abstract fusion features are finally transformed into quantitative test state confidence through a classifier. This series of technical processes realizes the transformation from discrete single-modal features to integrated fusion features, effectively capturing the multi-dimensional representation information of the functional state of the device under test and the dynamic change law of the temporal dimension. This allows the test results to be based on the quantitative judgment of multi-dimensional fusion features, which greatly improves the accuracy, objectivity and comprehensiveness of the judgment results, and can effectively identify complex logical defects such as audio-visual asynchrony and no response to operation.

[0089] In one possible implementation, the following is included after step S452: Step S46: Determine the test result type corresponding to the device under test based on the confidence level corresponding to each test state and the test result judgment condition of the device under test.

[0090] The test result determination criteria are the confidence level determination rules set in advance for the device under test, which may include: if the expected result is met and the confidence level is higher than the threshold, the test is considered passed; if a clear abnormal pattern is identified, the test fails and the defect type is output; if the confidence level is insufficient, a retry is triggered or the test is submitted for manual review.

[0091] The test result type can be the final test result obtained based on the judgment conditions, which can be divided into pass, failure (including specific defect types), and uncertain.

[0092] For example, if a confidence threshold is set (e.g., 0.8), and the confidence level for a picture without sound is 0.9 (greater than 0.8), the test result type is failure, meaning the picture is without sound.

[0093] Step S47: Determine the test task based on the test result type.

[0094] The test task is a subsequent operation task assigned to the device under test based on the test result type. Test tasks can include continuing the main process, executing branch processes, executing exception recovery processes, retrying, manual review, and terminating the test.

[0095] In one possible implementation, corresponding test task rules are pre-configured for each test result type, and the system automatically matches and determines the test task based on the test result type. For example, the test task rules could be: if the test passes, proceed to the next step of the main process; if it fails, terminate the test and record the defect; if uncertain, extend the identification time and retry; if a pop-up branch appears, execute the branch operation.

[0096] Step S48: Control the device under test to execute the test task.

[0097] In one possible implementation, the defined test task can be transformed into specific device operation instructions, sent to the device under test, and driven to execute. For example, operation control methods can include mouse and keyboard simulation, multi-touch simulation, voice command injection, virtual driver API calls, and cloud-based remote centralized control.

[0098] Therefore, by combining probabilistic test state confidence with preset quantitative judgment rules, the system completes the judgment from numerical confidence to structured test result type. Then, it matches targeted test tasks according to the actual test result type of the equipment, realizing adaptive dynamic adjustment of the test process. Finally, through multi-mode interactive control technology, it drives the device under test to execute the matched test tasks, opening up the technical link from test result judgment to actual test operation. This enables the equipment testing to form a complete closed loop of "feature analysis - result judgment - process adjustment - operation execution". The test process can autonomously adapt to different operations such as main process, branch process, and anomaly recovery according to the actual functional state of the device under test, greatly reducing manual intervention, improving the intelligence and automation level of equipment automated testing, while ensuring the continuity and integrity of the test process, effectively improving test efficiency and fault tolerance, and realizing intelligent closed-loop test management.

[0099] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the method for determining the test results of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0100] This application also provides a device for determining test results, please refer to... Figure 3 The device for determining the test results includes: The first determining module 301 is used to extract visual features from the video frame corresponding to each timestamp in the video stream of the device under test, and determine the video feature vector corresponding to each timestamp. The second determining module 302 is used to determine the semantic feature vector corresponding to the text in the text box when the video frame contains a text box; The third determining module 303 is used to extract audio features from the audio frames corresponding to each timestamp of the audio stream of the device under test, and determine the audio feature vector corresponding to each timestamp. The fourth determining module 304 is used to determine the test result based on the visual feature vector, the semantic feature vector and the audio feature vector corresponding to each timestamp.

[0101] In one possible implementation, the fourth determining module 304 is used for: Align the visual feature vector, the semantic feature vector, and the audio feature vector along the feature dimension; Add corresponding modality type labels to the aligned visual feature vector, semantic feature vector, and audio feature vector, respectively; According to the time sequence of the timestamps, the aligned visual feature vector, semantic feature vector, audio feature vector and modality type label corresponding to each timestamp are concatenated to obtain the time sequence input sequence; The time-series input sequence is fed into a multimodal feature fusion network to obtain contextual aggregation features; The context aggregation features are input into the classifier to obtain the test results.

[0102] In one possible implementation, the multimodal feature fusion network includes an encoding layer and a global pooling layer. The encoding layer includes a multi-head self-attention sublayer, a first residual connection sublayer, a first normalization sublayer, a feedforward network sublayer, a second residual connection sublayer, and a second normalization sublayer. The fourth determining module 304 is used for: The time-series input sequence is input into the multi-head self-attention sub-layer to obtain a first fusion feature sequence, which contains the fusion feature corresponding to each feature vector. The first fused feature sequence is input to the first residual connection sub-layer and the first normalization sub-layer to obtain the second fused feature sequence. The first residual connection sub-layer is used to add the first fused feature sequence to the time-series input sequence by residual addition. The first normalization sub-layer is used to normalize the output of the first residual connection sub-layer. The second fused feature sequence is input into the feedforward network sub-layer to obtain the third fused feature sequence. The feedforward network sub-layer is used to perform abstract extraction and feature enhancement on the second fused feature sequence. The third fused feature sequence is input into the second residual connection sub-layer and the second normalization sub-layer to obtain the fourth fused feature sequence. The second residual connection sub-layer is used to add the residual of the third fused feature sequence and the first fused feature sequence, and the second normalization sub-layer is used to normalize the output of the second residual connection sub-layer. The fourth fused feature sequence is input into the global pooling layer to obtain the context aggregation feature.

[0103] In one possible implementation, the classifier includes a fully connected sublayer and an activation function, and the fourth determining module 304 is used for: The context aggregation features are input into the fully connected sub-layer to obtain the test state score vector; The test state score vector is normalized by an activation function to obtain the confidence level corresponding to each test state, and the confidence level corresponding to each test state is the test result.

[0104] In one possible implementation, the fourth determining module 304 is used for: Based on the confidence level corresponding to each test state and the test result judgment conditions of the device under test, the test result type corresponding to the device under test is determined; Based on the test result type, determine the test task; Control the device under test to perform the test task.

[0105] In one possible implementation, a selection module is also included, used for: Depending on the video stream acquisition type supported by the device under test, select to acquire the video stream via a wired video interface or a wireless transmission protocol.

[0106] The test result determination apparatus provided in this application, employing the test result determination method in the above embodiments, can solve the technical problem of low accuracy in determining the test results related to the audio and video output performance of the device under test. Compared with the prior art, the beneficial effects of the test result determination apparatus provided in this application are the same as those of the test result determination method provided in the above embodiments, and other technical features in the test result determination apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0107] This application provides a test result determination device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the test result determination method in the above embodiment 1.

[0108] The following is for reference. Figure 4 The diagram illustrates a structural schematic of a device suitable for determining test results in accordance with embodiments of this application. The device for determining test results in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The device used to determine the test results shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0109] like Figure 4As shown, the device for determining test results may include a processing unit 401 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 402 or a program loaded from storage device 403 into random access memory (RAM) 404. RAM 404 also stores various programs and data required for the operation of the device for determining test results. The processing unit 401, ROM 402, and RAM 404 are interconnected via bus 405. Input / output (I / O) interface 406 is also connected to the bus. Typically, the following systems can be connected to I / O interface 406: input devices 407 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 408 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 403 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows the device for determining test results to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show devices for determining test results with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0110] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 403, or installed from ROM 402. When the computer program is executed by processing device 401, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0111] The test result determination device provided in this application, employing the test result determination method in the above embodiments, can solve the technical problem of low accuracy in determining the test results related to the audio and video output performance of the device under test. Compared with the prior art, the beneficial effects of the test result determination device provided in this application are the same as those of the test result determination method provided in the above embodiments, and other technical features in this test result determination device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0112] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0113] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0114] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the method for determining test results in the above embodiments.

[0115] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0116] The aforementioned computer-readable storage medium may be included in the device for determining the test results; or it may exist independently and not assembled into the device for determining the test results.

[0117] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by a device for determining test results, enable the device for determining test results to implement a method for determining test results.

[0118] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0119] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0120] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0121] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the method for determining the above-described test results. This addresses the technical problem of low accuracy in determining the test results related to the audio and video output performance of the device under test. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the method for determining test results provided in the above embodiments, and will not be elaborated upon here.

[0122] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method for determining test results as described above.

[0123] The computer program product provided in this application can solve the technical problem of low accuracy in test results for the determined audio and video output performance of the device under test. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the test result determination method provided in the above embodiments, and will not be repeated here.

[0124] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

[0125] It should be noted that the data collection, tag management, rule setting, and push decision-making processes involved in this application are designed to work with other technical features to solve technical problems. They do not involve or support any illegal activities. Any data processing that may violate laws and regulations (such as unauthorized collection of privacy data, generation of discriminatory tags, setting unfair rules, or pushing illegal information) is not within the scope of protection of this application's technical solution. Of course, the user data in this application will be encrypted, anonymized, or de-identified before storage to ensure user data security.

Claims

1. A method for determining test results, characterized in that, Includes the following steps: Visual features are extracted from the video frames corresponding to each timestamp in the video stream of the device under test, and the video feature vector corresponding to each timestamp is determined. If the video frame contains a text box, determine the semantic feature vector corresponding to the text in the text box; Audio features are extracted from the audio frames corresponding to each timestamp of the audio stream of the device under test, and the audio feature vector corresponding to each timestamp is determined. The test result is determined based on the visual feature vector, semantic feature vector, and audio feature vector corresponding to each timestamp.

2. The method according to claim 1, characterized in that, The step of determining the test result based on the visual feature vector, semantic feature vector, and audio feature vector corresponding to each timestamp includes: Align the visual feature vector, the semantic feature vector, and the audio feature vector along the feature dimension; Add corresponding modality type labels to the aligned visual feature vector, semantic feature vector, and audio feature vector, respectively; According to the time sequence of the timestamps, the aligned visual feature vector, semantic feature vector, audio feature vector and modality type label corresponding to each timestamp are concatenated to obtain the time sequence input sequence; The time-series input sequence is fed into a multimodal feature fusion network to obtain contextual aggregation features; The context aggregation features are input into the classifier to obtain the test results.

3. The method according to claim 2, characterized in that, The multimodal feature fusion network includes an encoding layer and a global pooling layer. The encoding layer includes a multi-head self-attention sublayer, a first residual connection sublayer, a first normalization sublayer, a feedforward network sublayer, a second residual connection sublayer, and a second normalization sublayer. The temporal input sequence is input into the multimodal feature fusion network to obtain contextual aggregated features, including: The time-series input sequence is input into the multi-head self-attention sub-layer to obtain a first fusion feature sequence, which contains the fusion feature corresponding to each feature vector. The first fused feature sequence is input to the first residual connection sub-layer and the first normalization sub-layer to obtain the second fused feature sequence. The first residual connection sub-layer is used to add the first fused feature sequence to the time-series input sequence by residual addition. The first normalization sub-layer is used to normalize the output of the first residual connection sub-layer. The second fused feature sequence is input into the feedforward network sub-layer to obtain the third fused feature sequence. The feedforward network sub-layer is used to perform abstract extraction and feature enhancement on the second fused feature sequence. The third fused feature sequence is input into the second residual connection sub-layer and the second normalization sub-layer to obtain the fourth fused feature sequence. The second residual connection sub-layer is used to add the residual of the third fused feature sequence and the first fused feature sequence, and the second normalization sub-layer is used to normalize the output of the second residual connection sub-layer. The fourth fused feature sequence is input into the global pooling layer to obtain the context aggregation feature.

4. The method according to claim 2, characterized in that, The classifier includes a fully connected sublayer and an activation function. The context aggregation features are input into the classifier to obtain test results, including: The context aggregation features are input into the fully connected sub-layer to obtain the test state score vector; The test state score vector is normalized by an activation function to obtain the confidence level corresponding to each test state, and the confidence level corresponding to each test state is the test result.

5. The method according to claim 4, characterized in that, After normalizing the test state score vector using an activation function to obtain the confidence level corresponding to each test state, the method further includes: Based on the confidence level corresponding to each test state and the test result judgment conditions of the device under test, the test result type corresponding to the device under test is determined; Based on the test result type, determine the test task; Control the device under test to perform the test task.

6. The method according to any one of claims 1-5, characterized in that, Before performing visual feature extraction on the video frames corresponding to each timestamp in the video stream of the device under test and determining the video feature vector corresponding to each timestamp, the method further includes: Depending on the video stream acquisition type supported by the device under test, select to acquire the video stream via a wired video interface or a wireless transmission protocol.

7. A device for determining test results, characterized in that, The device for determining the test results includes: The first determining module is used to extract visual features from the video frame corresponding to each timestamp in the video stream of the device under test, and determine the video feature vector corresponding to each timestamp. The second determining module is used to determine the semantic feature vector corresponding to the text in the text box when the video frame contains a text box; The third determining module is used to extract audio features from the audio frames corresponding to each timestamp of the audio stream of the device under test, and determine the audio feature vector corresponding to each timestamp. The fourth determining module is used to determine the test result based on the visual feature vector, the semantic feature vector, and the audio feature vector corresponding to each timestamp.

8. A device for determining test results, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the method for determining the test results as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the method for determining the test result as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the method for determining test results as described in any one of claims 1 to 6.

Images