Multimodal interaction testing method, apparatus, electronic device, and readable medium

Abstract

Embodiments of the present disclosure disclose a multimodal interaction test method, device, electronic equipment and readable medium. A specific embodiment of the method comprises: collecting multimodal interaction data generated in the execution process of a multimodal output test case; obtaining a multimodal interaction sample corresponding to the multimodal output test case; generating consistency detection information based on the multimodal interaction data, the multimodal interaction sample and a preset multimodal perception consistency detection model; and generating multimodal interaction test information based on the consistency detection information. This embodiment enables the test result to truly reflect the actual effect at the user perception level, reduces the occurrence of false positives (normal electrical signals but actual effect has defects) and false negatives (abnormal electrical signals but actual effect is acceptable), and improves the credibility of the interaction test.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computer technology, and more specifically to multimodal interaction testing methods, apparatus, electronic devices, and readable media. Background Technology

[0002] When mass-producing and configuring electronic devices, it is necessary to test the interactive effects, namely visual and auditory output effects, of the mass-produced electronic devices. Multimodal interaction testing is a technology for testing the interactive effects, namely visual and auditory output effects, of electronic devices (such as AR / VR display devices). Currently, the common methods for testing the interactive effects, namely visual and auditory output effects, of electronic devices (such as AR / VR display devices) are: using manual-assisted assertions or indirectly judging the test results by verifying log output or event triggering conditions.

[0003] However, when using the above methods to test the interactive effects, i.e., visual and auditory output effects, of electronic devices (such as AR / VR display devices), the following technical problems often arise: When using manual-assisted assertion, it relies on testers to make subjective comparisons through vision or hearing. The differences in sensory perception among different testers lead to inconsistent judgment standards, and the automated testing process is forced to be interrupted, making it difficult to achieve batch testing. When using log or event-based verification methods, only status flags or text logs at the electrical signal level can be obtained. Logs or event flags can only prove that the system or device "intends" to perform a certain operation, but cannot verify the final effect of the operation at the user perception level. For example, it is impossible to determine whether the actual screen has screen tearing or misalignment, or whether the actual sound has noise or sound field positioning errors, resulting in low credibility of interactive tests.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not form prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure provide multimodal interaction testing methods, apparatuses, electronic devices, and computer-readable media to address one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a multimodal interaction testing method, which includes: collecting multimodal interaction data generated during the execution of multimodal output test cases; obtaining multimodal interaction samples corresponding to the multimodal output test cases; generating consistency detection information based on the multimodal interaction data, the multimodal interaction samples, and a preset multimodal perception consistency detection model; and generating multimodal interaction test information based on the consistency detection information.

[0008] Secondly, some embodiments of this disclosure provide a multimodal interaction testing apparatus, the apparatus comprising: a collection unit configured to collect multimodal interaction data generated during the execution of multimodal output test cases; an acquisition unit configured to acquire multimodal interaction samples corresponding to the multimodal output test cases; a first generation unit configured to generate consistency detection information based on the multimodal interaction data, the multimodal interaction samples, and a preset multimodal perception consistency detection model; and a second generation unit configured to generate multimodal interaction test information based on the consistency detection information.

[0009] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0010] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above.

[0011] The above-described embodiments of this disclosure have the following beneficial effects: the multimodal interaction testing method of some embodiments of this disclosure improves the reliability of interaction testing. Specifically, the reason for the low reliability of interaction testing is that: the use of manual-assisted assertion relies on testers to make subjective comparisons through vision or hearing, which has a large subjective error, low efficiency of automated testing, and difficulty in batch testing. At the same time, when indirectly judging the test results by verifying log output or event triggering, only the status flags or text logs at the electrical signal level are obtained. The logs or event flags can only prove that the system or device "intends" to perform a certain operation (e.g., emits "play sound"), but cannot verify the final effect of the operation at the user perception level, which is prone to false positives (normal electrical signal but the actual picture or sound has defects that can be perceived by the human eye / ear) or false negatives (abnormal electrical signal but the actual output is acceptable), resulting in low reliability of interaction testing. Based on this, the multimodal interaction testing method of some embodiments of this disclosure first collects multimodal interaction data generated during the execution of multimodal output test cases. Thus, the interaction data (e.g., images or audio) generated during the test execution can be directly obtained, rather than indirectly from logs or status flags. Next, multimodal interaction samples corresponding to the aforementioned multimodal output test cases are acquired. This provides expected standard data for comparison, serving as a benchmark for determining test pass / fail. Then, based on the aforementioned multimodal interaction data, multimodal interaction samples, and a preset multimodal perception consistency detection model, consistency detection information is generated. Thus, the preset perception consistency detection model automatically and quantifiably compares the actual generated multimodal interaction data with the standard samples, calculating whether they are consistent at the perception level (e.g., whether visual features match, whether acoustic features align). Finally, based on the aforementioned consistency detection information, multimodal interaction test information is generated. This yields objective, quantifiable test assertion results (pass / fail) without human intervention. Because it employs the method of collecting actual multimodal interaction data and automatically comparing it with standard samples using a perception consistency detection model, it avoids the problems of inconsistent standards and process interruptions caused by relying on subjective human judgment. Meanwhile, because the visual or auditory signals ultimately presented to the user are directly verified, rather than just logs or event markers, the test results can truly reflect the actual effect at the user's perception level, reducing the occurrence of false positives (normal electrical signals but defective actual effects) and false negatives (abnormal electrical signals but acceptable actual effects), thus improving the credibility of interactive tests. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a flowchart of some embodiments of the multimodal interaction testing method according to the present disclosure; Figure 2 These are flowcharts of some other embodiments of the multimodal interaction testing method according to this disclosure; Figure 3 These are schematic diagrams illustrating the structure of some embodiments of the multimodal interaction testing apparatus according to this disclosure; Figure 4 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] Figure 1A flow 100 of some embodiments of a multimodal interaction testing method according to the present disclosure is shown. This multimodal interaction testing method includes the following steps: Step 101: Collect multimodal interaction data generated during the execution of multimodal output test cases.

[0021] In some embodiments, the execution entity of the multimodal interaction testing method (e.g., a computing device, such as an automated testing platform, AR / VR testing tool, graphical interface automated testing system, multimedia interactive system testing product, etc.) can collect multimodal interaction data generated during the execution of multimodal output test cases. In practice, the execution entity can call a preset interface of the device under test after the multimodal output test case execution is completed to capture an image of the current screen content of the device under test. This preset interface can be an API interface for screen capture. Subsequently, the execution entity can call the preset internal recording API interface of the device under test to collect the audio played during the execution of the multimodal output test case, obtaining audio data. Optionally, the execution entity can record audio during the execution of the multimodal output test case using an audio analyzer (such as a CRY6151B electroacoustic analyzer) to obtain audio data. Optionally, the audio signal played by the device under test during the execution of the multimodal output test case can be collected as audio data using a microphone placed at the speaker outlet of the device under test. The device under test can be an AR / VR device or a multimedia interactive system. The aforementioned preset internal recording API interface can be a standardized programming interface for capturing audio streams playing internally within the device. The aforementioned audio data can represent the sound played by the device under test during the execution of multimodal output test cases, and the audio data can be a time-domain audio signal. Finally, the acquired image and audio data of the current screen content are determined as multimodal interaction data. The aforementioned multimodal output test cases can be test cases designed for the various modal output effects generated by the device under test (such as AR / VR devices, multimedia interactive systems) during operation. The aforementioned multimodal output includes visual output and auditory output. The aforementioned visual output can include, but is not limited to, at least one of the following: images displayed on the screen, interface layout, animation effects, rendered images. The aforementioned auditory output can include, but is not limited to, at least one of the following: audio played by the device, voice prompts, sound effects, spatial audio. The aforementioned multimodal interaction data includes image data and audio data. The aforementioned image data can include images of the current screen content or extracted and recorded video frame sequences.

[0022] In some optional implementations of certain embodiments, the aforementioned execution entity may collect multimodal interaction data generated during the execution of multimodal output test cases through the following steps: The first step is to collect screen data of the currently displayed screen during the execution of the multimodal output test cases. In practice, the aforementioned execution entity can use Android's MediaProjection API to start a screen recording service during the execution of the multimodal output test cases and obtain a sequence of video frames as the screen data of the currently displayed screen.

[0023] The second step is to collect the audio played during the execution of the multimodal output test cases to obtain audio data.

[0024] The third step is to identify the aforementioned video data and audio data as multimodal interaction data.

[0025] In some optional implementations of certain embodiments, the aforementioned execution entity can collect screen data of the currently displayed screen during the execution of multimodal output test cases through the following steps: The first step is to start the screen recording service and record the screen changes during the execution of the above multimodal output test cases to obtain a sequence of recorded video frames.

[0026] The second step is to extract at least one recorded video frame from the aforementioned recorded video frame sequence to obtain the extracted recorded video frame sequence as the screen data for the current display. In practice, the aforementioned execution entity can extract at least one recorded video frame from the recorded video frame sequence at fixed intervals (e.g., 5 frames per second) and arrange the extracted at least one recorded video frame in sequence to obtain the extracted recorded video frame sequence.

[0027] Step 102: Obtain the multimodal interaction samples corresponding to the multimodal output test cases.

[0028] In some embodiments, the executing entity may obtain multimodal interaction samples corresponding to the multimodal output test cases. These multimodal interaction samples may be expected standard data serving as a judgment benchmark, corresponding to the multimodal output test cases. The multimodal interaction samples include standard screen data and standard audio data. The standard screen data may include standard screenshots or standard screen frame sequences. The standard screen frame sequences may represent reference images, interface screenshots, or rendered screens expected to be correctly displayed after executing the multimodal output test cases. The standard audio data may be reference audio, speech segments, or sound effect files expected to be correctly played after executing the multimodal output test cases. The multimodal interaction samples can be generated by recording with a standard reference device. In practice, multimodal interaction samples corresponding to the multimodal output test cases can be obtained from design standards or specification documents. Optionally, in practice, multimodal interaction samples corresponding to the multimodal output test cases can be obtained from a preset standard sample database. This preset standard sample database can be used to store multimodal interaction samples corresponding to the test cases. The multimodal interaction samples include standard screenshots and standard audio.

[0029] Step 103: Generate consistency detection information based on multimodal interaction data, multimodal interaction samples, and a preset multimodal perception consistency detection model.

[0030] In some embodiments, the executing entity may generate consistency detection information based on the multimodal interaction data, the multimodal interaction samples, and a preset multimodal perception consistency detection model. The consistency detection information may be information characterizing whether the multimodal interaction data and the multimodal interaction samples are similar.

[0031] In some optional implementations of certain embodiments, the aforementioned execution entity can generate consistency detection information based on the aforementioned multimodal interaction data, the aforementioned multimodal interaction samples, and a preset multimodal perception consistency detection model through the following steps: The first step involves inputting the aforementioned multimodal interaction data, multimodal interaction samples, and preset prompts into the aforementioned preset multimodal perceptual consistency detection model to obtain image similarity and audio similarity. Specifically, the image similarity is the similarity between the image data included in the multimodal interaction data and the standard image data included in the multimodal interaction samples; the audio similarity is the similarity between the audio data included in the multimodal interaction data and the standard audio data included in the multimodal interaction samples. The preset prompts can be predefined text instructions or parameter vectors used to guide the preset multimodal perceptual consistency detection model to perform specific judgment tasks (such as image similarity calculation or audio similarity calculation). The preset multimodal perceptual consistency detection model can be a pre-trained large model (e.g., GPT-4o large model, UI-TARS large model, etc.) capable of simultaneously processing visual and audio data to determine the similarity between the image data included in the multimodal interaction data and the standard image data included in the multimodal interaction samples, and to determine the similarity between the audio data included in the multimodal interaction data and the standard audio data included in the multimodal interaction samples.

[0032] The second step is to determine the above-mentioned image similarity and audio similarity as consistency detection information.

[0033] In addressing the aforementioned technical challenges by employing technical solutions, the following technical issues often arise in the interactive effect testing of the museum AR guide glasses application scenario: When directly comparing the collected complete video frame sequence with standard samples, the video frame sequence contains a large number of static redundant frames (such as static backgrounds in a standby state within a stable virtual scene when a visitor stops to view the same exhibit). Because consistency checks are required on all video frames (including static redundant frames), computational resources are consumed significantly, wasting computing power. To address the following requirements for this application scenario: It is necessary to automatically identify and remove static redundant frames from the video frame sequence, performing consistency checks only on key frames containing valid dynamic change information. Faced with these technical challenges, we have decided to adopt the following solution: The first step is to perform grayscale processing on the recorded video frame sequence included in the aforementioned multimodal interaction data to obtain a grayscale recorded video frame sequence. In practice, for each recorded video frame in the aforementioned recorded video frame sequence, the executing entity can sequentially record video frames and perform grayscale processing to obtain a grayscale recorded video frame sequence.

[0034] The second step is to obtain screenshots taken before the execution of the multimodal output test cases. These screenshots are static background images taken through the museum AR guide glasses before the execution of specific test operations (such as triggering the display of certain AR content), reflecting the current scene or standby state.

[0035] The third step is to perform grayscale processing on the screenshot to obtain a grayscale screenshot. In practice, the executing entity can perform grayscale processing on the screenshot to obtain a grayscale screenshot.

[0036] Fourth, for each grayscale recorded video frame in the above grayscale recorded video frame sequence, perform the following steps: Sub-step one: For each pixel in the grayscale recorded video frame above, perform the following steps: The first sub-step involves determining the position information of the pixels in the aforementioned grayscale recorded video frame as the target position information. This position information represents the location of the pixels within the grayscale recorded video frame. The position information can be coordinates; for example, it could be (2, 3).

[0037] The second sub-step is to determine the pixel value of the aforementioned grayscale recorded video frame as the first pixel value.

[0038] The third sub-step is to determine the pixel value of the pixel corresponding to the target location information in the grayscale screenshot as the second pixel value.

[0039] The fourth sub-step is to determine the absolute value of the difference between the first pixel value and the second pixel value.

[0040] The fifth sub-step involves determining, in response to the determination that the absolute value is greater than a preset threshold, the pixels of the grayscale recorded video frame are identified as pixels in the dynamically changing region.

[0041] Sub-step two: In response to the determination that the number of pixels in the determined dynamic change area is less than a preset threshold, the above grayscale recorded video frame is determined as a static frame.

[0042] Sub-step three involves deleting the recorded video frames in the recorded video frame sequence that correspond to the static frames mentioned above, in order to update the recorded video frame sequence.

[0043] The fifth step is to determine the updated recorded video frame sequence as the target recorded video frame sequence.

[0044] Step 6: Based on the audio data, the target recorded video frame sequence, the multimodal interaction samples, and the preset multimodal perception consistency detection model, generate consistency detection information.

[0045] The above-described technical solution and its related content, as an inventive point of this disclosure, solve the technical problem of "wasting computer computing resources." Factors leading to the waste of computer computing resources often include: when directly comparing a complete video frame sequence with a standard sample, the video frame sequence contains a large number of static redundant frames (such as the static background in a virtual scene where tourists are watching the same exhibit). Because consistency detection is required for all video frames (including static redundant frames), computational resources are consumed significantly, wasting computer computing resources. Solving these factors can reduce the waste of computer computing resources. To achieve this, firstly, the recorded video frame sequence included in the multimodal interaction data is grayscale processed to obtain a grayscale recorded video frame sequence. This converts the color image into a grayscale image, simplifying the color information of each pixel to a single brightness value. Then, a screenshot is acquired before the execution of the multimodal output test case. This yields a screenshot of the baseline static background image used for dynamic change detection. Next, the screenshot is grayscale processed to obtain a grayscale screenshot. Then, for each grayscale recorded video frame in the above grayscale recorded video frame sequence, the following steps are performed: First, for each pixel in the above grayscale recorded video frame, the following steps are performed: Step 1, determine the position information of the above grayscale recorded video frame pixel as target position information. Step 2, determine the pixel value of the above grayscale recorded video frame pixel as the first pixel value. Step 3, determine the pixel value of the pixel corresponding to the target position information in the grayscale screenshot as the second pixel value. Step 4, determine the absolute value of the difference between the first pixel value and the second pixel value. Step 5, in response to determining that the absolute value is greater than a preset threshold, determine the above grayscale recorded video frame pixel as a dynamically changing region pixel. Thus, through the above steps, the degree of difference between the current grayscale recorded video frame and the static background screenshot can be calculated pixel by pixel. Then, in response to determining that the number of determined dynamically changing region pixels is less than a preset threshold, determine the above grayscale recorded video frame as a static frame. Therefore, static redundant frames, i.e., frames with insufficient pixel count in dynamically changing areas, minimal difference from the static background, and almost no effective dynamic content, can be identified. Next, the recorded video frames corresponding to these static frames are deleted from the recorded video frame sequence to update the sequence. The updated recorded video frame sequence is then determined as the target recorded video frame sequence. Thus, the target recorded video frame sequence after removing static redundant frames is obtained. Then, based on the audio data included in the multimodal interaction data, the target recorded video frame sequence, the multimodal interaction samples, and the preset multimodal perception consistency detection model, consistency detection information is generated.Because it employs a method of first comparing pixel values ​​of grayscale recorded video frames with grayscale screenshots pixel by pixel, marking pixels in dynamically changing areas, and then determining and eliminating static redundant frames based on whether the number of pixels in dynamically changing areas is less than a preset threshold, the number of video frames required for consistency detection is reduced. This reduces the amount of video frame data that the subsequent consistency detection model needs to process, avoiding the waste of computing resources on processing a large number of repetitive static redundant frames with no effective change information, and reducing the waste of computer computing resources.

[0046] In addressing the aforementioned technical challenges by employing technical solutions, the application scenario—automatic audio-visual consistency detection in multimodal interactive systems during batch testing of intelligent audio-visual systems—often presents the following technical issues: When performing consistency detection on multimodal interaction data, video frame visual features exhibit multi-scale differences. Extracting only a single level of features cannot fully capture fine-grained and global features, resulting in poor accuracy and robustness of visual feature comparison and consequently, low reliability of the interaction test. Therefore, this application scenario requires the following: extraction of multi-scale visual features from video frames to achieve comprehensive visual feature comparison, ensuring reliable image similarity calculation, thereby improving the accuracy and robustness of multimodal interaction data consistency detection and meeting the practical needs of automated, batch testing.

[0047] In some optional implementations of certain embodiments, the aforementioned execution entity can generate consistency detection information based on the audio data included in the multimodal interaction data, the target recorded video frame sequence, the multimodal interaction samples, and a preset multimodal perception consistency detection model through the following steps: The first step involves inputting the target recorded video frame sequence and the standard frame sequence into the visual feature encoding layer of a pre-trained multimodal perceptual consistency detection model. This yields a first visual feature vector corresponding to the target recorded video frame sequence and a second visual feature vector corresponding to the standard frame sequence. The pre-trained multimodal perceptual consistency detection model includes the visual feature encoding layer, a feature projection layer, a feature pyramid extraction layer, a first output layer, an audio feature extraction layer, and a second output layer. The visual feature encoding layer can take the target recorded video frame sequence and the standard frame sequence as input, and output a high-dimensional numerical vector representing the visual attributes of the target recorded video frame sequence (i.e., the first visual feature vector) and a high-dimensional numerical vector representing the visual attributes of the standard frame sequence (i.e., the second visual feature vector).

[0048] The second step involves inputting the first visual feature vector and the second visual feature vector into the feature projection layer to obtain the first projected feature vector and the second shadow feature vector. The feature projection layer can be a projection layer that maps the input high-dimensional visual feature vector to a new space (that is, maps low-dimensional features to a higher dimension to enhance expressive power). The resulting first projected feature vector and second shadow feature vector represent the visual information of the target video frame and the standard frame, respectively.

[0049] The third step involves inputting the first projection feature vector and the second shadow feature vector into the feature pyramid extraction layer to obtain a first multi-scale feature vector set corresponding to the first projection feature vector and a second multi-scale feature vector set corresponding to the second shadow feature vector. The first multi-scale feature vector in the first multi-scale feature vector set corresponds one-to-one with the second multi-scale feature vector in the second multi-scale feature vector set. The feature pyramid extraction layer can be a neural network layer structure used to extract multi-scale spatial information from image features (i.e., the first projection feature vector and the second shadow feature vector), such as a spatial pyramid pooling network. Its core function is to sample the input feature vector (usually in the form of a two-dimensional feature map) through pooling windows of different sizes, generating multiple feature representations with different spatial resolutions, thereby simultaneously capturing the global semantic information (such as overall layout and color distribution) and local detail information (such as icon shape and text edges) of the image. The first multi-scale feature vector set can be a collection of multiple feature vectors at different scales output after processing the projection feature vectors of the target recorded video frame sequence through the feature pyramid extraction layer. Each scale feature vector corresponds to a specific spatial granularity, capturing visual information of the target recorded video frame sequence at different levels. The aforementioned second multi-scale feature vector set can be a collection of multiple feature vectors at different scales output after the projection feature vectors of a standard frame sequence have been processed through the same feature pyramid extraction layer. Its structure, dimensions, and scale division are completely consistent with the first multi-scale feature vector set.

[0050] The fourth step involves inputting the first multi-scale feature vector set and the second multi-scale feature vector set into the first output layer to generate image similarity. The first output layer can be used to determine the cosine similarity between the first and second multi-scale feature vector sets. The image similarity can be the cosine similarity between the first and second multi-scale feature vector sets (calculated by aligning the vectors one by one, then weighted and fused to obtain the cosine similarity). For example, the first multi-scale feature vector set includes at least one of the following: global layout feature vector (the overall interface structure and background environment corresponding to the target recorded video frame sequence), local region feature vector (the positions of icons and controls corresponding to the target recorded video frame sequence), detail texture feature vector (the edge sharpness and texture clarity corresponding to the target recorded video frame sequence), and animation timing feature vector (the animation timing features of the target recorded video frame sequence). The second multi-scale feature vector set includes at least one of the following feature vectors: global layout feature vector, local region feature vector, detail texture feature vector, and animation timing feature vector. The aforementioned global layout feature vector can be a vector formed by stacking or aggregating sub-vectors encoded with the overall structure and macroscopic attributes (e.g., interface layout structure, color distribution, lighting environment, etc.) of an image frame sequence (i.e., target recorded video frame sequence, standard frame sequence) in temporal order. The aforementioned local region feature vector can be a vector formed by stacking or aggregating sub-vectors encoded with the local features of the semantic regions (e.g., objects, icons, controls) of image frames in a temporal sequence in temporal order. The aforementioned detail texture feature vector can be a vector formed by stacking or aggregating sub-vectors encoded with the high-frequency detail information (e.g., edges, textures, corners) of image frames in a temporal sequence in temporal order. The aforementioned animation temporal feature vector can be a feature representation that encodes the dynamic change patterns of an image sequence in the time dimension.

[0051] The fifth step involves inputting the audio data included in the multimodal interaction data and the standard audio data included in the multimodal interaction samples into the audio feature extraction layer to obtain test audio feature vectors corresponding to the audio data and standard audio feature vectors corresponding to the standard audio data. The audio feature extraction layer can be a recurrent neural network (RNN) that outputs audio data and standard audio data, and outputs test audio feature vectors and standard audio feature vectors. The test audio feature vector can be a vector representing the time-domain or frequency-domain features of the audio data. The standard audio feature vector can be a vector representing the time-domain or frequency-domain features of the standard audio data.

[0052] The sixth step involves inputting the test audio feature vector and the standard audio feature vector into the second output layer to generate audio similarity. This second output layer can be used to determine the similarity between the test audio feature vector and the standard audio feature vector. The audio similarity can be represented using cosine similarity.

[0053] Step 7: The above-mentioned image similarity and audio similarity are determined as consistency detection information.

[0054] The above technical solution, combined with step 104 and related content, serves as an inventive point of this disclosure, solving the technical problem of "low credibility of interactive testing." Factors leading to low credibility of interactive testing often include: when performing consistency detection on multimodal interactive data, video frame visual features exhibit multi-scale differences. Single-level feature extraction alone cannot fully capture fine-grained and global features, resulting in poor accuracy of visual feature comparison and detection robustness, thus leading to low credibility of interactive testing. Solving these factors can improve the credibility of interactive testing. To achieve this, firstly, the target recorded video frame sequence and the standard screen frame sequence are input into the visual feature encoding layer of a pre-trained preset multimodal perception consistency detection model to obtain a first visual feature vector corresponding to the target recorded video frame sequence and a second visual feature vector corresponding to the standard screen frame sequence. The preset multimodal perception consistency detection model includes the visual feature encoding layer, feature projection layer, feature pyramid extraction layer, first output layer, audio feature extraction layer, and second output layer. Then, the first visual feature vector and the second visual feature vector are input into the feature projection layer to obtain the first projected feature vector and the second shadow feature vector. This allows visual feature vectors from different sources (test samples and standard samples) to be projected into a unified semantic space, eliminating inconsistencies in feature dimensions and obtaining the first projected feature vector and the second shadow feature vector. Next, the first projected feature vector and the second shadow feature vector are input into the feature pyramid extraction layer to obtain a first multi-scale feature vector set corresponding to the first projected feature vector and a second multi-scale feature vector set corresponding to the second shadow feature vector. The first multi-scale feature vector in the first multi-scale feature vector set corresponds one-to-one with the second multi-scale feature vector in the second multi-scale feature vector set. Thus, the feature pyramid extraction layer can separate and extract visual features at different spatial scales, obtaining feature vectors including global layout features (corresponding to the overall interface structure and background environment), local region features (corresponding to icon and control positions), detail texture features (corresponding to edge sharpness and texture clarity), and animation temporal features (corresponding to dynamic change trajectories). By employing a multi-scale separation extraction method, global features and local detail features no longer interfere with each other. Features at each scale can independently participate in subsequent similarity calculations, thus solving the technical deficiency of single-level feature extraction in failing to fully capture fine-grained and global features. Subsequently, the aforementioned first multi-scale feature vector set and the aforementioned second multi-scale feature vector set are input into the aforementioned first output layer to generate image similarity.Next, the audio data included in the multimodal interaction data and the standard audio data included in the multimodal interaction samples are input into the audio feature extraction layer to obtain the test audio feature vector corresponding to the audio data and the standard audio feature vector corresponding to the standard audio data. Then, the test audio feature vector and the standard audio feature vector are input into the second output layer to generate audio similarity. The image similarity and the audio similarity are determined as consistency detection information. Based on the consistency detection information from step 104, multimodal interaction test information is generated. Thus, objective and quantitative test assertion results, i.e., multimodal interaction test information, can be obtained. Because the feature pyramid extraction layer is used to extract visual features at multiple scales, visual features at different scales, such as global layout, local regions, detailed textures, and animation timing, can be extracted and compared independently. This avoids information loss or feature overload caused by single-scale feature extraction, thereby improving the accuracy of visual feature comparison and detection robustness, and ultimately increasing the credibility of the interaction test.

[0055] Step 104: Generate multimodal interaction test information based on consistency detection information.

[0056] In some embodiments, the aforementioned execution entity may generate multimodal interaction test information based on the aforementioned consistency detection information.

[0057] In some optional implementations of certain embodiments, the aforementioned execution entity may generate multimodal interaction test information based on the aforementioned consistency detection information through the following steps: The first step involves determining that the visual similarity of the consistency detection information is greater than a first preset similarity threshold and the audio similarity of the consistency detection information is greater than a preset second similarity threshold, thus identifying the information that passed the characterization test as multimodal interaction test information. The aforementioned information that passed the characterization test can be text information.

[0058] The second step involves determining that the similarity of the images included in the consistency detection information is less than or equal to a first preset similarity threshold, or the similarity of the audio included in the consistency detection information is less than or equal to a preset second similarity threshold, and classifying the information that failed the representation test as multimodal interaction test information. The aforementioned information that failed the representation test can be text information.

[0059] Optionally, the aforementioned implementing entity may also perform the following steps: The first step, in response to the determination that the multimodal interaction test information representation test failed, is to generate differential modality localization information based on the consistency detection information. In practice, the executing entity can use the consistency detection information as the differential modality localization information. This differential modality localization information includes visual similarity and audio similarity.

[0060] The second step involves generating and displaying at least one of the following based on the aforementioned differential modal localization information, multimodal interaction data, and multimodal interaction samples: a visual difference map and an audio difference spectrum. In practice, in response to determining that the screen similarity included in the differential modal localization information is less than or equal to a first preset similarity threshold, the screen data included in the modal interaction data is determined as the target screen data. The standard screen data included in the multimodal interaction samples is determined as the target standard screen data. In response to determining that the target screen data contains an image of the current screen content and the target standard screen data contains a standard screenshot, the image of the current screen content and the standard screenshot are stitched together, and the resulting image is determined as the visual difference map. In response to determining that the target screen data contains an extracted recorded video frame sequence and the target standard screen data contains a standard screen frame sequence, the following steps are performed: First, the extracted recorded video frames in the extracted recorded video frame sequence are stitched together sequentially from left to right according to their order in the extracted recorded video frame sequence to obtain a first stitched image. The second step involves stitching the standard frames in the standard frame sequence from left to right according to their order within the sequence to obtain the second stitched image. Then, the first and second stitched images are stitched together vertically to create the visual difference map. In response to the determination that the audio similarity of the differential modal localization information is less than or equal to a preset second similarity threshold, the audio data included in the modal interaction data is converted into a spectrogram as the first spectrogram using short-time Fourier transform (SFT). Similarly, the standard audio data included in the multimodal interaction samples is converted into a spectrogram as the second spectrogram using SFT. Finally, the first and second spectrograms are stitched together, and the resulting spectrogram is used to determine the audio difference spectrum.

[0061] The above-described embodiments of this disclosure have the following beneficial effects: the multimodal interaction testing method of some embodiments of this disclosure improves the reliability of interaction testing. Specifically, the reason for the low reliability of interaction testing is that: the use of manual-assisted assertion relies on testers to make subjective comparisons through vision or hearing, which has a large subjective error, low efficiency of automated testing, and difficulty in batch testing. At the same time, when indirectly judging the test results by verifying log output or event triggering, only the status flags or text logs at the electrical signal level are obtained. The logs or event flags can only prove that the system or device "intends" to perform a certain operation (e.g., emits "play sound"), but cannot verify the final effect of the operation at the user perception level, which is prone to false positives (normal electrical signal but the actual picture or sound has defects that can be perceived by the human eye / ear) or false negatives (abnormal electrical signal but the actual output is acceptable), resulting in low reliability of interaction testing. Based on this, the multimodal interaction testing method of some embodiments of this disclosure first collects multimodal interaction data generated during the execution of multimodal output test cases. Thus, the interaction data (e.g., images or audio) generated during the test execution can be directly obtained, rather than indirectly from logs or status flags. Next, multimodal interaction samples corresponding to the aforementioned multimodal output test cases are acquired. This provides expected standard data for comparison, serving as a benchmark for determining test pass / fail. Then, based on the aforementioned multimodal interaction data, multimodal interaction samples, and a preset multimodal perception consistency detection model, consistency detection information is generated. Thus, the preset perception consistency detection model automatically and quantifiably compares the actual generated multimodal interaction data with the standard samples, calculating whether they are consistent at the perception level (e.g., whether visual features match, whether acoustic features align). Finally, based on the aforementioned consistency detection information, multimodal interaction test information is generated. This yields objective, quantifiable test assertion results (pass / fail) without human intervention. Because it employs the method of collecting actual multimodal interaction data and automatically comparing it with standard samples using a perception consistency detection model, it avoids the problems of inconsistent standards and process interruptions caused by relying on subjective human judgment. Meanwhile, because the visual or auditory signals ultimately presented to the user are directly verified, rather than just logs or event markers, the test results can truly reflect the actual effect at the user's perception level, reducing the occurrence of false positives (normal electrical signals but defective actual effects) and false negatives (abnormal electrical signals but acceptable actual effects), thus improving the credibility of interactive tests.

[0062] Further reference Figure 2 This illustrates a flow 200 of another embodiment of a multimodal interaction testing method. Flow 200 of this multimodal interaction testing method includes the following steps: Step 201: Collect multimodal interaction data generated during the execution of multimodal output test cases.

[0063] In some embodiments, the execution entity of multimodal interaction testing (e.g., a computing device) can collect multimodal interaction data generated during the execution of multimodal output test cases.

[0064] Step 202: Obtain the multimodal interaction samples corresponding to the multimodal output test cases.

[0065] In some embodiments, the execution entity may obtain multimodal interaction samples corresponding to the multimodal output test cases.

[0066] Step 203: Generate consistency detection information based on multimodal interaction data, multimodal interaction samples, and a preset multimodal perception consistency detection model.

[0067] In some embodiments, the execution entity may generate consistency detection information based on the multimodal interaction data, the multimodal interaction samples, and the preset multimodal perception consistency detection model.

[0068] Step 204: Generate multimodal interaction test information based on consistency detection information.

[0069] In some embodiments, the aforementioned execution entity may generate multimodal interaction test information based on the aforementioned consistency detection information.

[0070] Step 205: In response to the determination that the multimodal interaction test information representation test fails, differential modality localization information is generated based on the consistency detection information.

[0071] In some embodiments, the aforementioned execution entity may, in response to determining that the multimodal interaction test information characterization test has failed, generate differential modality localization information based on the consistency detection information.

[0072] Step 206: Based on the differential modal localization information, multimodal interaction data, and multimodal interaction samples, generate and display at least one of the following: visual differential map and audio differential spectrum.

[0073] In some embodiments, the aforementioned executing entity may generate and display at least one of the following based on the aforementioned differential modal localization information, the aforementioned multimodal interaction data, and the aforementioned multimodal interaction samples: a visual differential map and an audio differential spectrum.

[0074] from Figure 2 It can be seen from this that, with Figure 1 Compared to the description of some corresponding embodiments, Figure 2 The multimodal interaction testing methods in some corresponding embodiments are further optimized based on the original process. Following... Figure 1 After the process generates multimodal interaction test information, Figure 2 The system adds a feature where, in response to a failure of the multimodal interaction test information representation test, differential modality localization information is generated based on consistency detection information. Based on this differential modality localization information, the multimodal interaction data, and the multimodal interaction samples, at least one of a visual difference map and an audio difference spectrum is generated and displayed. Generating a visual difference map visually presents the differences in image modalities in a graphical way, clearly showing differences in color, shape, texture, etc., helping testers quickly locate image problems. Generating an audio difference spectrum visualizes the differences in audio modalities in the form of a spectrum, making differences in audio frequency, amplitude, and other characteristics immediately apparent. This facilitates the discovery of distortion, noise, and other problems in the audio, helping testers locate and resolve issues and improving the user experience (e.g., testers).

[0075] Further reference Figure 3 As an implementation of the methods shown in the figures, this disclosure provides some embodiments of a multimodal interaction testing apparatus, which are similar to... Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0076] like Figure 3 As shown, a multimodal interaction testing device 300 in some embodiments includes: a data acquisition unit 301, an acquisition unit 302, a first generation unit 303, and a second generation unit 304. The data acquisition unit 301 is configured to acquire multimodal interaction data generated during the execution of multimodal output test cases; the acquisition unit 302 is configured to acquire multimodal interaction samples corresponding to the multimodal output test cases; the first generation unit 303 is configured to generate consistency detection information based on the multimodal interaction data, the multimodal interaction samples, and a preset multimodal perception consistency detection model; and the second generation unit 304 is configured to generate multimodal interaction test information based on the consistency detection information.

[0077] It is understandable that the units described in the device 300 are related to the reference. Figure 1 The steps in the method described above correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device 300 and the units contained therein, and will not be repeated here.

[0078] The following is for reference. Figure 4 It shows a schematic diagram of the structure of an electronic device 400 suitable for implementing some embodiments of the present disclosure. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0079] like Figure 4 As shown, electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 401, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 402 or a program loaded from storage device 408 into random access memory (RAM) 403. RAM 403 also stores various programs and data required for the operation of electronic device 400. Processing device 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0080] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows electronic device 400 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 An electronic device 400 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 4 Each box shown can represent a device or multiple devices as needed.

[0081] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 409, or installed from storage device 408, or installed from ROM 402. When the computer program is executed by processing device 401, it performs the functions defined in the methods of some embodiments of this disclosure.

[0082] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0083] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0084] A computer-readable medium may be contained within an electronic device or may exist independently, not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: collect multimodal interaction data generated during the execution of multimodal output test cases; acquire multimodal interaction samples corresponding to the aforementioned multimodal output test cases; generate consistency detection information based on the aforementioned multimodal interaction data, the aforementioned multimodal interaction samples, and a preset multimodal perception consistency detection model; and generate multimodal interaction test information based on the aforementioned consistency detection information.

[0085] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0086] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0087] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a data acquisition unit, an acquisition unit, a first generation unit, and a second generation unit. The names of these units do not necessarily limit the specific unit; for example, a generation unit may be described as "a unit that generates consistency detection information based on the aforementioned multimodal interaction data, the aforementioned multimodal interaction samples, and a preset multimodal perception consistency detection model."

[0088] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0089] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of technical features, but should also cover other technical solutions formed by arbitrary combinations of technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A multimodal interaction testing method, comprising: Collect multimodal interaction data generated during the execution of multimodal output test cases; Obtain the multimodal interaction samples corresponding to the multimodal output test cases; Based on the multimodal interaction data, the multimodal interaction samples, and the preset multimodal perception consistency detection model, consistency detection information is generated; Based on the consistency detection information, multimodal interaction test information is generated.

2. The method according to claim 1, wherein, The method further includes: In response to determining that the multimodal interaction test information characterization test fails, differential modality localization information is generated based on the consistency detection information; Based on the differential modal localization information, the multimodal interaction data, and the multimodal interaction samples, generate and display at least one of the following: a visual difference map and an audio difference spectrum.

3. The method according to claim 1, wherein, The multimodal interaction data generated during the execution of the multimodal output test cases includes: During the execution of multimodal output test cases, collect screen data of the currently displayed screen; During the execution of multimodal output test cases, the played audio is collected to obtain audio data; The video data and the audio data are identified as multimodal interaction data.

4. The method according to claim 3, wherein, The process of collecting screen data of the currently displayed screen during the execution of multimodal output test cases includes: Start the screen recording service to record the screen changes during the execution of the multimodal output test cases, and obtain the recorded video frame sequence; At least one recorded video frame is extracted from the recorded video frame sequence to obtain the extracted recorded video frame sequence as the screen data of the current display screen.

5. The method according to claim 4, wherein, The multimodal interaction samples include standard video data and standard audio data, and the consistency detection information generated based on the multimodal interaction data, the multimodal interaction samples, and a preset multimodal perception consistency detection model includes: The multimodal interaction data, the multimodal interaction samples, and the preset prompts are input into the preset multimodal perception consistency detection model to obtain image similarity and audio similarity. The image similarity is the similarity between the image data included in the multimodal interaction data and the standard image data included in the multimodal interaction samples, and the audio similarity is the similarity between the audio data included in the multimodal interaction data and the standard audio data included in the multimodal interaction samples. The image similarity and audio similarity are determined as consistency detection information.

6. The method according to claim 1, wherein, The consistency detection information includes image similarity and audio similarity, and the generation of multimodal interaction test information based on the consistency detection information includes: In response to determining that the similarity of the images included in the consistency detection information is greater than a first preset similarity threshold and the similarity of the audio included in the consistency detection information is greater than a preset second similarity threshold, the information representing the passing of the test is determined as multimodal interaction test information; In response to determining that the similarity of the images included in the consistency detection information is less than or equal to a first preset similarity threshold or the similarity of the audio included in the consistency detection information is less than or equal to a preset second similarity threshold, the information indicating that the test failed is identified as multimodal interaction test information.

7. A multimodal interaction testing device, comprising: The acquisition unit is configured to acquire multimodal interaction data generated during the execution of multimodal output test cases; The acquisition unit is configured to acquire multimodal interaction samples corresponding to the multimodal output test cases; The first generation unit is configured to generate consistency detection information based on the multimodal interaction data, the multimodal interaction samples, and a preset multimodal perception consistency detection model. The second generation unit is configured to generate multimodal interaction test information based on the consistency detection information.

8. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images