Electronic equipment testing method and equipment based on multi-modal feedback and storage medium

Abstract

The invention relates to the field of wireless product testing, and provides an electronic equipment testing method based on multi-modal feedback, and the method comprises the steps: receiving a test parameter set inputted by a user through a test platform interface; according to the test parameter set, an automatic execution mechanism is controlled to execute corresponding physical-level excitation operation on the tested electronic equipment; in the execution process of the physical-level excitation operation, response information generated by the tested electronic equipment is synchronously collected through at least two independent sensing channels; comparing a physical response signal acquired by an external response signal acquisition channel with an expected physical response in the expected equipment response, and comparing a logic state signal read by an equipment internal state feedback channel with an expected logic state in the expected equipment response; respectively obtaining a corresponding first comparison result and a second comparison result; and integrating the first comparison result and the second comparison result, and judging whether the current test case passes or not.

Description

Technical Field

[0001] This application relates to the field of wireless product testing, and in particular to a method, device, and storage medium for testing electronic devices based on multimodal feedback. Background Technology

[0002] With the rapid development of consumer electronics, smart home devices, and IoT devices, the functions of electronic devices are becoming increasingly complex, and their human-computer interaction methods are becoming more diversified, encompassing various forms such as physical buttons, indicator lights, touchscreens, and voice prompts. Ensuring the reliability and consistency of these human-computer interaction functions is a key aspect of product quality control, placing higher demands on automated testing technologies.

[0003] Currently, the mainstream methods for testing the human-computer interaction functions of electronic devices include traditional manual testing and single-modal automated testing. Manual testing relies on testers manually operating the device and observing its response, such as pressing a button, checking the screen display or indicator light status, listening to prompts, and judging whether the function is normal based on experience. To improve efficiency, the industry has also developed some automated testing methods, such as using robotic arms to simulate pressing buttons and reading the device's internal status codes through debugging interfaces such as serial ports (UART) for verification.

[0004] However, the aforementioned existing technical solutions have significant limitations. First, manual testing methods are highly subjective, inefficient, and prone to errors, and struggle to accurately reproduce complex interaction sequences such as long presses and double-clicks. Second, existing single-modal automated testing methods have blind spots. For example, verifying button functionality solely through internal status codes cannot detect external issues such as abnormal screen display content or missing prompt sounds. Summary of the Invention

[0005] This application provides a testing method, device, and storage medium for electronic devices based on multimodal feedback, which comprehensively solves the problem of perception blind spots in human-computer interaction function testing through synchronous detection and collaborative verification of internal and external states.

[0006] On the one hand, this application provides a testing method for electronic devices based on multimodal feedback, the method comprising: Receives the set of test parameters input by the user through the test platform interface; Based on the test parameter set, the automated actuator is controlled to perform corresponding physical-level excitation operations on the electronic device under test; During the physical-level excitation operation, the response information generated by the electronic device under test is synchronously acquired through at least two independent sensing channels, including an external response signal acquisition channel and an internal device status feedback channel. The physical response signal acquired by the external response signal acquisition channel is compared with the expected physical response in the expected device response, and the logic state signal read by the internal state feedback channel is compared with the expected logic state in the expected device response, to obtain the corresponding first comparison result and second comparison result respectively. Based on the combined results of the first and second comparisons, it is determined whether the current test case passes.

[0007] On the other hand, this application provides a testing device for electronic devices based on multimodal feedback, the device comprising: The receiving module is used to receive the set of test parameters input by the user through the test platform interface; The excitation module is used to control the automated actuator to perform corresponding physical-level excitation operations on the electronic device under test according to the test parameter set. The acquisition module is used to synchronously acquire the response information generated by the electronic device under test through at least two independent sensing channels during the execution of the physical-level excitation operation. The at least two independent sensing channels include an external response signal acquisition channel and an internal device status feedback channel. The comparison module is used to compare the physical response signal acquired by the external response signal acquisition channel with the expected physical response in the expected device response and to compare the logic state signal read by the internal state feedback channel with the expected logic state in the expected device response, so as to obtain the corresponding first comparison result and second comparison result respectively. The determination module is used to combine the first comparison result and the second comparison result to determine whether the current test case passes.

[0008] Thirdly, this application provides an electronic device, the device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described technical solution of the electronic device testing method based on multimodal feedback.

[0009] Fourthly, this application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described technical solution for testing electronic devices based on multimodal feedback.

[0010] As can be seen from the technical solution provided in this application, on the one hand, by adopting the technical means of synchronously collecting response information through at least two independent sensing channels, the testing process can cross-verify the internal and external states of the device, thereby overcoming the sensing blind spots of single-modal testing, discovering more potential defects that require inconsistencies between internal and external states to be exposed, and improving the comprehensiveness of test coverage and the reliability of results. Furthermore, the parameterized configuration steps allow the same testing system to quickly adapt to the testing needs of different products and different interaction scenarios by modifying configuration parameters, without changing the hardware architecture or core test code, thus improving the adaptability and flexibility of the testing process. On the other hand, by controlling the automated execution mechanism... Physical-level stimulation eliminates test deviations caused by inconsistent human operation force, rhythm, and angle, enabling the machine to accurately reproduce complex interaction sequences with high repeatability, thus ensuring the consistency of test case execution and the accuracy of stimulation operations. Thirdly, by comparing the physical response signals acquired through the external response signal acquisition channel with the expected physical responses in the expected device responses, and by comparing the logical state signals read from the internal device state feedback channel with the expected logical states in the expected device responses, the test system can make final decisions based on multi-dimensional evidence. This enhances the intelligence of the verification process, reduces test failures caused by false alarms from a single path, and improves test efficiency and accuracy. In summary, the technical solution of this application comprehensively solves the perception blind spot problem in human-computer interaction function testing through synchronous detection and collaborative verification of internal and external states. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a testing method for electronic devices based on multimodal feedback provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of the electronic device testing device based on multimodal feedback provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] In this specification, adjectives such as "first" and "second" are used only to distinguish one element or action from another, without necessarily requiring or implying any actual such relationship or order. Where circumstances permit, reference to an element or component or step (etc.) should not be construed as being limited to only one of the elements, components, or steps, but may be one or more of the elements, components, or steps, etc.

[0015] For ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale.

[0016] Currently, the mainstream methods for testing the human-computer interaction functions of electronic devices include traditional manual testing and single-modal automated testing. Manual testing relies on testers manually operating the device and observing its response, such as pressing a button, checking the screen display or indicator light status, listening to prompts, and judging whether the function is normal based on experience. To improve efficiency, the industry has also developed some automated testing methods, such as using robotic arms to simulate button presses and reading the device's internal status codes through debugging interfaces such as serial ports (UART) for verification; or capturing indicator light images with a camera and using simple image processing algorithms to analyze their color or on / off state. However, the above-mentioned existing technical solutions have significant limitations. First, manual testing methods are highly subjective, inefficient, and prone to errors, and it is difficult to accurately reproduce complex interaction sequences such as long presses and double-clicks. Second, existing single-modal automated testing methods have perceptual blind spots. For example, verifying button function solely through internal status codes cannot detect external performance problems such as abnormal screen display content or missing prompts; conversely, visual analysis of indicator lights alone cannot confirm whether their lighting logic is synchronized with the device's internal state machine. This separation of internal and external state verification results in incomplete test coverage, making it difficult to discover certain deep-seated interaction defects that require the linkage of internal and external states to be exposed, thus creating hidden dangers for product quality.

[0017] To address the aforementioned problems in the prior art, this application proposes a testing method for electronic devices based on multimodal feedback, the flowchart of which is attached. Figure 1 As shown, the main steps include S101 to S105, which are detailed below: Step S101: Receive the set of test parameters input by the user through the test platform interface.

[0018] In this embodiment, the test parameter set includes at least the test operation type, operation timing parameters, and expected device response corresponding to each test operation type for the human-computer interaction component of the device under test. These test parameter sets are loaded in the form of structured test case files, supporting dynamic updates during test execution without interrupting the entire test process.

[0019] Among them, the operation timing parameters refer to the specific parameters that define the operation action in the time dimension. They can ensure that the automated execution mechanism involved in subsequent steps can accurately reproduce various complex human interaction operations. For example, for the "short press" or "long press" operation, the timing parameter is the duration of the press. For the "double press" operation, the timing parameters include at least the duration of a single click and the interval between two clicks. For the "triple press" or more complex combination operation, the timing parameters are a set of parameters used to define the start and end times of each press. Regarding the expected device response corresponding to each test operation type, that is, the correct state that the user expects the device under test to exhibit after the test operation is successfully executed, specifically, for the "short press" operation, the expected device response includes two parts: the expected physical response (i.e., the screen interface changes from the "pause" icon to the "play" icon) and the expected logical state (i.e., receiving the "KEY_EVENT: PLAY_SHORT_PRESS" status code via the UART interface); for the "long press" operation, the expected device response may be that the device indicator light turns green and flashes twice per second and the system status changes to "charging"; for the "double press" operation, the expected response may be that the player jumps to the next song and the UART outputs "KEY_EVENT: NEXT_TRACK". The above embodiment solves the problems of existing test systems being rigid, inflexible, and unable to quickly adapt to different products and test scenarios by receiving the test parameter set input by the user through the test platform interface, thereby achieving decoupling between test logic and test execution and giving the test system higher flexibility and adaptability.

[0020] Step S102: Based on the test parameter set, control the automated actuator to perform the corresponding physical-level excitation operation on the electronic device under test.

[0021] Existing methods for testing electronic devices often involve manual button presses and operation by testers. Because the force, angle, and rhythm of these operations cannot be consistently maintained, this approach introduces significant random errors, resulting in unreliable test results and making continuous, 24 / 7 testing impossible. To address the inefficiencies, inconsistencies, fatigue, and inability to accurately reproduce complex timing sequences inherent in manual operation, this application proposes a technical solution that controls an automated actuator to perform corresponding physical-level excitation operations on the electronic device under test based on a set of test parameters. This solution ensures high repeatability, high accuracy, and high efficiency in the test excitation operations.

[0022] It should be noted that the automated actuator in the embodiments of this application can be a device such as a robotic arm that automatically performs some operations. The reason why the operation it performs is called a physical-level excitation operation is because the automated actuator applies an operation to the device entity in the real physical world that will produce mechanical contact or force effects.

[0023] As mentioned earlier, the test parameter set includes the test operation types for the human-machine interface components of the device under test (DUT). Here, the DUT's human-machine interface components can be physical buttons. Correspondingly, according to the test parameter set, controlling the automated actuator to perform corresponding physical-level excitation operations on the DUT can be: controlling the high-precision actuator module to perform pressing and releasing operations on the physical buttons, based on the test parameter set. Since the operation timing parameters define the specific parameters of the operation action in the time dimension, the pressing mode, pressing duration, and release interval of the high-precision actuator module on the physical buttons will all be performed according to the user's expected operation.

[0024] Step S103: During the physical-level excitation operation, the response information generated by the electronic device under test is collected synchronously through at least two independent sensing channels, wherein the at least two independent sensing channels include an external response signal acquisition channel and an internal status feedback channel of the device.

[0025] When collecting response information generated by the electronic device under test, if only the device status code is read through interfaces such as UART for verification, it is impossible to detect external faults such as the electronic device believing that the operation has been responded to, but the screen does not refresh, the indicator light does not turn on, or the speaker is silent. Furthermore, if only external responses are detected by sensors such as cameras or microphones, it is impossible to detect deep-seated logical errors where the device screen displays incorrect content due to driver issues, even though the internal status code is correct, or to distinguish between hardware faults and software logic errors. In other words, checking only the internal state cannot detect abnormal external behavior (e.g., screen display errors, lights not turning on, no sound, etc.), and checking only the external behavior cannot confirm whether the internal logic is correct (e.g., state machine disorder). To solve the problem of the perception blind spot of the single verification channel, the technical solution adopted in this application is to simultaneously collect the response information generated by the electronic device under test through at least two independent sensing channels during the physical-level excitation operation execution process. Here, at least two independent sensing channels include an external response signal acquisition channel and an internal device status feedback channel. The external response signal acquisition channel is used to acquire, via a non-invasive sensing device, physical response signals observable externally to the device, triggered by physical-level excitation operations. The internal device status feedback channel is used to read the internal logic status signals of the electronic device under test through its internal data interface. As one embodiment of this application, the internal device status feedback channel can be a data communication interface. Accordingly, reading the internal logic status signals of the electronic device under test through its internal data interface can be done via a UART, USB, or I2C interface, listening to or querying status codes or event messages issued by the device's internal system that identify its current functional status.

[0026] As an embodiment of this application, the external response signal acquisition channel can be a machine vision channel. Accordingly, acquiring the response information generated by the electronic device under test through at least two independent perception channels can be achieved by: using an image acquisition device to capture images of the human-computer interaction components or device display area through the machine vision channel; and performing recognition and analysis on the images of the human-computer interaction components or device display area. Specifically, performing recognition and analysis on the images of the human-computer interaction components or device display area can be achieved by: extracting the layout and texture features of interface elements from the images and inputting them into a pre-trained deep learning model for inference; the pre-trained deep learning model outputting a semantic-level description of the current interface functional state; and matching the semantic-level description with the expected state description in the expected device response.

[0027] In the above embodiments, semantic-level description is a textual or symbolic summary of the function, state, or meaning represented by the content of the current interface function state. Its essence is to enable the machine to understand the high-level abstract information of image content like a human, rather than staying at the low-level visual features such as pixels, colors, and lines. In addition, matching the semantic-level description with the expected state description in the expected device response can be achieved through the following steps S1 to S3: Step S1: Convert the semantic-level description text output by the deep learning model and the expected state description text preset in the parameterization configuration step into structured data formats with the same syntax and vocabulary (e.g., convert them into a set of keywords or attribute key-value pairs describing the device state); Step S2: Based on the structured data obtained in Step S1, calculate the semantic similarity score between the semantic-level description and the expected state description. This calculation process includes one or a combination of the following sub-steps: S2.1: Perform weighted matching calculation on keywords based on a predefined business rule dictionary; S2.2: Use a natural language processing model to vectorize the two description texts and calculate the cosine similarity; Step S3: Compare the semantic similarity score calculated in Step S2 with a preset confidence threshold. If the score is greater than or equal to the confidence threshold, the match is considered successful; otherwise, if the score is less than the confidence threshold, the match is considered unsuccessful.

[0028] In another embodiment of this application, the human-computer interaction component of the device under test can be an indicator light. Accordingly, the image recognition and analysis of the human-computer interaction component or the display area of ​​the device in the above embodiments can be achieved as follows: within a set sampling time window, acquire the image sequence of the indicator light; perform Region of Interest (ROI) analysis on each frame of the image, and calculate the mean and standard deviation of each region in the RGB and HSV color spaces; construct a brightness-time curve based on the image sequence of the indicator light, use an adaptive threshold algorithm to eliminate background noise, and accurately segment the bright and off state intervals by finding intersection points; perform Fast Fourier Transform analysis on the segmented bright and off state intervals to calculate their dominant frequency and harmonic components. In the above embodiments, the purpose of calculating the mean and standard deviation of each ROI in the RGB and HSV color spaces is to eliminate the influence of uneven ambient light spots, while the purpose of calculating the dominant frequency and harmonic components of the segmented bright and off state intervals through Fast Fourier Transform analysis is to determine the flicker frequency and duty cycle. As for constructing a brightness-time curve based on the image sequence of indicator lights, an adaptive threshold algorithm is used to eliminate background noise, and the on and off state intervals are accurately segmented by finding intersection points. This can be achieved through the following steps S1 to S3: Step S1: Based on the image sequence of indicator lights, extract the brightness feature values ​​of the region of interest of the indicator lights in each frame of the image in chronological order. Arrange the extracted brightness feature values ​​in chronological order to construct a brightness-time curve, where the brightness feature values ​​are the grayscale mean, brightness value, or intensity value of a specific color channel of all pixels in the region; Step S2: Traverse the entire curve with a sliding local time window and calculate the statistics of the data within the window (e.g., mean, Gaussian weighted mean, or median). And use this statistic as the dynamic segmentation threshold at the center point of the window, thereby generating a dynamically changing dynamic threshold curve corresponding to the brightness-time curve; Step S3: Compare the brightness-time curve obtained in step S1 with the dynamic threshold curve obtained in step S2 point by point, identify the intersection points of the two curves, mark each intersection point that crosses from below to above (its brightness value changes from below the threshold to above the threshold) as a bright state start point, and mark each intersection point that crosses from above to below (its brightness value changes from above the threshold to below the threshold) as an extinguished state start point. Based on these bright state start points and extinguished state start points, the entire time axis is divided into continuous bright state intervals and extinguished state intervals.

[0029] As another embodiment of this application, the external response signal acquisition channel in the above embodiment can be an audio signal acquisition channel. Accordingly, acquiring the response information generated by the electronic device under test through at least two independent sensing channels can be achieved by: using a microphone to acquire the sound emitted by the electronic device under test through the audio signal acquisition channel; performing feature analysis on the acquired audio signal to extract audio features, which include one or more of pitch, timbre, rhythm, duration, or speech content. Specifically, performing feature analysis on the acquired audio signal and extracting audio features can be achieved through steps S1031 to S1033, as detailed below: Step S1031: In the time domain, a time series elastic alignment algorithm is used to perform dynamic time warping to elastically match the shape of the measured audio sequence with the expected audio template and eliminate the influence of small fluctuations in playback rate.

[0030] Specifically, step S1031 can be implemented as follows: calculate the local distance between each point of the measured audio sequence and each point of the expected audio template sequence to form an M×N distance matrix D, where M and N are the lengths of the two sequences, respectively; starting from the upper left corner (1,1) of the distance matrix D, use a dynamic programming recursive formula to calculate the cumulative cost to reach each point in the matrix, and record the path selection to reach that point. This recursive process continues until the lower right corner (M,N) of the matrix, thereby finding the path from (1,1) to (M,N) with the minimum cumulative cost, i.e., the optimal regularization path; based on the optimal regularization path, align the measured audio sequence and the expected audio template sequence on the time axis; calculate the cumulative cost on the optimal regularization path, and use this cumulative cost or an index derived from it as the similarity score for temporal morphological matching.

[0031] Step S1032: In the frequency domain, extract the Mel frequency cepstral coefficients as acoustic fingerprints and perform similarity calculations with the pre-stored standard acoustic fingerprint database.

[0032] Step S1033: Combine the similarity scores in the time domain and frequency domain to form the final audio matching score.

[0033] Specifically, step S1033 can be implemented as follows: mapping the similarity scores in the time domain and frequency domain to comparable scaling ranges and normalizing them; combining the normalized time domain scores and frequency domain scores into a multi-dimensional feature vector; inputting the multi-dimensional feature vector into a predefined decision function to calculate a comprehensive matching score, wherein the decision function is a weighted summation function, a rule-based decision maker, or a trained machine learning classifier; comparing the comprehensive matching score with a global decision threshold; if the score is better than (greater than or equal to) the global decision threshold, a final decision of audio matching is formed; otherwise, a final decision of audio mismatch is formed.

[0034] Step S104: Compare the physical response signal acquired by the external response signal acquisition channel with the expected physical response in the expected device response, and compare the logic state signal read by the internal state feedback channel with the expected logic state in the expected device response, to obtain the corresponding first comparison result and second comparison result respectively.

[0035] Step S105: Based on the combined results of the first and second comparisons, determine whether the current test case passes.

[0036] If a test is considered passed if only one comparison result passes, or if only the comparison result of one acquisition channel is considered valid, it is impossible to determine the correlation between internal and external states, leading to misjudgments or omissions. For example, if only the comparison result of the internal state feedback channel is considered, external defects will be missed; if only the comparison result of the external response signal acquisition channel is considered, internal logic errors will be missed, negating the significance of cross-validation. To achieve intelligent and accurate decision-making based on multi-dimensional evidence and improve the reliability and authority of test judgments, this application compares the physical response signal with the expected physical response and simultaneously compares the logic state signal with the expected logic state, combining the results of the two comparisons to determine whether the test case passes or fails.

[0037] Specifically, steps S104 and S105 can be implemented as follows: matching the acquired physical response signal with the expected physical response within a preset tolerance range; performing precise matching or semantic matching of the read logic state signal with the expected logic state; using an "AND" logical relationship for comprehensive judgment, that is, only when both matches are successful is the current test case determined to be passed. If the test case fails, a retry is automatically triggered according to a preset strategy; the system records test failure data and adaptively optimizes the parameter tolerance range in the test parameter set or the retry strategy based on historical data using a machine learning model.

[0038] As another embodiment of this application, steps S104 and S105 can also be implemented as follows: constructing a multimodal evidence fusion decision model; using the matching degree of the physical response signal, the matching degree of the logic state signal, and the current test environment parameters as input feature vectors; inputting the input feature vectors into a trained machine learning classifier for decision reasoning; the machine learning classifier outputs a comprehensive confidence score and a final pass or fail decision result, the decision boundary of which is obtained by training with historical test data. When the verification results of the physical response signal and the logic state signal are inconsistent, the root cause analysis subprocess is initiated. The root cause analysis subprocess includes: triggering additional test cases to isolate faulty components, retrieving historical inconsistent cases for similarity matching, or generating a detailed diagnostic report containing signal time series graphs and deviation details.

[0039] From the above appendix Figure 1 As illustrated by the example of a multimodal feedback-based electronic device testing method, on the one hand, the use of a technique that synchronously acquires response information through at least two independent sensing channels enables cross-verification of the device's internal and external states during the testing process. This overcomes the sensing blind spots of single-modal testing, allowing for the discovery of more potential defects that only surface when internal and external states are inconsistent, thus improving the comprehensiveness of test coverage and the reliability of results. Furthermore, the parameterized configuration steps allow the same testing system to quickly adapt to the testing needs of different products and interaction scenarios by modifying configuration parameters, without changing the hardware architecture or core test code, thereby improving the adaptability and flexibility of the testing process. On the other hand, by controlling automated execution mechanisms… The system performs physical-level excitation operations, eliminating test deviations caused by inconsistent human operation force, rhythm, and angle. It can accurately reproduce complex interaction sequences with high repeatability, ensuring the consistency of test case execution and the accuracy of excitation operations. Thirdly, by comparing the physical response signals acquired by the external response signal acquisition channel with the expected physical responses in the expected device responses, and by comparing the logical state signals read from the internal device state feedback channel with the expected logical states in the expected device responses, the test system can make final decisions based on multi-dimensional evidence. This improves the intelligence of the verification process, reduces test failures caused by false alarms from a single path, and enhances test efficiency and accuracy. In summary, the technical solution of this application comprehensively solves the perception blind spot problem in human-computer interaction function testing through synchronous detection and collaborative verification of internal and external states.

[0040] Please see the appendix Figure 2 This application provides a testing device for electronic devices based on multimodal feedback. The device may include a receiving module 201, an excitation module 202, a data acquisition module 203, a comparison module 204, and a judgment module 205, as detailed below: The receiving module 201 is used to receive the set of test parameters input by the user through the test platform interface; The excitation module 202 is used to control the automated actuator to perform corresponding physical-level excitation operations on the electronic device under test according to the test parameter set; The acquisition module 203 is used to simultaneously acquire the response information generated by the electronic device under test through at least two independent sensing channels during the execution of the physical-level excitation operation. The at least two independent sensing channels include an external response signal acquisition channel and an internal device status feedback channel. The comparison module 204 is used to compare the physical response signal acquired by the external response signal acquisition channel with the expected physical response in the expected device response and to compare the logic state signal read by the internal state feedback channel with the expected logic state in the expected device response, so as to obtain the corresponding first comparison result and second comparison result respectively. The judgment module 205 is used to combine the first comparison result and the second comparison result to determine whether the current test case passes.

[0041] From the above appendix Figure 2 As illustrated by the example of a multimodal feedback-based electronic device testing apparatus, on the one hand, the use of a technique that synchronously acquires response information through at least two independent sensing channels enables cross-verification of the device's internal and external states during the testing process. This overcomes the sensing blind spots of single-modal testing, allowing for the discovery of more potential defects that only surface when internal and external states are inconsistent, thus improving the comprehensiveness of test coverage and the reliability of results. Furthermore, the parameterized configuration steps allow the same testing system to quickly adapt to the testing needs of different products and interaction scenarios by modifying configuration parameters, without changing the hardware architecture or core test code, thereby improving the adaptability and flexibility of the testing process. On the other hand, by controlling the automated execution mechanism… The system performs physical-level excitation operations, eliminating test deviations caused by inconsistent human operation force, rhythm, and angle. It can accurately reproduce complex interaction sequences with high repeatability, ensuring the consistency of test case execution and the accuracy of excitation operations. Thirdly, by comparing the physical response signals acquired by the external response signal acquisition channel with the expected physical responses in the expected device responses, and by comparing the logical state signals read from the internal device state feedback channel with the expected logical states in the expected device responses, the test system can make final decisions based on multi-dimensional evidence. This improves the intelligence of the verification process, reduces test failures caused by false alarms from a single path, and enhances test efficiency and accuracy. In summary, the technical solution of this application comprehensively solves the perception blind spot problem in human-computer interaction function testing through synchronous detection and collaborative verification of internal and external states.

[0042] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 3 As shown, the electronic device 3 in this embodiment mainly includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a program for a multimodal feedback-based electronic device testing method. When the processor 30 executes the computer program 32, it implements the steps described in the above embodiment of the multimodal feedback-based electronic device testing method, for example... Figure 1 The steps S101 to S105 are shown. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 2 The functions of the receiving module 201, the excitation module 202, the acquisition module 203, the comparison module 204, and the judgment module 205 are shown.

[0043] For example, the computer program 32 of the electronic device testing method based on multimodal feedback mainly includes: receiving a set of test parameters input by the user through the test platform interface; controlling an automated actuator to perform corresponding physical-level excitation operations on the electronic device under test according to the set of test parameters; synchronously acquiring response information generated by the electronic device under test through at least two independent sensing channels during the execution of the physical-level excitation operations, wherein the at least two independent sensing channels include an external response signal acquisition channel and an internal device state feedback channel; comparing the physical response signal acquired by the external response signal acquisition channel with the expected physical response in the expected device response and comparing the logic state signal read by the internal device state feedback channel with the expected logic state in the expected device response, respectively obtaining the corresponding first comparison result and second comparison result; combining the first comparison result and the second comparison result to determine whether the current test case passes. The computer program 32 can be divided into one or more modules / units, one or more modules / units are stored in the memory 31 and executed by the processor 30 to complete this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program 32 in the electronic device 3. For example, the computer program 32 can be divided into the functions of a receiving module 201, an excitation module 202, an acquisition module 203, a comparison module 204, and a judgment module 205 (a module in the virtual device). The specific functions of each module are as follows: The receiving module 201 is used to receive the test parameter set input by the user through the test platform interface; the excitation module 202 is used to control the automated actuator to perform corresponding physical-level excitation operations on the electronic device under test according to the test parameter set; the acquisition module 203 is used to simultaneously acquire the response information generated by the electronic device under test through at least two independent sensing channels during the execution of the physical-level excitation operation, wherein the at least two independent sensing channels include an external response signal acquisition channel and an internal device state feedback channel; the comparison module 204 is used to compare the physical response signal acquired by the external response signal acquisition channel with the expected physical response in the expected device response and to compare the logical state signal read by the internal device state feedback channel with the expected logical state in the expected device response, respectively obtaining the corresponding first comparison result and second comparison result; the judgment module 205 is used to combine the first comparison result and the second comparison result to determine whether the current test case passes.

[0044] Electronic device 3 may include, but is not limited to, processor 30 and memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0045] The processor 30 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0046] The memory 31 can be an internal storage unit of the electronic device 3, such as a hard disk or RAM. The memory 31 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 31 can include both internal and external storage units of the electronic device 3. The memory 31 is used to store computer programs and other programs and data required by the electronic device. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0047] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed. That is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above-described device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0048] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0049] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0050] In the embodiments provided in this application, it should be understood that the disclosed apparatus / device and method can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0051] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0052] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0053] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program for the electronic device testing method based on multimodal feedback can be stored in a storage medium. When the computer program is executed by a processor, it can implement the steps of the above method embodiments, namely, receiving a set of test parameters input by the user through the test platform interface; controlling the automated actuator to perform corresponding physical-level excitation operations on the electronic device under test according to the set of test parameters; during the execution of the physical-level excitation operation, synchronously collecting response information generated by the electronic device under test through at least two independent sensing channels, wherein the at least two independent sensing channels include an external response signal acquisition channel and an internal device state feedback channel; comparing the physical response signal acquired by the external response signal acquisition channel with the expected physical response in the expected device response and comparing the logic state signal read by the internal device state feedback channel with the expected logic state in the expected device response, respectively obtaining the corresponding first comparison result and second comparison result; combining the first comparison result and the second comparison result to determine whether the current test case passes. Computer programs include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of storage media can be appropriately added or removed according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, storage media do not include electrical carrier signals and telecommunication signals.

[0054] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application. The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the protection scope of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this invention.

Claims

1. A testing method for electronic devices based on multimodal feedback, characterized in that, The method includes: Receives the set of test parameters input by the user through the test platform interface; Based on the test parameter set, the automated actuator is controlled to perform corresponding physical-level excitation operations on the electronic device under test; During the physical-level excitation operation, the response information generated by the electronic device under test is simultaneously acquired through at least two independent sensing channels, including an external response signal acquisition channel and an internal device status feedback channel. The physical response signal acquired by the external response signal acquisition channel is compared with the expected physical response in the expected device response, and the logic state signal read by the internal state feedback channel is compared with the expected logic state in the expected device response, to obtain the corresponding first comparison result and second comparison result respectively. Based on the combined results of the first and second comparisons, it is determined whether the current test case passes.

2. The electronic device testing method based on multimodal feedback as described in claim 1, characterized in that, The external response signal acquisition channel is a machine vision channel. The acquisition of response information generated by the electronic device under test through at least two independent sensing channels includes: using an image acquisition device to capture an image of the human-computer interaction component or the display area of ​​the device through the machine vision channel; and performing recognition and analysis on the image of the human-computer interaction component or the display area of ​​the device.

3. The electronic device testing method based on multimodal feedback as described in claim 2, characterized in that, The human-machine interface component of the device under test is a physical button. The step of controlling the automated actuator to perform corresponding physical-level excitation operations on the electronic device under test according to the test parameter set includes: controlling the high-precision actuator module to perform pressing and releasing operations on the physical button according to the test parameter set. The image recognition and analysis of the display area of ​​the human-computer interaction component or device includes: The layout and texture features of the interface elements in the image are extracted and input into a pre-trained deep learning model for inference. The pre-trained deep learning model outputs a semantic-level description of the current interface functional state; The semantic-level description is matched with the expected state description in the expected device response.

4. The electronic device testing method based on multimodal feedback as described in claim 2, characterized in that, The human-computer interaction component of the device under test is an indicator light; the identification and analysis of the image of the human-computer interaction component or the display area of ​​the device includes: Within the set sampling time window, acquire the image sequence of the indicator lights; For each frame of the image, perform Region of Interest (ROI) analysis to divide the region and calculate the mean and standard deviation of each region in the RGB and HSV color spaces; A brightness-time curve is constructed based on the image sequence, an adaptive threshold algorithm is used to eliminate background noise, and the bright and dark state intervals are accurately segmented by finding the intersection points. Fast Fourier Transform analysis was performed on the segmented bright and dark state intervals to calculate their dominant frequency and harmonic components.

5. The electronic device testing method based on multimodal feedback as described in claim 1, characterized in that, The external response signal acquisition channel is an audio signal acquisition channel; The step of acquiring the response information generated by the electronic device under test through at least two independent sensing channels includes: using a microphone to acquire the sound emitted by the electronic device under test through the audio signal acquisition channel; The acquired audio signals are subjected to feature analysis to extract audio features.

6. The electronic device testing method based on multimodal feedback as described in claim 5, characterized in that, The audio features include one or more of pitch, timbre, rhythm, duration, or speech content.

7. The electronic device testing method based on multimodal feedback as described in claim 6, characterized in that, The feature analysis of the acquired audio signal and the extraction of audio features include: In the time domain, a time series elastic alignment algorithm is used for dynamic time warping to elastically match the shape of the measured audio sequence with the expected audio template and eliminate the influence of small fluctuations in playback rate. In the frequency domain, the Mel frequency cepstral coefficients are extracted as acoustic fingerprints, and similarity calculations are performed with a pre-stored standard acoustic fingerprint database. The final audio matching score is determined by combining the similarity scores in the time and frequency domains.

8. A testing device for electronic devices based on multimodal feedback, characterized in that, The device includes: The receiving module is used to receive the set of test parameters input by the user through the test platform interface; The excitation module is used to control the automated actuator to perform corresponding physical-level excitation operations on the electronic device under test according to the test parameter set. The acquisition module is used to synchronously acquire the response information generated by the electronic device under test through at least two independent sensing channels during the execution of the physical-level excitation operation. The at least two independent sensing channels include an external response signal acquisition channel and an internal device status feedback channel. The comparison module is used to compare the physical response signal acquired by the external response signal acquisition channel with the expected physical response in the expected device response and to compare the logic state signal read by the internal state feedback channel with the expected logic state in the expected device response, so as to obtain the corresponding first comparison result and second comparison result respectively. The determination module is used to combine the first comparison result and the second comparison result to determine whether the current test case passes.

9. An electronic device, the device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images