A performance test method, device, system and vehicle of an intelligent cockpit system

Abstract

The present application relates to the technical field of intelligent cockpit testing, and discloses a performance testing method, device and system of an intelligent cockpit system and a vehicle, wherein the present application sends a sequence of operation instructions to the intelligent cockpit system, synchronously triggers a video acquisition device to acquire a vehicle screen, obtains a video stream, performs semantic analysis on image frames, identifies attribute information of UI elements in each image frame, converts the sequence of image frames into a sequence of semantic states, performs key frame extraction, obtains an operation starting frame and a response completion frame, calculates the difference between the time stamps of the two key frames, obtains a response time length index, generates a performance testing report of the intelligent cockpit system in combination with resource load of the intelligent cockpit system and attribute information of user interface elements of each key frame, and through software instruction injection and visual post-analysis, the video stream is analyzed into a sequence of semantic states with business meaning, and key frames are extracted to accurately capture key nodes, thereby greatly improving the accuracy of response time quantification testing.

Description

Technical Field

[0001] This invention relates to the field of intelligent cockpit testing technology, and specifically to a performance testing method, apparatus, system, and vehicle for an intelligent cockpit system. Background Technology

[0002] As the automotive industry continues to evolve towards electrification, connectivity, and intelligence, vehicle intelligent cockpit systems have become the core carrier of human-machine interaction. Intelligent cockpit systems typically integrate voice interaction, touch display, multimedia entertainment, and various in-vehicle application (APP) ecosystems. During the product development and mass production stages, it is necessary to systematically test key dynamic performance indicators such as system cold and hot start-up time, application click response time, and interface switching latency.

[0003] Currently, mainstream automated testing solutions in the industry heavily rely on complex mechanical hardware. For example, a frame support assembly and a horizontal slide assembly are built on the passenger seat rail inside the vehicle, and a multi-axis robot drives a stylus to perform physical clicks. The test is completed with the help of pressure sensors and positioning cameras. It can be seen that the system is highly dependent on complex hardware such as drive motors and multi-axis robots. The hardware deployment cost is high and the structure is redundant. The physical bench usually needs to be rigidly connected to the actual vehicle seat rail and cumbersome physical alignment. In the dynamic driving test of the actual vehicle (such as on bumpy roads), the movement of the robotic arm is easily affected by the vibration of the vehicle body, resulting in physical inertial deviation, which leads to the distortion of the stylus click position and poses a safety hazard of scratching the vehicle's display screen. Moreover, the existing mechanical solutions usually calculate the "response time" from the "click action of the robotic arm" to the "overall pixel change detected by the industrial camera". Such coarse-grained detection is difficult to capture the semantic-level state changes inside the complex user interface (UI) and cannot accurately verify the real response time of each App under multi-task concurrency. It is difficult to meet the current needs of refined testing of smart cockpits. Summary of the Invention

[0004] This invention provides a performance testing method, apparatus, system, and vehicle for intelligent cockpit systems, to address the problem that physical testing cannot meet the requirements for refined testing of intelligent cockpits.

[0005] In a first aspect, the present invention provides a performance testing method for an intelligent cockpit system, applied to a performance testing system for an intelligent cockpit system. The performance testing system includes a video acquisition device and a control unit. The control unit establishes a low-level communication link with the intelligent cockpit system. The acquisition field of view of the video acquisition device covers the entire display area of ​​the vehicle screen. The method includes: according to preset test cases, sending an operation command sequence to the intelligent cockpit system through the control unit, and simultaneously triggering the video acquisition device to record the screen of the vehicle to obtain a video stream; after the test cases are executed, performing semantic parsing on the image frames in the video stream, identifying the attribute information of user interface elements in each image frame, and converting the image frame sequence into a time-ordered semantic state sequence based on the attribute information; extracting keyframes from the semantic state sequence to obtain an operation start frame and a response completion frame; calculating the difference between the timestamps corresponding to the operation start frame and the response completion frame to obtain a response duration index; and generating a performance test report for the intelligent cockpit system based on the response duration index, the resource load of the intelligent cockpit system, and the attribute information of the user interface elements corresponding to each key frame.

[0006] The performance testing method for an intelligent cockpit system provided by this invention sends an operation command sequence to the intelligent cockpit system according to preset test cases, and simultaneously triggers a video acquisition device to record the screen of the vehicle to obtain a video stream. After the test cases are executed, semantic parsing is performed on the image frames in the video stream to identify the attribute information of the user interface elements in each image frame. Based on the attribute information, the image frame sequence is converted into a semantic state sequence ordered by time. Keyframes are extracted from the semantic state sequence to obtain the operation start frame and response completion frame. The difference between the timestamps corresponding to the operation start frame and response completion frame is calculated to obtain the response time index. Based on the response time index, the resource load of the intelligent cockpit system, and the attribute information of the user interface elements corresponding to each key frame, a performance test report of the intelligent cockpit system is generated. By injecting software commands and combining visual post-hoc analysis, compared with physical touch simulation, hardware testing costs are significantly reduced, the lifespan of the vehicle screen is increased, and the video stream is parsed into a semantic state sequence with business meaning. Keyframes are extracted and key nodes are accurately captured, which greatly improves the accuracy of response time quantification testing.

[0007] In one optional implementation, the step of semantically parsing the image frames in the video stream to identify the attribute information of user interface elements in each image frame includes: splitting the video stream into an image frame sequence according to a preset frame rate; performing image preprocessing on the image frames in the image frame sequence, the image preprocessing including resolution normalization, distortion correction, and brightness compensation; calling a pre-trained deep learning model to perform semantic parsing on the image frames and outputting the attribute information of user interface elements in the image frames, the attribute information including at least element category, confidence level, and business semantic label, the business semantic label being used to characterize the business stage in which each image frame is during the execution of the test case.

[0008] This invention standardizes frame data by splitting video streams according to a preset frame rate, eliminates recognition interference caused by image distortion and brightness differences through preprocessing, extracts UI element attributes based on deep learning models, and uses business semantic tags to calibrate the test business stage, thereby improving the consistency and accuracy of interface state recognition and reducing misrecognition caused by environmental interference. It provides reliable underlying data support for subsequent construction of semantic state sequences and extraction of interaction key frames.

[0009] In one optional implementation, keyframe extraction is performed on the semantic state sequence to obtain the operation start frame, including: determining the image frame in the semantic state sequence that first appears with the operation feedback semantic state after the operation command is issued as the operation start frame, wherein the operation feedback semantic state is the semantic stage corresponding to the visual feedback signal generated by the vehicle interface after responding to the operation command.

[0010] This invention uses the initial visual feedback of the interface after the operation command is issued as the starting criterion, which is consistent with the real response logic of cockpit interaction and avoids the timing deviation caused by judging based on the moment of touch click.

[0011] In an optional implementation, the method further includes: if an operation feedback semantic state exists in multiple consecutive frame images, selecting the frame image with the earliest timestamp and corresponding confidence level reaching a preset threshold from the multiple frame images as the operation start frame.

[0012] In one optional implementation, keyframe extraction is performed on the semantic state sequence to obtain a response completion frame, including: determining the first image frame in the semantic state sequence in which the target content region has been presented and the target content region satisfies a preset stability condition within a preset number of consecutive image frames as the response completion frame, wherein the stability condition is determined by calculating the sum of squared pixel differences of the target content region in adjacent image frames and the structural similarity index.

[0013] This invention extracts response completion frames based on interface semantic state combined with multi-frame stability verification. Unlike traditional methods that rely solely on global pixel changes, it ensures complete rendering of the target interface functional areas by relying on UI semantic recognition. Furthermore, it verifies screen stability using dual indicators such as the sum of squared pixel differences and structural similarity index. This effectively avoids misjudgments caused by loading animations and screen jitter, eliminates single-frame judgment errors, accurately locks in the stable and usable moments of the cockpit interface, and significantly improves the accuracy of key frame extraction and the reliability of test results.

[0014] In one optional implementation, the target content region is a region within the vehicle interface that is prone to state changes and is related to the test case. Image frames in the semantic state sequence where the target content region has appeared and where the target content region satisfies a preset stability condition within a series of consecutive image frames are determined as response completion frames. This includes: calculating the sum of squared first pixel differences between the target content region in the current image frame and the previous image frame, and a first structural similarity index; if the sum of squared first pixel differences exceeds a first preset threshold and the first structural similarity index is lower than a second preset threshold, it is determined that the target content region has experienced an interface change state; calculating the sum of squared second pixel differences and a second structural similarity index corresponding to the target content region in subsequent adjacent image frames; when the sum of squared second pixel differences corresponding to the target content region in multiple consecutive adjacent image frames is lower than a third preset threshold and the second structural similarity index is higher than a fourth preset threshold, it is determined that the stability condition is met; and the current image frame is determined as a response completion frame.

[0015] In an optional implementation, before determining the current image frame as a response completion frame, the method further includes: inputting the current image frame into a pre-trained deep learning model, outputting attribute information of the user interface element corresponding to the current image frame, wherein the attribute information includes at least element category, confidence level, and semantic label; if the current image frame includes a semantic label of the target content region and the confidence level reaches a preset threshold, the current image frame is determined as a response completion frame.

[0016] In an optional implementation, the method further includes: monitoring the communication link status between the control unit and the intelligent cockpit system; resending the control command if no response is detected; and initiating emergency handling operations if no response is detected for a preset number of consecutive control commands, or if the communication link is disconnected, or if the resource load of the intelligent cockpit system is in an abnormal state.

[0017] Secondly, this invention provides a performance testing device for an intelligent cockpit system, applied to a performance testing system for an intelligent cockpit system. The performance testing system includes a video acquisition device and a control unit. The control unit establishes a low-level communication link with the intelligent cockpit system. The acquisition field of the video acquisition device covers the entire display area of ​​the vehicle's infotainment screen. The device includes: an instruction sending module, used to send a sequence of operation instructions to the intelligent cockpit system through the control unit according to preset test cases, and simultaneously trigger the video acquisition device to record the screen of the vehicle to obtain a video stream; a semantic parsing module, used to perform semantic parsing on the image frames in the video stream after the test cases are executed, identify the attribute information of user interface elements in each image frame, and convert the image frame sequence into a time-ordered semantic state sequence based on the attribute information; a key frame extraction module, used to extract key frames from the semantic state sequence to obtain an operation start frame and a response completion frame; and a performance testing module, used to calculate the difference between the timestamps corresponding to the operation start frame and the response completion frame to obtain a response duration index, and generate a performance test report for the intelligent cockpit system based on the response duration index, the resource load of the intelligent cockpit system, and the attribute information of the user interface elements corresponding to each key frame.

[0018] Thirdly, this invention provides a performance testing system for an intelligent cockpit system. The system includes a video acquisition device, a control unit, and a video analysis unit. The video acquisition device's field of view covers the entire display area of ​​the vehicle's infotainment screen. The control unit establishes a low-level communication link with the intelligent cockpit system. Specifically: the control unit sends an operation command sequence to the intelligent cockpit system according to preset test cases and simultaneously triggers the video acquisition device to record the screen's content, obtaining a video stream; the video analysis unit performs semantic analysis on the image frames in the video stream after the test cases are executed, identifies the attribute information of user interface elements in each image frame, and converts the image frame sequence into a time-ordered semantic state sequence based on the attribute information; the video analysis unit extracts keyframes from the semantic state sequence to obtain operation start frames and response completion frames; the video analysis unit calculates the difference between the timestamps corresponding to the operation start frame and the response completion frame to obtain a response duration index, and generates a performance test report for the intelligent cockpit system based on the response duration index, the resource load of the intelligent cockpit system, and the attribute information of the user interface elements corresponding to each key frame.

[0019] In one optional embodiment, the video acquisition device is mounted on a sliding bracket assembly in the adjustment bracket. The sliding bracket assembly is adjusted so that the video acquisition device faces the display center of the vehicle screen and its field of view covers the entire display area of ​​the vehicle screen. The adjustment bracket also includes a stabilizing component assembly, which cooperates with the sliding bracket assembly to securely clamp the vehicle screen.

[0020] In one alternative implementation, the stabilizing component includes fixtures distributed at four locations on the vehicle infotainment screen for securing the screen.

[0021] Fourthly, the present invention provides a vehicle that includes the intelligent cockpit system of the third aspect above or any corresponding embodiment thereof.

[0022] Fifthly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the performance testing method of the intelligent cockpit system described in the first aspect or any corresponding embodiment thereof.

[0023] In a sixth aspect, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the performance testing method of the intelligent cockpit system described in the first aspect or any corresponding embodiment thereof.

[0024] In a seventh aspect, the present invention provides a computer program product, including computer instructions, which are used to cause a computer to execute the performance testing method of the intelligent cockpit system described in the first aspect or any corresponding embodiment thereof.

[0025] The present invention has the following technical effects: The performance testing method for an intelligent cockpit system provided by this invention sends an operation command sequence to the intelligent cockpit system according to preset test cases, and simultaneously triggers a video acquisition device to record the screen of the vehicle to obtain a video stream. After the test cases are executed, semantic parsing is performed on the image frames in the video stream to identify the attribute information of the user interface elements in each image frame. Based on the attribute information, the image frame sequence is converted into a semantic state sequence ordered by time. Keyframes are extracted from the semantic state sequence to obtain the operation start frame and response completion frame. The difference between the timestamps corresponding to the operation start frame and response completion frame is calculated to obtain the response time index. Based on the response time index, the resource load of the intelligent cockpit system, and the attribute information of the user interface elements corresponding to each key frame, a performance test report of the intelligent cockpit system is generated. By injecting software commands and combining visual post-hoc analysis, compared with physical touch simulation, hardware testing costs are significantly reduced, the lifespan of the vehicle screen is increased, and the video stream is parsed into a semantic state sequence with business meaning. Keyframes are extracted and key nodes are accurately captured, which greatly improves the accuracy of response time quantification testing. Attached Figure Description

[0026] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of the architecture of a performance testing system for an intelligent cockpit system according to an embodiment of the present invention; Figure 2a This is a three-dimensional schematic diagram of the structure of the adjustment bracket according to an embodiment of the present invention; Figure 2b This is a front view of the adjustment bracket according to an embodiment of the present invention; Figure 2c This is a top view of the adjustment bracket according to an embodiment of the present invention; Figure 2d This is an enlarged example diagram of the fixator according to an embodiment of this aspect; Figure 3 This is a schematic diagram of the first process of a performance testing method for an intelligent cockpit system according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a second process for a performance testing method of an intelligent cockpit system according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the keyframe verification process according to an embodiment of the present invention; Figure 6This is a structural block diagram of a vehicle according to an embodiment of the present invention; Figure 7 This is a structural block diagram of a performance testing device for an intelligent cockpit system according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention.

[0028] The components include: adjustment bracket 100, cube frame bracket 1, longitudinal slide rail assembly 2, transverse slide rail assembly 3, video acquisition device 4, vehicle screen 5, transverse movable guide rail 6, four fasteners 7, and support column and fasteners 8. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0031] According to an embodiment of the present invention, a performance testing method for an intelligent cockpit system is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] This embodiment provides a performance testing method for an intelligent cockpit system, applied to a performance testing system for intelligent cockpit systems, such as... Figure 1As shown, the performance testing system includes a video acquisition device, a control unit, and a video analysis unit. The video acquisition device's field of view covers the entire display area of ​​the vehicle's infotainment screen. The control unit establishes a low-level communication link with the intelligent cockpit system to send operation command sequences to the intelligent cockpit system. Specifically, the control unit can communicate via debugging interfaces (such as Universal Serial Bus (USB) and Android Debug Bridge Protocol (Android Debug Bridge Protocol)... The DebugBridge (ADB) establishes a low-level communication link with the intelligent cockpit system. Based on preset test cases, it sends a sequence of operation commands to the intelligent cockpit system and simultaneously triggers a video capture device to record the screen, generating a video stream. This sequence of operation commands is a purely software-level, non-physical operation command sequence (including but not limited to app launch commands, page switching commands, etc.). After the test cases are executed, the video analysis unit performs semantic parsing on the image frames in the video stream, identifies the attribute information of user interface elements in each image frame, and converts the image frame sequence into a time-ordered semantic state sequence based on the attribute information. Keyframes are extracted from the semantic state sequence to obtain the operation start frame and response completion frame. The difference between the timestamps corresponding to the operation start frame and response completion frame is calculated to obtain the response duration index. Based on the response duration index, the resource load of the intelligent cockpit system, and the attribute information of the user interface elements corresponding to each keyframe, a performance test report for the intelligent cockpit system is generated. The video capture device supports a sampling rate of at least 60fps to ensure millisecond-level UI mutation capture; this is just an example.

[0033] Furthermore, the video acquisition device is mounted on the sliding bracket assembly in the adjustment bracket. By adjusting the sliding bracket assembly, the video acquisition device is positioned directly in front of the display center of the vehicle screen, and its field of view covers the entire display area of ​​the vehicle screen. The adjustment bracket also includes a stabilizing component assembly, which works in conjunction with the sliding bracket assembly to securely clamp the vehicle screen. The stabilizing component assembly includes fixers distributed in four directions on the vehicle screen for fixing the vehicle screen.

[0034] The adjustable bracket 100 designed in this embodiment of the invention is a multi-degree-of-freedom adjustable equipment fixture. It serves as the physical base of the entire device, supporting the vehicle-mounted display screen and video acquisition device. This eliminates the need for redundant components such as traditional rack supports and horizontal slides. The lightweight bracket with multi-degree-of-freedom adjustment capabilities provides a stable viewing angle for the video acquisition unit (such as a high-frame-rate industrial camera or mobile camera). The vehicle-mounted display screen is identical to the display screen used in the smart cockpit; it is only for testing and can be moved freely. This allows the adjustable bracket to be flexibly anchored to specific locations within the vehicle (such as the floor or trunk) without needing to be connected to the vehicle seat rails. This supports stable video acquisition under dynamic conditions such as road bumps and sharp turns. The vehicle-mounted display screen can be powered by an additional independent charging power supply and uses the same signal harness as the original vehicle screen. The smart cockpit host display output port is directly connected to the vehicle-mounted display screen driver board to achieve complete interactive signal transmission. Figure 2a , Figure 2b , Figure 2c As shown, the adjustable bracket 100 integrates a sliding bracket assembly and a stabilizing component assembly. The sliding bracket assembly supports multi-dimensional adjustment along a preset track to adapt to vehicle-mounted displays of different sizes. The sliding bracket assembly includes a longitudinal slide rail assembly 2, a transverse slide rail assembly 3, and a transverse movable guide rail 6. The longitudinal slide rail assembly 2 is z-oriented and has a slider to adjust the height of the video acquisition device 4, i.e., its relative distance to the vehicle-mounted screen 5, ensuring that the video acquisition device 4's field of view covers the entire display area of ​​the vehicle-mounted screen. The transverse slide rail assembly 3 is x-oriented, and there are two transverse movable guide rails 6, adjustable in their front-to-back direction, used to adjust the transverse slide rail assembly 3 and / or the transverse movable guide rails 6, ensuring that the optical axis of the video acquisition device 4 is aligned with the center of the vehicle-mounted screen 5 and remains perpendicular. The stabilizing component assembly includes a cubic frame bracket 1, four fixtures 7, and support columns and fixtures 8. For example... Figure 2d As shown, the four retainers are 1 / 4 arc structures with internal cushioning and anti-slip features, and enclosed sidewalls to prevent the screen from detaching. The positions of the four retainers can also be used to accurately determine the center point of the vehicle screen 5, facilitating manual adjustment of the sliding bracket to ensure that the optical axis of the video acquisition device 4 is aligned with the center of the vehicle screen 5 and remains perpendicular. These multiple measures work in parallel to effectively avoid image distortion or incomplete display caused by clamp vibration or axis misalignment, significantly reducing initial acquisition errors from a physical perspective. The support column and fixing component 8 are bolt structures, ensuring that the adjustment bracket can be placed stably on the vehicle, ensuring clamping stability during actual vehicle dynamic driving (including bumpy and turning conditions). This is just an example.

[0035] Figure 3 This is a flowchart of a performance testing method for an intelligent cockpit system according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps: Step S301: According to the preset test cases, the control unit sends an operation command sequence to the intelligent cockpit system and simultaneously triggers the video acquisition device to record the screen of the vehicle to obtain a video stream.

[0036] The control unit designed in this embodiment of the invention serves as the control core of the system and integrates a virtual instruction execution module. It can use ADB instruction sequences to simulate human touch behavior (including but not limited to single-finger click, drag, long press, etc.). It has more than 200 standardized test cases built-in, covering high-frequency interaction scenarios such as navigation path planning, air conditioning system adjustment, and concurrent switching of multiple applications. Compared with manual testing, the system control accuracy is greatly improved, the spatial positioning error can be controlled within a very small range, and human error interference is eliminated. Meanwhile, the system features a "failure re-execution" mechanism, which can automatically handle occasional communication interruptions during actual vehicle operation, improving the system redundancy and continuity of automated testing. Developers can establish a low-level communication connection between the control unit and the intelligent cockpit system, load preset test cases, and adjust the bracket position to ensure that the video capture device is directly facing the vehicle screen and in clear focus. At the start of the test, the control unit injects a non-physical contact operation command sequence into the intelligent cockpit system according to the preset test cases, and simultaneously triggers the video capture device to record the vehicle screen's image at a high frame rate throughout, obtaining a video stream. Furthermore, while injecting operation commands such as clicks, swipes, and application launches into the intelligent cockpit system, the control unit can write the command number, command issuance timestamp, and test case number into the test log, ensuring that the video frame sequence and the operation command sequence are on the same time reference. The video capture device is controlled by the synchronization signal issued by the control unit, achieving precise capture that records immediately upon command issuance and stops upon test case completion. The video capture device supports flexible adjustment of sampling resolution and frame rate according to test accuracy requirements. Generally speaking, the higher the resolution or the higher the frame rate, the clearer the captured image and the smaller the time error. However, it will also increase the data processing burden and slow down the processing speed. This invention takes effect directly at the software level through operation commands, eliminating the risk of irreversible scratches and damage to the vehicle screen and mechanical accidental touches that may be caused by the physical stylus when testing on bumpy roads.

[0037] Step S302: After the test cases are executed, semantic parsing is performed on the image frames in the video stream to identify the attribute information of the user interface elements in each image frame, and the image frame sequence is converted into a semantic state sequence ordered by time based on the attribute information.

[0038] In this embodiment of the invention, after the test cases are executed, the video analysis unit can split the video stream recorded by the video acquisition device to obtain several consecutive image frames. Then, semantic analysis can be performed on the image frames in the video. For example, a UI element recognition model can be used to detect user interface elements such as function buttons, loading animations, and pop-up prompts in the image frames one by one. At the same time, the element attribute information corresponding to the image frames can be identified, including but not limited to element status, such as click trigger, page loading, page rendering completion, and other status information. Semantic analysis can be performed on all image frames in the video sequentially. Alternatively, invalid noise frames in the video can be filtered in advance, such as black screen, overexposure, and other distorted frames, as well as noise frames with only slight camera disturbances and minor noise in the image, but whose core interactive UI element attributes have not changed. Then, semantic analysis can be performed on the image frames in the filtered video. Subsequently, the discrete single-frame UI semantic information can be integrated based on the acquisition timestamp of each image frame to form a semantic state sequence that changes continuously over time, so as to completely record the complete business state transition process of the vehicle interface from operation initiation to response completion during the test case execution.

[0039] Step S303: Extract keyframes from the semantic state sequence to obtain the operation start frame and response completion frame.

[0040] This invention can intelligently extract keyframes from semantic state sequences, identify key nodes where interface states change abruptly, and determine the image frame initiating the operation and triggering the loading action as the operation start frame. It continues to match states until the image frame in which the screen is completely rendered, the loading animation disappears, and the interface elements are stably displayed without flickering is determined as the response completion frame. This is just an example.

[0041] Step S304: Calculate the difference between the timestamps corresponding to the operation start frame and the response completion frame to obtain the response duration index. Based on the response duration index, the resource load of the intelligent cockpit system, and the attribute information of the user interface elements corresponding to each key frame, generate a performance test report of the intelligent cockpit system.

[0042] This invention can read the timestamps corresponding to the operation start frame and the response completion frame, calculate the difference between the two timestamps to obtain the system response time index, and simultaneously obtain the resource load data of the intelligent cockpit within this time interval (including but not limited to the CPU and memory utilization of the intelligent cockpit). Combined with the user interface element attributes corresponding to the key frames, a performance test report of the intelligent cockpit system is generated. The test report includes, but is not limited to, test time, test vehicle number, test case number, operation type, response time, CPU utilization, GPU load, and memory utilization. Furthermore, the performance test supports exporting in Excel, CSV, or PDF formats. This invention abandons the complex multi-axis robotic arm and horizontal sliding stage actuator, completely replacing the traditional "physical touch simulation" with "software instruction injection combined with visual post-hoc analysis," significantly reducing hardware procurement and daily maintenance costs.

[0043] The performance testing method for an intelligent cockpit system provided by this invention sends an operation command sequence to the intelligent cockpit system according to preset test cases, and simultaneously triggers a video acquisition device to record the screen of the vehicle to obtain a video stream. After the test cases are executed, semantic parsing is performed on the image frames in the video stream to identify the attribute information of the user interface elements in each image frame. Based on the attribute information, the image frame sequence is converted into a semantic state sequence ordered by time. Keyframes are extracted from the semantic state sequence to obtain the operation start frame and response completion frame. The difference between the timestamps corresponding to the operation start frame and response completion frame is calculated to obtain the response time index. Based on the response time index, the resource load of the intelligent cockpit system, and the attribute information of the user interface elements corresponding to each key frame, a performance test report of the intelligent cockpit system is generated. By injecting software commands and combining visual post-hoc analysis, compared with physical touch simulation, hardware testing costs are significantly reduced, the lifespan of the vehicle screen is increased, and the video stream is parsed into a semantic state sequence with business meaning. Keyframes are extracted and key nodes are accurately captured, which greatly improves the accuracy of response time quantification testing.

[0044] This embodiment provides a performance testing method for an intelligent cockpit system, applied to an intelligent cockpit system performance testing system. The performance testing system includes a video acquisition device and a control unit. The control unit establishes a low-level communication link with the intelligent cockpit system, and the video acquisition device's field of view covers the entire display area of ​​the vehicle's infotainment screen. Figure 4 This is a flowchart of a performance testing method for an intelligent cockpit system according to an embodiment of the present invention, such as... Figure 4 As shown, the process includes the following steps: Step S401: According to the preset test cases, the control unit sends a sequence of operation commands to the intelligent cockpit system, and simultaneously triggers the video capture device to record the screen of the vehicle, obtaining a video stream. For details, please refer to [link to relevant documentation]. Figure 3Step S301 of the illustrated embodiment will not be described again here.

[0045] Step S402: After the test cases are executed, semantic parsing is performed on the image frames in the video stream to identify the attribute information of the user interface elements in each image frame, and the image frame sequence is converted into a semantic state sequence ordered by time based on the attribute information.

[0046] Specifically, step S402 includes: Step S4021: The video stream is split into image frame sequences according to a preset frame rate.

[0047] Step S4022: Perform image preprocessing on the image frames in the image frame sequence. Image preprocessing includes resolution normalization, distortion correction, and brightness compensation.

[0048] Step S4023: Call the pre-trained deep learning model to perform semantic parsing on the image frames and output the attribute information of the user interface elements in the image frames. The attribute information includes at least element category, confidence level, and business semantic label. The business semantic label is used to characterize the business stage of each image frame during the execution of the test cases.

[0049] The video analysis unit designed in this embodiment of the invention may include a video decoding module, an image preprocessing module, a UI object detection module, a keyframe extraction module, and a timestamp calculation module. The video decoding module is used to split the acquired video stream into continuous image frames according to a preset frame rate. The image preprocessing module is used to perform image preprocessing on the image frames, including but not limited to resolution normalization, distortion correction, brightness compensation, and contrast enhancement (this is just an example). Then, the UI object detection module can call a pre-trained deep learning model (which may be an object detection model trained based on the YOLOv8 network structure) to perform semantic parsing on the image frames and output the user interface in the image frames. The attribute information of the interface elements includes, but is not limited to, application icons, navigation map areas, music playback controls, loading animations, video playback windows, status bars, function buttons, pop-up prompts, etc. Furthermore, the attribute information of the interface elements includes, but is not limited to, element category, confidence level, and business semantic tags. Among them, business semantic tags are used to characterize the business stage that each image frame is in during the execution of the test case. For example, business semantic tags can be "main interface has been displayed", "navigation layer has appeared", "music playback control has been loaded", "loading animation has appeared", "pop-up prompt has appeared", "target button is in an operable state", etc., which are just examples.

[0050] After semantic-level analysis is completed, the embodiments of the present invention can convert continuous image frames into a UI semantic state sequence ordered by time. For example, in the test case of "starting the navigation application", the system converts the original image frame sequence into a semantic state sequence of "desktop navigation icon visible - click feedback animation appears - loading animation appears - map container appears - road layer first frame rendering - map control stable display". This semantic state sequence is used to characterize the business process of the vehicle interface from operation to response completion.

[0051] This invention standardizes frame data by splitting video streams according to a preset frame rate, eliminates recognition interference caused by image distortion and brightness differences through preprocessing, extracts UI element attributes based on deep learning models, and uses business semantic tags to calibrate the test business stage, thereby improving the consistency and accuracy of interface state recognition and reducing misrecognition caused by environmental interference. It provides reliable underlying data support for subsequent construction of semantic state sequences and extraction of interaction key frames.

[0052] Step S403: Extract keyframes from the semantic state sequence to obtain the operation start frame and response completion frame.

[0053] Specifically, the operation start frame is obtained by extracting keyframes from the semantic state sequence through the following steps: the image frame in the semantic state sequence that first appears the operation feedback semantic state after the operation command is issued is determined as the operation start frame. The operation feedback semantic state is the semantic stage corresponding to the visual feedback signal generated by the vehicle interface after responding to the operation command.

[0054] The keyframe extraction module designed in this embodiment of the invention can take the semantic state sequence as input and perform frame-level positioning in combination with the preset test case state transition rules. For the operation start frame, the image frame in which the operation feedback semantic state appears for the first time after the operation command is issued can be selected as the operation start frame. The operation feedback semantic state is the semantic stage corresponding to the visual feedback signal generated by the vehicle interface after responding to the operation command, including but not limited to the operation feedback semantic state such as the first appearance of the click feedback animation, the switching of the interface focus, the change of button color or the change of transparency.

[0055] This invention uses the initial visual feedback of the interface after the operation command is issued as the starting criterion, which is consistent with the real response logic of cockpit interaction and avoids the timing deviation caused by judging based on the moment of touch click.

[0056] Furthermore, if the operation feedback semantic state exists in multiple consecutive frame images, the frame image with the earliest timestamp and the corresponding confidence level reaching a preset threshold is selected from the multiple frame images as the operation start frame.

[0057] If the present invention detects that the above-mentioned operation feedback semantic state exists in multiple consecutive frame images, the image frame with the earliest timestamp and the confidence level reaching the preset threshold can be selected from the multiple frame images as the operation start frame.

[0058] This invention solves the problems of misidentification of state in multiple consecutive frames and jitter interference by using confidence screening and the earliest timestamp selection mechanism. It effectively filters out invalid state frame interference caused by road bumps and slight noise in the image, greatly improves the positioning accuracy and consistency of the operation start frame, ensures accurate and reproducible calculation of subsequent response time, and improves the accuracy and reliability of intelligent cockpit performance testing.

[0059] In an optional implementation, the semantic state sequence can be keyframe extracted to obtain the response completion frame by the following steps: the first image frame in the semantic state sequence in which the target content region has been presented and the target content region satisfies the preset stability condition within a preset number of consecutive image frames is determined as the response completion frame. The stability condition is determined by calculating the sum of squared pixel differences of the target content region in adjacent image frames and the structural similarity index.

[0060] In this embodiment of the invention, when extracting the response completion frame, the end criterion is not solely based on a change in the image. Instead, it requires that the semantic state of the target content area has appeared, and that the target content area meets preset stability conditions within a subsequent preset number of consecutive image frames. The semantic state of the target content area includes, but is not limited to, the appearance of the target application's main interface or the completion of rendering of the target area. For example, a navigation application needs to detect that target semantic objects such as map containers, road layers, or positioning controls have appeared; a music application needs to detect that playback controls, song information areas, or album art areas have been displayed; and an air conditioning control interface needs to detect that target temperature, fan speed, or mode controls have entered an operable state. The first frame that meets the above semantic conditions and passes the consecutive frame stability verification is determined as the response completion key frame.

[0061] Specifically, stability conditions are determined by calculating the sum of squared pixel differences and the structural similarity index of the target content region in adjacent image frames. The sum of squared differences (SSD) measures the cumulative change in pixel differences between adjacent image frames of the target content region, while the structural similarity index (SSIM) measures the similarity of adjacent image frames in terms of brightness, contrast, and structural information. For example, a preset number of consecutive frames is selected as the verification interval, and the sum of squared pixel differences (SSD) and structural similarity index (SSIM) of adjacent image frames of the target content region are calculated frame by frame. When the sum of squared pixel differences is lower than a preset difference threshold and the structural similarity index is higher than a preset similarity threshold, it is determined that there is no flickering, jitter, or dynamic loading change in the image, and the stability conditions are met. Finally, the first image frame that simultaneously meets the conditions of complete rendering of the target content region and stability of multiple consecutive frames is determined as the response completion frame of this interaction. This is just an example.

[0062] In this embodiment of the invention, the image frame in which the semantic state of the target interface appears and passes the continuous frame stability verification is determined as the response completion frame. Thus, the key frame extraction is based on the semantic-level analysis results, rather than relying solely on global pixel changes.

[0063] This invention extracts response completion frames based on interface semantic state combined with multi-frame stability verification. Unlike traditional methods that rely solely on global pixel changes, it ensures complete rendering of the target interface functional areas by relying on UI semantic recognition. Furthermore, it verifies screen stability using dual indicators such as the sum of squared pixel differences and structural similarity index. This effectively avoids misjudgments caused by loading animations and screen jitter, eliminates single-frame judgment errors, accurately locks in the stable and usable moments of the cockpit interface, and significantly improves the accuracy of key frame extraction and the reliability of test results.

[0064] Furthermore, the target content region is a region within the vehicle interface that is prone to state changes and is related to the test case. Image frames in the semantic state sequence where the target content region has appeared and where the target content region meets preset stability conditions within a series of consecutive image frames are identified as response completion frames. This includes: calculating the sum of squared first pixel differences between the target content region in the current image frame and the previous image frame, and a first structural similarity index; if the sum of squared first pixel differences exceeds a first preset threshold and the first structural similarity index is lower than a second preset threshold, it is determined that the target content region has experienced an interface change state; calculating the sum of squared second pixel differences and a second structural similarity index corresponding to the target content region in subsequent adjacent image frames; when the sum of squared second pixel differences corresponding to the target content region in multiple consecutive adjacent image frames is lower than a third preset threshold and the second structural similarity index is higher than a fourth preset threshold, it is determined that the stability condition is met; and the current image frame is identified as the response completion frame.

[0065] This invention addresses the challenges of uncertain interface rendering and difficulty in determining response completion frames in network-connected applications (such as online Amap, in-car QQ Music, online video, or online voice assistants) due to network jitter, asynchronous rendering, or dynamic content refresh. It introduces a dual-compensation identification mechanism combining sensitive area monitoring and continuous frame mutation detection. In this embodiment, the in-vehicle screen can be divided into a global display area and one or more target sensitive areas (i.e., target content areas). The analysis strategy pre-defines areas within the in-vehicle screen (also called the in-vehicle interface) that are prone to state mutations and directly related to test cases. These include, but are not limited to, the core display area of ​​the navigation map, video playback window, music playback control area, loading animation area, and pop-up prompt area. This embodiment only performs subsequent mutation detection and stability judgment on the target content areas and can mask non-target areas such as the status bar, system time, vehicle signal icons, and background reflective areas, thereby reducing interference from irrelevant screen changes on response completion frame determination.

[0066] This invention introduces a combined algorithm of sum of squared pixel differences (SSD) and structural similarity (SSIM) in the target content region. For two adjacent frames (i.e., the current image frame Ft and the previous image frame Ft-1), the sum of squared pixel differences between the target content region R in the current image frame and the previous image frame can be calculated using the following formula: SSD(R)=Σ[It(x,y)-It-1(x,y)]² Wherein, SSD(R) represents the sum of squares of the first pixel difference corresponding to the target content region, which is used to quantify the pixel change amplitude in the region; It(x,y) represents the pixel gray value at coordinate (x,y) in the current frame image; It-1(x,y) represents the pixel gray value at coordinate (x,y) in the previous frame image.

[0067] Simultaneously, the first structural similarity index SSIM(Ft-1,Ft) is calculated to determine the similarity of adjacent image frames in terms of brightness, contrast, and structural information. When SSD exceeds the first preset threshold and SSIM is lower than the second preset threshold within a short period of time, the system determines that a significant structural change has occurred in the target content region. Subsequently, the system can calculate the second sum of squared pixel differences and the second structural similarity index corresponding to the target content region in subsequent adjacent image frames. When the second sum of squared pixel differences corresponding to the target content region in consecutive adjacent image frames is lower than the third preset threshold and the second structural similarity index is higher than the fourth preset threshold, the stability condition is met, and the current image frame can be determined as the response completion frame. The above thresholds can be calibrated according to screen resolution, acquisition frame rate, test case type, and vehicle dynamic conditions.

[0068] Calculate the sum of squared first pixel differences between the target content region in the current image frame and the previous image frame, as well as the first structural similarity index. If the sum of squared first pixel differences exceeds a first preset threshold and the first structural similarity index is lower than a second preset threshold, it is determined that the target content region has entered a sudden interface state. Calculate the sum of squared second pixel differences and the second structural similarity index corresponding to the target content region in subsequent adjacent image frames. When the sum of squared second pixel differences corresponding to the target content region in multiple consecutive adjacent image frames is lower than a third preset threshold and the second structural similarity index is higher than a fourth preset threshold, it is determined that the stability condition is met. The current image frame is determined as the response completion frame.

[0069] This invention does not rely solely on pixel mutation results to determine response completion frames. Instead, it jointly determines response completion frames by combining candidate mutation frames detected by SSD / SSIM with the UI semantic state output by the deep learning model. A candidate frame is only identified as a response completion frame when the target application's main interface, target layer, target control, or target content area has appeared, and subsequent consecutive frames remain stable. This avoids misjudging non-target changes such as screen reflections, vehicle vibrations, status bar time jumps, or ad carousels as response completion. Furthermore, the combination of SSD and SSIM is not a simple superposition. SSD is sensitive to rapid local changes, enabling timely capture of instantaneous changes such as loading completion, pop-up appearance, and screen switching. SSIM focuses on image structure consistency, eliminating interference such as brightness fluctuations, slight camera shake, and screen reflections that only cause pixel value changes without altering the interface structure. When used in conjunction with semantic-level target detection results, both can filter out truly business-meaning response completion nodes in complex scenarios such as "semantic objects not yet stably appearing," "network content being rendered in segments," and "vehicle movement causing screen shake."

[0070] In a specific embodiment, in the test case of "starting the online map and entering the main navigation interface", after the automatic control unit injects the navigation application start command into the vehicle system, the video acquisition device continuously captures the interface video. If the core map display area suddenly changes from a blank background map or loading animation to a map screen containing road layers, positioning points, and zoom controls, then the adjacent image frames typically show a rapid increase in the SSD value and a significant decrease in the SSIM value. If, in the following 3 to 5 frames, the SSD value of the core map display area falls back to a low level while the SSIM value remains at a high level, and the YOLOv8 model recognizes semantic objects such as "map layer", "positioning control", or "navigation control", then the system determines the image frame that first meets the stability condition after the mutation as the candidate response completion frame.

[0071] For example, in the test case of "opening the car music application and entering the playback interface," if the song cover, play / pause controls, and song information area change from an undisplayed state to a fully displayed state within the target sensitive area, SSD can capture the pixel abrupt change in this local area, and SSIM can determine whether this abrupt change is accompanied by a change in the interface structure. When the playback control area maintains structural stability for several consecutive frames after the abrupt change, and the semantic recognition result shows that the target control is in an operable state, the system identifies it as a candidate frame for response completion. For continuously changing content such as lyrics scrolling and progress bar micro-movements, the system can limit the target sensitive area or reduce its weight to avoid misjudging continuous small refreshes as new response completion events.

[0072] The unexpected technical effects of this are as follows: First, compared to simply using global pixel differences as the basis for response completion, this embodiment can distinguish between non-target area disturbances and target UI responses, significantly reducing the misjudgment rate under dynamic driving conditions; Second, compared to simply relying on deep learning target detection results, this embodiment can provide quantifiable inter-frame mutation evidence through SSD / SSIM when the confidence of the target semantic object fluctuates or network content is loaded asynchronously, thereby improving the stability of key frame localization; Third, the system can capture the moment when "the first frame of the target interface is effectively rendered and enters a short period of stability", making the response time closer to the usable response time actually perceived by the user, thus improving the engineering realism of the test indicators.

[0073] Optionally, before determining the current image frame as the response completion frame, the method further includes: inputting the current image frame into a pre-trained deep learning model, outputting the attribute information of the user interface element corresponding to the current image frame, the attribute information including at least element category, confidence level and semantic label; if the current image frame includes the semantic label of the target content region and the confidence level reaches a preset threshold, the current image frame is determined as the response completion frame.

[0074] In this embodiment of the invention, after determining candidate image frames through frame stability verification and before finally determining them as response completion frames, the selected stable candidate frames can be input into a pre-trained deep learning model. The model can then parse and output the UI element category, recognition confidence, and semantic label attributes of the frame. The model can verify whether the current image frame carries the exclusive semantic label of the target content region and determine whether the model's recognition confidence meets a preset threshold. If the current image frame includes the semantic label of the target content region and the confidence reaches the preset threshold, the current image frame is determined as a response completion frame.

[0075] In one alternative implementation, such as Figure 5As shown, if the video analysis unit verifies that the current image frame does not carry a dedicated semantic label for the target content region or that the confidence level does not meet the preset threshold, it can mark the current image frame as an anomaly, labeling it "to be confirmed as a valid response," and synchronize it to the visualization decision terminal. Testers can use tools provided by the terminal, such as frame-by-frame tracing, local image magnification, precise timeline search, SSD / SSIM curve display, and semantic recognition box overlay display, to quickly review and confirm the marked frame. After review and confirmation, the current image frame is determined as a response completion frame. This mechanism effectively solves the problem of misjudgment caused by dynamic content loading in complex network environments, ensuring that test results are accurate, interpretable, and traceable under complex working conditions.

[0076] Step S404: Calculate the difference between the timestamps corresponding to the operation start frame and the response completion frame to obtain the response duration index. Based on the response duration index, the resource load of the intelligent cockpit system, and the attribute information of the user interface elements corresponding to each key frame, generate a performance test report of the intelligent cockpit system.

[0077] The timestamp calculation module designed in this embodiment of the invention is used to extract the timestamps corresponding to the operation start frame and the response completion frame, and calculate the time difference between the two. The time difference is used as the response duration of the corresponding test case. The data output unit is used to generate a structured test report. The test report includes: test time, test vehicle number, test case number, operation type, response duration, CPU utilization, GPU load, and memory usage. Furthermore, the test report supports export in Excel, CSV, or PDF format.

[0078] In one optional implementation, the communication link status between the control unit and the intelligent cockpit system is monitored; if no response is detected to the control command, the control command is resent; if no response is detected to the control command for a preset number of consecutive times, or if the communication link is disconnected, or if the resource load of the intelligent cockpit system is in an abnormal state, an emergency handling operation is initiated.

[0079] like Figure 1As shown, the performance testing system for the intelligent cockpit system also includes a monitoring unit. The monitoring unit is used to monitor the communication link status between the control unit and the intelligent cockpit system in real time or at a preset frequency (every 10 seconds), and simultaneously collect the resource load of the vehicle system (such as CPU, memory usage, etc.). If a control command is detected as unresponsive, the control command is resent. If a control command is detected as unresponsive for a preset number of consecutive times (up to 3 times), or if the communication link is detected as disconnected, or if the resource load of the intelligent cockpit system is in an abnormal state, emergency handling operations are initiated. These emergency handling operations include, but are not limited to, triggering voice alarms and remote log upload mechanisms. For example, triggering the text-to-speech (TTS) module to issue a warning broadcast of "Please check the vehicle connection" and simultaneously sending an alarm email to the remote backend. Backend administrators can then remotely guide the driver to take parking measures in a safe area, and subsequently perform system reset or firmware repair. This is just an example.

[0080] This embodiment also provides a vehicle, such as Figure 6 As shown, the vehicle includes an intelligent cockpit system; for details, please refer to the above embodiments, which will not be repeated here.

[0081] This embodiment also provides a performance testing device for an intelligent cockpit system. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0082] This embodiment provides a performance testing device for an intelligent cockpit system, applied to a performance testing system for intelligent cockpit systems. The performance testing system includes a video acquisition device and a control unit. The control unit establishes a low-level communication link with the intelligent cockpit system. The video acquisition device's field of view covers the entire display area of ​​the vehicle's infotainment screen. Figure 7As shown, the system includes: an instruction sending module 701, used to send a sequence of operation instructions to the intelligent cockpit system through the control unit according to preset test cases, and simultaneously trigger the video acquisition device to record the screen of the vehicle to obtain a video stream; a semantic parsing module 702, used to perform semantic parsing on the image frames in the video stream after the test cases are executed, identify the attribute information of the user interface elements in each image frame, and convert the image frame sequence into a semantic state sequence ordered by time based on the attribute information; a key frame extraction module 703, used to extract key frames from the semantic state sequence to obtain the operation start frame and the response completion frame; and a performance testing module 704, used to calculate the difference between the timestamps corresponding to the operation start frame and the response completion frame to obtain the response duration index, and generate a performance test report of the intelligent cockpit system based on the response duration index, the resource load of the intelligent cockpit system, and the attribute information of the user interface elements corresponding to each key frame.

[0083] In some optional implementations, the semantic parsing module 702 includes: a video stream splitting unit, used to split the video stream into an image frame sequence according to a preset frame rate; an image processing unit, used to perform image preprocessing on the image frames in the image frame sequence, including resolution normalization, distortion correction, and brightness compensation; and an image parsing unit, used to call a pre-trained deep learning model to perform semantic parsing on the image frames and output attribute information of user interface elements in the image frames, the attribute information including at least element category, confidence level, and business semantic label, the business semantic label being used to characterize the business stage of each image frame during the execution of the test case.

[0084] In some optional implementations, the key frame extraction module 703 includes: a start frame extraction unit, used to determine the image frame in the semantic state sequence that first appears with the operation feedback semantic state after the operation command is issued as the operation start frame, where the operation feedback semantic state is the semantic stage corresponding to the visual feedback signal generated by the vehicle interface after responding to the operation command.

[0085] In some optional implementations, the keyframe extraction module 703 further includes an image selection unit, used to select the frame image with the earliest timestamp and corresponding confidence level reaching a preset threshold from multiple consecutive frame images if an operation feedback semantic state exists in multiple consecutive frame images, as the operation start frame.

[0086] In some optional implementations, the keyframe extraction module 703 includes: a completion frame extraction unit, used to determine the first image frame in the semantic state sequence in which the target content region has been presented and the target content region satisfies a preset stability condition within a preset number of consecutive image frames as the response completion frame. The stability condition is determined by calculating the sum of squared pixel differences of the target content region in adjacent image frames and the structural similarity index.

[0087] In some optional implementations, the target content region is a region within the vehicle interface that is prone to state changes and is related to the test case. The frame extraction unit includes: a parameter calculation subunit, used to calculate the first pixel difference sum of squares and the first structural similarity index between the target content region in the current image frame and the previous image frame; a change state determination subunit, used to determine that the target content region has an interface change state if the first pixel difference sum of squares exceeds a first preset threshold and the first structural similarity index is lower than a second preset threshold; a parameter calculation subunit, used to calculate the second pixel difference sum of squares and the second structural similarity index corresponding to the target content region in subsequent adjacent image frames; a stability determination subunit, used to determine that the stability condition is met when the second pixel difference sum of squares is lower than a third preset threshold and the second structural similarity index is higher than a fourth preset threshold in multiple consecutive adjacent image frames; and a completion frame determination subunit, used to determine the current image frame as the response completion frame.

[0088] In some optional implementations, before determining the current image frame as a response completion frame, the device further includes: a model output module, used to input the current image frame into a pre-trained deep learning model and output attribute information of the user interface element corresponding to the current image frame, the attribute information including at least element category, confidence level and semantic label; and an image verification module, used to determine the current image frame as a response completion frame if the current image frame includes the semantic label of the target content region and the confidence level reaches a preset threshold.

[0089] In some optional embodiments, the device further includes: a status monitoring module for monitoring the communication link status between the control unit and the intelligent cockpit system; a command retransmission module for retransmitting control commands if no response is detected; and an emergency handling module for initiating emergency handling operations if no response is detected for a preset number of consecutive control commands, or if the communication link is disconnected, or if the resource load of the intelligent cockpit system is in an abnormal state.

[0090] The performance testing device for the intelligent cockpit system provided in this embodiment of the invention can execute the performance testing method for the intelligent cockpit system provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0091] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0092] The following is a detailed reference. Figure 8This diagram illustrates a suitable structural schematic for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 801, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 802 or a program loaded from memory 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device. The processor 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0093] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0094] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a memory 808, or installed from a ROM 802. When the computer program is executed by the processor 801, it performs the functions defined in the performance testing method of the intelligent cockpit system of the embodiments of the present invention.

[0095] Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0096] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the performance testing method for the intelligent cockpit system shown in the above embodiments is implemented.

[0097] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0098] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined herein.

Claims

1. A performance testing method for an intelligent cockpit system, characterized in that, A performance testing system for intelligent cockpit systems, comprising a video acquisition device and a control unit, wherein the control unit establishes a low-level communication link with the intelligent cockpit system, and the video acquisition device's field of view covers the entire display area of ​​the vehicle's infotainment screen, the method comprising: According to the preset test cases, the control unit sends a sequence of operation commands to the intelligent cockpit system and simultaneously triggers the video acquisition device to record the screen of the vehicle to obtain a video stream; After the test cases are executed, semantic parsing is performed on the image frames in the video stream to identify the attribute information of the user interface elements in each image frame, and the image frame sequence is converted into a semantic state sequence ordered by time based on the attribute information. Keyframe extraction is performed on the semantic state sequence to obtain the operation start frame and response completion frame; The difference between the timestamps corresponding to the operation start frame and the response completion frame is calculated to obtain the response duration index. Based on the response duration index, the resource load of the intelligent cockpit system, and the user interface element attribute information corresponding to each key frame, a performance test report of the intelligent cockpit system is generated.

2. The method according to claim 1, characterized in that, The step of semantically parsing the image frames in the video stream to identify the attribute information of user interface elements in each image frame includes: The video stream is split into image frame sequences according to a preset frame rate; Image preprocessing is performed on image frames in the image frame sequence. The image preprocessing includes resolution normalization, distortion correction, and brightness compensation. A pre-trained deep learning model is invoked to perform semantic parsing on the image frame, and the attribute information of the user interface elements in the image frame is output. The attribute information includes at least element category, confidence level and business semantic label. The business semantic label is used to characterize the business stage that each image frame is in during the execution of the test case.

3. The method according to claim 1, characterized in that, Keyframe extraction is performed on the semantic state sequence to obtain the operation start frame, including: The image frame in the semantic state sequence that first shows the operation feedback semantic state after the operation command is issued is determined as the operation start frame. The operation feedback semantic state is the semantic stage corresponding to the visual feedback signal generated by the vehicle interface after responding to the operation command.

4. The method according to claim 3, characterized in that, The method further includes: If the operation feedback semantic state exists in multiple consecutive frame images, select the frame image with the earliest timestamp and the corresponding confidence level reaching the preset threshold from the multiple frame images as the operation start frame.

5. The method according to claim 1, characterized in that, Keyframe extraction is performed on the semantic state sequence to obtain the response completion frame, including: The first image frame in the semantic state sequence in which the target content region has been presented and the target content region satisfies the preset stability condition within a preset number of consecutive image frames is determined as the response completion frame. The stability condition is determined by calculating the sum of squared pixel differences of the target content region in adjacent image frames and the structural similarity index.

6. The method according to claim 5, characterized in that, The target content region is a region within the vehicle interface that is prone to sudden state changes and is related to the test case. Image frames in the semantic state sequence where the target content region has appeared and where the target content region meets a preset stability condition within a series of consecutive image frames are identified as response completion frames, including: Calculate the sum of squares of the first pixel differences between the target content region in the current image frame and the previous image frame, as well as the first structural similarity index; If the sum of squares of the first pixel difference exceeds the first preset threshold and the first structural similarity index is lower than the second preset threshold, it is determined that the target content area has an interface mutation state. Calculate the sum of squared second pixel differences and the second structural similarity index corresponding to the target content region in subsequent adjacent image frames; When the sum of squared differences of the second pixel difference corresponding to the target content region in multiple consecutive adjacent image frames is lower than the third preset threshold and the second structural similarity index is higher than the fourth preset threshold, the stability condition is determined to be met. The current image frame is determined as the response completion frame.

7. The method according to claim 6, characterized in that, Before determining the current image frame as the response completion frame, the method further includes: The current image frame is input into a pre-trained deep learning model, and the attribute information of the user interface element corresponding to the current image frame is output. The attribute information includes at least element category, confidence level and semantic label. If the current image frame includes a semantic tag for the target content region and the confidence level reaches a preset threshold, the current image frame is determined as a response completion frame.

8. The method according to claim 1, characterized in that, The method further includes: Monitor the communication link status between the control unit and the intelligent cockpit system; If no response is detected to the control command, resend the control command. If a control command fails to respond for a preset number of consecutive times, or if the communication link is disconnected, or if the resource load of the intelligent cockpit system is abnormal, an emergency response operation will be initiated.

9. A performance testing device for an intelligent cockpit system, characterized in that, A performance testing system for intelligent cockpit systems, comprising a video acquisition device and a control unit, wherein the control unit establishes a low-level communication link with the intelligent cockpit system, and the video acquisition device has a field of view covering the entire display area of ​​the vehicle's infotainment screen; the device includes: The instruction sending module is used to send a sequence of operation instructions to the intelligent cockpit system through the control unit according to preset test cases, and simultaneously trigger the video acquisition device to record the screen of the vehicle to obtain a video stream; The semantic parsing module is used to perform semantic parsing on the image frames in the video stream after the test cases are executed, identify the attribute information of the user interface elements in each image frame, and convert the image frame sequence into a semantic state sequence ordered by time based on the attribute information. The keyframe extraction module is used to extract keyframes from the semantic state sequence to obtain the operation start frame and the response completion frame. The performance testing module is used to calculate the difference between the timestamps corresponding to the operation start frame and the response completion frame to obtain the response duration index. Based on the response duration index, the resource load of the intelligent cockpit system, and the user interface element attribute information corresponding to each key frame, a performance test report of the intelligent cockpit system is generated.

10. A performance testing system for an intelligent cockpit system, characterized in that, The system includes a video acquisition device, a control unit, and a video analysis unit. The video acquisition device's field of view covers the entire display area of ​​the vehicle's infotainment screen. The control unit establishes a low-level communication link with the intelligent cockpit system, wherein: The control unit sends a sequence of operation commands to the intelligent cockpit system according to preset test cases, and simultaneously triggers the video acquisition device to record the screen of the vehicle to obtain a video stream; The video analysis unit is used to perform semantic parsing on the image frames in the video stream after the test cases are executed, identify the attribute information of the user interface elements in each image frame, and convert the image frame sequence into a semantic state sequence ordered by time based on the attribute information. The video analysis unit is used to extract keyframes from the semantic state sequence to obtain the operation start frame and the response completion frame. The video analysis unit is used to calculate the difference between the timestamps corresponding to the operation start frame and the response completion frame to obtain the response duration index. Based on the response duration index, the resource load of the intelligent cockpit system, and the user interface element attribute information corresponding to each key frame, a performance test report of the intelligent cockpit system is generated.

11. The system according to claim 10, characterized in that, The video acquisition device is mounted on a sliding bracket assembly in the adjustment bracket. By adjusting the sliding bracket assembly, the video acquisition device is positioned directly facing the display center of the vehicle screen, and its field of view covers the entire display area of ​​the vehicle screen. The adjustment bracket also includes a stabilizing component assembly, which works in conjunction with the sliding bracket assembly to securely clamp the vehicle screen.

12. The system according to claim 11, characterized in that, The stabilizing component includes fixtures distributed at four locations on the vehicle infotainment screen for securing the screen.

13. A vehicle, characterized in that, The vehicle includes the intelligent cockpit system as described in any one of claims 10-12.

14. An electronic device, characterized in that, include: A memory and a processor are interconnected, the memory stores computer instructions, and the processor executes the computer instructions to perform the performance testing method of the intelligent cockpit system according to any one of claims 1 to 8.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the performance testing method of the intelligent cockpit system according to any one of claims 1 to 8.

16. A computer program product, characterized in that, Includes computer instructions for causing a computer to execute the performance testing method for the intelligent cockpit system according to any one of claims 1 to 8.

Images