Method, device, processor and electronic equipment for detecting interface of car machine system

Abstract

The application discloses a detection method and device of an in-vehicle infotainment system interface, a processor and electronic equipment. The method comprises the following steps: obtaining picture data and video data of the in-vehicle infotainment system interface; using a static analysis module to analyze the picture data to obtain a picture analysis result, and using a dynamic analysis module to analyze the video data to obtain a video analysis result, wherein the picture analysis result is used to represent text information and / or icon information in the picture data, the video analysis result is used to represent motion characteristics between different video frames in the video data, the dynamic analysis module and the static analysis module are triggered by the same operation event, and the operation event is at least one interactive behavior in the in-vehicle infotainment system interface; and based on the picture analysis result and the video analysis result, the in-vehicle infotainment system interface is detected to obtain a detection result. The application solves the technical problem that the in-vehicle infotainment system interface cannot be effectively detected.

Description

Technical Field

[0001] This application relates to the field of vehicle detection technology, and more specifically, to a method, apparatus, processor, and electronic device for detecting vehicle infotainment system interfaces. Background Technology

[0002] Currently, automated detection technologies for vehicle infotainment system interfaces, such as vehicle user interfaces (UI), suffer from problems such as limited detection dimensions, poor robustness, and lack of automated closed-loop systems.

[0003] In related technologies, the detection of in-vehicle infotainment system interfaces often relies on overall image pixel comparison, or only uses Optical Character Recognition (OCR) to detect text anomalies, or only monitors the page switching sequence. However, overall image pixel comparison is easily affected by dynamic content such as time displays and lighting reflections, resulting in a high false alarm rate. Detecting text anomalies solely through OCR cannot identify non-text defects such as icon misalignment and overlap. Monitoring only the page switching sequence ignores the frame rate smoothness of dynamic operations such as list scrolling.

[0004] In summary, the methods described above cannot achieve comprehensive coverage of various types of in-vehicle infotainment system (IVS) defects; furthermore, each method operates in isolation, lacking semantic-level precision positioning capabilities, resulting in severe false positives and false negatives, leading to incomplete test coverage and delayed problem fixing. Therefore, the technical challenge of effectively detecting IVS interfaces remains.

[0005] There is currently no effective solution to the aforementioned technical problems. Summary of the Invention

[0006] This application provides a method, apparatus, processor, and electronic device for detecting vehicle infotainment system interfaces, in order to at least solve the technical problem of the inability to effectively detect vehicle infotainment system interfaces.

[0007] According to one aspect of the embodiments of this application, a method for detecting a vehicle infotainment system interface is provided. The method may include: acquiring image data and video data of the vehicle infotainment system interface; using a static analysis module to parse the image data to obtain image parsing results, and using a dynamic analysis module to parse the video data to obtain video parsing results, wherein the image parsing results are used to characterize text information and / or icon information in the image data, and the video parsing results are used to characterize motion features between different video frames in the video data; the dynamic analysis module and the static analysis module are triggered by the same operation event, which is at least one interactive behavior in the vehicle infotainment system interface; and based on the image parsing results and the video parsing results, detecting the vehicle infotainment system interface to obtain detection results, wherein the detection results are used to indicate whether there are abnormalities in the text information and / or icon information, and whether there are abnormalities in the motion features.

[0008] Optionally, the static analysis module is used to parse the image data to obtain image parsing results, including: using the static analysis module to preprocess the image data to obtain preprocessed image data; segmenting the regions of the preprocessed image data to obtain multiple different semantic regions, where different semantic regions correspond to different types of elements; and parsing the semantic regions to obtain image parsing results.

[0009] Optionally, the image parsing result includes semantic anomaly information and / or positional offset information. The semantic region is parsed to obtain the image parsing result, including: in response to the semantic region being a text region, the text region is identified to obtain a recognition result, wherein the recognition result represents the text content in the text information within the text region and the confidence level of the text content; the recognition result is compared with standard text in the template image library to obtain a comparison result; in response to the comparison result having a confidence level less than a confidence threshold, or the edit distance of the text content being less than the edit distance of the standard text, semantic anomaly information is generated; in response to the semantic region being an icon region, feature points in the icon region are matched with feature points of the corresponding icons in the template image library to obtain positional offset information corresponding to the icon information.

[0010] Optionally, the video data is parsed using a dynamic analysis module to obtain video parsing results, including: acquiring initial feature points from the initial frames of the video data, wherein the initial feature points are used to characterize the spatial motion reference positions of the content on the vehicle infotainment system screen; parsing the initial feature points using the dynamic analysis module to obtain the time required for the initial feature points to generate effective displacement between adjacent frames; and determining the video parsing results based on the time.

[0011] Optionally, the video analysis results include a presentation time series and rendering smoothness. Based on time, the video analysis results are determined by: arranging the time in sequence to obtain a presentation time series; determining the average frame rate and standard deviation of the presentation time series; and using the average frame rate and standard deviation as quantitative indicators of rendering smoothness. The method further includes: detecting abnormal feature points in the presentation time series that are greater than or equal to a threshold of the presentation time series, wherein the abnormal feature points are used to identify the time and frequency of rendering stuttering.

[0012] Optionally, the method further includes: classifying the abnormal data in the image parsing results and / or video parsing results to obtain classification results; labeling the severity of the abnormal data according to the classification results to obtain labeling results, wherein the labeling results are used to indicate the severity level of the abnormal data; and generating a detection report based on the classification results and labeling results.

[0013] According to another aspect of the embodiments of this application, a detection device for a vehicle infotainment system interface is also provided. The device may include: an acquisition unit for acquiring image data and video data of the vehicle infotainment system interface; a parsing unit for parsing the image data using a static analysis module to obtain image parsing results, and parsing the video data using a dynamic analysis module to obtain video parsing results, wherein the image parsing results are used to characterize text information and / or icon information in the image data, and the video parsing results are used to characterize motion features between different video frames in the video data; the dynamic analysis module and the static analysis module are triggered by the same operation event, which is at least one interactive behavior in the vehicle infotainment system interface; and a detection unit for detecting the vehicle infotainment system interface based on the image parsing results and the video parsing results to obtain detection results, wherein the detection results are used to indicate whether there are abnormalities in the text information and / or icon information, and whether there are abnormalities in the motion features.

[0014] According to another aspect of the embodiments of this application, a processor is also provided. The processor is used to run a program, wherein the program executes the methods of the embodiments of this application during runtime.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the method of the embodiments of this application when it runs.

[0016] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided. The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the method of the embodiments of this application.

[0017] According to another aspect of the embodiments of this application, a vehicle is also provided. The vehicle includes a memory and a processor. The memory stores an executable program; the processor is used to run the program, which, when running, implements the methods described in the embodiments of this application.

[0018] In this embodiment, if it is necessary to detect the vehicle infotainment system interface, image data and video data of the vehicle infotainment system interface can be acquired. Then, a static analysis module is used to parse the image data, obtaining image parsing results representing text information and / or icon information in the image data. Similarly, a dynamic analysis module is used to parse the video data, obtaining video parsing results representing motion characteristics between different video frames in the video data. This allows for the detection of the vehicle infotainment system interface, determining whether there are anomalies in the text information and / or icon information, and whether there are anomalies in the motion characteristics. Since the dynamic analysis module and the static analysis module are triggered by the same operation event, which is at least one interactive behavior in the vehicle infotainment system interface, triggering the parsing of static screenshots and dynamic videos through the same operation event overcomes the limitations of single-dimensional detection methods in related technologies that isolate text processing, overall image comparison, or simple frame rate statistics. This solves the technical problem of ineffective detection of the vehicle infotainment system interface and achieves the technical effect of effectively detecting the vehicle infotainment system interface. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a flowchart of a method for detecting a vehicle infotainment system interface according to an embodiment of this application;

[0021] Figure 2 This is a flowchart of an automated detection method for vehicle UI rendering anomalies based on image recognition, according to an embodiment of this application.

[0022] Figure 3 This is a flowchart illustrating the working method of a static analysis submodule according to an embodiment of this application;

[0023] Figure 4 This is a flowchart illustrating a method for operating a dynamic analysis submodule according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of a vehicle infotainment system interface detection device according to an embodiment of this application;

[0025] Figure 6 This is a structural diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] According to an embodiment of this application, an embodiment of a method for detecting a vehicle infotainment system interface is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] Figure 1 This is a flowchart of a method for detecting a vehicle infotainment system interface according to an embodiment of this application, such as... Figure 1 As shown, the method may include the following steps.

[0030] Step S102: Obtain image and video data from the vehicle infotainment system interface.

[0031] In the technical solution provided in step S102 of this application, in order to simultaneously capture the complete visual presentation of the vehicle system interface in a static and stable state, as well as the continuous rendering performance during dynamic interaction, in real user operation scenarios, and to provide real, consistent, and comparable raw data input for subsequent static anomaly detection and dynamic performance analysis, image data and video data of the vehicle system interface can be obtained.

[0032] In this embodiment, in response to the detection of a preset user operation command, dual-modal data (image data and video data) acquisition can be performed automatically and accurately.

[0033] For example, when the test process of the vehicle infotainment system interface reaches a certain interface operation node (e.g., clicking the "navigation" icon, scrolling through a list, switching to the air conditioning settings page, etc.), the system waits for the interface to complete rendering and reach a stable state. By detecting signals such as whether the layout of elements in the interface converges, whether dynamic animations have ended, and whether the load is stable, it is confirmed that the current screen is the final rendering result. Subsequently, two acquisition channels can be simultaneously started with millisecond-level precision: a static image acquisition channel and a dynamic video stream acquisition channel.

[0034] Optionally, in the static image acquisition channel, a high-resolution, uncompressed, true-color static screenshot of the complete interface can be obtained as image data by connecting a High-Definition Multimedia Interface (HDMI) capture card to the vehicle's infotainment system or by directly reading the frame buffer. The image data fully preserves the pixel information of visual elements such as text, icons, buttons, background, and shadows of the vehicle's infotainment system interface. The content can cover the visible area of ​​the vehicle's infotainment system interface, including but not limited to font content, icon shape, layout position, color distribution, and transparency effects.

[0035] Optionally, in the dynamic video stream acquisition channel, high frame rate video recording can be started immediately after the same operation command is triggered, continuously capturing the continuous changes in the vehicle system interface caused by user operation within several seconds. For example, the smooth transition of the video stream list scrolling, the animation effects of page switching, and the gradual process of pop-up window appearance, etc., are recorded as video data to ensure complete recording of dynamic behavior from the start of the operation to the stabilization of the interface.

[0036] In this embodiment of the application, by synchronously collecting image data and video data, it can be ensured that image data (e.g., static screenshots) and video data (e.g., dynamic videos) originate from the same operation event and the same interface state, fundamentally solving the problem of analysis misalignment caused by the asynchrony between screenshots and screen recordings in traditional testing.

[0037] Step S104: The static analysis module is used to parse the image data to obtain the image parsing result, and the dynamic analysis module is used to parse the video data to obtain the video parsing result.

[0038] In the technical solution provided in step S104 of this application, the image analysis result can be used to characterize the text information and / or icon information in the image data, and the video analysis result can be used to characterize the motion characteristics between different video frames in the video data. The dynamic analysis module and the static analysis module are triggered by the same operation event, and the operation event can be at least one interactive behavior in the vehicle system interface.

[0039] In this embodiment, parsing the picture data and video data essentially converts the simultaneously collected picture data and video data into quantifiable, classifiable, and locatable semantic anomaly signals, providing structured input for subsequent detection of the in-vehicle system interface.

[0040] Optionally, the static analysis module can perform semantic-level structural decomposition and content verification on the picture data. For example, the static analysis module can be used to segment the picture data and divide the interface into multiple semantic regions. Then, different semantic regions can be parsed separately to obtain the text information and / or icon information in the picture data.

[0041] Optionally, through an image segmentation algorithm, the interface can be divided into multiple semantic regions, such as text regions, icon regions, background regions, control borders, etc. The text region can be a rectangular block with continuous gray-scale changes or clear edges, and the icon region can be a non-text graphic with a fixed geometric shape or a sudden change in contrast. For the text region, the static analysis module can call the OCR algorithm to extract the character sequence and confidence level. For example, "Favorites" is recognized as "收蒇夹" (a misspelling in Chinese), and it is determined whether there are anomalies such as an edit distance greater than zero from the standard text (such as "Favorites") pre-stored in the reference picture library and a recognition confidence level lower than 97%. For the icon region, the static analysis module can adopt a feature point detection and matching algorithm to extract the coordinates, scale, and orientation information of the icon key points, and then perform a spatial transformation comparison with the feature point template of the corresponding icon in the reference picture library to calculate the offset, rotation angle, and scaling ratio in the horizontal and vertical directions. If the offset exceeds 2 pixels or there is obvious deformation, it is determined that the icon is misaligned or deformed. Finally, the picture parsing result output by the static analysis module is a structured data set containing specific semantic defects such as incorrect text content, text truncation, icon offset, and element overlap. Each defect can be accompanied by position coordinates, defect type, and confidence score, rather than a simple vague conclusion of image inconsistency.

[0042] Optionally, the dynamic analysis module can quantify the motion features of the dynamic video stream. For example, it can employ feature-point-based optical flow tracing technology to accurately capture the real physical motion of the video data during interaction. Hundreds of visual feature points with high gradient response and strong trackability are automatically selected in the first frame of the video. These feature points can be evenly distributed on stable structures such as icons, buttons, and text edges, representing motion reference anchor points for the interface content. Subsequently, in each subsequent frame, the dynamic analysis module can track the pixel displacement of these feature points using an optical flow algorithm, retaining only successfully tracked points with stable displacement, and excluding mismatches caused by reflections, noise, or occlusion. Afterward, the dynamic analysis module can calculate the average time required for effective feature points to generate meaningful displacements between two adjacent frames; this time is the frame rendering time. The aforementioned frame rendering time reflects the latency required for the vehicle system's graphics rendering engine to complete a frame update. The presentation times of each frame are arranged chronologically to form a presentation time series. The average frame rate (the inverse mean of frame presentation times), frame rate standard deviation (reflecting the stability of inter-frame fluctuations), and stuttering index (the percentage of frames with a presentation time exceeding 66 milliseconds) are then calculated to generate the video analysis result. This video analysis result is no longer a binary judgment of whether there is stuttering, but a quantitative representation of multi-dimensional performance indicators including average frame rate, fluctuation amplitude, and stuttering frequency. It can objectively reproduce the subjective experience of users such as lag and stuttering when scrolling through lists or switching pages.

[0043] Optionally, the static analysis module and the dynamic analysis module are triggered by the same operation event, which is any interactive behavior in the vehicle system interface, such as clicking a navigation icon, sliding a music playlist, or switching the air conditioning temperature adjustment interface. This means that the static analysis module and the dynamic analysis module do not run independently, but are activated uniformly within milliseconds after the same test command is issued, collecting the same interface response process triggered by the same user behavior. This design ensures that the final state represented by the static screenshot is completely aligned with the change process recorded by the dynamic video in terms of time, space, and causality. The icon offset found in the static screenshot is the final result of the icon being misaligned due to rendering delay during the sliding process in the dynamic video; the sudden drop in frame rate in the dynamic video will also inevitably be reflected in the static screenshot for whether there is element overlap or ghosting. This architecture of "same source triggering, synchronous collection, and same scene analysis" breaks the traditional fragmented mode of "static comparison is static and dynamic monitoring is dynamic" in testing, thereby realizing a unified diagnosis of visible errors and perceptible stuttering in the vehicle system interface in the context of real interaction.

[0044] In the embodiments of this application, through the above steps, more than ten sub-categories of anomalies, including static defects and dynamic performance defects, can be covered within a single testing cycle, achieving an improvement from point detection to surface analysis.

[0045] Step S106: Detect the in-vehicle system interface based on the picture parsing result and the video parsing result, and obtain the detection result.

[0046] In the technical solution of step S106 of the present application, the detection result can be used to indicate whether there are abnormalities in text information and / or icon information, and whether there are abnormalities in motion characteristics.

[0047] In this embodiment, detecting the in-vehicle system interface based on the picture parsing result and the video parsing result mainly involves semantic fusion, logical reasoning, and comprehensive judgment of two types of heterogeneous information produced by static semantic parsing and dynamic performance quantification, so as to form a structured, operable, and traceable final defect conclusion, rather than presenting multiple scattered analysis signals in isolation.

[0048] Optionally, the picture parsing result may include static semantic defects such as abnormal text content (for example, OCR recognizes "设罝" instead of "设置" and the confidence level is lower than 97%), icon position offset (for example, the horizontal offset of the icon center point is 7 pixels and the rotation angle is 3.5°), and element overlap (for example, the overlapping area between the text and the icon boundary exceeds the threshold). The video parsing result may include dynamic performance indicators such as presentation time series, average frame rate (for example, 22fps), standard deviation of frame rate fluctuation (for example, 8.2ms), and stuttering index (for example, 19%). The picture parsing result and the video parsing result already have clear semantic labels and quantization values, but have not yet formed a systematic defect judgment. Therefore, the in-vehicle system interface can be detected based on the picture parsing result and the video parsing result to obtain the detection result.

[0049] Optionally, the above detection method is not a simple superposition or averaging, but can perform multi-dimensional association reasoning through a preset, rule-driven intelligent analysis engine. This intelligent analysis engine can build a set of judgment rule libraries that are strictly aligned with the design specifications and user experience standards of the in-vehicle system interface. For example, when "text content recognition error" is detected in the picture parsing result and the confidence level is lower than 97%, the intelligent analysis engine automatically classifies the picture parsing result as a "text typo" defect and marks it as a "blocking level" abnormality. When the icon offset exceeds 10 pixels or the rotation angle is greater than 5°, it can be determined as "icon misalignment" and classified as a "severe level".

[0050] Optionally, when the average frame rate in the video parsing result is lower than 30fps and the stuttering index is higher than 10%, the intelligent analysis engine can determine that there is a "rendering stutter" defect and classify it as a "severe level". If icon offset and text truncation occur simultaneously, a "element overlap" composite exception is triggered, which can be automatically marked as a "high-priority problem" and a spatial overlap heat map of the two abnormal areas is associated and displayed.

[0051] In this embodiment, the above steps enable the coordinated judgment of static and dynamic defects. For example, if only the icon is offset but the frame rate is normal, it may be due to a layout configuration error; if the icon is offset and accompanied by a low frame rate, it may be due to excessive computational load causing layout errors when the graphics engine is drawing the element. A unified intelligent analysis engine ensures the standardization and reproducibility of defect classification, avoids subjective bias in manual judgment, and improves the consistency of test results across versions and vehicle models.

[0052] In steps S102 to S106 of this application, if it is necessary to detect the vehicle infotainment system interface, image data and video data of the vehicle infotainment system interface can be obtained. Then, the static analysis module is used to parse the image data to obtain image parsing results representing text information and / or icon information in the image data, and the dynamic analysis module is used to parse the video data to obtain video parsing results representing motion characteristics between different video frames in the video data. This allows for the detection of the vehicle infotainment system interface, determining whether there are anomalies in the text information and / or icon information, and whether there are anomalies in the motion characteristics. Since the dynamic analysis module and the static analysis module are triggered by the same operation event, which is at least one interactive behavior in the vehicle infotainment system interface, triggering the parsing of static screenshots and dynamic videos through the same operation event overcomes the limitations of single-dimensional detection methods in related technologies that isolate text processing, overall image comparison, or simple frame rate statistics. This solves the technical problem of ineffective detection of the vehicle infotainment system interface and achieves the technical effect of effectively detecting the vehicle infotainment system interface.

[0053] The method described in this embodiment will be further described below.

[0054] As an optional embodiment, step S104 involves using a static analysis module to parse the image data and obtain image parsing results. This includes: using the static analysis module to preprocess the image data to obtain preprocessed image data; segmenting the regions of the preprocessed image data to obtain multiple different semantic regions, where different semantic regions correspond to different types of elements; and parsing the semantic regions to obtain image parsing results.

[0055] In this embodiment, the static analysis module is used to parse the image data, which can accurately extract semantically meaningful text and icon elements from the original interface image with severe interference, providing reliable and quantifiable basic data for subsequent defect judgment.

[0056] Optionally, due to interference factors in the actual use environment of the vehicle system interface, such as strong light reflection, changes in ambient color temperature, color shifts caused by screen aging, and fingerprints on the glass cover, directly performing OCR or feature point matching on image data can easily lead to misidentification and missed detection. Therefore, image data can be preprocessed, and multi-level image enhancement algorithms can be introduced in the preprocessing stage. For example, Contrast Limited Adaptive Histogram Equalization (CLAHE) can be used to enhance the local contrast of image data, significantly improving the clarity of text edges and icon outlines while preserving overall color information, especially for text in low-brightness areas. Subsequently, Gaussian blurring can be used to suppress high-frequency noise, and adaptive threshold binarization can be combined to transform the image data into a grayscale contour structure that is more conducive to edge detection. Finally, to reduce the computational burden and improve matching efficiency, a perceptual hashing algorithm can be introduced to quickly screen suspected icon areas in the image data, retaining only candidate areas whose similarity to icon structures in the benchmark image library is higher than the similarity threshold, thereby skipping a large number of invalid analyses of background or irrelevant controls. The preprocessed image data has significantly improved the signal-to-noise ratio and structural clarity, providing a foundation for subsequent accurate segmentation.

[0057] Optionally, based on the preprocessing, semantic region segmentation can be performed on the image data. The aim is to decompose the cluttered overall screen image into several sub-regions with clear functional semantics, rather than processing them uniformly. This segmentation process can combine contour recognition based on edge detection with layout prior knowledge based on template matching. For example, by using edge detection and morphological closing operations, rectangular or circular button outlines in the image data can be identified; text block regions can be located using text line detection algorithms; and then, combined with typical layout patterns of the vehicle system interface, such as the top navigation bar, the middle information area, and the bottom function menu, rule-based or lightweight classification models can be used to determine the semantic type of each region. Finally, the image data is divided into several non-overlapping semantic regions, such as "navigation title text area," "main function icon area," "menu list text area," "dynamic time display area," and "system status icon area." Each semantic region can be assigned a type label and bound to the coordinate boundaries in the image data to achieve a precise mapping of "region-type-position."

[0058] Optionally, after region segmentation, a refined parsing process can be performed on each semantic region using an adapted parsing strategy. For regions identified as "text regions," a high-precision OCR engine can be invoked to adaptively optimize the model based on font type, font size, and background contrast, extracting the character sequences and confidence scores, and comparing them with the standard text at that location on the interface in the benchmark image library. For regions identified as "icon regions," the algorithm switches to feature point matching to extract the scale-invariant feature transform (SIFT) or oriented accelerated segment test features and rotated binary robust independent elementary features (ORB) key points within the region. These key points are then spatially transformed and aligned with the standard feature templates of the corresponding icons in the benchmark image library, accurately calculating translation, rotation, and scaling parameters. For elements expected to change, such as "dynamic time regions," exclusion rules are automatically added to avoid misclassifying these elements as abnormal.

[0059] Optionally, the image parsing results can be output in a structured format, such as "Text area - Navigation title: identified as 'Setting up traps', confidence level 95%, edit distance from standard 'Settings' = 1", "Icons area - Air conditioner button: average feature point offset Δx = 6.2px, Δy = 0.8px, rotation angle 2.1°", which are only examples and no specific restrictions are imposed here.

[0060] In this embodiment of the application, through the above steps, a differentiated analysis strategy of "precise positioning, classification processing, and intelligent identification" is realized in the complex interference environment inherent in the vehicle system interface.

[0061] As an optional embodiment, the image parsing result includes semantic anomaly information and / or positional offset information. Parsing the semantic region yields the image parsing result, including: in response to the semantic region being a text region, recognizing the text region to obtain a recognition result, wherein the recognition result represents the text content in the text information within the text region and the confidence level of the text content; comparing the recognition result with standard text in a template image library to obtain a comparison result; in response to the comparison result having a confidence level less than a confidence threshold, or the edit distance of the text content being less than the edit distance of the standard text, generating semantic anomaly information; and in response to the semantic region being an icon region, matching the feature points in the icon region with the feature points of the corresponding icon in the template image library to obtain positional offset information corresponding to the icon information.

[0062] In this embodiment, based on the type of semantic region, an adapted algorithm engine can be called respectively to ensure that each type of exception can be accurately captured, clearly defined, and quantitatively expressed.

[0063] Optionally, when a certain semantic region is identified as a text region, the parsing process starts the Optical Character Recognition (OCR) module. The OCR module adaptively optimizes the text content in the text information of the text region, such as features like font style, background contrast, text size, etc., calls a high-precision OCR engine to recognize each character in the character sequence one by one, and outputs a confidence score for each recognized character, and finally summarizes the recognition results for the entire text region. The recognition results can include two core pieces of information: one is the specific text content recognized, for example, "设罝", "导航", "音量", etc.; the other is the confidence of the text content, such as 95.3%. Subsequently, the recognition results can be compared with the standard text pre-stored for this interface and this position in the template image library (such as the reference image library).

[0064] Optionally, the above comparison content can include two dimensions. If there are character differences between the recognized text content and the standard text (such as "设罝" ≠ "设置"), the edit distance (Levenshtein Distance) is calculated, that is, the minimum number of operations required to make the two consistent by inserting, deleting, or replacing characters. If the recognition confidence is less than the confidence threshold (such as 97%), it indicates that the current recognition result is not reliable enough and may be misrecognized due to blurring, incompleteness, or interference. When either condition is met - that is, the confidence is less than the confidence threshold, or the edit distance of the text content is less than the edit distance of the standard text, a semantic exception can be determined and the corresponding semantic exception information is generated.

[0065] Optionally, when the semantic region is identified as an icon region, the parsing process switches to the feature point matching mode. A set of highly distinguishable visual feature points, such as corner points, edge intersection points, texture key points, etc., can be extracted from the current icon region, and algorithms such as SIFT or ORB are used to encode the scale, direction, and local gradient of these points to form the feature vector of the icon. Then, the standard feature template of the icon in the reference version can be retrieved from the template image library, and the current icon feature points are matched bidirectionally with the standard template feature points. After eliminating mis-matched points, the affine transformation parameters between the two are calculated, including the horizontal offset Δx, vertical offset Δy, rotation angle θ, and scaling ratio s. If the offset Δx or Δy exceeds the preset tolerance threshold (such as 2 pixels), or the rotation angle θ exceeds ±3°, it can be determined that there is a spatial displacement anomaly of the icon, and position offset information is generated, clearly marked as "icon misalignment", and the specific offset, rotation angle, matching similarity score, and abnormal region coordinates are recorded.

[0066] In this embodiment of the application, the above steps enable accurate, efficient, and automated assessment of the visual compliance of the vehicle system interface, providing technical support for ensuring a high-quality human-computer interaction experience in the smart cockpit.

[0067] As an optional embodiment, step S104 involves using a dynamic analysis module to parse the video data and obtain video parsing results, including: collecting initial feature points from the initial frames of the video data, wherein the initial feature points are used to characterize the spatial motion reference positions of the content on the vehicle infotainment system screen; using the dynamic analysis module to parse the initial feature points and obtain the time required for the initial feature points to generate effective displacement between adjacent frames; and determining the video parsing results based on the time.

[0068] In this embodiment, a dynamic analysis module is used to analyze video data. Based on the physical motion trajectory of visual feature points, a high-precision, interference-resistant, and quantifiable dynamic evaluation of the smoothness of the vehicle system interface's rendering during interaction is performed. This approach does not simply count "how many frames per second," but rather starts from the essential experience of "whether the motion is smooth" perceived by the user, constructing a dynamic performance analysis system with time as the measurement scale.

[0069] Optionally, a set of stable and traceable initial feature points can be collected from the initial frames of the video data. These initial feature points are not randomly selected, but are automatically identified and filtered by algorithms to identify visual anchor points with high gradient changes, strong texture structure, and stable positions. Examples include icon edge intersections, text stroke corners, control outline corners, and background texture repeating structures. These initial feature points have a high probability of repeatable matching in subsequent frames, accurately reflecting the physical movement of the interface content, rather than pseudo-motion such as noise, reflections, or dynamic time displays. The collection range of the initial feature points can cover key areas of the entire interface that may change, such as the borders of items in a scrolling list, the center of buttons in navigation transitions, and the outlines of animated pop-ups, ensuring that the selected initial feature points fully represent the overall motion behavior of the interface. These initial feature points can be encoded into vectors with position, scale, and orientation information, forming a spatial reference benchmark for subsequent motion analysis.

[0070] Optionally, after obtaining the initial feature points, the dynamic analysis module can initiate a frame-by-frame optical flow tracing algorithm to continuously track the initial feature points in each frame. When an initial feature point undergoes a smooth displacement that conforms to the laws of physical motion between two adjacent frames (i.e., the displacement direction is continuous, the amplitude is within a reasonable range, and it is not occluded or lost), it is determined as a valid tracking point (that is, the initial feature point generates a valid displacement between adjacent frames). The time taken for this point to move from the previous frame to the current frame is then calculated, and the video parsing result can be determined based on the time.

[0071] Related technologies only determine whether the page transition is complete, but cannot distinguish between "fast but laggy transitions" and "slow but smooth transitions." However, this application's embodiment accurately identifies which frames are truly not updated by tracking the update delay of effective tracking points. Even if the screen appears to be moving, if the displacement time of feature points is too long, it indicates a rendering bottleneck. This allows for the accurate identification of fine-grained performance issues that are difficult to capture by related technologies, such as "stuttering during list scrolling," "frame drops in transition animations," and "delayed appearance of dynamic pop-ups."

[0072] As an optional implementation, the video analysis results include a presentation time series and rendering smoothness. Based on time, the video analysis results are determined by: arranging the time in sequence to obtain a presentation time series; determining the average frame rate and standard deviation of the presentation time series; and using the average frame rate and standard deviation as quantitative indicators of rendering smoothness. The method further includes: detecting abnormal feature points in the presentation time series that are greater than or equal to a threshold of the presentation time series, wherein the abnormal feature points are used to identify the time and frequency of rendering stuttering.

[0073] In this embodiment, by systematically analyzing the actual rendering latency distribution of each frame, a quantitative evaluation system that reflects the overall performance level and can accurately locate local stuttering can be constructed, thereby achieving in-depth diagnosis of the dynamic rendering quality of the vehicle system interface.

[0074] Optionally, the time (e.g., frame rendering time) acquired by the dynamic analysis module during frame-by-frame video tracking—that is, the actual time required for effective feature points to complete physical displacement between adjacent frames—is arranged chronologically to form a continuous rendering time series. This rendering time series is not a theoretical value, but rather millisecond-level latency data calculated based on the actual movement of visual feature points, representing the time consumed by each frame update. The timestamps of the rendering time series are strictly aligned with user operation events, completely recording all rendering latency changes from the start of the swipe to the point where the interface stabilizes. Subsequently, statistical analysis can be performed on this rendering time series. For example, the average value of the rendering time series can be calculated to obtain the average frame rendering time, and the reciprocal of the average frame rendering time can be taken to obtain the average frame rate. Simultaneously, the standard deviation of the rendering time series can be calculated, which reflects the dispersion of latency fluctuations between different frames.

[0075] Optionally, the average frame rate represents the overall response speed, while the standard deviation reveals the stability of the experience. For example, an average frame rate of 35fps but a standard deviation as high as 12ms indicates that although the overall speed is relatively fast, there are frequent bursts of latency, and users will still perceive a "stuttering" or "laggy" experience. Conversely, if the average frame rate is only 28fps but the standard deviation is extremely low (e.g., 2ms), it represents a stable but slightly slower smooth experience. This two-dimensional evaluation breaks through the one-sidedness of traditional methods that only consider "average frame rate," truly reflecting the user's subjective perception of "smoothness" rather than "speed."

[0076] Optionally, further anomaly detection can be performed on the presentation time series to identify the specific moments and frequencies of stuttering. A dynamic threshold mechanism can be used, setting a presentation time series threshold that matches the sensitivity of human visual perception, for example, 66 milliseconds (corresponding to 15 fps). This presentation time series threshold represents the critical point at which most users begin to noticeably perceive "latency" or "unresponsiveness." By traversing the entire presentation time series, frame presentation time points greater than or equal to this presentation time series threshold are marked; these points are the anomaly feature points.

[0077] Optionally, each anomalous feature point not only identifies the specific time and location of the stutter (e.g., "stutter occurs at 3.2 seconds"), but also counts the total number of times it appears in the video, forming a "stutter frequency" metric. A "stutter index," which is the percentage of anomalous feature points out of the total number of frames, can be further calculated as a comprehensive indicator of the overall severity of the stutter. For example, if a 5-second scrolling video has 300 frames, and 45 of those frames have a presentation time ≥66ms, then the stutter index is 15%, which can be classified as "severe stutter" and automatically marked as a "high-priority defect."

[0078] As an optional embodiment, the method further includes: classifying the abnormal data in the image parsing results and / or video parsing results to obtain classification results; labeling the severity of the abnormal data according to the classification results to obtain labeling results, wherein the labeling results are used to indicate the severity level of the abnormal data; and generating a detection report based on the classification results and labeling results.

[0079] In this embodiment, the image parsing results may include anomalies such as "text typos", "text truncation", "icon offset", and "element overlap", while the video parsing results may output performance defects such as "rendering stutter", "animation frame drops", and "sudden drop in frame rate".

[0080] Optionally, by integrating a rule-driven classification engine, which is not based on machine learning models but rather a set of interpretable and auditable decision rules built according to in-vehicle system interface design specifications, functional safety standards, and user experience guidelines. For example, when the OCR in the image parsing result recognizes "设罝" instead of "设置" and the confidence level is lower than 97%, the rule "The edit distance between the OCR recognition content and the standard text > 0 and the confidence level < 97%" can be automatically matched, and it is determined as "text error". When the offset of the icon feature points Δx ≥ 5 pixels or the rotation angle ≥ 3°, the "icon misalignment" classification is triggered. When in the video parsing result, the frame presentation time of more than three consecutive frames ≥ 66 ms, or the stuttering index ≥ 10%, it is classified as "rendering stutter". Each piece of abnormal data can be assigned a unique defect type label, achieving an accurate mapping from "pixel deviation" to "defect category".

[0081] Optionally, after classification, the severity of each type of anomaly can be further labeled based on its impact on functional safety, user operations, and brand experience. For example, if a "text typo" appears in key information such as the navigation destination name, emergency braking prompt, or airbag status, it is labeled as "blocking level". If "icon misalignment" causes the user to accidentally touch other function keys, it is labeled as "severe level", and if it only affects aesthetics, it is labeled as "general level". If "rendering stutter" occurs during the main control interface switch or voice response process, affecting the continuity of human-vehicle interaction, it is labeled as "severe level". If it only appears on secondary pages and there is no operation delay, it is labeled as "general level". The labeling process automatically associates with the context, such as the interface path where the anomaly occurs, the operation type, and the user role (driver / passenger), ensuring that the level determination fits the real usage scenario.

[0082] Optionally, based on the classification results and labeling results, a structured inspection report can be generated. This inspection report is not a simple list but a visual and interactive document for development, testing, and quality teams. It can be in the format of HyperText Markup Language (HTML for short), and the inspection report can contain multi-dimensional information. For example, each abnormal entry is accompanied by a comparison of the original screenshot and the reference diagram (highlighting the difference area), a video stuttering timeline waveform diagram, an animation of the feature point movement trajectory, defect type and severity level labels, the occurrence timestamp, the所属 interface module, and the recommended repair direction (such as "optimize icon layout adaptation", "reduce the rendering load of the graphics processor"). The inspection report can also automatically generate summary statistics, such as "This test found 1 blocking-level defect, 7 severe-level defects, and 15 general-level defects in total", and statistically calculate the defect density by module to assist the team in identifying high-risk modules.

[0083] In this embodiment, if it is necessary to detect the vehicle infotainment system interface, image data and video data of the vehicle infotainment system interface can be acquired. Then, a static analysis module is used to parse the image data, obtaining image parsing results representing text information and / or icon information in the image data. Similarly, a dynamic analysis module is used to parse the video data, obtaining video parsing results representing motion characteristics between different video frames in the video data. This allows for the detection of the vehicle infotainment system interface, determining whether there are anomalies in the text information and / or icon information, and whether there are anomalies in the motion characteristics. Since the dynamic analysis module and the static analysis module are triggered by the same operation event, which is at least one interactive behavior in the vehicle infotainment system interface, triggering the parsing of static screenshots and dynamic videos through the same operation event overcomes the limitations of single-dimensional detection methods in related technologies that isolate text processing, overall image comparison, or simple frame rate statistics. This solves the technical problem of ineffective detection of the vehicle infotainment system interface and achieves the technical effect of effectively detecting the vehicle infotainment system interface.

[0084] The technical solutions of the embodiments of this application will be illustrated below with reference to preferred embodiments.

[0085] Currently, with the accelerated pace of vehicle intelligence and connectivity, the intelligent cockpit has become the core of human-vehicle interaction. The complexity and iteration speed of intelligent cockpit software systems are growing exponentially. The user interface of the in-vehicle infotainment system carries a massive number of functions such as navigation, entertainment, and vehicle control. The accuracy, smoothness, and consistency of the content displayed on the user interface are directly related to driving safety and user experience.

[0086] In related technologies, testing methods for user interface display content are mainly divided into three categories.

[0087] Overall image comparison. The structural similarity index and pixel value variance between the test image and the standard design image are calculated, but it cannot accurately locate specific abnormal elements and is easily affected by dynamic content such as time display, leading to false alarms.

[0088] Text-specific detection. It uses OCR to recognize the text on the interface and compares it with theoretical text information to test whether there are anomalies such as truncation or overlap and output the type. However, it cannot detect rendering anomalies of non-text elements such as icons and graphics.

[0089] Dynamic process monitoring. By extracting consecutive image frames from the in-vehicle infotainment system's page switching video frame by frame, calculating the Structural Similarity Index Method (SSIM) value between adjacent frames, and determining whether the change in the SSIM value is less than a set threshold, the system automatically identifies the start and end frames of page switching, thus achieving automated monitoring of the page switching logic. The primary focus is on the timing of the switching, rather than the smoothness of operations such as list scrolling—a core performance indicator.

[0090] In summary, the above solutions suffer from a lack of diversity in detection dimensions, mostly targeting only a specific type of defect and each only addressing a portion of UI rendering defects. The root cause lies in the diverse types of UI rendering anomalies, whose causes and manifestations involve multiple technical fields such as image processing, text recognition, and performance analysis. Furthermore, the complex in-vehicle UI environment, with its reflections, color differences, and interference from dynamic elements, leads to insufficient recognition accuracy and robustness, resulting in high rates of false positives and false negatives.

[0091] To address the aforementioned issues, this application proposes an automated detection system and method for in-vehicle UI rendering anomalies based on image recognition. The detection system comprises four core modules: a benchmark image library construction module, an automated testing and data acquisition module, an intelligent analysis engine module, and an anomaly classification and report generation module. These modules work collaboratively to form a complete closed loop from test preparation to result output.

[0092] The embodiments of this application will be further described below.

[0093] Figure 2 This is a flowchart of an automated detection method for vehicle infotainment UI rendering anomalies based on image recognition, according to an embodiment of this application. Figure 2 As shown, the method may include the following steps.

[0094] Step S201: Build a standard UI database using the benchmark library building module.

[0095] In this embodiment, the benchmark library construction module aims to establish a reliable and complete comparison benchmark, which is a prerequisite for ensuring the accuracy of detection.

[0096] Optionally, in a controlled testing environment, the vehicle's infotainment system can be driven by a benchmark acquisition control script. This script sends precise command sequences to the system via an Adaptive Debug Bridge (ADB) or automotive Ethernet, automatically traversing the user-accessible interface layers, including the main desktop, various settings menus, and all pre-installed and test applications. After each target interface has been stably rendered (by determining if the interface element layout is stable), the benchmark image library construction module can directly read from the frame buffer or connect to a high-precision industrial camera to capture the pixel data of the current screen and save it as a high-fidelity red, green, and blue (RGB) image. Each image can be automatically tagged with metadata (e.g., interface identifier, interface name, vehicle resolution, theme mode, software version number, and acquisition timestamp) and stored in a structured image library database.

[0097] Optionally, the above steps establish a pixel-level truth standard, avoiding errors and inconsistencies from manual screenshots. Automated traversal and metadata tagging ensure the integrity and traceability of the benchmark image library, providing accurate references for subsequent intelligent comparisons and achieving high reliability in improving detection results.

[0098] Step S202: Use the automated testing and data acquisition module to control the vehicle's infotainment system to traverse and synchronously acquire screen data.

[0099] In this embodiment, the automated testing and data acquisition module is responsible for reproducing user operations and synchronously acquiring screen output during the testing process, serving as a bridge connecting the real vehicle system and the testing logic.

[0100] Optionally, the automated testing and data acquisition module can adopt a two-layer architecture. The upper layer is the test case executor, which reads the test scenario description file (such as traversal path) and converts the test scenario description file into specific device operation instructions. The lower layer is the device control and data acquisition layer, which connects both the vehicle control unit and the screen data acquisition unit.

[0101] Optionally, the vehicle infotainment control unit can inject simulated touch events, button events (such as steering wheel buttons), and simulated voice signals into the vehicle infotainment system via physical wiring or wireless connection to precisely control the UI. The screen data acquisition unit focuses on synchronous and multimodal acquisition. After the vehicle infotainment system executes an operation command (such as clicking the "Settings" icon), the detection system waits for a preset interface rendering timeout (e.g., 500ms), and then simultaneously triggers two acquisition methods: Static screenshot acquisition: Similar to the baseline construction phase, a complete single-frame interface image is acquired. Dynamic video stream acquisition: When the test case is described as a "scrolling list" or "transition animation," a high frame rate video recording process (e.g., 60fps) can be started to continuously capture screen changes over a subsequent period, generating a video file.

[0102] Optionally, the above steps of instruction-driven and multi-modal data synchronous acquisition precisely simulate the real user interaction scenario. Static screenshots are used for element-level comparison, and dynamic videos are used for performance analysis. The combination of the two ensures complete coverage of the detection scenario. This design can achieve a high degree of automation and repeatability in the testing process and is the basis for improving the overall testing efficiency.

[0103] Step S203: Analyze using the intelligent analysis engine module.

[0104] In this step, the intelligent analysis engine module is responsible for deeply analyzing the collected data. It consists of two sub-modules (static analysis sub-module and dynamic analysis sub-module) that work in parallel to solve the detection of static defects and dynamic defects respectively. Analyzing using the intelligent analysis engine module includes: Step S2031: Compare graphic and text elements using the static analysis sub-module. Step S2032: Analyze the rendering performance using the dynamic analysis sub-module.

[0105] Step S204: Classify defects & generate a visualization report.

[0106] In this embodiment, an anomaly classification and report generation module can be used to classify defects & generate a visualization report. This anomaly classification and report generation module can have a built-in rule classifier that receives the original anomaly data from the intelligent analysis engine (such as "the text 'Settings' is recognized as '设罝' with a confidence of 95%, coordinates (x1, y1, x2, y2)", "the average offset of the icon feature points is Δx = 5 pixels, Δy = 0 pixels"). The classifier automatically classifies and marks the severity level (such as blocking, severe, general) according to a preset rule library (such as OCR confidence < 97% and edit distance > 0 → "text error"; offset > 2 pixels → "element offset"; average frame rate < 30fps → "rendering lag"). Finally, the report generator integrates the classification results, original screenshots, comparison with the baseline map (highlighting the different regions), performance curve graphs, etc., to generate an interactive visualization report in HTML format.

[0107] Figure 3 is a flowchart of a working method of a static analysis sub-module according to an embodiment of the present application. As Figure 3 shown, this method may include the following steps.

[0108] Step S301: Input real-time test screenshots.

[0109] Step S302: Perform image preprocessing.

[0110] Step S303: Perform region segmentation and recognition.

[0111] Step S304: Perform multi-channel comparison.

[0112] In this embodiment, the multi-channel comparison includes: step S3041, text integrity analysis; step S3042, UI element position analysis.

[0113] Step S305: Perform result fusion.

[0114] In this embodiment, the UI is deconstructed into different semantic regions using region segmentation technology, enabling the application of adaptive algorithms to different types of elements. The text channel employs OCR combined with semantic and layout comparison, solving the problem that overall pixel comparison methods cannot identify "text content errors" and "local overlaps," significantly improving the detection accuracy of text-related defects. The icon channel uses feature point matching, which is insensitive to changes in lighting and slight color differences, and can calculate specific offsets and rotation angles, providing quantitative and precise positioning information rather than simple inconsistency judgments. Through these steps, high-precision, localizable, and categorizable detection of complex UI static defects is achieved.

[0115] Figure 4 This is a flowchart illustrating a method for operating a dynamic analysis submodule according to an embodiment of this application, such as... Figure 4 As shown, the method may include the following steps.

[0116] Step S401: Receive screen operation video.

[0117] In this embodiment, a video clip of screen operation (such as list scrolling) can be received. Then, it can be decoded frame by frame using an algorithm, and a timestamp can be calculated for each frame.

[0118] Step S402: Select hundreds of feature points in the starting frame of the video and track the movement of the feature points in each subsequent frame.

[0119] In this embodiment, the algorithm does not compare all pixels between frames, but selects hundreds of feature points in the starting frame of the video and tracks the movement of these points in each subsequent frame.

[0120] Step S403: Determine the frame rendering time data.

[0121] In this embodiment, after tracking the movement of feature points in each subsequent frame, analyzing the smoothness and continuity of the movement trajectories of a large number of feature points allows for a more robust determination of whether the image is truly "effectively updated." The algorithm can calculate the average time required for all tracked feature points to produce effective displacement between two adjacent frames, using this as the frame rendering time between those two frames. By traversing the entire video, a series of frame rendering time data can be obtained.

[0122] Step S404: Detect the frame presentation time data to obtain the detection result.

[0123] In this embodiment, the average frame rate and frame rate standard deviation of the rendering time data (rendering time series) can be calculated, and anomalies where the frame rendering time exceeds a preset threshold (e.g., 100ms, i.e., severe stuttering) can be detected. Simultaneously, the frame rendering time trend of multiple consecutive frames is analyzed to determine if there is a persistent low frame rate.

[0124] Unlike related technologies that only monitor whether page switching is "completed", this application embodiment quantifies smoothness by analyzing the effective motion time of feature points between frames. It can accurately distinguish between "the screen is still and not updated" and "the screen should be updated but is lagging", thereby objectively and quantitatively diagnosing dynamic rendering performance defects such as "list scrolling lag" and "animation frame drops".

[0125] In this embodiment, the fragmented algorithm outputs are transformed into problem descriptions with clear semantics and priorities through a rule engine. The generated visual report makes the test results clear at a glance, allowing developers to locate problems without interpreting the original logs. This greatly improves the efficiency of problem investigation and repair, completing the final step from problem discovery to guiding repair, and maximizing the value of the test.

[0126] The following explanation will use a specific testing system (detection system) as an example.

[0127] Test system configuration: The test system can consist of a test host, a vehicle-mounted infotainment system prototype under test, and an in-vehicle Ethernet and HDMI capture card connecting the two. The host runs automated control, image recognition, and detection programs.

[0128] The implementation process is as follows.

[0129] Database construction: The test system automatically controls the vehicle's infotainment system to traverse all interfaces, captures standard images through the acquisition card, and builds a benchmark image library with metadata.

[0130] Testing: On the new software version, the test system reproduced the same operation. For example, when testing the "scrolling navigation list," a static screenshot of the interface was captured first, then the scrolling command was executed while simultaneously recording a 60fps video of the operation.

[0131] Analysis: Static Comparison: The test screenshot was compared with the baseline image. OCR identified that "Favorites" was incorrectly rendered as "Collection," and overlap between text and icon areas was detected, which was determined to be "text typo" and "element overlap." Dynamic Analysis: Analyzing the scrolling video, calculations showed that the average frame rate was only 12fps (below the 30fps threshold), indicating "severe scrolling stuttering."

[0132] Report: Each defect is automatically categorized, generating a visual HTML report that includes highlighted annotations, comparison charts, and performance curves.

[0133] In this embodiment, in a real vehicle environment, screen brightness and ambient light reflection may interfere with the comparison. Before feature point matching, an image preprocessing step can be introduced. For example, the CLAHE algorithm can be used for local contrast equalization, and a perceptual hash algorithm can be used to initially screen the icon area. Only areas with hash value differences greater than a threshold are accurately matched for feature points, reducing unnecessary calculations and improving stability under varying lighting conditions.

[0134] The system also adds supplementary metrics for dynamic performance analysis. In addition to frame rate, the testing system can calculate a "stuttering index," which is the percentage of frames whose rendering time exceeds a predetermined threshold (e.g., 66ms) out of the total number of frames. This quantitative metric can be better correlated with the subjective "unresponsiveness" experience, providing more refined data support for performance optimization.

[0135] Based on the above, the embodiments of this application solve the following problems.

[0136] The current detection methods are limited by their single dimension, failing to cover the complete spectrum of UI defects. Related technologies either focus only on overall image differences or detect only text anomalies, unable to simultaneously address both static text and image anomalies and dynamic performance stuttering. This application integrates three technologies—OCR-based text integrity checking, feature point detection-based UI element position comparison, and screen recording-based dynamic rendering performance analysis—into a unified automated process. This allows the application to concurrently detect all types of rendering defects, including "text truncation," "icon misalignment," "element overlap," and "rendering stuttering," achieving a multi-dimensional and comprehensive evaluation of in-vehicle UI quality and significantly expanding the breadth and depth of testing coverage.

[0137] Secondly, the embodiments of this application overcome the problems of low recognition accuracy and high false positive and false negative rates in related technologies. Addressing the shortcomings of overall comparison methods being susceptible to interference and text recognition methods failing in complex scenarios, the intelligent comparison module of this application does not perform simple pixel comparison. Through OCR and feature point detection algorithms, it directly extracts and compares the semantic content (text) and structural features (icon position, shape) of UI elements independently, thereby producing higher accuracy and stronger robustness. The testing system can effectively distinguish between dynamic content updates and real rendering errors, and can accurately locate the specific elements where anomalies occur, thus significantly reducing the false positive and false negative rates and ensuring the reliability and accuracy of the detection results.

[0138] Finally, this embodiment of the application can realize an intelligent closed loop throughout the entire process from test execution to result analysis, solving the bottleneck of automation efficiency. By seamlessly connecting "benchmark image library management," "automatic traversal and screenshotting," "intelligent comparison analysis," and "automatic classification and report generation," this embodiment of the application constitutes an end-to-end automated system. The automated system can automatically perform a full interface traversal and automatically generate a structured report with visual comparison evidence, defect classification, and quantitative data, effectively improving testing efficiency and providing developers with clear and direct repair basis, shortening the problem investigation cycle.

[0139] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0140] According to an embodiment of this application, a device for detecting a vehicle infotainment system interface is also provided. It should be noted that this device can be used to execute the vehicle infotainment system interface detection method described in the embodiments.

[0141] Figure 5 This is a schematic diagram of a vehicle infotainment system interface detection device according to an embodiment of this application, such as... Figure 5 As shown, the detection device 500 of the vehicle system interface may include: an acquisition unit 502, a parsing unit 504, and a detection unit 506.

[0142] The acquisition unit 502 is used to acquire image data and video data from the vehicle infotainment system interface.

[0143] The parsing unit 504 is used to parse image data using the static analysis module to obtain image parsing results, and to parse video data using the dynamic analysis module to obtain video parsing results. The image parsing results are used to characterize the text information and / or icon information in the image data, and the video parsing results are used to characterize the motion characteristics between different video frames in the video data. The dynamic analysis module and the static analysis module are triggered by the same operation event, which is at least one interactive behavior in the vehicle system interface.

[0144] The detection unit 506 is used to detect the vehicle system interface based on the image analysis results and video analysis results, and obtain the detection results. The detection results are used to indicate whether there are abnormalities in the text information and / or icon information, and whether there are abnormalities in the motion features.

[0145] Optionally, the parsing unit 504 includes: a processing subunit for preprocessing the image data using a static analysis module to obtain preprocessed image data; a segmentation subunit for segmenting the regions of the preprocessed image data to obtain multiple different semantic regions, wherein different semantic regions correspond to different types of elements; and a parsing subunit for parsing the semantic regions to obtain image parsing results.

[0146] Optionally, the image parsing result includes semantic anomaly information and / or positional offset information. The parsing subunit includes: a recognition subunit, used to recognize the text region in response to the semantic region being a text region, and obtain a recognition result, wherein the recognition result is used to represent the text content in the text information in the text region and the confidence level of the text content; a comparison subunit, used to compare the recognition result with standard text in the template image library, and obtain a comparison result; a first generation subunit, used to generate semantic anomaly information in response to the comparison result being a confidence level less than a confidence threshold, or the edit distance of the text content being less than the edit distance of the standard text; and a matching subunit, used to match the feature points in the icon region with the feature points of the corresponding icon in the template image library in response to the semantic region being an icon region, and obtain the positional offset information corresponding to the icon information.

[0147] Optionally, the parsing unit 504 includes: an acquisition subunit for acquiring initial feature points from the initial frame of the video data, wherein the initial feature points are used to characterize the spatial motion reference position of the content on the vehicle system screen; a first parsing subunit for parsing the initial feature points using a dynamic analysis module to obtain the time required for the initial feature points to generate effective displacement between adjacent frames; and a determination subunit for determining the video parsing result based on time.

[0148] Optionally, the video analysis results include a presentation time series and rendering smoothness. The determination sub-unit includes: an arrangement sub-unit, used to arrange the time in order to obtain the presentation time series; a first determination sub-unit, used to determine the average frame rate and standard deviation of the presentation time series; and a second determination sub-unit, used to determine the average frame rate and standard deviation as quantitative indicators of rendering smoothness. The detection device 500 of the vehicle system interface also includes: a detection sub-unit, used to detect the presentation time series and obtain abnormal feature points in the presentation time series that are greater than or equal to the presentation time series threshold, wherein the abnormal feature points are used to identify the time and frequency of rendering stuttering.

[0149] Optionally, the detection device 500 of the vehicle system interface further includes: a classification subunit, used to classify abnormal data in the image parsing results and / or video parsing results to obtain classification results; a labeling subunit, used to label the severity of abnormal data according to the classification results to obtain labeling results, wherein the labeling results are used to indicate the severity level of abnormal data; and a second generation subunit, used to generate a detection report based on the classification results and labeling results.

[0150] In this embodiment, the acquisition unit 502 acquires image data and video data of the vehicle infotainment system interface; the parsing unit 504 uses a static analysis module to parse the image data to obtain image parsing results, and uses a dynamic analysis module to parse the video data to obtain video parsing results. The image parsing results characterize text and / or icon information in the image data, and the video parsing results characterize motion features between different video frames in the video data. The dynamic analysis module and the static analysis module are triggered by the same operation event, which is at least one interactive behavior in the vehicle infotainment system interface. The detection unit 506 detects the vehicle infotainment system interface based on the image and video parsing results to obtain detection results. These detection results indicate whether there are any anomalies in the text and / or icon information, and whether there are any anomalies in the motion features. This solves the technical problem of not being able to effectively detect the vehicle infotainment system interface and achieves the technical effect of effectively detecting the vehicle infotainment system interface.

[0151] This application also provides an electronic device, please refer to... Figure 6 , Figure 6 This is a structural diagram of an electronic device according to an embodiment of this application. The electronic device 60 includes a processor 610 and a memory 620, wherein the memory 620 is used to store computer programs; the processor 610 is used to execute the programs stored in the memory 620 to implement the methods described in any embodiment of this application.

[0152] According to another aspect of the embodiments of this application, a processor is also provided. The processor is used to run a program, wherein the program executes the methods of the embodiments of this application during runtime.

[0153] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided. The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the method of the embodiments of this application.

[0154] According to another aspect of the embodiments of this application, a vehicle is also provided. The vehicle includes a memory and a processor. The memory stores an executable program; the processor is used to run the program, which, when running, implements the methods described in the embodiments of this application.

[0155] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0157] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0159] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0160] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for detecting the interface of a vehicle infotainment system, characterized in that, include: Acquire image and video data from the vehicle infotainment system interface; The static analysis module is used to parse the image data to obtain image parsing results, and the dynamic analysis module is used to parse the video data to obtain video parsing results. The image parsing results are used to characterize the text information and / or icon information in the image data, and the video parsing results are used to characterize the motion features between different video frames in the video data. The dynamic analysis module and the static analysis module are triggered by the same operation event, which is at least one interactive behavior in the vehicle system interface. Based on the image analysis results and the video analysis results, the vehicle system interface is inspected to obtain inspection results, wherein the inspection results are used to indicate whether there are any abnormalities in the text information and / or the icon information, and whether there are any abnormalities in the motion features.

2. The method according to claim 1, characterized in that, The static analysis module is used to parse the image data to obtain image parsing results, including: The static analysis module is used to preprocess the image data to obtain the preprocessed image data. The preprocessed image data is segmented to obtain multiple different semantic regions, where different semantic regions correspond to different types of elements; The semantic region is parsed to obtain the image parsing result.

3. The method according to claim 2, characterized in that, The image parsing result includes semantic anomaly information and / or positional offset information. The semantic region is parsed to obtain the image parsing result, which includes: In response to the semantic region being a text region, the text region is identified to obtain an identification result, wherein the identification result is used to represent the text content in the text information in the text region and the confidence level of the text content; The recognition results are compared with the standard text in the template library to obtain the comparison results; In response to the comparison result being that the confidence level is less than the confidence threshold, or the edit distance of the text content is less than the edit distance of the standard text, the semantic anomaly information is generated; In response to the semantic region being an icon region, feature points in the icon region are matched with feature points of the corresponding icons in the template library to obtain the position offset information corresponding to the icon information.

4. The method according to claim 1, characterized in that, The video data is parsed using a dynamic analysis module to obtain video parsing results, including: Initial feature points are acquired from the initial frame of the video data, wherein the initial feature points are used to characterize the spatial motion reference position of the content on the in-vehicle system screen; Using the dynamic analysis module, the initial feature points are analyzed to obtain the time required for the initial feature points to generate effective displacement between adjacent frames; The video parsing result is determined based on the time.

5. The method according to claim 4, characterized in that, The video parsing results include a presentation time series and rendering smoothness. Based on the time, the video parsing results are determined, including: The times are arranged in order to obtain the presented time sequence; Determine the average frame rate and standard deviation of the presentation time series; The average frame rate and the standard deviation are determined as quantitative indicators of the rendering smoothness; The method further includes: The rendering time series is detected to obtain abnormal feature points in the rendering time series that are greater than or equal to the rendering time series threshold. The abnormal feature points are used to identify the time and frequency of rendering stuttering.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: The abnormal data in the image parsing results and / or the video parsing results are classified to obtain classification results; Based on the classification results, the severity of the abnormal data is labeled to obtain a labeling result, wherein the labeling result is used to represent the severity level of the abnormal data; A detection report is generated based on the classification results and the annotation results.

7. A detection device for a vehicle infotainment system interface, characterized in that, include: The acquisition unit is used to acquire image data and video data of the vehicle system interface; The parsing unit is used to parse the image data using a static analysis module to obtain image parsing results, and to parse the video data using a dynamic analysis module to obtain video parsing results. The image parsing results are used to characterize the text information and / or icon information in the image data, and the video parsing results are used to characterize the motion features between different video frames in the video data. The dynamic analysis module and the static analysis module are triggered by the same operation event, which is at least one interactive behavior in the vehicle system interface. The detection unit is used to detect the vehicle system interface based on the image parsing results and the video parsing results, and obtain detection results, wherein the detection results are used to indicate whether there are any abnormalities in the text information and / or the icon information, and whether there are any abnormalities in the motion features.

8. A processor, characterized in that, The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 6 when it runs.

9. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 6.

11. A vehicle, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 6.

Images