Multimodal large model car-machine touch test button recognition positioning method and device

Abstract

This invention relates to the field of vehicle infotainment touch testing, and discloses a multimodal large-scale model method and apparatus for button recognition and positioning in vehicle infotainment touch testing. The method includes: constructing a vehicle infotainment UI-related knowledge base; acquiring the UI image of the current touchscreen and the depth image of the corresponding area; retrieving and matching the interface identifier of the current UI image based on the association data between visual feature vectors and interface identifiers in the knowledge base; combining touch test instructions with the interface hierarchy and the corresponding data between the interface and visible controls in the knowledge base to obtain the unique identifier and interface path of the target button; using a preset multimodal model, inputting the current UI image, the standard control image of the target button, and a text description to obtain the bounding box coordinate information of the target button in the current UI image; and converting the bounding box coordinate information into three-dimensional position coordinates and attitude information in the touch actuator coordinate system based on the depth image, a preset image acquisition device intrinsic parameter matrix, and the coordinate transformation matrix between the image acquisition device and the touch actuator.

Description

Technical Field

[0001] This invention relates to the field of vehicle infotainment touch testing technology, specifically to a method and apparatus for identifying and locating buttons in multimodal large-scale vehicle infotainment touch testing. Background Technology

[0002] Since 2024-2025, when automotive cockpits rapidly transitioned from traditional architecture to intelligent systems, mainstream passenger vehicles have generally been equipped with multiple touch terminals such as central control screens, instrument panels, HUDs, and rear entertainment screens. The number of interactive function buttons in a single vehicle has increased from dozens in the early days to hundreds, and the interface themes and functions are continuously iterated with OTA updates. To meet the testing needs of mass production, software upgrades, and quality acceptance, the industry has developed a variety of typical in-vehicle touch interaction testing solutions, including manual touch testing (testers click, observe, and record feedback according to preset test cases), mechanical finger or collaborative robot-coordinate script solutions (clicks are triggered by screen coordinates through scripts such as ADB and Appium), template or feature matching visual solutions (target buttons are located using OpenCV template matching, color features, or ORB key points), and convolutional neural network-based object detection solutions (object detection is performed on RGB images using CNN models such as YOLO and Faster-RCNN to obtain the bounding box coordinates of buttons).

[0003] While the aforementioned existing in-vehicle infotainment touch interaction testing solutions have been applied in the industry, they are generally unable to simultaneously meet the comprehensive requirements of "efficient coverage, generalized adaptation to multiple vehicle models and versions, deep UI semantic parsing, and multi-screen collaborative verification" in in-vehicle infotainment touch testing scenarios. This core issue restricts the improvement of in-vehicle infotainment touch testing in terms of efficiency, adaptability, and verification capabilities in complex scenarios, and cannot adequately address the current testing needs of multiple touch terminals, multiple button counts, and frequent iterations of interface functions in in-vehicle infotainment systems. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a multimodal large-scale vehicle infotainment touch test button recognition and positioning method and device to solve the aforementioned technical problems.

[0005] Firstly, a multimodal, large-scale in-vehicle infotainment touchscreen test button recognition and localization method is provided, including: Construct a knowledge base related to vehicle infotainment system UI, wherein the knowledge base includes the association data between the visual feature vectors of the vehicle infotainment system UI interface and the interface identifiers, the interface hierarchy of the vehicle infotainment system UI, and the corresponding data between the interface and the visible controls; Obtain the UI image of the current touchscreen of the vehicle's infotainment system and the depth image of the corresponding area; Based on the association data between visual feature vectors and interface identifiers in the knowledge base, retrieve the interface identifiers that match the current UI image; By combining the touch test instructions with the interface hierarchy and the corresponding data between the interface and visible controls in the knowledge base, the unique identifier and interface path of the target button are obtained. Using a preset multimodal model, input the current UI image, the standard control image of the target button, and the text description, and obtain the bounding box coordinate information of the target button in the current UI image; Based on the depth image, the preset image acquisition device intrinsic parameter matrix, and the coordinate transformation matrix between the image acquisition device and the touch actuator, the bounding box coordinate information is converted into three-dimensional position coordinates and attitude information in the touch actuator coordinate system; The three-dimensional position coordinates and attitude information are sent to the touch actuator, which drives it to perform the operation corresponding to the touch test command on the vehicle touch screen.

[0006] Furthermore, the construction of the vehicle infotainment system UI-related knowledge base includes: The resolution and brightness of the vehicle infotainment UI images of different vehicle models and software versions are standardized and normalized to obtain the processed vehicle infotainment UI images. Visual feature vectors are extracted from each of the processed vehicle UI images to obtain the visual feature vectors. Each visual feature vector is associated with its corresponding vehicle UI identifier and stored in a preset vector database; Based on the interface hierarchy of the vehicle's UI, the logic of navigation between interfaces, and the relationship between visible controls under each interface, the correspondence between interfaces and visible controls and the navigation paths between interfaces are recorded.

[0007] Furthermore, acquiring the UI image of the current touchscreen of the vehicle system and the depth image of the corresponding area includes: Industrial cameras or depth cameras are used to capture color images of the current touchscreen of the vehicle and depth images of the corresponding areas; The timestamps of the color image and the depth image are calibrated to obtain the UI image of the current touch screen and the depth image of the corresponding area.

[0008] Furthermore, the step of retrieving interface identifiers matching the current UI image based on the association data between visual feature vectors and interface identifiers in the knowledge base includes: The UI image of the current touch screen of the vehicle is processed by resolution standardization and brightness normalization to obtain the processed UI image; Extract visual feature vectors from the processed UI image to obtain the query feature vector; Calculate the cosine similarity between the query feature vector and the visual feature vectors of each vehicle UI interface image in the knowledge base, and obtain each cosine similarity value; The visual feature vector with the highest cosine similarity value and that cosine similarity value is greater than a preset threshold is selected, and the interface identifier associated with the visual feature vector is used as the interface identifier corresponding to the current UI image.

[0009] Furthermore, the step of combining touch test commands with the interface hierarchy and corresponding data between the interface and visible controls in the knowledge base to obtain the unique identifier and interface path of the target button includes: Obtain a touch test instruction, wherein the touch test instruction includes descriptive information of the target to be touched; By using the interface hierarchy of the vehicle's UI in the knowledge base, the corresponding data of the interface and visible controls, and the interface identifier corresponding to the current UI image, we can obtain the information of the visible controls under the current interface and the information of the subsequent interfaces that the current interface can jump to. By combining the description of the target to be touched in the touch test command, the visible control information of the current interface, and the subsequent interface information that the current interface can jump to, the unique identifier of the target button and the interface path where the target button is located are obtained.

[0010] Furthermore, the step of employing a preset multimodal model, inputting the current UI image, the standard control image of the target button, and a text description, to obtain the bounding box coordinate information of the target button in the current UI image, includes: Based on the unique identifier of the target button, the standard control image corresponding to the target button is retrieved from the preset control image library. The control image library stores the standard image of each control and its corresponding control identifier. Based on the function and appearance characteristics of the target button, a text description of the target button is obtained; The UI image of the current touch screen of the vehicle system, the standard control image of the target button, and the text description of the target button are input into the preset multimodal model to obtain the bounding box coordinate information of the target button in the current UI image.

[0011] Furthermore, the conversion of bounding box coordinate information into three-dimensional position coordinates and attitude information in the touch actuator coordinate system based on the depth image, a preset image acquisition device intrinsic parameter matrix, and a coordinate transformation matrix between the image acquisition device and the touch actuator includes: Based on the bounding box coordinate information of the target button, the center pixel coordinates of the bounding box are calculated, wherein the center pixel coordinates are the average of the pixel coordinates of the upper left corner and the pixel coordinates of the lower right corner of the bounding box. The target depth value is obtained by extracting the depth value corresponding to the coordinates of the center pixel from the depth image. Based on the preset intrinsic parameter matrix of the image acquisition device, the coordinates of the center pixel and the corresponding target depth value are back-projected to obtain the three-dimensional coordinates in the coordinate system of the image acquisition device. A preset coordinate transformation matrix between the image acquisition device and the touch actuator is used to convert the three-dimensional coordinates in the coordinate system of the image acquisition device into the three-dimensional position coordinates in the coordinate system of the touch actuator.

[0012] Furthermore, the three-dimensional position coordinates and attitude information are sent to the touch actuator to drive it to perform operations corresponding to the touch test commands on the vehicle's touchscreen, including: The three-dimensional position coordinates and attitude information of the touch actuator in the coordinate system are sent to the control unit of the touch actuator; The control unit plans the motion trajectory of the touch actuator components based on three-dimensional position coordinates and attitude information; The actuator of the touch control mechanism moves to the target position according to the planned motion trajectory and performs the operation corresponding to the touch test command; During the operation, the position and speed information of the touch actuator components are collected in real time to form an operation execution record.

[0013] Secondly, a multimodal large-scale vehicle infotainment touch test button recognition and positioning device is provided, based on the multimodal large-scale vehicle infotainment touch test button recognition and positioning method described in any of the preceding items, including: The construction module is configured to build a knowledge base related to the vehicle infotainment system UI. The knowledge base includes the association data between the visual feature vectors of the vehicle infotainment system UI and the interface identifiers, the interface hierarchy of the vehicle infotainment system UI, and the corresponding data between the interface and the visible controls. The acquisition module is configured to acquire the UI image of the current touchscreen of the vehicle system and the depth image of the corresponding area; The retrieval module is configured to retrieve interface icons that match the current UI image based on the association data between visual feature vectors and interface icons in the knowledge base. The module is configured to combine touch test commands with the corresponding data of interface hierarchy and visible controls in the knowledge base to obtain the unique identifier and interface path of the target button. The input module is configured to use a preset multimodal model to input the current UI image, the standard control image of the target button, and the text description, and obtain the bounding box coordinate information of the target button in the current UI image; The conversion module is configured to convert the bounding box coordinate information into three-dimensional position coordinates and attitude information in the coordinate system of the touch actuator based on the depth image, the preset image acquisition device intrinsic parameter matrix, and the coordinate transformation matrix between the image acquisition device and the touch actuator. The driving module is configured to send the three-dimensional position coordinates and attitude information to the touch actuator, driving it to perform operations corresponding to the touch test commands on the vehicle touch screen.

[0014] Thirdly, a terminal is provided, including a processor, an input device, an output device, and a memory, wherein the processor, input device, output device, and memory are interconnected, wherein the memory is used to store a computer program, the computer program including program instructions, and the processor is configured to call the program instructions to execute the multimodal large-scale vehicle touch test button recognition and positioning method as described in any of the preceding claims.

[0015] The invention employing the above technical solution has the following advantages: 1. This invention proposes a smart cockpit touch interaction testing and positioning scheme that integrates a UI jump text knowledge base, a vehicle UI image embedding vector knowledge base, a large language model text retrieval reasoning, and a multimodal large model icon understanding and positioning: The system first generates a visual embedding from a real-time screenshot and retrieves the matching interface from the vector knowledge base. Combined with LMM, it completes the semantic positioning of the interface and the determination of the target control. Then, through multimodal large model recognition, it outputs accurate button coordinates and drives the mechanical actuator for closed-loop verification. This improves testing efficiency and positioning accuracy while significantly reducing maintenance and annotation costs in multi-vehicle, multi-version, and multi-screen scenarios.

[0016] 2. When a new vehicle model or UI version is added, this invention only requires offline collection of a small number of interface screenshots and writing them into the vector database, and updating the jump relationship entries to take effect. The maintenance workload is reduced from "re-annotation / regression script" to "adding data entries". The decoupling between modules means that subsequent replacement of the visual model or language model does not require modification of the entire test link.

[0017] 3. This invention uses "standard image of target icon + semantic description" as one-shot prompt input Qwen-2.5-VL-72B. The multimodal large model utilizes visual features and semantic priors simultaneously, and can output pixel-level bounding boxes in scenarios with complex lighting, occlusion, and multiple instances on the same screen (such as multiple seat heating buttons). Combined with depth 3D back projection, the touch point positioning error can be reduced to sub-millimeter level, which is significantly better than traditional template matching or pure CNN methods. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention, the accompanying drawings used in the specific embodiments will be briefly described below. In all the drawings, the elements or parts are not necessarily drawn to scale.

[0019] Figure 1 This is a schematic diagram of the multimodal large-scale vehicle infotainment touch test button recognition and positioning method and device of the present invention; Figure 2 This is a flowchart of the multimodal large-scale vehicle infotainment touch test button recognition and positioning method of the present invention; Figure 3This is a flowchart of the multimodal large-scale vehicle touch test button recognition and positioning device of the present invention. Detailed Implementation

[0020] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are merely illustrative of the technical solution of the present invention and are therefore intended to limit the scope of protection of the present invention.

[0021] like Figures 1-3 As shown, the multimodal large-scale vehicle touch test button recognition and positioning method of the present invention includes: Step S01: Construct a knowledge base related to the vehicle infotainment system UI. The knowledge base includes the association data between the visual feature vectors of the vehicle infotainment system UI and the interface identifiers, the interface hierarchy of the vehicle infotainment system UI, and the corresponding data between the interface and the visible controls. Step S02: Obtain the UI image of the current touch screen of the vehicle system and the depth image of the corresponding area; Step S03: Based on the association data between visual feature vectors and interface identifiers in the knowledge base, retrieve the interface identifiers that match the current UI image; Step S04: Combine the touch test instructions with the interface hierarchy and the corresponding data between the interface and visible controls in the knowledge base to obtain the unique identifier and interface path of the target button; Step S05: Using a preset multimodal model, input the current UI image, the standard control image of the target button, and the text description to obtain the bounding box coordinate information of the target button in the current UI image; Step S06: Based on the depth image, the preset intrinsic parameter matrix of the image acquisition device, and the coordinate transformation matrix between the image acquisition device and the touch actuator, the bounding box coordinate information is converted into three-dimensional position coordinates and attitude information in the coordinate system of the touch actuator. Step S07: Send the three-dimensional position coordinates and attitude information to the touch actuator to drive it to perform the operation corresponding to the touch test command on the vehicle touch screen.

[0022] Specifically, the principle of this invention is to first construct a knowledge base containing visual features and interface logic in the offline stage to provide data and logical support for online testing; in the online stage, through image acquisition, interface matching, semantic reasoning, multimodal localization, coordinate transformation and execution control, the accurate recognition and positioning test of vehicle touch buttons is achieved, without the need for repeated calibration or large-scale sample retraining, and it is compatible with multiple vehicle models and multiple versions of vehicle infotainment systems.

[0023] Hardware equipment: 3D Industrial Camera: Employs a Mech-Mind Nano depth camera, supporting simultaneous acquisition of color images (up to 1280×1024 resolution) and depth images (depth measurement range 0.3-0.6m, accuracy ±0.1mm), used to obtain visual and spatial depth information from the vehicle's touchscreen. Touch actuator: The robot adopts the Titan T170A collaborative robot, which is equipped with a flexible touch finger at the end, supports a repeatability accuracy of ±0.5mm, and can perform touch operations such as clicking, long pressing, and swiping. Control terminal: An industrial computer running Ubuntu 22.04 LTS operating system, used to run knowledge base construction, model reasoning, coordinate transformation and robot control programs.

[0024] Software Models and Databases: Visual feature extraction model: The BGE-VL-ScreenShot model is used to extract 2048-dimensional visual embedding vectors from the vehicle's UI interface images; Semantic reasoning model: The DeepSeek-7B large language model is used to combine test commands and UI logic to reason about target key information; Multimodal localization model: The Qwen-2.5-VL-72B multimodal large model is adopted, which supports one-shot input of "image + standard control diagram + text description" and outputs the coordinates of the button bounding box. Vector Database: The FAISS vector database is used to store the association data between the visual feature vectors of the vehicle's UI and the interface identifiers, and supports efficient cosine similarity retrieval. Control Image Library: Locally stores standard control images (such as icons for seat heating, air conditioning adjustment, volume control, etc.) for each vehicle model's infotainment system, indexed by the control's unique identifier.

[0025] In this embodiment, a knowledge base related to the vehicle's infotainment system UI is constructed, including: The resolution and brightness of the vehicle infotainment UI images of different vehicle models and software versions are standardized and normalized to obtain the processed vehicle infotainment UI images. Visual feature vectors are extracted from each of the processed vehicle UI images to obtain the visual feature vectors. Each visual feature vector is associated with its corresponding vehicle UI identifier and stored in a preset vector database; Based on the interface hierarchy of the vehicle's UI, the logic of navigation between interfaces, and the relationship between visible controls under each interface, the correspondence between interfaces and visible controls and the navigation paths between interfaces are recorded.

[0026] Specifically, the preprocessing of the vehicle's UI interface images: We collected images of the UI interfaces of mainstream car models and different software versions of each model, covering typical functional interfaces such as the main interface, seat control interface, air conditioning interface, and entertainment control interface.

[0027] Preprocessing is performed on each frame of the image: first, the resolution is standardized to 1280×1024 (to adapt to the resolution of most in-vehicle touch screens), and then the brightness is normalized by histogram equalization to eliminate the difference in image brightness under different lighting conditions, thus obtaining the processed in-vehicle UI interface image.

[0028] Visual feature vector extraction and storage: Each processed UI image frame is input into the BGE-VL-ScreenShot model, and the model outputs a 2048-dimensional visual embedding vector (i.e., visual feature vector). Assign a unique interface identifier to each frame of image (e.g., “A-ModelX-V1.2.0 - Seat Control Interface”), associate the visual feature vector with the corresponding interface identifier, and write them in batches into the FAISS vector database to construct a visual feature database associated with “visual features - interface identifier”. This database supports millisecond-level nearest neighbor retrieval.

[0029] UI navigation to knowledge base construction: Based on the product design prototypes of the in-vehicle infotainment systems of various models, we have sorted out the hierarchical relationship of the UI interface (such as "main interface → seat control interface" and "main interface → air conditioning interface"), the jump logic between interfaces (such as clicking the "seat icon" to jump from the main interface to the seat control interface), and the relationship between the visible controls under each interface (such as the visible "driver's seat heating", "passenger seat heating", and "seat ventilation" controls under the seat control interface). The above information is recorded in the form of a directed graph, such as "Main interface (interface identifier A001) → Click 'seat icon' (control identifier C003) → Seat control interface (interface identifier A005)", forming a UI jump knowledge base. This knowledge base can be stored in Excel or JSON format and supports quick retrieval during subsequent semantic reasoning.

[0030] In this embodiment, acquiring the UI image of the current touchscreen of the vehicle system and the depth image of the corresponding area includes: Industrial cameras or depth cameras are used to capture color images of the current touchscreen of the vehicle and depth images of the corresponding areas; The timestamps of the color image and the depth image are calibrated to obtain the UI image of the current touch screen and the depth image of the corresponding area.

[0031] Specifically, image acquisition parameter configuration: Fix the Mech-Mind Nano depth camera in front of the vehicle's touchscreen and enable the camera's "timestamp synchronization" function so that each frame of color image and the corresponding depth image of the area carry the same timestamp, avoiding subsequent depth value matching deviations due to asynchronous acquisition. Synchronize the acquisition of color images and depth images through the camera SDK.

[0032] Image acquisition and output: The camera is controlled to capture color and depth images of the vehicle's current touchscreen. The color image serves as the "UI image of the vehicle's current touchscreen," while the depth image is used to subsequently obtain the spatial depth information of the touch point. The acquired UI and depth images are transmitted to an industrial computer and stored in a local temporary folder for further processing.

[0033] In this embodiment, based on the association data between visual feature vectors and interface identifiers in the knowledge base, the interface identifiers matching the current UI image are retrieved, including: The UI image of the current touch screen of the vehicle is processed by resolution standardization and brightness normalization to obtain the processed UI image; Extract visual feature vectors from the processed UI image to obtain the query feature vector; Calculate the cosine similarity between the query feature vector and the visual feature vectors of each vehicle UI interface image in the knowledge base, and obtain each cosine similarity value; The visual feature vector with the highest cosine similarity value and that cosine similarity value is greater than a preset threshold is selected, and the interface identifier associated with the visual feature vector is used as the interface identifier corresponding to the current UI image.

[0034] Specifically, specifically, the current UI image preprocessing: For the acquired in-vehicle UI image, perform the same operations as "In-vehicle UI Image Preprocessing": standardize the resolution to 1280×1024 and normalize the brightness to ensure that the processed image features are comparable to the features stored in the knowledge base.

[0035] Query feature vector extraction: The preprocessed current UI image is input into the BGE-VL-ScreenShot model to extract a 2048-dimensional visual embedding vector, which is used as the "query feature vector".

[0036] Cosine similarity calculation and interface matching: The FAISS vector database retrieval interface is called to calculate the similarity between the query feature vector and all visual feature vectors in the database (the similarity calculation formula is: ,in, To query feature vectors, (the i-th visual feature vector in the database). Select the visual feature vector with the highest cosine similarity and use its associated interface identifier (such as "A-ModelX-V1.2.0-seat control interface") as the interface identifier corresponding to the current UI image.

[0037] In this embodiment, by combining touch test instructions with the interface hierarchy and the corresponding data between the interface and visible controls in the knowledge base, the unique identifier and interface path of the target button are obtained, including: Obtain touch test instructions, wherein the touch test instructions include descriptive information of the target to be touched; By using the interface hierarchy of the vehicle's UI in the knowledge base, the corresponding data of the interface and visible controls, and the interface identifier corresponding to the current UI image, we can obtain the information of the visible controls under the current interface and the information of the subsequent interfaces that the current interface can jump to. By combining the description of the target to be touched in the touch test command, the visible control information of the current interface, and the subsequent interface information that the current interface can jump to, the unique identifier of the target button and the interface path where the target button is located are obtained.

[0038] Specifically, touch test command acquisition: The system retrieves the touch test instructions to be executed from the test case library of the vehicle infotainment touch test system. The instructions are expressed in natural language and include descriptive information about the target to be touched, such as "click the driver's seat heating button".

[0039] UI logic information retrieval: Based on the obtained current interface identifier, the constructed UI jump knowledge base is called to obtain the visible control information under the current interface (such as “Driver’s seat heating (C010)”, “Passenger’s seat heating (C011)”, “Seat ventilation (C012)”) and the subsequent interface information that the current interface can jump to (such as clicking the “Back” control to jump to the main interface).

[0040] Semantic reasoning yields the target key information: The touch test command ("Click the driver's seat heating button"), the current interface identifier, and the obtained UI logic information are input into the DeepSeek-7B model. The model infers the unique identifier (C010) of the target button and the interface path ("A-ModelX-V1.2.0-Main Interface → Seat Control Interface") through semantic matching (e.g., matching "driver's seat heating" with the description of the control identifier C010) and interface path verification (e.g., confirming that the "driver's seat heating" control is visible on the current interface and does not need to jump to other interfaces).

[0041] In this embodiment, a preset multimodal model is used. The current UI image, the standard control image of the target button, and a text description are input to obtain the bounding box coordinates of the target button in the current UI image, including: Based on the unique identifier of the target button, the standard control image corresponding to the target button is retrieved from the preset control image library. The control image library stores the standard image of each control and its corresponding control identifier. Based on the function and appearance characteristics of the target button, a text description of the target button is obtained; The UI image of the current touch screen of the vehicle system, the standard control image of the target button, and the text description of the target button are input into the preset multimodal model to obtain the bounding box coordinate information of the target button in the current UI image.

[0042] Specifically, based on the obtained target button unique identifier (C010), the local control image library is accessed, and the standard control image corresponding to the identifier is retrieved (i.e., the standard icon image of "driver's seat heating", with a red circular background and a white seat heating symbol inside).

[0043] Target button text description generation: Combining the function (driver's seat heating control) and appearance characteristics of the target button, the text description of the target button is generated as follows: "Red circular control with a white seat heating pattern inside, used to control the driver's seat heating function, located in the left area of ​​the seat control interface."

[0044] Multimodal model localization bounding box: The current UI image of the vehicle system, the standard control image retrieved, and the generated text description are used as inputs and passed into the Qwen-2.5-VL-72B multimodal model. The model outputs the bounding box coordinates of the target button in the current UI image through visual feature comparison (feature matching between the standard control image and the candidate region in the current UI image) and text semantic constraints (text descriptions limit the position and appearance of the control), in the format (x_min, y_min, x_max, y_max), for example (500, 300, 550, 350), and also outputs the confidence score.

[0045] In this embodiment, based on the depth image, a preset intrinsic parameter matrix of the image acquisition device, and a coordinate transformation matrix between the image acquisition device and the touch actuator, the bounding box coordinate information is converted into three-dimensional position coordinates and attitude information in the coordinate system of the touch actuator, including: Based on the bounding box coordinate information of the target button, calculate the center pixel coordinates of the bounding box, where the center pixel coordinates are the average of the pixel coordinates of the top left corner and the bottom right corner of the bounding box. Extract the depth value corresponding to the center pixel coordinates from the depth image to obtain the target depth value; Based on the preset intrinsic parameter matrix of the image acquisition device, the coordinates of the center pixel and the corresponding target depth value are back-projected to obtain the three-dimensional coordinates in the coordinate system of the image acquisition device. A preset coordinate transformation matrix between the image acquisition device and the touch actuator is used to convert the three-dimensional coordinates in the coordinate system of the image acquisition device into the three-dimensional position coordinates in the coordinate system of the touch actuator.

[0046] Specifically, the calculation of the center pixel coordinates of the bounding box: Based on the obtained bounding box coordinates (500, 300, 550, 350), the center pixel coordinates are calculated as (525, 325) using the formula "center pixel coordinates = ((x_min+x_max) / 2, (y_min+y_max) / 2)".

[0047] Target depth value extraction: Open the acquired depth image and read the depth value corresponding to the center pixel coordinates (525, 325) (i.e., the distance from the center of the target button in the current UI image to the camera).

[0048] Calculation of 3D coordinates in the camera coordinate system: Call the intrinsic parameter matrix of the Mech-Mind Nano camera (obtained in advance through camera calibration, intrinsic parameter matrix) ; in, =1000 pixels =1000 pixels =960 pixels =540 pixels; The 3D coordinates (Xc, Yc, Zc) in the camera coordinate system are calculated by back-projection using the intrinsic parameter matrix. The calculation formula is as follows: , (Where, U=525 and V=325 are the center pixel coordinates, and Z=150mm is the target depth value).

[0049] Coordinate system pose transformation of touch actuator: Call the pre-calibrated "camera-robot" coordinate transformation matrix (i.e., the coordinate transformation matrix between the image acquisition device and the touch actuator). (A 4×4 homogeneous matrix, obtained through hand-eye calibration). Substitute the 3D coordinates (Xc, Yc, Zc) in the camera coordinate system into the transformation matrix. The three-dimensional position coordinates (Xe=200mm, Ye=150mm, Ze=300mm) of the end effector of the Titan Tiger robot T170A were calculated. Based on the operation type corresponding to the touch test command ("click" in this embodiment), the robot end-effector posture information is determined: the touch finger is perpendicular to the vehicle touch screen, and the posture angle Re=0° (rotation angle around the Z-axis of the end-effector). Finally, the target pose in the coordinate system of the touch actuator is obtained (Xe=200mm, Ye=150mm, Ze=300mm, Re=0°).

[0050] In this embodiment, the three-dimensional position coordinates and attitude information are sent to the touch actuator to drive it to perform the operation corresponding to the touch test command on the vehicle's touch screen, including: The three-dimensional position coordinates and attitude information of the touch actuator in the coordinate system are sent to the control unit of the touch actuator; The control unit plans the motion trajectory of the touch actuator components based on three-dimensional position coordinates and attitude information; The actuator of the touch control mechanism moves to the target position according to the planned motion trajectory and performs the operation corresponding to the touch test command; During the operation, the position and speed information of the touch actuator components are collected in real time to form an operation execution record.

[0051] Specifically, the target pose is transmitted: The obtained target pose (Xe=200mm, Ye=150mm, Ze=300mm, Re=0°) is sent to the control unit of the Titan T170A robot via EtherCAT bus.

[0052] Motion trajectory planning: Based on the target pose, the robot control unit plans the motion trajectory using a joint space interpolation algorithm: from the robot's current position (Xe=0mm, Ye=0mm, Ze=500mm) to the target position (Xe=200mm, Ye=150mm, Ze=300mm) at a preset speed. During the motion, the attitude angle Re=0° remains unchanged to avoid interference between the touch finger and other components of the vehicle.

[0053] Touch operation execution: After the robot moves to the target position according to the planned trajectory, the control end touch finger performs a "click" operation with a preset pressure (adapted to the trigger pressure requirements of the vehicle's touch screen) and click duration to complete the touch test instruction requirements; If the test command is "slide" (such as "slide from left to right to adjust volume"), then control the touch finger to perform the operation according to the preset sliding trajectory.

[0054] Execution process record: During operation, the robot's joint encoders collect the position and speed information of the executing components in real time. The industrial computer records the start time, end time, operation type, and execution result (such as "click successful" or "slide to position"), forming an operation execution record, which is stored in the test log database for subsequent test result traceability and analysis.

[0055] This embodiment, through the above steps, can achieve automated recognition and positioning testing of touch buttons for multiple vehicle models and versions; the testing efficiency is significantly improved compared to manual testing. When the UI version is updated, only 30-50 frames of interface images need to be added to the knowledge base for adaptation, effectively solving the problems of low efficiency, poor generalization, and semantic loss in existing solutions, and meeting the rapid testing needs after batch generation and OTA upgrade of vehicle infotainment systems.

[0056] In other embodiments, a multimodal large-scale vehicle infotainment touch test button recognition and positioning device is provided, and a multimodal large-scale vehicle infotainment touch test button recognition and positioning method based on any of the preceding embodiments includes: The module is configured to build a knowledge base related to the vehicle infotainment system UI. The knowledge base includes the association data between the visual feature vectors of the vehicle infotainment system UI and the interface identifiers, the interface hierarchy of the vehicle infotainment system UI, and the corresponding data between the interface and the visible controls. The acquisition module is configured to acquire the UI image of the current touchscreen of the vehicle system and the depth image of the corresponding area; The retrieval module is configured to retrieve interface icons that match the current UI image based on the association data between visual feature vectors and interface icons in the knowledge base. The module is configured to combine touch test commands with the interface hierarchy and corresponding data between the interface and visible controls in the knowledge base to obtain the unique identifier and interface path of the target button. The input module is configured to use a preset multimodal model to input the current UI image, the standard control image of the target button, and the text description, and obtain the bounding box coordinate information of the target button in the current UI image; The conversion module is configured to convert bounding box coordinate information into three-dimensional position coordinates and attitude information in the coordinate system of the touch actuator based on depth images, preset image acquisition device intrinsic parameter matrix, and coordinate transformation matrix between image acquisition device and touch actuator. The drive module is configured to send three-dimensional position coordinates and attitude information to the touch actuator, driving it to perform operations corresponding to touch test commands on the vehicle's touch screen.

[0057] In other embodiments, a terminal is provided, including a processor, an input device, an output device, and a memory, which are interconnected. The memory is used to store a computer program, which includes program instructions. The processor is configured to invoke the program instructions to execute the multimodal large-scale vehicle touch test button recognition and positioning method as described above.

[0058] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A method for button recognition and positioning in multimodal large-scale vehicle infotainment touch testing, characterized in that, include: Construct a knowledge base related to vehicle infotainment system UI, wherein the knowledge base includes the association data between the visual feature vectors of the vehicle infotainment system UI interface and the interface identifiers, the interface hierarchy of the vehicle infotainment system UI, and the corresponding data between the interface and the visible controls; Obtain the UI image of the current touchscreen of the vehicle's infotainment system and the depth image of the corresponding area; Based on the association data between visual feature vectors and interface identifiers in the knowledge base, retrieve the interface identifiers that match the current UI image; By combining the touch test instructions with the interface hierarchy and the corresponding data between the interface and visible controls in the knowledge base, the unique identifier and interface path of the target button are obtained. Using a preset multimodal model, input the current UI image, the standard control image of the target button, and the text description, and obtain the bounding box coordinate information of the target button in the current UI image; Based on the depth image, the preset image acquisition device intrinsic parameter matrix, and the coordinate transformation matrix between the image acquisition device and the touch actuator, the bounding box coordinate information is converted into three-dimensional position coordinates and attitude information in the touch actuator coordinate system; The three-dimensional position coordinates and attitude information are sent to the touch actuator, which drives it to perform the operation corresponding to the touch test command on the vehicle touch screen; The construction of the vehicle infotainment UI-related knowledge base includes: The resolution and brightness of the vehicle infotainment UI images of different vehicle models and software versions are standardized and normalized to obtain the processed vehicle infotainment UI images. Visual feature vectors are extracted from each of the processed vehicle UI images to obtain the visual feature vectors. Each visual feature vector is associated with its corresponding vehicle UI identifier and stored in a preset vector database; Based on the interface hierarchy of the vehicle's UI, the logic of navigation between interfaces, and the relationship between visible controls under each interface, the correspondence between interfaces and visible controls and the navigation path between interfaces are recorded. The process employs a preset multimodal model, inputting the current UI image, the standard control image of the target button, and a text description, to obtain the bounding box coordinates of the target button within the current UI image, including: Based on the unique identifier of the target button, the standard control image corresponding to the target button is retrieved from the preset control image library. The control image library stores the standard image of each control and its corresponding control identifier. Based on the function and appearance characteristics of the target button, a text description of the target button is obtained; The UI image of the current touch screen of the vehicle system, the standard control image of the target button, and the text description of the target button are input into the preset multimodal model to obtain the bounding box coordinate information of the target button in the current UI image.

2. The method for identifying and locating touch control buttons in a multimodal large-scale vehicle system according to claim 1, characterized in that, The acquisition of the UI image of the current touchscreen of the vehicle system and the depth image of the corresponding area includes: Industrial cameras or depth cameras are used to capture color images of the current touchscreen of the vehicle and depth images of the corresponding areas; The timestamps of the color image and the depth image are calibrated to obtain the UI image of the current touch screen and the depth image of the corresponding area.

3. The method for identifying and locating touch control buttons in a multimodal large-scale vehicle system according to claim 1, characterized in that, The step of retrieving interface identifiers matching the current UI image based on the association data between visual feature vectors and interface identifiers in the knowledge base includes: The UI image of the current touch screen of the vehicle is processed by resolution standardization and brightness normalization to obtain the processed UI image; Extract visual feature vectors from the processed UI image to obtain the query feature vector; Calculate the cosine similarity between the query feature vector and the visual feature vectors of each vehicle UI interface image in the knowledge base, and obtain each cosine similarity value; The visual feature vector with the highest cosine similarity value and that cosine similarity value is greater than a preset threshold is selected, and the interface identifier associated with the visual feature vector is used as the interface identifier corresponding to the current UI image.

4. The method for identifying and locating touch control buttons in a multimodal large-scale vehicle system according to claim 1, characterized in that, The process of combining touch test commands with the interface hierarchy and corresponding data between the interface and visible controls in the knowledge base to obtain the unique identifier and interface path of the target button includes: Obtain a touch test instruction, wherein the touch test instruction includes descriptive information of the target to be touched; By using the interface hierarchy of the vehicle's UI in the knowledge base, the corresponding data of the interface and visible controls, and the interface identifier corresponding to the current UI image, we can obtain the information of the visible controls under the current interface and the information of the subsequent interfaces that the current interface can jump to. By combining the description of the target to be touched in the touch test command, the visible control information of the current interface, and the subsequent interface information that the current interface can jump to, the unique identifier of the target button and the interface path where the target button is located are obtained.

5. The method for identifying and locating touch control buttons in a multimodal large-scale vehicle system according to claim 1, characterized in that, The method, based on the depth image, a preset intrinsic parameter matrix of the image acquisition device, and a coordinate transformation matrix between the image acquisition device and the touch actuator, converts the bounding box coordinate information into three-dimensional position coordinates and attitude information in the coordinate system of the touch actuator, including: Based on the bounding box coordinate information of the target button, the center pixel coordinates of the bounding box are calculated, wherein the center pixel coordinates are the average of the pixel coordinates of the upper left corner and the pixel coordinates of the lower right corner of the bounding box. The target depth value is obtained by extracting the depth value corresponding to the coordinates of the center pixel from the depth image. Based on the preset intrinsic parameter matrix of the image acquisition device, the coordinates of the center pixel and the corresponding target depth value are back-projected to obtain the three-dimensional coordinates in the coordinate system of the image acquisition device. A preset coordinate transformation matrix between the image acquisition device and the touch actuator is used to convert the three-dimensional coordinates in the coordinate system of the image acquisition device into the three-dimensional position coordinates in the coordinate system of the touch actuator.

6. The method for identifying and locating touch control buttons in a multimodal large-scale vehicle system according to claim 1, characterized in that, The three-dimensional position coordinates and attitude information are sent to the touch actuator to drive it to perform operations corresponding to the touch test commands on the vehicle's touch screen, including: The three-dimensional position coordinates and attitude information of the touch actuator in the coordinate system are sent to the control unit of the touch actuator; The control unit plans the motion trajectory of the touch actuator components based on three-dimensional position coordinates and attitude information; The actuator of the touch control mechanism moves to the target position according to the planned motion trajectory and performs the operation corresponding to the touch test command; During the operation, the position and speed information of the touch actuator components are collected in real time to form an operation execution record.

7. A multimodal large-scale vehicle infotainment touch control test button recognition and positioning device, characterized in that, The in-vehicle infotainment touch test button recognition and positioning method based on any one of claims 1 to 6 includes: The construction module is configured to build a knowledge base related to the vehicle infotainment system UI. The knowledge base includes the association data between the visual feature vectors of the vehicle infotainment system UI and the interface identifiers, the interface hierarchy of the vehicle infotainment system UI, and the corresponding data between the interface and the visible controls. The acquisition module is configured to acquire the UI image of the current touchscreen of the vehicle system and the depth image of the corresponding area; The retrieval module is configured to retrieve interface icons that match the current UI image based on the association data between visual feature vectors and interface icons in the knowledge base. The module is configured to combine touch test commands with the corresponding data of interface hierarchy and visible controls in the knowledge base to obtain the unique identifier and interface path of the target button. The first input module is configured to use a preset multimodal model to input the current UI image, the standard control image of the target button, and the text description, and obtain the bounding box coordinate information of the target button in the current UI image; The conversion module is configured to convert the bounding box coordinate information into three-dimensional position coordinates and attitude information in the coordinate system of the touch actuator based on the depth image, the preset image acquisition device intrinsic parameter matrix, and the coordinate transformation matrix between the image acquisition device and the touch actuator. The driving module is configured to send the three-dimensional position coordinates and attitude information to the touch actuator, driving it to perform the operation corresponding to the touch test command on the vehicle touch screen; The processing module is configured to perform resolution standardization and brightness normalization processing on the vehicle infotainment UI interface images of different vehicle models and different software versions to obtain the processed vehicle infotainment UI interface images. The extraction module is configured to extract visual feature vectors from each processed vehicle UI interface image to obtain each visual feature vector. The identifier association module is configured to associate each visual feature vector with the corresponding vehicle UI interface identifier and store it in a preset vector database. The recording module is configured to record the interface hierarchy, the jump logic between interfaces, and the relationship between visible controls under each interface based on the vehicle's UI, as well as the correspondence between interfaces and visible controls and the jump path between interfaces. The retrieval module is configured to retrieve the standard control image corresponding to the target button from a preset control image library based on the unique identifier of the target button. The control image library stores the standard image of each control and its corresponding control identifier. The text description module is configured to obtain a text description of the target button based on its function and appearance features; The second input module is configured to input the UI image of the current touch screen of the vehicle system, the standard control image of the target button, and the text description of the target button into a preset multimodal model to obtain the bounding box coordinate information of the target button in the current UI image.

8. A terminal, characterized in that, The system includes a processor, an input device, an output device, and a memory, which are interconnected. The memory stores a computer program, which includes program instructions. The processor is configured to invoke the program instructions to execute the multimodal large-scale vehicle touch test button recognition and positioning method as described in any one of claims 1 to 6.

Images