A testing method and device for vehicle-mounted screen based on a robotic arm

Abstract

The present invention relates to the field of vehicle testing technology, and more specifically, to a method and apparatus for testing an in-vehicle screen using a robotic arm. The method comprises: controlling the distal end of the robotic arm to enter the vehicle through a window; acquiring initial point cloud data of the in-vehicle screen using a depth camera; completing the initial point cloud data based on pre-collected complete point cloud data of the in-vehicle screen to obtain final point cloud data; and performing a screen test based on the final point cloud data. The robotic arm of the present invention can automatically move toward the screen in a variety of scenarios and complete the point cloud data of the in-vehicle screen to ensure the validity of the test data.

Description

Technical Field

[0001] The present invention relates to the field of vehicle testing technology, and in particular to a testing method and device for a vehicle-mounted screen based on a robotic arm. Background Art

[0002] Robotic arm-based in-vehicle screen testing is an efficient and precise automated testing technology widely used for performance verification of intelligent cockpits, central control screens, and in-vehicle infotainment (IVI) systems. This method uses a robotic arm to simulate human operation and combines sensors, image recognition, and data analysis technologies to comprehensively test touchscreen response speed, sensitivity, anti-interference capabilities, and interactive functions.

[0003] In the existing technology, the robotic arm needs to set a fixed motion path to move toward the screen, and is unable to identify the validity of the vehicle-mounted screen.

[0004] In view of this, the present invention is proposed. Summary of the Invention

[0005] The purpose of the present invention is to provide a testing method and equipment for a vehicle-mounted screen based on a robotic arm. The robotic arm can automatically move toward the screen in a variety of scenarios and complete the point cloud data of the vehicle-mounted screen to ensure the validity of the test data.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for testing a vehicle screen based on a robotic arm, which is applicable to a robotic arm equipped with a depth camera, and the method comprises:

[0008] Control the end of the robotic arm to enter the car through the window;

[0009] Collect the initial point cloud data of the vehicle screen through the depth camera;

[0010] Completing the initial point cloud data based on the pre-collected complete point cloud data of the vehicle screen to obtain final point cloud data;

[0011] Performing a screen test based on the final point cloud data;

[0012] Before completing the initial point cloud data based on the pre-collected complete point cloud data of the vehicle screen to obtain the final point cloud data, the method further includes:

[0013] Perform coarse registration between the complete point cloud data and the initial point cloud data;

[0014] The complete point cloud data is precisely aligned with the initial point cloud data.

[0015] In a second aspect, the present invention provides a vehicle-mounted screen testing device based on a robotic arm, comprising: a depth camera, a robotic arm, and an electronic device;

[0016] The electronic device comprises:

[0017] at least one processor, and a memory communicatively coupled to the at least one processor;

[0018] The memory stores instructions that can be executed by at least one of the processors. The instructions are executed by at least one of the processors to enable the at least one processor to perform a method for testing a vehicle-mounted screen based on a robotic arm.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] In this embodiment, a depth camera guides a robotic arm to automatically inspect the in-vehicle display. Based on position calibration and point cloud registration, and finally completing the screen point cloud, the automated testing function is implemented. The combination of a robotic arm and visual inspection technology can significantly improve the automation level of in-vehicle display inspection, reduce manual intervention, and mitigate the impact of human error. Automated inspection not only improves inspection accuracy but also shortens inspection cycles, thereby increasing overall production efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 1 is a flow chart of a method for testing a vehicle-mounted screen based on a robotic arm provided by an embodiment of the present invention;

[0023] Figure 2 1 is a schematic structural diagram of a vehicle-mounted screen testing device based on a robotic arm provided by an embodiment of the present invention;

[0024] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] Example 1

[0027] Figure 1 This is a flowchart of a robotic arm-based vehicle screen testing method provided in this embodiment. This embodiment is applicable to situations where the robotic arm moves from outside the vehicle toward the vehicle screen and photographs the screen. The method provided in this embodiment is applicable to execution by robotic arm-based vehicle screen testing equipment.

[0028] The test equipment includes a depth camera, a robotic arm, and electronic equipment. The robotic arm itself can move in six degrees of freedom, including telescopic motion. Furthermore, it can be moved via slide rails, extending its range of motion. A stylus at the end of the robotic arm can be used to perform motion operations such as tapping, sliding, and pressing, based on the position of the vehicle's screen and the angles of the robotic arm's joints.

[0029] A structured light depth camera is a device that projects a specific light pattern (usually stripes or dots) into a scene and then analyzes the deformation of the light pattern to obtain three-dimensional depth information. It is an important tool in computer vision and 3D reconstruction, and is widely used in fields such as face recognition, gesture recognition, and industrial inspection. It can generate point cloud information to guide robotic arms. This depth camera can capture both three-dimensional point cloud data and two-dimensional images of the environment.

[0030] The depth camera is mounted on a robotic arm. This includes: 1) the depth camera is fixed to the robotic arm and moves with the arm, i.e., "hand-on-eye"; 2) the depth camera is external to the robotic arm, in a fixed position and does not move with the arm, i.e., "hand-off-eye". Before formal testing, position calibration using a calibration plate is required to achieve the conversion between the depth camera coordinate system and the robotic arm's base coordinate system. This process is commonly referred to as "hand-eye calibration" and its purpose is to establish a precise relationship between the visual system and the robotic arm's kinematics. Specifically, the hand-on-eye calibration algorithm solves the depth camera's extrinsic parameters. For the depth camera mounted on the robotic arm, the coordinate transformation between the robotic arm's end coordinate system and the depth camera's coordinate system must be solved. This coordinate transformation, known as the extrinsic parameter, is a rotation and translation matrix. The hand-off-eye calibration algorithm solves the depth camera's extrinsic parameters. For the depth camera mounted externally to the robotic arm, the coordinate transformation between the robotic arm's base coordinate system and the depth camera's coordinate system must be solved. This coordinate transformation, known as the extrinsic parameter, is a rotation and translation matrix.

[0031] During the calibration process, a depth camera is used to capture an image of the calibration plate. Ensure that the calibration plate covers the center of the camera's field of view, and rotate or move the plate multiple times to obtain rich perspective information. Taking advantage of the depth camera's ability to obtain depth information, the depth point cloud information is fused with the image pixel information. The depth camera is used to obtain the 3D coordinates of the calibration plate's corner points and match them with the 2D coordinates in the image to form corresponding point pairs. During data processing, the calibration results can be optimized using least squares or singular value decomposition (SVD) methods to reduce reprojection error, resulting in more accurate information and a more precise final result. Reprojection error is the result of projecting corner points identified in an image onto the same coordinate system. The accuracy of corner identification can be determined by the error between the projected points of the same corner point after projection into different images.

[0032] During the calibration process, the robot's motion path needs to be set, including the depth camera's depth calibration range parameters. These parameters should be determined based on the recommended working distance of the depth camera and the robot's workspace size. You can set the path type to a pyramid structure (InHand) with a narrow top and wide bottom, and specify parameters such as the pyramid's height range, number of layers, and bottom layer size to ensure that the calibration range is covered and accuracy requirements are met. Convert the coordinate points on the pyramid structure to coordinate points in the robot's base coordinate system, and output the series of coordinate points in sequence to obtain a series of path points for automatic calibration.

[0033] The present invention first selects an appropriate hand-eye calibration algorithm based on the depth camera's installation position. This calibration algorithm calculates the rotation and translation matrices, or extrinsic parameters. These extrinsic parameters remain unchanged in the external environment and need not be altered. The calibration algorithm should be rerun each time the relative position of the robotic arm and the depth camera changes. These extrinsic parameters, combined with the camera's intrinsic parameters and the robotic arm's pose during camera capture, yield the three-dimensional coordinate transformation relationship between the two-dimensional coordinates of the camera's image coordinate system (also known as the pixel coordinate system) and the robotic arm's base coordinate system. The specific transformation relationship is described in the prior art and will not be elaborated here.

[0034] The method provided in this embodiment is executed by an electronic device. Figure 1 As shown, this embodiment provides a method for testing a vehicle-mounted screen based on a robotic arm, comprising the following steps:

[0035] S110. Control the end of the robotic arm to enter the vehicle through the vehicle window.

[0036] The base of the robotic arm is located outside the car. The end of the robotic arm needs to be moved through the car window into the car through the movement of each joint of the robotic arm to operate the on-board screen, including: identifying the window position based on the window point cloud data collected by the depth camera; and controlling the robotic arm to enter the car through the window.

[0037] Multiple sets of vehicle window point cloud data collected at different times are transformed from the depth camera coordinate system to a global coordinate system, such as the ground coordinate system. Through this coordinate transformation, the coordinates of each point are recalculated to reflect its position in the global coordinate system, thereby achieving the overall transformation and merging of the vehicle window point cloud. This process is repeated for all point clouds, ultimately forming a unified, transformed 3D point cloud model that can be saved and used for subsequent 3D data processing and feature recognition to guide the robot arm.

[0038] The window point cloud is identified from the three-dimensional point cloud model, and then the position of the window is identified. The position of the window is used as the motion path of the robotic arm, and the robotic arm is controlled to enter the car through the window.

[0039] Optionally, in the process of controlling the robotic arm to enter the car, the joint limitation of the robotic arm and the rapid exploration random tree algorithm are used to plan the movement path of the robotic arm to avoid touching obstacles such as car doors and steering wheels.

[0040] Specifically, collision avoidance is a key issue in robotic arm motion control. Its goal is to ensure that each joint remains within a safe range during the robot's execution, avoiding interference between joints or collisions with other objects. This obstacle avoidance algorithm employs two approaches: joint constraints and trajectory planning simulation based on a rapidly exploring random tree algorithm (RRT algorithm). With joint constraints, each joint angle typically has physical limitations stemming from its mechanical structure and range of motion. By obtaining the extreme angles (maximum and minimum) for each joint, a reasonable operating range can be set for each joint. Within this range, the joint can operate smoothly without exceeding the extreme angles, which could lead to mechanical damage or control system errors. The RRT algorithm performs path planning in joint space, exploring the space through random sampling and gradually building a tree structure until a collision-free path from the starting position to the target position is found. This approach is suitable for solving robotic arm path planning problems in high-dimensional spaces and complex environments. While avoiding obstacles, the robot arm is guided to the inspection station.

[0041] S120: Collect initial point cloud data of the vehicle screen through the depth camera.

[0042] The end of the robotic arm (i.e., the actuator, such as a stylus) is used to touch and click the in-vehicle screen. At the same time, the in-vehicle screen is photographed by a depth camera to obtain initial point cloud data.

[0043] Specifically, the coordinates of the screen pixels where the robotic arm performs click, slide, and press operations on the vehicle screen are determined. These pixel coordinates are converted to three-dimensional coordinates in the robotic arm's coordinate system. These three-dimensional coordinates are used to guide the robotic arm to complete the click, slide, and press operations, completing the automatic detection of the robotic arm.

[0044] S130 , completing the initial point cloud data based on the pre-collected complete point cloud data of the vehicle-mounted screen to obtain final point cloud data.

[0045] Because the screen may be reflective or contaminated, the initial point cloud data may not represent the entire screen. Therefore, a diffuse reflective film is applied to the closed screen and the screen is photographed with a depth camera to obtain complete point cloud data. From this complete point cloud data, missing points in the initial point cloud data are selected to complete the final point cloud data.

[0046] S140: Perform screen testing based on the final point cloud data.

[0047] After point cloud completion, the depth camera's internal parameters are used to convert the final point cloud data into a depth map for subsequent testing, such as whether the screen responds as expected and the response time. After the test is complete, the window position recognition is re-executed, the exit path is calculated, and the vehicle is withdrawn from the window.

[0048] In this embodiment, a depth camera guides a robotic arm to automatically inspect the in-vehicle display. Based on position calibration and point cloud registration, and finally completing the screen point cloud, the automated testing function is implemented. The combination of a robotic arm and visual inspection technology can significantly improve the automation level of in-vehicle display inspection, reduce manual intervention, and mitigate the impact of human error. Automated inspection not only improves inspection accuracy but also shortens inspection cycles, thereby increasing overall production efficiency.

[0049] Example 2

[0050] This embodiment optimizes the process of point cloud completion based on the above embodiment. Optionally, the initial point cloud data is completed based on the complete point cloud data of the vehicle screen collected in advance, and before obtaining the final point cloud data, the process of coarse and fine registration of the complete point cloud data and the initial point cloud data is also included. After the registration is completed, the complete point cloud data and the initial point cloud data are compared, and the target point cloud data that is missing from the initial point cloud data and exists in the complete point cloud data is determined, and the target point cloud data is supplemented to the initial point cloud data to form the final point cloud data. In the actual test process, the parts of the screen that are affected by light and other positions are generally the corners of the screen or the background part of the screen; while the effective information on the screen, such as icons and text, is often displayed in a relatively clear appearance style, and there will be no situation where it cannot be recognized. Therefore, this embodiment uses pre-stored complete point cloud data to complete the initial point cloud data, which will not affect the actual response of the screen and will not affect the test results.

[0051] Optionally, a coarse registration is performed on the complete point cloud data and the initial point cloud data; a fine registration is performed on the complete point cloud data and the initial point cloud data. The following describes the coarse registration process in detail, which includes the following four steps:

[0052] The first step is to extract key points with both color and geometric characteristics from the initial point cloud data.

[0053] Depth cameras can capture point cloud data containing both geometric and color information. Keypoint algorithms can significantly speed up subsequent matching. However, existing keypoint algorithms focus solely on geometric information. This invention utilizes the color gradients within the point cloud's color information to extract texture keypoint sets where significant changes occur, and utilizes the curvature changes within the geometric information to obtain geometric keypoint sets. The union of these texture and geometric keypoint sets is then extracted. This embodiment utilizes keypoints with both color and geometric characteristics for matching, allowing color information to be integrated into subsequent algorithms to improve overall computational speed and accuracy.

[0054] Specifically, based on the color gradient between a point in the point cloud and its neighborhood, points whose color gradient exceeds a customizable threshold are extracted and added to the texture keypoint set. Based on the curvature of a point in the point cloud and its neighborhood, points whose curvature exceeds a customizable threshold are extracted and added to the geometry keypoint set. The color gradient and curvature are calculated based on the point cloud and its neighborhood points. They represent the local texture and geometric transformation trends of the point cloud. Whether this trend exceeds the threshold determines whether the point cloud is representative.

[0055] Optionally, for point clouds that exist in both the texture keypoint set and the geometric keypoint set, the point clouds can be filtered using an evenly spaced sampling method and proceed to the subsequent descriptor construction step. For point clouds that exist only in the texture keypoint set or only in the geometric keypoint set, all of them can be retained and proceed to the subsequent descriptor construction step. This method preserves keypoints with sufficient characteristics while avoiding redundancy.

[0056] The second step is to construct a descriptor based on the color and geometric characteristics of the key points.

[0057] For any key point P i and the neighboring point P k , twisted by the normal vector , normal vector angle , the angle between the normal vector and the connecting line Constructing geometric descriptors . Among them, the normal vector torsion angle , normal vector angle , the angle between the normal vector and the connecting line The existing SPFH geometric feature characterizes the local geometric structure of the point cloud from three perspectives. The creativity of this embodiment lies in the introduction of a descriptor constructed based on color characteristics.

[0058] According to the key point P i and the neighboring point P k Color differences between And the color gradient direction , construct a color descriptor , see the following formula:

[0059] ;

[0060] in, RGB / HSV color vectors of key points and domain points respectively.

[0061] Compute the color gradient direction on the local surface of a point cloud:

[0062] ;

[0063] in, They are the color gradients in the x and y directions of the local coordinate system. The local coordinate system is based on the key point P i The color gradient is calculated based on the pixel difference / first-order derivative between the key point and its neighboring points. Specifically, the pixel difference / first-order derivative between the key point and its neighboring points in the x-direction is used as the color gradient in the x-direction. Similarly, the pixel difference / first-order derivative between the key point and its neighboring points in the y-direction is used as the color gradient in the y-direction.

[0064] According to the normal vector twist angle , normal vector angle , the angle between the normal vector and the connecting line , color difference And the color gradient direction , construct a comprehensive descriptor .

[0065] The third step is to perform a rough registration of the complete point cloud data with the initial point cloud data based on the fast point feature histogram FPFH feature search algorithm in combination with the descriptor.

[0066] The following formula is used to calculate the fast point feature histogram FPFH of each key point neighborhood and perform feature matching;

[0067] ;

[0068] Among them, g and c are weight coefficients, n is the number of neighborhood points, i is the key point index, k is the neighborhood point index, Indicates the distance between the key point and its neighboring points.

[0069] The process of fine registration is described in detail below.

[0070] Because most of the screen test scenes have obvious surface features, this embodiment adopts a point-surface iterative nearest point algorithm, which completes the precise matching work by iterating the error formula between the point and the surface to the minimum, and finally obtains accurate results.

[0071] Example 3

[0072] See also Figure 2 This embodiment also provides a vehicle screen testing device based on a robotic arm, including: a depth camera, a robotic arm and an electronic device. Figure 3 For other parts shown, please refer to the description of the above embodiment and will not be repeated here.

[0073] like Figure 3 As shown, this embodiment provides an electronic device, including:

[0074] at least one processor; and

[0075] a memory communicatively connected to at least one of the processors; wherein,

[0076] The memory stores instructions executable by at least one processor, and the instructions are executed by at least one processor to enable at least one processor to perform the above method. The at least one processor in the electronic device is capable of performing the above method, thereby having at least the same advantages as the above method.

[0077] Optionally, the electronic device also includes interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of a GUI (Graphical User Interface) on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors can be used with multiple memories, and / or multiple buses can be used with multiple memories. Similarly, multiple electronic devices can be connected (for example, as a server array, a group of blade servers, or a multi-processor system), with each device providing part of the necessary operations. Figure 3 A processor 301 is taken as an example.

[0078] Memory 302, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the robotic arm-based vehicle-mounted display testing method in the embodiments of the present invention. Processor 301 executes the software programs, instructions, and modules stored in memory 302 to execute various functional applications and data processing of the device, thereby implementing the aforementioned robotic arm-based vehicle-mounted display testing method.

[0079] The memory 302 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the terminal, etc. Furthermore, the memory 302 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 302 may further include a memory remotely located relative to the processor 301, and these remote memories may be connected to the device via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0080] The electronic device may further include: an input device 303 and an output device 304. The processor 301, the memory 302, the input device 303 and the output device 304 may be connected via a bus or other means. Figure 3 The bus connection is taken as an example.

[0081] The input device 303 can receive input digital or character information, and the output device 304 can include a display device, an auxiliary lighting device (e.g., an LED), a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0082] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0083] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present invention can be achieved. This is not limited herein.

[0084] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A testing method for a vehicle-mounted screen based on a robotic arm, characterized in that: Applicable to a robotic arm equipped with a depth camera, the method includes: Control the end of the robotic arm to enter the car through the window; Collect the initial point cloud data of the vehicle screen through the depth camera; The initial point cloud data is completed based on the pre-collected complete point cloud data of the vehicle screen to obtain final point cloud data; wherein, a diffuse reflection film is attached to the closed screen, and the screen is photographed with a depth camera to obtain the complete point cloud data; Performing a screen test based on the final point cloud data; Before completing the initial point cloud data based on the pre-collected complete point cloud data of the vehicle screen to obtain the final point cloud data, the method further includes: The complete point cloud data and the initial point cloud data are roughly aligned, including: according to the color gradient in the color information of the point cloud and the neighborhood, the point cloud with a color gradient exceeding the threshold is extracted and put into the texture key point set; according to the curvature of the point cloud and the neighborhood, the point cloud with a curvature exceeding the threshold is extracted and put into the geometric key point set; the union of the texture key point set and the geometric key point set is extracted; for any key point P i and the neighboring point P k , the geometric descriptor SPFH is constructed by the normal vector torsion angle α, the normal vector angle θ, and the angle β between the normal vector and the connecting line geo (p k ) = [α, β, θ]; According to the key point P i and the neighboring point P k The color difference ΔC and the color gradient direction θ c , construct color descriptor SPFH color (p k )=[ΔC,θ c ]; According to the normal vector torsion angle α, the normal vector angle θ, the angle β between the normal vector and the connecting line, the color difference ΔC and the color gradient direction θ c , construct comprehensive descriptor SPFH(p i )=[α,β,θ,ΔC,θ c ]; The following formula is used to calculate the fast point feature histogram FPFH of each key point neighborhood and perform feature matching; Among them, g and c are weight coefficients, n is the number of neighborhood points, i is the index of the key point, and k is the index of the neighborhood point; The complete point cloud data is precisely aligned with the initial point cloud data.

2. The method according to claim 1, characterized in that Control the end of the robotic arm to enter the car through the window, including: The window position is identified based on the window point cloud data collected by the depth camera; and the robotic arm is controlled to enter the car through the window.

3. The method according to claim 1, characterized in that Control the end of the robotic arm to enter the car through the window, including: In the process of controlling the robotic arm to enter the vehicle, the joint limitation of the robotic arm and the rapid exploration random tree algorithm are used to plan the operation path of the robotic arm.

4. A vehicle-mounted screen testing device based on a robotic arm, characterized in that: include: depth cameras, robotic arms, and electronics; The electronic device comprises: at least one processor, and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by at least one of the processors, and the instructions are executed by at least one of the processors to enable at least one of the processors to execute the method for testing a vehicle-mounted screen based on a robotic arm according to any one of claims 1 to 3.

Images