A device and method for detecting the response time of a curved screen of an automobile

Abstract

The present invention relates to the technical field of automobile curved screen detection, and in particular to a device and method for detecting the response time of an automobile curved screen, the method comprising: optimizing and affine transforming pixels on a two-dimensional plane image obtained by a 2D camera to obtain corresponding three-dimensional position information of the curved screen; determining the three-dimensional position information of a target icon on the automobile curved screen by a 3D camera, controlling a mechanical arm to accurately click the target icon on the automobile curved screen according to navigation coordinates, synchronously starting high-frame rate cameras set in different directions, obtaining continuous images and time information of changes in different areas of the curved screen when the mechanical arm operates on the automobile curved screen; inputting into a response time detection algorithm for analysis, and evaluating the response time performance of the automobile curved screen according to the output result of the response time detection algorithm. The present invention can realize accurate detection of the response time of an automobile curved screen.

Description

Technical Field

[0001] The present invention relates to the technical field of automobile screen detection, and in particular to a device and method for detecting the response time of an automobile curved screen. Background Art

[0002] With the continuous development of automobile technology, car screens have gradually evolved from traditional flat screens to curved screens, providing drivers with a more immersive visual experience. However, the introduction of curved screens also brings new challenges, especially the problem of screen response time. Response time refers to the time it takes for the pixels of an LCD screen to switch from one state to another. Faster response time helps avoid blur, smearing or interference in dynamic images. Fast and accurate screen response can significantly improve the driver's operating experience and help reduce potential safety risks caused by screen delays.

[0003] At present, the methods for detecting the response time of automotive curved screens mainly rely on manual measurement or the use of external equipment. The manual measurement method evaluates the response time of the screen by manually operating the screen and recording the time difference from operation to response. However, this method is easily affected by the subjective factors of the tester, resulting in great uncertainty in the accuracy and reliability of the test results. In addition, the manual measurement process is time-consuming and labor-intensive, and is not suitable for large-scale applications and long-term monitoring.

[0004] On the other hand, although using external devices (such as high-speed cameras or sensors) to measure screen response time improves the accuracy of the test to a certain extent, this method requires additional hardware investment, increasing costs and complexity. At the same time, the external device method may require professional equipment and cannot be easily deployed in the existing test environment, limiting its promotion in practical applications.

[0005] In view of the defects of the prior art, it is necessary to develop a more accurate, efficient and cost-effective device and method to accurately detect the response time of a car's curved screen. Summary of the invention

[0006] One purpose of the present invention is to provide a device for detecting the response time of a car curved screen, so as to achieve accurate detection of the response time of the car curved screen.

[0007] Another object of the present invention is to provide a method for detecting the response time of a curved screen of an automobile, using a device for detecting the response time of a curved screen of an automobile as described above.

[0008] To achieve this object, the present invention adopts the following technical solutions:

[0009] A device for detecting the response time of a curved screen of an automobile, comprising:

[0010] A two-dimensional plane image acquisition module is used to capture the image of the car's curved screen to obtain a two-dimensional plane image;

[0011] The 3D robotic arm module is connected to the control module and the 2D plane image acquisition module, and is used to obtain the 3D position information of the target icon in the 2D plane image and the operation information of the control module, and move to the front of the car curved screen to perform corresponding operations;

[0012] The image acquisition module is connected to the control module and is used to capture continuous images of changes in different areas of the curved screen when the robotic arm module operates on the curved screen of the car from different directions according to the operation information of the control module.

[0013] The image processing and analysis module is used to receive the two-dimensional plane image acquired by the two-dimensional plane image acquisition module, and execute the affine transformation algorithm to obtain the 3D position information of the car curved screen; and receive and analyze multiple groups of continuous images acquired by the image acquisition module, and execute the response method detection algorithm to obtain the response method detection result;

[0014] The control module is used to receive the user's operation instructions and send corresponding operation information to the 3D robot arm module and the analysis and image acquisition module;

[0015] The output module is used to output the identified response method detection results for evaluating the response method performance of the curved screen.

[0016] Preferably, the two-dimensional plane image acquisition module includes a 2D camera and a 2D camera control unit;

[0017] The 2D camera is used to capture the image of the curved screen to obtain a two-dimensional plane image;

[0018] The 2D camera control unit is used to adjust the shooting parameters of the 2D camera.

[0019] Preferably, the 3D robotic arm module comprises a 3D camera, a robotic arm and a robotic arm control unit;

[0020] The 3D camera is used to obtain the three-dimensional position information of the target icon on the current car curved screen;

[0021] The mechanical arm is used to move in front of the car curved screen according to the three-dimensional position information;

[0022] The robotic arm control unit is used to accurately control the movement of the robotic arm.

[0023] Preferably, the analysis image acquisition module includes a plurality of high frame rate cameras;

[0024] The multiple high frame rate cameras are respectively located at different positions in front of the car curved screen, and the multiple high frame rate cameras respectively shoot different areas of the car curved screen. The multiple high frame rate cameras are used to shoot continuous images of changes in different areas of the curved screen from different directions when the robotic arm module operates on the car curved screen.

[0025] A method for detecting the response time of a curved screen of an automobile, using the device for detecting the response time of a curved screen of an automobile as described above, comprises the following steps:

[0026] S1, using a 2D camera to obtain a two-dimensional plane image of the car's curved screen;

[0027] S2, by optimizing and affine transforming the pixels on the two-dimensional plane image, mapping the positions of the pixels on the 2D image to the three-dimensional space of the 3D camera, and obtaining the corresponding three-dimensional position information of the curved screen;

[0028] S3, determining the three-dimensional position information of the target icon on the car surface screen through the 3D camera, and converting it into the navigation coordinates of the robot arm;

[0029] S4, the robotic arm accurately clicks the target icon on the car's curved screen according to the navigation coordinates, and synchronously starts high-frame rate cameras set in different directions to obtain continuous images and time information of changes in different areas of the curved screen when the robotic arm operates on the car's curved screen;

[0030] S5, stitching the images in the same frame in the multiple sets of continuous images to form a continuous stitching image of the screen changes when the robot arm clicks on the entire car curved screen;

[0031] S6. Input the continuous spliced ​​images and time information into a response time detection algorithm for analysis, and evaluate the response time performance of the car curved screen according to the output result of the response time detection algorithm.

[0032] Preferably, in S2, the optimizing and affine transforming of the pixels on the two-dimensional plane image comprises the following steps:

[0033] S21, accurately positioning pixels on the two-dimensional plane image using an edge detection method or a corner detection method;

[0034] S22, calibrate the 2D camera in stereo, and use the least squares method to obtain the internal parameter covariance matrix of the 3D camera and the external parameter covariance matrix

[0035] S23. Using uncertainty propagation to calculate the internal parameter covariance matrix and the external parameter covariance matrix Evaluate the reprojection error between the final 2D camera and the 3D camera:

[0036]

[0037] Among them, P u Represents the coordinates of a two-dimensional plane image point, usually expressed as P u = (u, v), is the point projected onto the 2D image plane by the 3D camera, or written as a column vector Represents the pixel coordinates on a two-dimensional plane image;

[0038] K represents the camera's intrinsic parameter matrix, which describes the camera's focal length, principal point position, and possible pixel ratio. Usually, the intrinsic parameter matrix is ​​a 3*3 matrix, expressed as:

[0039]

[0040] R represents the rotation matrix of the camera coordinate system relative to the world coordinate system;

[0041] T represents the translation matrix of the camera coordinate system relative to the world coordinate system;

[0042] X w Represents a point in a three-dimensional world coordinate system, usually represented as a column vector:

[0043]

[0044] i means that the reprojection error of the i-th point is being calculated;

[0045] S24, when the reprojection error meets the preset threshold, the pixel coordinate system of the pixel position on the 2D camera is converted into the image coordinate system, and the coordinate of the pixel position in the pixel coordinate system is set to (P u ,P v ), the coordinates of the corresponding pixel in the image coordinate system are (x, y), then the relationship is:

[0046]

[0047] Among them, dx, dy represent the actual physical size of each pixel in the pixel coordinate system, in mm; u0, v0 represent the translation distance of each pixel in the pixel coordinate system;

[0048] By translating the pixel coordinate system, the pixel coordinate system can be converted to the image coordinate system. The calculation formula is:

[0049]

[0050] S25, convert the image coordinate system into the 3D camera coordinate system, according to the triangle similarity principle, let the three-dimensional coordinates be (X c ,Y c ,Z c ), the focal length of the 3D camera is f, then the relationship is:

[0051]

[0052] Convert the above relationship into:

[0053]

[0054] Then convert the pixel coordinate system to the coordinate system of the 3D image. The conversion formula is:

[0055]

[0056] in is the intrinsic parameter matrix of the 3D camera, set to K;

[0057] S26, convert the 3D camera coordinate system into the world coordinate system (X w ,Y w ,Z w ), the conversion formula is:

[0058]

[0059] Among them, R is the rotation matrix and T is the translation matrix.

[0060] Preferably, in S4, the robotic arm accurately clicks the target icon on the car curved screen according to the navigation coordinates, so that the control system performs control algorithms and motion planning on the robotic arm, specifically including the following steps:

[0061] S41, the control algorithm is controlled by a PID controller, specifically:

[0062] The output u(t) of the PID controller is a weighted sum based on the error e(t) = r(t) - y(t);

[0063] Where r(t) is the set point and y(t) is the output;

[0064] First, calculate the error e(t) = r(t) - y(t);

[0065] Substituting into the mathematical form of the PID controller:

[0066]

[0067] in,

[0068] u(t): control signal (controller output);

[0069] e(t): Error, the difference between the set point and the actual output;

[0070] K p : Proportional gain, which determines the immediate response to the error;

[0071] K i : Integral gain, which determines the effect of error accumulation over time;

[0072] K d : Differential gain, which determines the response of the error change rate;

[0073] The integral term takes into account the accumulation of all past errors and is mainly used to eliminate steady-state errors;

[0074] The derivative term predicts future error changes and helps reduce overshoot;

[0075] Finally, u(t) is obtained and applied to the control object;

[0076] S42. Motion planning includes path planning and trajectory planning: path planning is to calculate the optimal path of the robot arm from the starting point to the target point in 3D space; trajectory planning is to define the speed and acceleration of the robot arm during the movement.

[0077] Preferably, in S6, the continuous spliced ​​images and time information are input into a response time detection algorithm for analysis, which specifically includes the following steps:

[0078] S61. Use the frame difference method to detect the difference of a certain pixel in two consecutive stitched images: define two consecutive stitched images as I1(x, y) and I2(x, y), then the frame difference D(x, y) is expressed as:

[0079] D(x,y)=|I1(x,y)-I2(x,y)|

[0080] S62, setting a pixel difference threshold to distinguish and define the change area and change situation of two frames of continuous spliced ​​images:

[0081] If the frame difference D(x, y) is greater than the pixel difference threshold, the pixel is considered to have changed in the two consecutive stitched images, and the adjacent pixels that are greater than the set pixel difference threshold are merged and defined as the changed area;

[0082] Otherwise, the pixel is considered to have not changed in two consecutive stitched images;

[0083] S63. Retrieve time information, including the high-frame camera startup time image_start_time_, the robot arm click time robot_click_time_, and the shooting time of two consecutive stitched images, and calculate the time by the frame rate:

[0084] Time=((diff_frame_index+1)*perframetime).

[0085] The frame rate is calculated as: perframetime = 1 / frame interval time

[0086] The frame interval time is calculated by the shooting time of two consecutive stitched images: frame interval time = current frame time - previous frame timestamp, then the response time delay_time = Time - (robot_click_time_ - image_start_time_).

[0087] One of the above technical solutions has the following beneficial effects:

[0088] 1. Improve detection accuracy: By combining 2D and 3D imaging technologies, accurate positioning and click operations on target icons on curved screens can be achieved, significantly improving the accuracy of response time detection.

[0089] 2. Enhanced detection efficiency: The use of high frame rate cameras can capture subtle changes in screen response. Combined with an efficient response time detection algorithm, it can quickly and accurately evaluate the screen response time, thereby improving detection efficiency.

[0090] 3. Broaden the scope of application: It is not only suitable for flat screens, but can also effectively detect the response time of curved screens, greatly broadening the application scope of detection technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 It is a physical schematic diagram of a device for detecting the response time of a curved screen of an automobile according to the present invention;

[0092] Figure 2 It is a schematic diagram of time information parameters of a method for detecting the response time of a curved screen of an automobile according to the present invention. DETAILED DESCRIPTION

[0093] The technical solution of the present invention is further described below with reference to the accompanying drawings and through specific implementation methods.

[0094] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0095] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0096] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0097] A device for detecting the response time of a curved screen of an automobile, comprising:

[0098] A two-dimensional plane image acquisition module is used to capture the image of the car's curved screen to obtain a two-dimensional plane image;

[0099] The 3D robotic arm module is connected to the control module and the 2D plane image acquisition module, and is used to obtain the 3D position information of the target icon in the 2D plane image and the operation information of the control module, and move to the front of the car curved screen to perform corresponding operations;

[0100] The image acquisition module is connected to the control module and is used to capture continuous images of changes in different areas of the curved screen when the robotic arm module operates on the curved screen of the car from different directions according to the operation information of the control module.

[0101] The image processing and analysis module is used to receive the two-dimensional plane image acquired by the two-dimensional plane image acquisition module, and execute the affine transformation algorithm to obtain the 3D position information of the car curved screen; and receive and analyze multiple groups of continuous images acquired by the image acquisition module, and execute the response method detection algorithm to obtain the response method detection result;

[0102] The control module is used to receive the user's operation instructions and send corresponding operation information to the 3D robot arm module and the analysis and image acquisition module;

[0103] The output module is used to output the identified response method detection results for evaluating the response method performance of the curved screen.

[0104] Preferably, the two-dimensional plane image acquisition module includes a 2D camera and a 2D camera control unit;

[0105] The 2D camera is used to capture the image of the curved screen to obtain a two-dimensional plane image;

[0106] The 2D camera control unit is used to adjust the shooting parameters of the 2D camera.

[0107] Preferably, the 3D robotic arm module comprises a 3D camera, a robotic arm and a robotic arm control unit;

[0108] The 3D camera is used to obtain the three-dimensional position information of the target icon on the current car curved screen;

[0109] The mechanical arm is used to move to the front of the car curved screen according to the three-dimensional position information;

[0110] The robotic arm control unit is used to accurately control the movement of the robotic arm.

[0111] Preferably, the analysis image acquisition module includes a plurality of high frame rate cameras;

[0112] The multiple high frame rate cameras are respectively located at different positions in front of the car curved screen, and the multiple high frame rate cameras respectively shoot different areas of the car curved screen. The multiple high frame rate cameras are used to shoot continuous images of changes in different areas of the curved screen from different directions when the robotic arm module operates on the car curved screen.

[0113] The working principle of the device of the present invention is to achieve accurate detection of the response time of the car curved screen by integrating multiple sensors and the coordinated work of the mechanical arm. The specific steps are as follows:

[0114] Initialization and calibration: Before starting the inspection, the 2D camera, 3D camera and three high frame rate cameras are calibrated to ensure that they can accurately capture the image information of the curved screen and are consistent with the coordinate system of the robotic arm. The robotic arm also needs to be initialized to ensure that it can move accurately along the preset path.

[0115] Image capture and 3D mapping: 2D cameras capture 2D images of curved screens, providing basic data for subsequent image processing and affine transformation. 3D cameras use stereo vision technology or structured light technology to obtain 3D spatial information of curved screens. This information will be used to build a 3D model of the curved screen and provide navigation for subsequent precise operation of the robotic arm.

[0116] Robotic arm operation and image acquisition: The control system commands the robotic arm to move to a specific position on the curved screen (such as in front of the target icon) according to the preset test script. The robotic arm performs a click operation to simulate the action of a user touching the screen.

[0117] Among them, the robot arm has kinematic basis and posture control;

[0118] The kinematics includes forward kinematics and inverse kinematics: forward kinematics refers to calculating the position and posture of the end effector in 3D space given the angles of each joint, by using a homogeneous transformation matrix to describe the relationship between joints; inverse kinematics refers to calculating the required angle of each joint based on the target position and posture;

[0119] The posture control is to accurately control the position and posture of the end effector according to the 3D position information, by: expressing the target position and posture as a 4x4 posture matrix for calculation; using linear interpolation or spline interpolation to plan the motion trajectory.

[0120] At the same time as the robotic arm clicks, three high-frame rate cameras start synchronously to capture continuous images of screen changes at a high frame rate, covering different areas of the screen and capturing more subtle movements and changes.

[0121] Image processing and response time analysis: The captured continuous images are sent to the image processing module of the control system for preprocessing and feature extraction. Subsequently, these images are input into the response time detection algorithm, which analyzes parameters such as the response speed of the screen during the robot arm click and the smoothness of the screen switching. Finally, a response time evaluation result is output to evaluate the response time performance of the curved screen.

[0122] In summary, the device of the present invention has the following beneficial effects:

[0123] 1. Accuracy: By using the position information obtained by 2D high-definition cameras and 3D cameras, combined with the precise click operation of the robot arm, and the high-speed acquisition of screen images by multiple high-frame cameras, the accurate detection of the car screen response time is achieved. In addition, the multiple data collection and analysis obtained ensure the accuracy of the test results.

[0124] 2. Automation: Utilize automated equipment such as robotic arms and high-frame cameras to automatically detect the response time of the car screen, avoiding the subjectivity and uncertainty in traditional manual measurement methods while greatly improving the efficiency of the test.

[0125] 3. Parallelism: Since the shooting ends of the three high-frame rate cameras are facing the curved screen, their imaging of the curved screen is similar. If there are multiple curved screens, the relative position relationship of each curved screen in the three high-frame rate cameras is the same. The images of the car's curved screen are collected in parallel by three high-frame cameras. Each high-frame camera can identify which area in the image the specific curved screen corresponds to, and combined with the parallel clicking operation of the robotic arm, the response time of different curved screens corresponding to different areas is analyzed. The response time detection of multiple curved screens is realized at the same time, which improves the parallelism and efficiency of the test.

[0126] 4. Practicality: The equipment and technologies used are relatively mature and practical, such as 2D and 3D cameras, robotic arms, etc., which can be easily integrated into existing automobile production lines or testing systems, improving the practicality and operability of the technology.

[0127] In summary, the technical effects of the device of the present invention include improving the accuracy, automation and parallelism of the test, making the detection of the response time of the car screen more convenient, efficient and reliable.

[0128] A method for detecting the response time of a curved screen of an automobile, using the device for detecting the response time of a curved screen of an automobile as described above, comprises the following steps:

[0129] S1, using a 2D camera to obtain a two-dimensional plane image of the car's curved screen;

[0130] S2, by optimizing and affine transforming the pixels on the two-dimensional plane image, mapping the positions of the pixels on the 2D image to the three-dimensional space of the 3D camera, and obtaining the corresponding three-dimensional position information of the curved screen;

[0131] S3, determining the three-dimensional position information of the target icon on the car surface screen through the 3D camera, and converting it into the navigation coordinates of the robot arm;

[0132] S4, the robotic arm accurately clicks the target icon on the car's curved screen according to the navigation coordinates, and synchronously starts high-frame rate cameras set in different directions to obtain continuous images and time information of changes in different areas of the curved screen when the robotic arm operates on the car's curved screen;

[0133] S5, stitching the images in the same frame in the multiple sets of continuous images to form a continuous stitching image of the screen changes when the robot arm clicks on the entire car curved screen;

[0134] S6. Input the continuous spliced ​​images and time information into a response time detection algorithm for analysis, and evaluate the response time performance of the car curved screen according to the output result of the response time detection algorithm.

[0135] The method of the present invention aims to accurately detect the response time of the car's curved screen. Its working principle deeply integrates two-dimensional and three-dimensional imaging technology, precise operation of the robotic arm, and high-speed image analysis technology. The specific process is described as follows:

[0136] S1 is mainly used for 2D image acquisition: using a high-precision 2D camera to shoot the curved screen of the car to be tested and obtain its accurate 2D image. This step lays a solid foundation for subsequent 3D mapping and position information extraction.

[0137] S2 is mainly for image optimization and 3D mapping: it optimizes 2D images at the pixel level, including removing noise and enhancing contrast, to improve image quality. Then, using advanced mathematical methods such as affine transformation, the optimized 2D image pixel position is accurately mapped to the 3D space coordinate system of the 3D camera, thereby obtaining the 3D position information of the curved screen.

[0138] S3 is mainly used for 3D position information extraction and navigation coordinate conversion:

[0139] With the help of a 3D camera, the three-dimensional position information of the target icon (such as buttons, icons, etc.) on the car's curved screen is accurately located. Subsequently, through complex algorithm processing, these three-dimensional position information is converted into navigation coordinates that can be recognized by the robotic arm, ensuring that the robotic arm can accurately click on the target icon.

[0140] S4 is mainly used for robot arm operation and high-speed image capture: the robot arm accurately clicks the target icon on the car's curved screen according to the navigation coordinates. At the same time, high-frame rate cameras set in different directions are synchronously started to capture continuous images of screen changes when the robot arm clicks at ultra-high speed, and record the time information of each frame of the image. The high-frame rate camera can capture subtle changes in the screen response and provide detailed data support for subsequent response time analysis.

[0141] S5 mainly obtains continuous stitching images of screen changes when the robotic arm clicks on the entire car curved screen, so as to improve the analysis efficiency for the next step of response time detection and analysis.

[0142] S6 is mainly for response time detection and analysis: the continuous spliced ​​images and their corresponding time information are input into the response time detection algorithm. The algorithm can analyze the time difference of screen changes in the image, and combine the time information to accurately calculate the response time of the screen. Finally, based on the output results of the algorithm, the response time performance of the car's curved screen is objectively and accurately evaluated.

[0143] In summary, the method of the present invention has the following beneficial effects:

[0144] 1. Improve detection accuracy: By combining 2D and 3D imaging technologies, accurate positioning and click operations on target icons on curved screens can be achieved, significantly improving the accuracy of response time detection.

[0145] 2. Enhanced detection efficiency: The use of high frame rate cameras can capture subtle changes in screen response. Combined with an efficient response time detection algorithm, it can quickly and accurately evaluate the screen response time, thereby improving detection efficiency.

[0146] 3. Broaden the scope of application: It is not only suitable for flat screens, but can also effectively detect the response time of curved screens, greatly broadening the application scope of detection technology.

[0147] In summary, this method not only improves the detection accuracy and efficiency, but also broadens the scope of application of the detection technology, which is of great significance to promoting the advancement of automotive display technology.

[0148] To further illustrate, in S2, the optimization processing and affine transformation of the pixels on the two-dimensional plane image include the following steps:

[0149] S21, accurately locating pixels on the two-dimensional plane image using an edge detection method or a corner detection method;

[0150] Since there are errors in the pixel positions, the above optimization steps can reduce the uncertainty of the pixel positions.

[0151] S22, calibrate the 2D camera in stereo, and use the least squares method to obtain the internal parameter covariance matrix of the 3D camera and the external parameter covariance matrix

[0152] In the 3D reconstruction process, uncertainty comes from many aspects, such as inaccurate camera calibration, measurement errors, ambient lighting changes, etc. When using the least squares method to obtain the intrinsic covariance matrix of the 3D camera and the external parameter covariance matrix The uncertainty propagation method can be reused to estimate the reprojection error between the final 2D camera and the 3D camera.

[0153] S23. Using uncertainty propagation to calculate the internal parameter covariance matrix and the external parameter covariance matrix Evaluate the reprojection error between the final 2D camera and the 3D camera:

[0154]

[0155] Among them, P u Represents the coordinates of a two-dimensional plane image point, usually expressed as P u = (u, v), is the point projected onto the 2D image plane by the 3D camera, or written as a column vector Represents the pixel coordinates on a two-dimensional plane image;

[0156] K represents the camera's intrinsic parameter matrix, which describes the camera's focal length, principal point position, and possible pixel ratio. Usually, the intrinsic parameter matrix is ​​a 3*3 matrix, expressed as:

[0157]

[0158] R represents the rotation matrix of the camera coordinate system relative to the world coordinate system;

[0159] T represents the translation matrix of the camera coordinate system relative to the world coordinate system;

[0160] X w Represents a point in a three-dimensional world coordinate system, usually represented as a column vector:

[0161]

[0162] i means that the reprojection error of the i-th point is being calculated;

[0163] In the camera model, the reprojection error is an important metric for evaluating the difference between a 2D point and its projected 3D point in the image. When the reprojection error does not meet the preset threshold, using a robust estimation method such as RANSAC can effectively exclude outliers from the data, thereby improving the accuracy of the affine transformation.

[0164] S24, when the reprojection error meets the preset threshold, the pixel coordinate system of the pixel position on the 2D camera is converted into the image coordinate system, and the coordinate of the pixel position in the pixel coordinate system is set to (P u ,P v ), the coordinates of the corresponding pixel in the image coordinate system are (x, y), then the relationship is:

[0165]

[0166] Among them, dx, dy represent the actual physical size of each pixel in the pixel coordinate system, in mm; u0, v0 represent the translation distance of each pixel in the pixel coordinate system;

[0167] By translating the pixel coordinate system, the pixel coordinate system can be converted to the image coordinate system. The calculation formula is:

[0168]

[0169] S25, convert the image coordinate system into the 3D camera coordinate system, according to the triangle similarity principle, let the three-dimensional coordinates be (X c ,Y c ,Z c ), the focal length of the 3D camera is f, then the relationship is:

[0170]

[0171] Convert the above relationship into:

[0172]

[0173] Then convert the pixel coordinate system to the coordinate system of the 3D image. The conversion formula is:

[0174]

[0175] in is the intrinsic parameter matrix of the 3D camera, set to K;

[0176] S26, convert the 3D camera coordinate system into the world coordinate system (X w ,Y w ,Z w ), the conversion formula is:

[0177]

[0178] Among them, R is the rotation matrix and T is the translation matrix.

[0179] It should be noted that the optimization process may also include using filters to process dynamic scenes and adopting data fusion methods:

[0180] In dynamic scenes, using techniques such as Kalman filtering and particle filtering can help us track objects in time series and deal with noise and errors in time series data. These filters can continuously update state estimates based on new observations, thereby improving overall accuracy.

[0181] In multiple viewpoints or multi-sensor systems, data fusion can effectively reduce uncertainty. For example, combining lidar and camera data can obtain a more accurate 3D point cloud, and sensor fusion technology (such as extended Kalman filter, EKF) can be used to combine the measurement information of different sensors.

[0182] Further explanation, in S4, the robotic arm accurately clicks the target icon on the car surface screen according to the navigation coordinates, and the control system performs control algorithm and motion planning on the robotic arm, which specifically includes the following steps:

[0183] S41, the control algorithm is controlled by a PID controller, specifically:

[0184] The output u(t) of the PID controller is a weighted sum based on the error e(t) = r(t) - y(t);

[0185] Where r(t) is the set point and y(t) is the output;

[0186] First, calculate the error e(t) = r(t) - y(t);

[0187] Substituting into the mathematical form of the PID controller:

[0188]

[0189] in,

[0190] u(t): control signal (controller output);

[0191] e(t): Error, the difference between the set point and the actual output;

[0192] K p : Proportional gain, which determines the immediate response to the error;

[0193] K i : Integral gain, which determines the effect of error accumulation over time;

[0194] K d : Differential gain, which determines the response of the error change rate;

[0195] The integral term takes into account the accumulation of all past errors and is mainly used to eliminate steady-state errors;

[0196] The derivative term predicts future error changes and helps reduce overshoot;

[0197] Finally, u(t) is obtained and applied to the control object;

[0198] Furthermore, robust control can be adopted: external disturbances and changes in system parameters are taken into account to ensure accurate control under uncertainty.

[0199] S42. Motion planning includes path planning and trajectory planning: path planning is to calculate the optimal path of the robot arm from the starting point to the target point in 3D space to avoid obstacles; trajectory planning is to define the speed and acceleration of the robot arm during the movement so that the movement is smooth and meets physical limitations.

[0200] In addition, the robot arm can also add a sensor system to provide real-time feedback and environmental perception, mainly including position sensors and force and tactile sensors. Position sensors include encoders: installed on each joint, real-time feedback of the angular position of the joint; laser rangefinders or depth cameras: used to detect the precise position of the end effector in 3D space, especially in dynamic environments. Force and tactile sensors include force sensors: monitor the force applied by the end effector to ensure proper contact and prevent damage during operation; tactile sensors: provide tactile feedback to help the robot arm be more precise when handling flexible objects or complex surfaces.

[0201] Furthermore, the control system also has repeatability, which refers to the ability of the robot to reach the same position when repeating the same action multiple times. It can be evaluated by calculating the standard deviation of the end effector position through repeated experiments. The use of high-quality servo motors and feedback systems can ensure high repeatability. The current accuracy of the robot and subsequent optimization strategies are measured based on the estimated deviation results.

[0202] To further illustrate, in S6, the continuous splicing images and time information are input into the response time detection algorithm for analysis, which specifically includes the following steps:

[0203] S51. Use the frame difference method to detect the difference of a certain pixel in two consecutive stitched images: define two consecutive stitched images as I1(x, y) and I2(x, y), then the frame difference D(x, y) is expressed as:

[0204] D(x,y)=|I1(x,y)-I2(x,y)|

[0205] By calculating the pixel differences between adjacent frames, changes in the scene can be identified;

[0206] S52, setting a pixel difference threshold to distinguish and define the change area and change situation of two frames of continuous spliced ​​images:

[0207] If the frame difference D(x, y) is greater than the pixel difference threshold, the pixel is considered to have changed in the two consecutive stitched images, and the adjacent pixels that are greater than the set pixel difference threshold are merged and defined as the changed area;

[0208] Otherwise, the pixel is considered to have not changed in two consecutive stitched images;

[0209] S53, retrieve time information, including the high-frame camera startup time image_start_time_, the robot arm click time robot_click_time_, and the shooting time of two consecutive stitched images, and calculate the time by the frame rate:

[0210] Time=((diff_frame_index+1)*perframetime).

[0211] The frame rate is calculated as: perframetime = 1 / frame interval time

[0212] The frame interval time is calculated by the shooting time of two consecutive stitched images: frame interval time = current frame time - previous frame timestamp, then the response time delay_time = Time - (robot_click_time_ - image_start_time_).

[0213] It should be noted that the pixel difference threshold value is determined by comparing multiple scenes or test sets, and gradually adjusting the threshold until the best balance between distinguishing changes and keeping unchanged is found.

[0214] The technical principle of the present invention is described above in conjunction with specific embodiments. These descriptions are only for explaining the principle of the present invention and cannot be interpreted as limiting the scope of protection of the present invention in any way. Based on the explanations herein, those skilled in the art can associate other specific embodiments of the present invention without creative work, and these equivalent variations or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A device for detecting the response time of a car curved screen, characterized in that: include: A two-dimensional plane image acquisition module is used to capture the image of the car's curved screen to obtain a two-dimensional plane image; The 3D robotic arm module is connected to the control module and the 2D plane image acquisition module, and is used to obtain the 3D position information of the target icon in the 2D plane image and the operation information of the control module, and move to the front of the car curved screen to perform corresponding operations; The image acquisition module is connected to the control module and is used to capture continuous images of changes in different areas of the curved screen when the robotic arm module operates on the curved screen of the car from different directions according to the operation information of the control module. An image processing and analysis module is used to receive the two-dimensional plane image acquired by the two-dimensional plane image acquisition module, and execute an affine transformation algorithm to obtain the 3D position information of the car curved screen; And, receiving and analyzing multiple groups of continuous images collected by the image collection module, and executing the response method detection algorithm to obtain the response method detection result; The control module is used to receive the user's operation instructions and send corresponding operation information to the 3D robot arm module and the analysis and image acquisition module; The output module is used to output the identified response method detection results for evaluating the response method performance of the curved screen.

2. The device for detecting the response time of a curved screen of an automobile according to claim 1, characterized in that: The two-dimensional plane image acquisition module includes a 2D camera and a 2D camera control unit; The 2D camera is used to capture the image of the curved screen to obtain a two-dimensional plane image; The 2D camera control unit is used to adjust the shooting parameters of the 2D camera.

3. The device for detecting the response time of a curved screen of an automobile according to claim 1, characterized in that: The 3D robotic arm module includes a 3D camera, a robotic arm and a robotic arm control unit; The 3D camera is used to obtain the three-dimensional position information of the target icon on the current car curved screen; The mechanical arm is used to move to the front of the car curved screen according to the three-dimensional position information; The robotic arm control unit is used to accurately control the movement of the robotic arm.

4. The device for detecting the response time of a curved screen of an automobile according to claim 1, characterized in that: The analysis image acquisition module includes a plurality of high frame rate cameras; The multiple high frame rate cameras are respectively located at different positions in front of the car curved screen, and the multiple high frame rate cameras respectively shoot different areas of the car curved screen. The multiple high frame rate cameras are used to shoot continuous images of changes in different areas of the curved screen from different directions when the robotic arm module operates on the car curved screen.

5. A method for detecting the response time of a car curved screen, characterized in that: The device for detecting the response time of a curved screen of an automobile as claimed in any one of claims 1 to 4 comprises the following steps: S1, using a 2D camera to obtain a two-dimensional plane image of the car's curved screen; S2, by optimizing and affine transforming the pixels on the two-dimensional plane image, mapping the positions of the pixels on the 2D image to the three-dimensional space of the 3D camera, and obtaining the corresponding three-dimensional position information of the curved screen; S3, determining the three-dimensional position information of the target icon on the car surface screen through the 3D camera, and converting it into the navigation coordinates of the robot arm; S4, the robotic arm accurately clicks the target icon on the car's curved screen according to the navigation coordinates, and synchronously starts high-frame rate cameras set in different directions to obtain continuous images and time information of changes in different areas of the curved screen when the robotic arm operates on the car's curved screen; S5, stitching the images in the same frame in the multiple sets of continuous images to form a continuous stitching image of the screen changes when the robot arm clicks on the entire car curved screen; S6. Input the continuous spliced ​​images and time information into a response time detection algorithm for analysis, and evaluate the response time performance of the car curved screen according to the output result of the response time detection algorithm.

6. The method for detecting the response time of a curved screen of an automobile according to claim 5, characterized in that: In S2, the optimization processing and affine transformation of the pixels on the two-dimensional plane image include the following steps: S21, accurately locating pixels on the two-dimensional plane image using an edge detection method or a corner detection method; S22, calibrate the 2D camera in stereo, and use the least squares method to obtain the internal parameter covariance matrix of the 3D camera and the external parameter covariance matrix S23. Using uncertainty propagation to calculate the internal parameter covariance matrix and the external parameter covariance matrix Evaluate the reprojection error between the final 2D camera and the 3D camera: Among them, P u Represents the coordinates of a two-dimensional plane image point, usually expressed as P u = (u, v), is the point projected onto the 2D image plane by the 3D camera, or written as a column vector Represents the pixel coordinates on a two-dimensional plane image; K represents the camera's intrinsic parameter matrix, which describes the camera's focal length, principal point position, and possible pixel ratio. Usually, the intrinsic parameter matrix is ​​a 3*3 matrix, expressed as: R represents the rotation matrix of the camera coordinate system relative to the world coordinate system; T represents the translation matrix of the camera coordinate system relative to the world coordinate system; X w Represents a point in a three-dimensional world coordinate system, usually represented as a column vector: i means that the reprojection error of the i-th point is being calculated; S24, when the reprojection error meets the preset threshold, the pixel coordinate system of the pixel position on the 2D camera is converted into the image coordinate system, and the coordinate of the pixel position in the pixel coordinate system is set to (P u ,P v ), the coordinates of the corresponding pixel in the image coordinate system are (x, y), then the relationship is: Among them, dx, dy represent the actual physical size of each pixel in the pixel coordinate system, in mm; u0, v0 represent the translation distance of each pixel in the pixel coordinate system; By translating the pixel coordinate system, the pixel coordinate system can be converted to the image coordinate system. The calculation formula is: S25, convert the image coordinate system into the 3D camera coordinate system, according to the triangle similarity principle, let the three-dimensional coordinates be (X c ,Y c ,Z c ), the focal length of the 3D camera is f, then the relationship is: Convert the above relationship into: Then convert the pixel coordinate system to the coordinate system of the 3D image. The conversion formula is: in is the intrinsic parameter matrix of the 3D camera, set to K; S26, convert the 3D camera coordinate system into the world coordinate system (X w ,Y w ,Z w ), the conversion formula is: Among them, R is the rotation matrix and T is the translation matrix.

7. The method for detecting the response time of a curved screen of an automobile according to claim 5, characterized in that: In S4, the robotic arm accurately clicks the target icon on the car surface screen according to the navigation coordinates, and the control system performs control algorithms and motion planning on the robotic arm, which specifically includes the following steps: S41, the control algorithm is controlled by a PID controller, specifically: The output u(t) of the PID controller is a weighted sum based on the error e(t) = r(t) - y(t); Where r(t) is the set point and y(t) is the output; First, calculate the error e(t) = r(t) - y(t); Substituting into the mathematical form of the PID controller: in, u(t): control signal (controller output); e(t): Error, the difference between the set point and the actual output; K p : Proportional gain, which determines the immediate response to the error; K i : Integral gain, which determines the effect of error accumulation over time; K d : Differential gain, which determines the response of the error change rate; The integral term takes into account the accumulation of all past errors and is mainly used to eliminate steady-state errors; The derivative term predicts future error changes and helps reduce overshoot; Finally, u(t) is obtained and applied to the control object; S42. Motion planning includes path planning and trajectory planning: path planning is to calculate the optimal path of the robot arm from the starting point to the target point in 3D space; trajectory planning is to define the speed and acceleration of the robot arm during the movement.

8. The method for detecting the response time of a curved screen of an automobile according to claim 5, characterized in that: In S6, the continuous spliced ​​images and time information are input into a response time detection algorithm for analysis, which specifically includes the following steps: S61. Use the frame difference method to detect the difference of a certain pixel in two consecutive stitched images: define two consecutive stitched images as I1(x, y) and I2(x, y), then the frame difference D(x, y) is expressed as: D(x,y)=|I1(x,y)-I2(x,y)| S62, setting a pixel difference threshold to distinguish and define the change area and change situation of two frames of continuous spliced ​​images: If the frame difference D(x, y) is greater than the pixel difference threshold, the pixel is considered to have changed in the two consecutive stitched images, and the adjacent pixels that are greater than the set pixel difference threshold are merged and defined as the changed area; Otherwise, the pixel is considered to have not changed in two consecutive stitched images; S63. Retrieve time information, including the high-frame camera startup time image_start_time_, the robot arm click time robot_click_time_, and the shooting time of two consecutive stitched images, and calculate the time by the frame rate: Time=((diff_frame_index+1)*perframetime). The frame rate is calculated as: perframetime = 1 / frame interval time The frame interval time is calculated by the shooting time of two consecutive stitched images: frame interval time = current frame time - previous frame timestamp, then the response time delay_time = Time - (robot_click_time_ - image_start_time_).

Images