A vehicle door assembly error compensation method and device based on multi-source data fusion

Abstract

The present invention discloses a vehicle door assembly error compensation method and device based on multi-source data fusion, relating to the field of vehicle assembly technology. The method is used to compensate for errors during vehicle door assembly. A corresponding laser radar, image acquisition device, and inertial measurement unit are deployed in the vehicle door assembly area. The method comprises: during vehicle door assembly, acquiring vehicle door assembly geometry data, vehicle door assembly image data, and vehicle door assembly motion data in real time, performing spatial unification and temporal synchronization, and fusing these data to obtain semantically complementary target geometry data, target image data, and target motion data; identifying geometric errors, position errors, and motion errors during vehicle door assembly, and determining corresponding compensation parameters based on the geometric errors, position errors, and motion errors to compensate for errors during vehicle door assembly. The present invention addresses the problem of inaccurate door assembly compensation in the prior art, resulting in poor door assembly accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle assembly, and in particular to a vehicle door assembly error compensation method and device based on multi-source data fusion. Background Art

[0002] In recent years, the new energy vehicle industry has flourished, driving the demand for increasingly precise, stable, and intelligent body assembly. This is particularly true in the assembly of electric vehicle doors, where the quality of assembly and adjustment directly impacts the safety, noise control, and service life of the vehicle. To meet these demands for high-precision assembly, the industry has gradually introduced multi-degree-of-freedom hinge structures, intelligent drive systems, sensor technology, and closed-loop control feedback mechanisms. These intelligent drive systems utilize a combination of servo motors and hydraulic cylinders, using a robotic arm to assemble the doors. The system actively controls the assembly position and uses real-time feedback from sensors during the door assembly process to provide control compensation, ensuring accurate door assembly.

[0003] However, most of the current methods rely on data collected by a single sensor to accurately control the door assembly process. In fact, a single sensor has certain limitations. For example, the data is relatively one-sided and unreliable, resulting in inaccurate compensation and ultimately poor door assembly accuracy. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a vehicle door assembly error compensation method and device based on multi-source data fusion, aiming to solve the problem in the prior art that vehicle door assembly compensation is not accurate enough resulting in poor vehicle door assembly accuracy.

[0005] In one aspect, the present invention provides a door assembly error compensation method based on multi-source data fusion, which is used to perform error compensation during door assembly. Corresponding laser radars, image acquisition devices, and inertial measurement units are deployed in the door assembly area. The method includes:

[0006] When assembling the door, the door assembly geometry data, door assembly image data, and door assembly motion data collected by the laser radar, image acquisition device, and inertial measurement unit are respectively acquired in real time;

[0007] The door assembly geometry data, door assembly image data and door assembly motion data are spatially unified and temporally synchronized, and then fused according to preset rules to obtain semantically complementary target geometry data, target image data and target motion data;

[0008] According to the target geometric data, target image data and target motion data, the geometric error, position error and motion error during the assembly of the vehicle door are respectively identified, and the corresponding compensation parameters are determined according to the geometric error, position error and motion error to compensate for the error during the assembly of the vehicle door.

[0009] Furthermore, in the above-mentioned door assembly error compensation method based on multi-source data fusion, the step of spatially unifying and temporally synchronizing the door assembly geometric data, the door assembly image data, and the door assembly motion data comprises:

[0010] The coordinate transformation relationship between the laser radar, image acquisition device, and inertial measurement unit is established through external parameter calibration, and the door assembly geometry data, door assembly image data, and door assembly motion data are spatially unified using the coordinate transformation relationship;

[0011] By aligning the timestamps of the laser radar, image acquisition device, and inertial measurement unit with synchronous clocks, the time synchronization of the door assembly geometry data, door assembly image data, and door assembly motion data is achieved.

[0012] Furthermore, in the above-mentioned door assembly error compensation method based on multi-source data fusion, the step of fusing according to preset rules to obtain semantically complementary target geometric data, target image data, and target motion data includes:

[0013] A vibration model is constructed using the door assembly motion data, the door assembly geometry data is reversely corrected, and the door assembly geometry data is given texture information using the door assembly image data to obtain the target geometry data;

[0014] Performing transformation compensation on the door assembly image data based on the door assembly motion data, extracting the background area to obtain the target image data;

[0015] The door assembly geometry data, door assembly image data and door assembly motion data are input into the extended Kalman filter, and the door assembly motion data drift is corrected by constraint conditions to obtain the target motion data.

[0016] Furthermore, in the above-mentioned door assembly error compensation method based on multi-source data fusion, the step of respectively identifying the geometric error, position error and motion error during door assembly based on the target geometric data, target image data and target motion data includes:

[0017] Extracting the three-dimensional point cloud data of the car door from the target geometric data, and matching the three-dimensional point cloud data of the car door with the point cloud data of the standard car door model to obtain the geometric error including translation error and rotation error;

[0018] Extract the pixel coordinates of the assembly mark from the target image data, convert them into world coordinates using the calibration matrix, and compare the world coordinates with the theoretical coordinates of the assembly mark in the standard door model. Calculate the pixel-level offset and convert it into physical dimensions to obtain the position error.

[0019] The motion parameters contained in the target motion data are compared with a preset motion parameter threshold to obtain a motion error.

[0020] Furthermore, in the above-mentioned door assembly error compensation method based on multi-source data fusion, the step of performing transformation compensation on the door assembly image data based on the door assembly motion data and extracting the background area to obtain the target image data includes:

[0021] Based on the door assembly motion data, an external parameter model of the image acquisition device is established to predict the real-time position and orientation of the image acquisition device in the world coordinate system;

[0022] In the initial state of the door assembly robot arm, a static background image is captured. When the robot arm moves, the static background image is affine transformed using the pose transformation obtained by the predicted real-time position and orientation of the image acquisition device to generate a theoretical background image at the current moment.

[0023] Perform a differential operation on the door assembly image data and the theoretical background image, and extract the area with significant grayscale difference as a dynamic foreground mask;

[0024] The target image data is obtained by extracting the background area from the door assembly motion data according to the dynamic foreground mask.

[0025] Furthermore, in the above-mentioned door assembly error compensation method based on multi-source data fusion, the expression of the dynamic foreground mask is:

[0026] ;

[0027] in, M= 1 is the foreground area in the dynamic foreground mask, M =0 is the background area in the dynamic foreground mask, Assemble image data for the car door, This is the theoretical background image.

[0028] Furthermore, in the above-mentioned door assembly error compensation method based on multi-source data fusion, the step of determining corresponding compensation parameters according to geometric error, position error and motion error to compensate for errors during door assembly includes:

[0029] Use historical geometric errors, position errors, motion errors and corresponding compensation parameters to train the DNN model to obtain an error compensation model;

[0030] The geometric error, position error and motion error are input into the trained error compensation model to obtain the corresponding compensation parameters.

[0031] Another object of the present invention is to provide a vehicle door assembly error compensation device based on multi-source data fusion, which is used to perform error compensation during vehicle door assembly. A corresponding laser radar, an image acquisition device, and an inertial measurement unit are deployed in the vehicle door assembly area. The device includes:

[0032] An acquisition module is used to acquire, in real time, vehicle door assembly geometric data, vehicle door assembly image data, and vehicle door assembly motion data collected by the laser radar, the image acquisition device, and the inertial measurement unit during vehicle door assembly;

[0033] A complementary module is used to spatially unify and temporally synchronize the door assembly geometry data, door assembly image data, and door assembly motion data, and then fuse them according to preset rules to obtain semantically complementary target geometry data, target image data, and target motion data;

[0034] The compensation module is used to identify the geometric error, position error and motion error during door assembly based on the target geometric data, target image data and target motion data, and determine corresponding compensation parameters based on the geometric error, position error and motion error to compensate for the error during door assembly.

[0035] Another object of the present invention is to provide a readable storage medium having a computer program stored thereon, wherein the program implements the steps of the above method when executed by a processor.

[0036] Another object of the present invention is to provide an electronic device comprising a memory, a processor and a computer program stored in the memory and running on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0037] This invention deploys a laser radar, an image acquisition device, and an inertial measurement unit to collect different data, and then integrates these data to semantically complement the collected data, integrating the multi-source data into a unified representation that is consistent in time and space and semantically complementary. For example, image texture features are used to supplement the lidar semantic information, and dynamic data from the inertial measurement unit is used to correct for motion noise in the lidar data. This reduces the inherent errors of a single sensor and improves data integrity and accuracy. This solves the problem of poor door assembly accuracy caused by inaccurate door assembly compensation in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Flowchart of a vehicle door assembly error compensation method based on multi-source data fusion in a first embodiment of the present invention;

[0039] Figure 2 4 is a structural block diagram of a vehicle door assembly error compensation device based on multi-source data fusion in the third embodiment of the present invention.

[0040] The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0041] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The drawings illustrate several embodiments of the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present invention.

[0042] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be an intermediate element. When an element is referred to as being "connected to" another element, it may be directly connected to the other element or there may be an intermediate element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0044] Example 1

[0045] See also Figure 1 , shown is a vehicle door assembly error compensation method based on multi-source data fusion in the first embodiment of the present invention, which is used to perform error compensation during vehicle door assembly. Corresponding laser radars, image acquisition devices, and inertial measurement units are deployed in the vehicle door assembly area. The method includes steps S10 to S12.

[0046] Step S10 , when assembling the vehicle door, obtaining the vehicle door assembly geometric data, vehicle door assembly image data and vehicle door assembly motion data collected by the laser radar, the image acquisition device and the inertial measurement unit in real time respectively.

[0047] During the door assembly process, corresponding laser radars, image acquisition devices (such as industrial cameras), and inertial measurement units are deployed in the door assembly area. The laser radars can be positioned around the door, with multiple sensors configured to collect 3D point cloud data. The image acquisition devices can be positioned in the area directly facing the door to collect image data, enabling identification of relevant elements, such as assembly markers on the door. The inertial measurement unit can be installed on the robotic arm that grips the door assembly to collect dynamic data during door assembly.

[0048] In step S11 , the door assembly geometry data, the door assembly image data and the door assembly motion data are spatially unified and temporally synchronized, and then fused according to preset rules to obtain semantically complementary target geometry data, target image data and target motion data.

[0049] Among them, in order to achieve more comprehensive and accurate monitoring and analysis of the vehicle door assembly process, it is necessary to integrate and process the different types of data involved in the vehicle door assembly process. Specifically, the three types of data, namely, vehicle door assembly geometry data, vehicle door assembly image data, and vehicle door assembly motion data, must first be unified in the spatial dimension and synchronized in the temporal dimension. Then, they are fused according to pre-set rules to obtain target geometry data, target image data, and target motion data after semantic complementarity, providing multimodal input for subsequent error recognition.

[0050] Specifically, a coordinate transformation relationship between the laser radar, image acquisition device, and inertial measurement unit is established through external parameter calibration, and the door assembly geometry data, door assembly image data, and door assembly motion data are spatially unified using the coordinate transformation relationship;

[0051] By aligning the timestamps of the laser radar, image acquisition device, and inertial measurement unit with synchronous clocks, the time synchronization of the door assembly geometry data, door assembly image data, and door assembly motion data is achieved.

[0052] For example, through specific calibration algorithms and calibration tools, such as calibration plates and calibration objects, the relative position and rotation relationship between the coordinate systems of each device is determined, and then the coordinate transformation matrix is ​​obtained. Using this coordinate transformation relationship, the door assembly geometry data collected by the lidar, the door assembly image data collected by the image acquisition device, and the door assembly motion data collected by the inertial measurement unit can be uniformly converted to the same coordinate system to achieve spatial unification, so that the position information of each door component in the data of different devices can be accurately corresponded, laying a spatial foundation for subsequent data fusion analysis. At the same time, it is also necessary to align the timestamps of the lidar, image acquisition device and inertial measurement unit through synchronized clocks. The synchronized clock uses the Network Time Protocol (NTP) or hardware synchronization signals to ensure that the time base of each device is consistent, and the timestamps of data collected by different devices are adjusted to the same time scale. In this way, the time synchronization of the door assembly geometry data, door assembly image data and door assembly motion data can be achieved, ensuring that different types of data collected at the same time can be accurately associated. For example, at a certain precise moment, the door geometric position information recorded by the lidar, the door appearance image taken by the image acquisition device and the door motion state recorded by the inertial measurement unit can correspond to each other, thereby providing an accurate time basis for subsequent time series-based data analysis and fusion. Ultimately, these spatially unified and time-synchronized data can be comprehensively utilized to more comprehensively and deeply analyze the door assembly process.

[0053] In step S12, geometric errors, position errors, and motion errors during door assembly are identified based on the target geometric data, target image data, and target motion data, and corresponding compensation parameters are determined based on the geometric errors, position errors, and motion errors to compensate for errors during door assembly.

[0054] The error compensation model is obtained by training the DNN model using historical geometric errors, position errors, motion errors and corresponding compensation parameters;

[0055] The geometric error, position error and motion error are input into the trained error compensation model to obtain the corresponding compensation parameters.

[0056] Specifically, the collected target geometry data, target image data, and target motion data are used to identify geometric errors, position errors, and motion errors during door assembly, respectively. The three-dimensional point cloud data of the door is extracted from the target geometry data and matched with the point cloud data of a standard door model to obtain geometric errors, including translation and rotation errors. The pixel coordinates of the assembly mark are extracted from the target image data and converted to world coordinates using a calibration matrix. The world coordinates are then compared with the theoretical coordinates of the assembly mark in the standard door model, and the pixel-level offset is calculated and converted to physical dimensions to obtain the position error. The motion parameters contained in the target motion data are compared with preset motion parameter thresholds to obtain the motion error. In specific implementation, a deep neural network (DNN) model is trained using historically collected geometric error, position error, and motion error data, along with corresponding compensation parameters. After repeated iterations to optimize the model parameters, an error compensation model is ultimately obtained that accurately maps the relationship between error and compensation parameters. After geometric errors, position errors, and motion errors are identified during the actual door assembly process, these error data are input into a trained error compensation model. The model will output corresponding compensation parameters based on the rules learned during training, thereby achieving error compensation in the door assembly process and effectively improving the accuracy and quality of door assembly.

[0057] In summary, the vehicle door assembly error compensation method based on multi-source data fusion in the above-mentioned embodiments of the present invention collects different data through the deployment of a lidar, an image acquisition device, and an inertial measurement unit. This data is then integrated into a unified representation that is temporally and spatially consistent and semantically complementary. For example, image texture features are used to supplement lidar semantic information, and dynamic data from the inertial measurement unit is used to correct motion noise in lidar data. This reduces the inherent error of a single sensor and improves data integrity and accuracy. This method solves the problem of inaccurate door assembly compensation in the prior art, which leads to poor door assembly accuracy.

[0058] Example 2

[0059] This embodiment also proposes a door assembly error compensation method based on multi-source data fusion. The difference between the door assembly error compensation method based on multi-source data fusion in this embodiment and the door assembly error compensation method based on multi-source data fusion in the first embodiment is that:

[0060] The step of fusing according to preset rules to obtain semantically complementary target geometric data, target image data, and target motion data comprises:

[0061] A vibration model is constructed using the door assembly motion data, the door assembly geometry data is reversely corrected, and the door assembly geometry data is given texture information using the door assembly image data to obtain the target geometry data;

[0062] Performing transformation compensation on the door assembly image data based on the door assembly motion data, extracting the background area to obtain the target image data;

[0063] The door assembly geometry data, door assembly image data and door assembly motion data are input into the extended Kalman filter, and the door assembly motion data drift is corrected by constraint conditions to obtain the target motion data.

[0064] Different strategies are used to optimize the door assembly geometry, image, and motion data to achieve semantic complementarity. Specifically, after acquiring the door assembly motion data, a vibration model is first constructed using this data. By analyzing the vibration characteristics and motion patterns during the door assembly process, the door assembly geometry data is reversely inferred and corrected to compensate for deviations in the geometry caused by dynamic changes during the assembly process. Simultaneously, using the door assembly image data, surface texture, color, and other information contained in the image are mapped onto the geometry data, allowing the geometry data to include not only shape and size information but also appearance details, thereby obtaining the target geometry data. For the door assembly image data, based on the motion state reflected by the assembly motion data, the image is subjected to transformations such as translation, rotation, and scaling to eliminate image distortion caused by assembly motion. Then, using techniques such as image segmentation, background regions are extracted to remove background interference and highlight the effective image information related to the door assembly, thus obtaining the target image data. Finally, the door assembly geometry data, image data and motion data are input into the extended Kalman filter algorithm together. The spatial position, shape and other information provided by the geometry data and image data are used as constraints to correct the drift phenomenon generated during the acquisition and processing of the door assembly motion data, making the motion data more accurate and reliable. Finally, the target motion data is obtained, completing the fusion optimization and semantic complementarity of the three types of data.

[0065] Exemplarily, the step of performing transformation compensation on the door assembly image data based on the door assembly motion data and extracting the background area to obtain the target image data includes:

[0066] Based on the door assembly motion data, an external parameter model of the image acquisition device is established to predict the real-time position and orientation of the image acquisition device in the world coordinate system;

[0067] In the initial state of the door assembly robot arm, a static background image is captured. When the robot arm moves, the static background image is affine transformed using the pose transformation obtained by the predicted real-time position and orientation of the image acquisition device to generate a theoretical background image at the current moment.

[0068] Perform a differential operation on the door assembly image data and the theoretical background image, and extract the area with significant grayscale difference as a dynamic foreground mask;

[0069] The target image data is obtained by extracting the background area from the door assembly motion data according to the dynamic foreground mask.

[0070] Specifically, an extrinsic parameter model for the image acquisition device is constructed based on the door assembly motion data, combined with image acquisition principles and geometric relationships. This model predicts the real-time position and orientation of the image acquisition device in the world coordinate system based on the motion data, thereby determining the dynamic pose information of the image acquisition device during the assembly process. Next, in the initial state of the door assembly robot, before it moves, a static background image is captured to serve as a reference for subsequent processing. When the robot begins to move, an affine transformation is applied to the static background image using the pose transformation relationship determined by the previously predicted real-time position and orientation of the image acquisition device. Through operations such as translation, rotation, and scaling, the static background image is transformed to simulate the state that the background should present under dynamic changes at the current moment, thereby generating a theoretical background image at the current moment. Subsequently, a difference operation is performed between the door assembly image data and the generated theoretical background image, comparing the grayscale values ​​of each pixel in the two images to identify areas with significant grayscale differences. These areas are then extracted as dynamic foreground masks, which identify the foreground portions of the image that have changed. Finally, based on the dynamic foreground mask, the foreground area is accurately eliminated from the door assembly image data, and only the background area is retained. This obtains the target image data that removes foreground interference and contains only background information, providing a purer and more accurate data basis for subsequent image data-based analysis and processing.

[0071] More specifically, the expression of the dynamic foreground mask is:

[0072] ;

[0073] in, M= 1 is the foreground area in the dynamic foreground mask, M =0 is the background area in the dynamic foreground mask, Assemble image data for the car door, This is the theoretical background image.

[0074] The dynamic foreground mask is essentially a binary mask image of the same size as the door assembly image data. It uses specific rules to divide the image region into foreground and background. In this expression, when the value M of a pixel in the mask image is 1, it indicates that the pixel is located in the foreground region identified by the dynamic foreground mask. This region is subject to significant changes during the door assembly process and differs significantly from the background. When M = 0, the pixel is in the background region defined by the dynamic foreground mask, which is relatively stable and does not undergo significant changes during the assembly process. By performing a certain operation (in practical applications, grayscale difference calculations are often used to calculate the difference in grayscale values ​​between the corresponding pixels in the two images), if the difference exceeds a preset threshold, the corresponding pixel is assigned a value of 1 in the dynamic foreground mask, indicating that it is in the foreground region. Pixels with a difference less than the threshold are assigned a value of 0 and are classified as background. Therefore, the dynamic foreground mask can clearly outline the boundary between the foreground and background in the image, providing an accurate selection basis for the subsequent precise extraction of the background area from the door assembly image data and the acquisition of the target image data.

[0075] In summary, the vehicle door assembly error compensation method based on multi-source data fusion in the above-mentioned embodiments of the present invention collects different data through the deployment of a lidar, an image acquisition device, and an inertial measurement unit. This data is then integrated into a unified representation that is temporally and spatially consistent and semantically complementary. For example, image texture features are used to supplement lidar semantic information, and dynamic data from the inertial measurement unit is used to correct motion noise in lidar data. This reduces the inherent error of a single sensor and improves data integrity and accuracy. This method solves the problem of inaccurate door assembly compensation in the prior art, which leads to poor door assembly accuracy.

[0076] Example 3

[0077] See also Figure 2 , shown is a vehicle door assembly error compensation device based on multi-source data fusion proposed in the third embodiment of the present invention, which is used to perform error compensation during vehicle door assembly. Corresponding laser radars, image acquisition devices, and inertial measurement units are deployed in the vehicle door assembly area. The device includes:

[0078] An acquisition module 100 is used to acquire, in real time, vehicle door assembly geometric data, vehicle door assembly image data, and vehicle door assembly motion data collected by a laser radar, an image acquisition device, and an inertial measurement unit during vehicle door assembly;

[0079] The complementation module 200 is used to spatially unify and temporally synchronize the door assembly geometry data, the door assembly image data, and the door assembly motion data, and then fuse them according to preset rules to obtain semantically complementary target geometry data, target image data, and target motion data;

[0080] The compensation module 300 is used to identify the geometric error, position error and motion error during the assembly of the vehicle door based on the target geometric data, target image data and target motion data, and determine corresponding compensation parameters based on the geometric error, position error and motion error to compensate for the error during the assembly of the vehicle door.

[0081] The functions or operation steps implemented when the above modules are executed are substantially the same as those in the above method embodiments and will not be repeated here.

[0082] Example 4

[0083] Another aspect of the present invention further provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the above-mentioned embodiments 1 to 2.

[0084] Example 5

[0085] On the other hand, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the program, the steps of the method described in any one of the above-mentioned embodiments 1 to 2 are implemented.

[0086] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0087] Those skilled in the art will appreciate that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device), or for use in conjunction with such instruction execution system, apparatus, or device. For purposes of this specification, a "computer-readable storage medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, apparatus, or device, or in conjunction with such instruction execution system, apparatus, or device.

[0088] More specific examples (a non-exhaustive list) of computer-readable storage media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable storage medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0089] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the aforementioned embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following technologies known in the art may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0090] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0091] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A door assembly error compensation method based on multi-source data fusion, characterized in that: For error compensation during door assembly, corresponding laser radars, image acquisition devices, and inertial measurement units are deployed in the door assembly area. The method includes: When assembling the door, the door assembly geometry data, door assembly image data, and door assembly motion data collected by the laser radar, image acquisition device, and inertial measurement unit are respectively acquired in real time; The door assembly geometry data, door assembly image data and door assembly motion data are spatially unified and temporally synchronized, and then fused according to preset rules to obtain semantically complementary target geometry data, target image data and target motion data; According to the target geometric data, target image data and target motion data, the geometric error, position error and motion error during the assembly of the vehicle door are respectively identified, and the corresponding compensation parameters are determined according to the geometric error, position error and motion error to compensate for the error during the assembly of the vehicle door.

2. The door assembly error compensation method based on multi-source data fusion according to claim 1 is characterized in that: The step of spatially unifying and temporally synchronizing the door assembly geometric data, the door assembly image data, and the door assembly motion data comprises: The coordinate transformation relationship between the laser radar, image acquisition device, and inertial measurement unit is established through external parameter calibration, and the door assembly geometry data, door assembly image data, and door assembly motion data are spatially unified using the coordinate transformation relationship; By aligning the timestamps of the laser radar, image acquisition device, and inertial measurement unit with synchronous clocks, the time synchronization of the door assembly geometry data, door assembly image data, and door assembly motion data is achieved.

3. The door assembly error compensation method based on multi-source data fusion according to claim 2 is characterized in that: The step of fusing according to preset rules to obtain semantically complementary target geometric data, target image data, and target motion data comprises: A vibration model is constructed using the door assembly motion data, the door assembly geometry data is reversely corrected, and the door assembly geometry data is given texture information using the door assembly image data to obtain the target geometry data; Performing transformation compensation on the door assembly image data based on the door assembly motion data, extracting the background area to obtain the target image data; The door assembly geometry data, door assembly image data and door assembly motion data are input into the extended Kalman filter, and the door assembly motion data drift is corrected by constraint conditions to obtain the target motion data.

4. The door assembly error compensation method based on multi-source data fusion according to claim 3 is characterized in that: The step of respectively identifying the geometric error, position error and motion error during door assembly based on the target geometric data, the target image data and the target motion data comprises: Extracting the three-dimensional point cloud data of the car door from the target geometric data, and matching the three-dimensional point cloud data of the car door with the point cloud data of the standard car door model to obtain the geometric error including translation error and rotation error; Extract the pixel coordinates of the assembly mark from the target image data, convert them into world coordinates using the calibration matrix, and compare the world coordinates with the theoretical coordinates of the assembly mark in the standard door model. Calculate the pixel-level offset and convert it into physical dimensions to obtain the position error. The motion parameters contained in the target motion data are compared with a preset motion parameter threshold to obtain a motion error.

5. The door assembly error compensation method based on multi-source data fusion according to claim 4 is characterized in that: The step of performing transformation compensation on the door assembly image data based on the door assembly motion data and extracting the background area to obtain the target image data comprises: Based on the door assembly motion data, an external parameter model of the image acquisition device is established to predict the real-time position and orientation of the image acquisition device in the world coordinate system; In the initial state of the door assembly robot arm, a static background image is captured. When the robot arm moves, the static background image is affine transformed using the pose transformation obtained by the predicted real-time position and orientation of the image acquisition device to generate a theoretical background image at the current moment. Perform a differential operation on the door assembly image data and the theoretical background image, and extract the area with significant grayscale difference as a dynamic foreground mask; The target image data is obtained by extracting the background area from the door assembly motion data according to the dynamic foreground mask.

6. The door assembly error compensation method based on multi-source data fusion according to claim 5 is characterized in that: The expression of the dynamic foreground mask is: ； in, M= 1 is the foreground area in the dynamic foreground mask, M =0 is the background area in the dynamic foreground mask, Assemble image data for the car door, This is the theoretical background image.

7. The door assembly error compensation method based on multi-source data fusion according to claim 1, characterized in that: The step of determining corresponding compensation parameters according to the geometric error, position error and motion error to compensate for the error during door assembly includes: Use historical geometric errors, position errors, motion errors and corresponding compensation parameters to train the DNN model to obtain an error compensation model; The geometric error, position error and motion error are input into the trained error compensation model to obtain the corresponding compensation parameters.

8. A door assembly error compensation device based on multi-source data fusion, characterized in that: For error compensation during door assembly, corresponding laser radar, image acquisition device and inertial measurement unit are arranged in the door assembly area, and the device includes: An acquisition module is used to acquire, in real time, vehicle door assembly geometric data, vehicle door assembly image data, and vehicle door assembly motion data collected by the laser radar, the image acquisition device, and the inertial measurement unit during vehicle door assembly; A complementary module is used to spatially unify and temporally synchronize the door assembly geometry data, door assembly image data, and door assembly motion data, and then fuse them according to preset rules to obtain semantically complementary target geometry data, target image data, and target motion data; The compensation module is used to identify the geometric error, position error and motion error during door assembly based on the target geometric data, target image data and target motion data, and determine corresponding compensation parameters based on the geometric error, position error and motion error to compensate for the error during door assembly.

9. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the steps of the method according to any one of claims 1 to 7 are implemented when the processor executes the program.

Images