Vehicle door assembly error compensation method and device based on multi-source data fusion

Abstract

The invention discloses a vehicle door assembly error compensation method and device based on multi-source data fusion, relates to the technical field of vehicle assembly, is used for performing error compensation during vehicle door assembly, and is characterized in that a corresponding laser radar, an image acquisition device and an inertial measurement unit are arranged in a vehicle door assembly area. Vehicle door assembly geometric data, vehicle door assembly image data and vehicle door assembly motion data are obtained in real time, then space unification and time synchronization are carried out, and fusion is carried out to obtain target geometric data, target image data and target motion data after semantic complementation; the geometric error, the position error and the motion error are identified when the vehicle door is assembled, and corresponding compensation parameters are determined according to the geometric error, the position error and the motion error so as to carry out error compensation when the vehicle door is assembled. The problem that in the prior art, vehicle door assembly compensation is not accurate enough, and consequently the vehicle door assembly accuracy is poor is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle assembly, and particularly relates to a method and device for compensating door assembly errors based on multi-source data fusion. Background Art

[0002] In recent years, the new energy vehicle industry has developed vigorously, and the requirements for the accuracy, stability and intelligent level of body assembly are getting higher and higher. Especially in the electric door assembly process, the assembly and adjustment quality directly affects the safety, noise control and service life of the whole vehicle. In order to meet the requirements of high-precision assembly, multi-degree-of-freedom hinge structures, intelligent drive systems, sensor technologies and closed-loop control feedback mechanisms have been gradually introduced in the industry. The intelligent drive system combines a servo motor and a hydraulic cylinder, uses a robotic arm to assemble the door, and actively controls the assembly position, and performs control compensation based on the real-time feedback data of the sensor during the door assembly process, so as to ensure the accuracy of the door assembly.

[0003] However, at present, most of the accuracy control of the door assembly process is based on the data collected by a single sensor. In fact, a single sensor has certain limitations, such as one-sided data and insufficient reliability, resulting in inaccurate compensation and ultimately poor accuracy of the door assembly. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method and device for compensating door assembly errors based on multi-source data fusion, aiming to solve the problem of poor accuracy of door assembly caused by inaccurate door assembly compensation in the prior art.

[0005] On the one hand, the present invention proposes a method for compensating door assembly errors based on multi-source data fusion, which is used for error compensation during door assembly. A corresponding lidar, image acquisition device and inertial measurement unit are arranged in the door assembly area. The method includes: During door assembly, respectively and real-time obtain the door assembly geometric data, door assembly image data and door assembly motion data collected by the lidar, image acquisition device and inertial measurement unit; Perform spatial unification and time synchronization on the door assembly geometric data, door assembly image data and door assembly motion data, and then fuse them according to a preset rule to obtain target geometric data, target image data and target motion data after semantic complementation; Identify geometric errors, position errors and motion errors during door assembly according to the target geometric data, target image data and target motion data, and determine corresponding compensation parameters according to the geometric errors, position errors and motion errors to perform error compensation during door assembly.

[0006] Further, for the above-mentioned door assembly error compensation method based on multi-source data fusion, the step of spatially unifying and temporally synchronizing the door assembly geometric data, door assembly image data, and door assembly motion data includes: Establish the coordinate transformation relationship between the lidar, image acquisition device, and inertial measurement unit through external parameter calibration, and use the coordinate transformation relationship to spatially unify the door assembly geometric data, door assembly image data, and door assembly motion data; Align the timestamps of the lidar, image acquisition device, and inertial measurement unit through a synchronous clock to achieve the temporal synchronization of the door assembly geometric data, door assembly image data, and door assembly motion data.

[0007] Further, for the above-mentioned door assembly error compensation method based on multi-source data fusion, the step of fusing according to preset rules to obtain the target geometric data, target image data, and target motion data after semantic complementarity includes; Construct a vibration model using the door assembly motion data, inversely correct the door assembly geometric data, and endow the door assembly geometric data with texture information using the door assembly image data to obtain the target geometric data; Perform transformation compensation on the door assembly image data based on the door assembly motion data, and extract the background area to obtain the target image data; Input the door assembly geometric data, door assembly image data, and door assembly motion data into the extended Kalman filter, and correct the drift of the door assembly motion data through constraint conditions to obtain the target motion data.

[0008] Further, for the above-mentioned door assembly error compensation method based on multi-source data fusion, the step of respectively identifying the geometric error, position error, and motion error during door assembly according to the target geometric data, target image data, and target motion data includes: Extract the three-dimensional point cloud data of the door from the target geometric data, and match the three-dimensional point cloud data of the door with the point cloud data of the standard door model to obtain the geometric error including translation error and rotation error; Extract the pixel coordinates of the assembly mark from the target image data, convert them to world coordinates through the calibration matrix, and compare the world coordinates with the theoretical coordinates of the assembly mark in the standard door model, calculate the pixel-level offset and convert it to physical dimensions to obtain the position error; Compare the motion parameters included in the target motion data with the preset motion parameter threshold to obtain the motion error.

[0009] Further, for the above-mentioned door assembly error compensation method based on multi-source data fusion, wherein the step of performing transformation compensation on the door assembly image data based on the door assembly motion data and extracting the background region to obtain the target image data includes: According to the door assembly motion data, establish an external parameter model of the image acquisition device, and predict the real-time position and orientation of the image acquisition device in the world coordinate system; In the initial state where the robotic arm for door assembly does not move, acquire a static background image, and when the robotic arm moves, perform an affine transformation on the static background image using the pose transformation obtained from the predicted real-time position and orientation of the image acquisition device to generate the theoretical background image at the current moment; Perform a difference operation between the door assembly image data and the theoretical background image, and extract the region with significant gray-scale difference as the dynamic foreground mask; Extract the background region from the door assembly motion data according to the dynamic foreground mask to obtain the target image data.

[0010] Further, for the above-mentioned door assembly error compensation method based on multi-source data fusion, wherein the expression of the dynamic foreground mask is: ; Wherein, M= 1 is the foreground region in the dynamic foreground mask, M =0 is the background region in the dynamic foreground mask, is the door assembly image data, is the theoretical background image.

[0011] Further, for the above-mentioned door assembly error compensation method based on multi-source data fusion, wherein the step of determining the corresponding compensation parameters according to the geometric error, position error, and motion error to perform error compensation during door assembly includes: Use the historical geometric error, position error, motion error, and corresponding compensation parameters to train a DNN model to obtain an error compensation model; Input the geometric error, position error, and motion error into the trained error compensation model to obtain the corresponding compensation parameters.

[0012] Another object of the present invention is to provide a door assembly error compensation device based on multi-source data fusion, which is used to perform error compensation during door assembly. Corresponding lidar, image acquisition device, and inertial measurement unit are arranged in the door assembly area. The device includes: An acquisition module, which is used to respectively and real-time acquire the door assembly geometric data, door assembly image data, and door assembly motion data collected by the lidar, image acquisition device, and inertial measurement unit during door assembly; A complementary module for spatially unifying and temporally synchronizing door assembly geometric data, door assembly image data, and door assembly motion data, and then fusing them according to preset rules to obtain target geometric data, target image data, and target motion data after semantic complementarity; A compensation module for respectively identifying geometric errors, position errors, and motion errors during door assembly based on the target geometric data, target image data, and target motion data, and determining corresponding compensation parameters according to the geometric errors, position errors, and motion errors to compensate for errors during door assembly.

[0013] Another object of the present invention is to provide a readable storage medium with a computer program stored thereon, and when the program is executed by a processor, the steps of the above method are implemented.

[0014] Another object of the present invention is to provide an electronic device, including a memory, a processor, and a computer program stored on the memory and running on the processor, and when the processor executes the program, the steps of the above method are implemented.

[0015] In the present invention, lidar, an image acquisition device, and an inertial measurement unit are arranged to collect different data, and these data are fused to perform semantic complementarity on the collected data, integrating multi-source data into a unified expression that is spatio-temporally consistent and semantically complementary. For example, using image texture features to supplement lidar semantic information, relying on the dynamic data of the inertial measurement unit to correct the motion noise of the data collected by lidar, etc., reducing the inherent errors of a single sensor and improving data integrity and accuracy. It solves the problem in the prior art that the door assembly compensation is not accurate enough, resulting in poor door assembly accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flowchart of the method for compensating door assembly errors based on multi-source data fusion in the first embodiment of the present invention; Figure 2 It is a structural block diagram of the device for compensating door assembly errors based on multi-source data fusion in the third embodiment of the present invention.

[0017] The following specific embodiments will further illustrate the present invention in conjunction with the above drawings. SPECIFIC EMBODIMENTS

[0018] To facilitate the understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided to make the disclosure of the present invention more thorough and comprehensive.

[0019] It should be noted that when an element is referred to as "fixedly provided on" another element, it can be directly on the other element or there may be an intermediate element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there may be an intermediate element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used in this article are only for the purpose of illustration.

[0020] Unless otherwise defined, all technical and scientific terms used in this article have the same meaning as commonly understood by those skilled in the technical field to which this invention belongs. The terms used in the description of the present invention in this article are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in this article includes any and all combinations of one or more of the related listed items.

[0021] Embodiment 1 Please refer to Figure 1 , which shows a method for compensating for door assembly errors based on multi-source data fusion in the first embodiment of the present invention, used for error compensation during door assembly. A corresponding lidar, image acquisition device, and inertial measurement unit are arranged in the door assembly area. The method includes steps S10 to S12.

[0022] Step S10, during door assembly, respectively and in real time obtain the door assembly geometric data, door assembly image data, and door assembly motion data collected by the lidar, image acquisition device, and inertial measurement unit.

[0023] Among them, during the door assembly process, a corresponding lidar, image acquisition device (such as an industrial camera), and inertial measurement unit are arranged in the door assembly area. The lidar can be set around the door facing it, and multiple are set to collect the three-dimensional point cloud data of the door. The image acquisition device can be set in the area facing the door to collect the image data of the door, so as to identify relevant elements, such as the assembly marking points on the door, while the inertial measurement unit can be set on the robotic arm clamping the door assembly to collect the dynamic data during door assembly.

[0024] Step S11, perform spatial unification and time synchronization on the door assembly geometric data, door assembly image data, and door assembly motion data, and then fuse them according to a preset rule to obtain target geometric data, target image data, and target motion data after semantic complementation.

[0025] Among them, to achieve a more comprehensive and accurate monitoring and analysis of the door assembly process, it is necessary to integrate and process different types of data involved in the door assembly process. Specifically, first, the three types of data, namely, door assembly geometric data, door assembly image data, and door assembly motion data, need to be unified in the spatial dimension and synchronized in the time dimension, and then they are fused according to pre-set rules to obtain the target geometric data, target image data, and target motion data after semantic complementarity, providing multi-modal input for subsequent error identification.

[0026] Specifically, through external parameter calibration, the coordinate transformation relationship between the lidar, image acquisition device, and inertial measurement unit is established, and the coordinate transformation relationship is used to spatially unify the door assembly geometric data, door assembly image data, and door assembly motion data. By synchronizing the timestamps of the lidar, image acquisition device, and inertial measurement unit with a synchronous clock, the time synchronization of the door assembly geometric data, door assembly image data, and door assembly motion data is achieved.

[0027] Exemplarily, through specific calibration algorithms and calibration tools, such as using calibration plates, calibration objects, etc., the relative positions and rotation relationships between the coordinate systems of each device are determined, and then the coordinate transformation matrix is obtained. Using this coordinate transformation relationship, the door assembly geometric data collected by the lidar, the door assembly image data collected by the image acquisition device, and the door assembly motion data collected by the inertial measurement unit can be uniformly transformed into the same coordinate system, achieving spatial unity, so that the position information of each door component in the data of different devices can be accurately corresponding, laying a spatial foundation for subsequent data fusion analysis. At the same time, it is also necessary to synchronize the timestamps of the lidar, image acquisition device, and inertial measurement unit through a synchronous clock. The synchronous clock uses methods such as the Network Time Protocol (NTP) or hardware synchronization signals to ensure that the time bases of each device are consistent, and adjusts the timestamps of the data collected by different devices to the same time scale, so as to achieve the time synchronization of the door assembly geometric data, door assembly image data, and door assembly motion data, ensuring that different types of data collected at the same moment can be accurately associated. For example, at a precise moment, the door geometric position information recorded by the lidar, the door appearance image captured by the image acquisition device, and the door motion state recorded by the inertial measurement unit can correspond to each other, thus providing an accurate time basis for subsequent time-series data analysis and fusion. Finally, these spatially unified and time-synchronized data can be comprehensively utilized to analyze the door assembly process more comprehensively and deeply.

[0028] Step S12: Identify the geometric error, position error, and motion error during door assembly based on the target geometric data, target image data, and target motion data respectively, and determine the corresponding compensation parameters according to the geometric error, position error, and motion error to compensate for the errors during door assembly.

[0029] Among them, use the historical geometric error, position error, motion error, and corresponding compensation parameters to train the DNN model to obtain an error compensation model. Input the geometric error, position error, and motion error into the trained error compensation model to obtain the corresponding compensation parameters.

[0030] Specifically, the collected target geometric data, target image data, and target motion data are respectively used to identify the geometric error, position error, and motion error during door assembly. Among them, extract the three-dimensional point cloud data of the door from the target geometric data, and match the three-dimensional point cloud data of the door with the point cloud data of the standard door model to obtain the geometric error including translation error and rotation error; extract the pixel coordinates of the assembly mark from the target image data, convert them into world coordinates through the calibration matrix, and compare the world coordinates with the theoretical coordinates of the assembly mark in the standard door model, calculate the pixel-level offset and convert it into a physical size to obtain the position error; compare the motion parameters included in the target motion data with the preset motion parameter threshold to obtain the motion error. In specific implementation, use the historical collected geometric error, position error, motion error data, and corresponding compensation parameters to train the deep neural network (DNN) model. After repeatedly iterating and optimizing the model parameters, finally obtain an error compensation model that can accurately map the relationship between errors and compensation parameters. When the geometric error, position error, and motion error are identified during the actual door assembly process, input these error data into the trained error compensation model, and the model will output the corresponding compensation parameters based on the rules learned during training, so as to achieve error compensation for the door assembly process and effectively improve the accuracy and quality of door assembly.

[0031] In summary, the door assembly error compensation method based on multi-source data fusion in the above embodiments of the present invention collects different data by arranging lidar, image acquisition devices, and inertial measurement units, and fuses these data to perform semantic complementarity on the collected data, integrating multi-source data into a unified expression that is consistent in time and space and semantically complementary. For example, use image texture features to supplement lidar semantic information, and rely on the dynamic data of the inertial measurement unit to correct the motion noise of the data collected by the lidar, etc., reducing the inherent error of a single sensor and improving data integrity and accuracy. It solves the problem that the door assembly compensation in the prior art is not accurate enough, resulting in poor door assembly accuracy.

[0032] Embodiment 2 This embodiment also proposes a method for compensating door assembly errors based on multi-source data fusion. The difference between the method for compensating door assembly errors based on multi-source data fusion in this embodiment and the method for compensating door assembly errors based on multi-source data fusion in Embodiment 1 is as follows: The step of performing fusion according to a preset rule to obtain the target geometric data, target image data, and target motion data after semantic complementation includes: Construct a vibration model using the door assembly motion data, perform inverse correction on the door assembly geometric data, and endow the door assembly geometric data with texture information using the door assembly image data to obtain the target geometric data; Perform transformation compensation on the door assembly image data based on the door assembly motion data, and extract the background region to obtain the target image data; Input the door assembly geometric data, door assembly image data, and door assembly motion data into the extended Kalman filter, and correct the drift of the door assembly motion data through constraint conditions to obtain the target motion data.

[0033] Among them, the geometric, image, and motion data of the door assembly are optimized through different strategies to achieve semantic complementation. Specifically, after obtaining the door assembly motion data, first use these data to construct a vibration model. By analyzing the vibration characteristics and motion laws during the door assembly process, reverse calculate and correct the door assembly geometric data to make up for the deviation of the geometric data caused by the dynamic changes during the assembly process; at the same time, with the help of the door assembly image data, map the surface texture, color, and other information in the image to the geometric data, so that the geometric data not only has shape and size information, but also has appearance details, thus obtaining the target geometric data. For the door assembly image data, according to the motion state reflected by the assembly motion data, perform transformation compensation on the image, such as translation, rotation, scaling, etc., to eliminate the image distortion caused by the assembly motion; then use image segmentation and other technologies to extract the background region, remove the background interference, and highlight the effective image information related to the door assembly to obtain the target image data. Finally, input the door assembly geometric data, image data, and motion data into the extended Kalman filter algorithm together, and use the spatial position, shape, and other information provided by the geometric data and image data as constraint conditions to correct the drift phenomenon generated during the acquisition and processing of the door assembly motion data, make the motion data more accurate and reliable, and finally obtain the target motion data, completing the fusion optimization and semantic complementation of the three types of data.

[0034] Exemplarily, the step of performing transformation compensation on the door assembly image data based on the door assembly motion data, and extracting the background region to obtain the target image data includes: According to the door assembly motion data, establish an external parameter model of the image acquisition device, and predict the real-time position and orientation of the image acquisition device in the world coordinate system; In the initial state where the robotic arm for door assembly does not move, a static background image is collected. When the robotic arm moves, an affine transformation is performed on the static background image using the pose transformation obtained from the predicted real-time position and orientation of the image acquisition device to generate the theoretical background image at the current moment. Perform a differential operation on the door assembly image data and the theoretical background image, and extract the regions with significant gray-scale differences as the dynamic foreground mask. Extract the background region from the door assembly motion data according to the dynamic foreground mask to obtain the target image data.

[0035] Specifically, based on the door assembly motion data, combined with the image acquisition principle and geometric relationships, an external parameter model of the image acquisition device is constructed. This model can predict the real-time position and orientation of the image acquisition device in the world coordinate system according to the motion data, thereby determining the dynamic pose information of the image acquisition device during the assembly process. Then, in the initial state where the robotic arm for door assembly has not moved, a static background image is collected as the basic reference for subsequent processing. When the robotic arm starts to move, an affine transformation is performed on the static background image using the pose transformation relationship determined by the previously predicted real-time position and orientation of the image acquisition device. Through operations such as translation, rotation, and scaling, the state that the background should present under dynamic changes at the current moment is simulated, thereby generating the theoretical background image at the current moment. Subsequently, a differential operation is performed on the door assembly image data and the generated theoretical background image, comparing the gray-scale values of each pixel point in the two images, finding the regions with significant gray-scale differences, and extracting these regions as the dynamic foreground mask. This mask identifies the foreground part that has changed in the image. Finally, according to the dynamic foreground mask, the foreground region is accurately removed from the door assembly image data, and only the background region is retained, thereby obtaining the target image data that removes foreground interference and only contains background information, providing a cleaner and more accurate data basis for subsequent analysis and processing based on the image data.

[0036] More specifically, the expression of the dynamic foreground mask is: ; where M= 1 is the foreground region in the dynamic foreground mask, M =0 is the background region in the dynamic foreground mask, is the door assembly image data, is the theoretical background image.

[0037] Among them, the dynamic foreground mask is essentially a binary mask image with the same size as the door assembly image data. The image area is divided into foreground and background by setting specific rules. In this expression, when the value of a certain pixel point in the mask image is M = 1, it means that the pixel point is located in the foreground area identified by the dynamic foreground mask, that is, this area changes during the door assembly process and has a significant difference from the background; when M = 0, it indicates that the pixel point is in the background area defined by the dynamic foreground mask, that is, the area that is relatively stable and does not change significantly during the assembly process. By performing a certain operation on the door assembly image data and the theoretical background image (in practical applications, gray value difference operation is often used to calculate the difference in gray values of the corresponding pixel points of the two), when the difference exceeds the preset threshold, the corresponding pixel point is assigned a value of 1 in the dynamic foreground mask and marked as the foreground; the pixel points with a difference less than the threshold are assigned a value of 0 and classified into the background area. Thus, the dynamic foreground mask can clearly outline the foreground and background boundaries in the image, providing an accurate selection basis for accurately extracting the background area from the door assembly image data and obtaining the target image data later.

[0038] In summary, the door assembly error compensation method based on multi-source data fusion in the above embodiments of the present invention collects different data by arranging lidar, image acquisition devices, and inertial measurement units, and fuses these data to perform semantic complementarity on the collected data, integrating multi-source data into a unified expression that is spatio-temporally consistent and semantically complementary. For example, using image texture features to supplement lidar semantic information, relying on the dynamic data of the inertial measurement unit to correct the motion noise of the data collected by the lidar, etc., reducing the inherent error of a single sensor and improving data integrity and accuracy. It solves the problem that the door assembly accuracy is poor due to inaccurate door assembly compensation in the prior art.

[0039] Embodiment III Please refer to Figure 2 , which shows the door assembly error compensation device based on multi-source data fusion proposed in the third embodiment of the present invention, used for error compensation during door assembly. A corresponding lidar, image acquisition device, and inertial measurement unit are arranged in the door assembly area. The device includes: An acquisition module 100, configured to respectively and real-time acquire the door assembly geometric data, door assembly image data, and door assembly motion data collected by the lidar, image acquisition device, and inertial measurement unit during door assembly; A complementary module 200, configured to perform spatial unification and time synchronization on the door assembly geometric data, door assembly image data, and door assembly motion data, and then fuse them according to preset rules to obtain target geometric data, target image data, and target motion data after semantic complementarity; The compensation module 300 is configured to respectively identify geometric errors, position errors, and motion errors during door assembly based on target geometric data, target image data, and target motion data, and determine corresponding compensation parameters according to the geometric errors, position errors, and motion errors to compensate for the errors during door assembly.

[0040] The functions or operation steps implemented when the above modules are executed are substantially the same as those in the above method embodiments, and will not be described in detail here.

[0041] Embodiment 4 On the other hand, the present invention also provides a readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in any one of the above Embodiments 1 to 2 are implemented.

[0042] Embodiment 5 On the other hand, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and running on the processor. When the processor executes the program, the steps of the method described in any one of the above Embodiments 1 to 2 are implemented.

[0043] The technical features of the above various embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0044] Those skilled in the art can understand that the logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable storage medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0045] More specific examples (nonexhaustive list) of computer-readable storage media include the following: electrical connection parts (electronic devices) with one or more wirings, portable computer disk cartridges (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber devices, and portable compact disc read-only memory (CDROM). Additionally, the computer-readable storage media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, then editing, interpreting, or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0046] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with suitable combinational logic gate circuits, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0047] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0048] The above-described embodiments merely represent several implementation manners of the present invention, and their descriptions are relatively specific and detailed, but should not be construed as limiting the scope of the patent of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention should be subject to the appended claims.

Claims

1. A method for compensating door assembly errors based on multi-source data fusion, characterized in that For error compensation during door assembly, corresponding lidar, image acquisition device, and inertial measurement unit are arranged in the door assembly area. The method includes: During door assembly, respectively and in real-time obtain the door assembly geometric data, door assembly image data, and door assembly motion data collected by the lidar, image acquisition device, and inertial measurement unit; Perform spatial unification and time synchronization on the door assembly geometric data, door assembly image data, and door assembly motion data, and then fuse them according to preset rules to obtain the target geometric data, target image data, and target motion data after semantic complementarity; Respectively identify the geometric error, position error, and motion error during door assembly based on the target geometric data, target image data, and target motion data, and determine corresponding compensation parameters according to the geometric error, position error, and motion error to perform error compensation during door assembly.

2. The method for compensating door assembly errors based on multi-source data fusion according to claim 1, characterized in that The step of performing spatial unification and time synchronization on the door assembly geometric data, door assembly image data, and door assembly motion data includes: Establish the coordinate transformation relationship between the lidar, image acquisition device, and inertial measurement unit through external parameter calibration, and use the coordinate transformation relationship to perform spatial unification on the door assembly geometric data, door assembly image data, and door assembly motion data; Align the timestamps of the lidar, image acquisition device, and inertial measurement unit through a synchronous clock to achieve the time synchronization of the door assembly geometric data, door assembly image data, and door assembly motion data.

3. The method for compensating the assembly error of the vehicle door based on multi-source data fusion according to claim 2, wherein The step of fusing according to preset rules to obtain the target geometric data, target image data, and target motion data after semantic complementarity includes; Use the door assembly motion data to construct a vibration model, perform inverse correction on the door assembly geometric data, and endow the door assembly geometric data with texture information using the door assembly image data to obtain the target geometric data; Perform transformation compensation on the door assembly image data based on the door assembly motion data, and extract the background area to obtain the target image data; Input the door assembly geometric data, door assembly image data, and door assembly motion data into an extended Kalman filter, and correct the drift of the door assembly motion data through constraint conditions to obtain the target motion data.

4. The method for compensating the door assembly error based on multi-source data fusion according to claim 3, wherein, The step of respectively identifying the geometric error, position error, and motion error during door assembly based on the target geometric data, target image data, and target motion data includes: Extract the three-dimensional point cloud data of the door from the target geometric data, and match the three-dimensional point cloud data of the door with the point cloud data of the standard door model to obtain the geometric error including translation error and rotation error; Extract the pixel coordinates of the assembly identifier from the target image data, convert them to world coordinates through a calibration matrix, and compare the world coordinates with the theoretical coordinates of the assembly identifier in the standard door model, calculate the pixel-level offset and convert it to a physical size to obtain the position error; Compare the motion parameters included in the target motion data with the preset motion parameter threshold to obtain the motion error.

5. The method for compensating the door assembly error based on multi-source data fusion according to claim 4, wherein The steps of performing transformation compensation on the door assembly image data based on the door assembly motion data and extracting the background region to obtain the target image data include: According to the door assembly motion data, establish an external parameter model of the image acquisition device, and predict the real-time position and orientation of the image acquisition device in the world coordinate system; In the initial state where the robotic arm for door assembly does not move, capture a static background image, and when the robotic arm moves, perform an affine transformation on the static background image using the pose transformation obtained from the predicted real-time position and orientation of the image acquisition device to generate the theoretical background image at the current moment; Perform a difference operation on the door assembly image data and the theoretical background image, and extract the region with significant gray-scale difference as the dynamic foreground mask; Extract the background region from the door assembly motion data according to the dynamic foreground mask to obtain the target image data.

6. The method for compensating the door assembly error based on multi-source data fusion according to claim 5, characterized in that The expression of the dynamic foreground mask is: ； Among them, M= 1 is the foreground area in the dynamic foreground mask, M =0 is the background area in the dynamic foreground mask, is the door assembly image data, is the theoretical background image.

7. The method for compensating door assembly errors based on multi-source data fusion according to claim 1, characterized in that, The steps of determining the corresponding compensation parameters according to the geometric error, position error, and motion error to perform error compensation during door assembly include: Use the historical geometric error, position error, motion error, and corresponding compensation parameters to train a DNN model to obtain an error compensation model; Input the geometric error, position error, and motion error into the trained error compensation model to obtain the corresponding compensation parameters.

8. An error compensation device for door assembly based on multi-source data fusion, characterized in that For performing error compensation during door assembly, corresponding lidar, image acquisition device, and inertial measurement unit are arranged in the door assembly area. The device includes: An acquisition module for respectively and real-time acquiring the door assembly geometric data, door assembly image data, and door assembly motion data collected by the lidar, image acquisition device, and inertial measurement unit during door assembly; A complementary module for spatially unifying and temporally synchronizing the door assembly geometric data, door assembly image data, and door assembly motion data, and then fusing them according to a preset rule to obtain the target geometric data, target image data, and target motion data after semantic complementarity; A compensation module for respectively identifying the geometric error, position error, and motion error during door assembly according to the target geometric data, target image data, and target motion data, and determining the corresponding compensation parameters according to the geometric error, position error, and motion error to perform error compensation during door assembly.

9. A readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method described in any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and running on the processor. When the processor executes the program, it implements the steps of the method described in any one of claims 1 to 7.

Images