A method of guiding the assembly of a part and related apparatus

Abstract

The application discloses a part assembly guiding method and related equipment, the method comprises the following steps: obtaining the preset installation operation information of the part, the first source point cloud data and the scene data under the current assembly scene, the first source point cloud data is the point cloud data corresponding to the part; processing the scene data to obtain the first target point cloud data; registering the first target point cloud data with the first source point cloud data corresponding to the current part to be assembled to obtain the real-time pose information of the current part to be assembled; under the condition that the current part to be assembled meets the preset installation condition based on the real-time pose information of the current part to be assembled and the preset installation operation information, the next assembly information is displayed to guide the assembly of the next part to be assembled, and the next assembly information comprises the name of the next part to be assembled and the assembly pose information of the next part to be assembled. Through the above method, the application can reduce the probability of assembly error and improve the assembly efficiency.

Description

Technical Field

[0001] This application relates to the field of mechanical assembly technology, specifically to a method for guiding the assembly of parts and related equipment. Background Technology

[0002] In the traditional assembly process of mechanical equipment, assemblers rely on reading paper documents or electronic documents to perform assembly work, which has the following problems: Although production assembly documents use text as a supplement and assembly drawings as the main method to describe the assembly process, the visualization effect is not intuitive, and assemblers need to spend a certain amount of time to obtain effective information, resulting in low assembly efficiency; In addition, there is a lack of supervision during the assembly process, and on-site assembly may result in omissions or misassemblies due to insufficient understanding by assemblers, leading to unqualified product quality. Summary of the Invention

[0003] This application provides a method and related equipment for guiding the assembly of parts, which can reduce the probability of assembly errors and improve assembly efficiency.

[0004] To solve the above-mentioned technical problems, the technical solution adopted in this application is: to provide a method for guiding the assembly of parts, which is used to guide the assembly of parts of mechanical equipment, the mechanical equipment including at least two parts, the method including: acquiring preset installation operation information of the parts, first source point cloud data, and scene data in the current assembly scene, wherein the first source point cloud data is the point cloud data corresponding to the parts; processing the scene data to obtain first target point cloud data; registering the first target point cloud data with the first source point cloud data corresponding to the parts to be assembled to obtain the real-time pose information of the parts to be assembled, wherein the parts to be assembled are the parts currently being installed in the target assembly scene; and, based on the real-time pose information of the parts to be assembled and the preset installation operation information, determining that the parts to be assembled meet the preset installation conditions, displaying the next assembly information to guide the assembly of the next parts to be assembled, wherein the next assembly information includes the name of the next parts to be assembled and the assembly pose information of the next parts to be assembled.

[0005] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide an assembly guidance device, which includes a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the part assembly guidance method in the above-mentioned technical solution.

[0006] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide an assembly guidance system, which includes an assembly guidance device and a data acquisition device. The assembly guidance device is connected to the data acquisition device and is used to receive scene data collected by the data acquisition device and process the data acquisition device to obtain virtual assembly information. The assembly guidance device is the assembly guidance device in the above-mentioned technical solution.

[0007] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium for storing a computer program, which, when executed by a processor, is used to implement the part assembly guidance method in the above-mentioned technical solution.

[0008] The beneficial effects of this application through the above scheme are as follows: First, obtain the preset installation operation information related to the part, the first source point cloud data, and the scene data under the current assembly scenario. The first source point cloud data is the point cloud data corresponding to the part. Then, process the scene data to obtain the first target point cloud data. Register the first target point cloud data with the first source point cloud data corresponding to the part to be assembled to obtain the real-time pose information of the part to be assembled. The part to be assembled is the part currently being installed in the target assembly scenario. Then, using the real-time pose information of the part to be assembled and the preset installation operation information, determine whether the part to be assembled meets the preset installation conditions. If so, display the next assembly information to guide the assembly of the next part to be assembled. The assembly information includes the name of the next part to be assembled and its assembly pose information. This solution can match and display virtual assembly information with real parts without adding labels to the parts, making it easier for assemblers to install according to the prompts and reducing the chance of installation errors. Compared with manually reading the installation manual, it can quickly find the next part to be installed, making the assembly more efficient. Moreover, because it uses point cloud data registration, compared with contour matching, it can identify parts and solve the pose of parts in working scenes with low light intensity, cluttered backgrounds and few texture features, achieving virtual-real integration and can be applied to real production assembly environments. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0010] Figure 1This is a schematic diagram of an embodiment of the assembly guidance system provided in this application;

[0011] Figure 2 This is a schematic diagram of another embodiment of the assembly guidance system provided in this application;

[0012] Figure 3 This is a flowchart illustrating an embodiment of the part assembly guidance method provided in this application;

[0013] Figure 4 This is a flowchart illustrating another embodiment of the part assembly guidance method provided in this application;

[0014] Figure 5 This is a schematic diagram of another embodiment of the assembly guidance system provided in this application;

[0015] Figure 6 This is a schematic diagram of the structure of an embodiment of the assembly guiding device provided in this application;

[0016] Figure 7 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0017] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the application. Similarly, the following embodiments are only some, not all, embodiments of the present application, and all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the present application.

[0018] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0019] It should be noted that the terms "first," "second," and "third" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0020] First, let me introduce the technical terms used in this application:

[0021] Assembly refers to the process of joining several parts and components into a product according to technical requirements, and then debugging, inspecting and testing it to make it a qualified product.

[0022] Augmented Reality (AR) uses computer graphics and data interaction technology to map virtual information onto real-world scenes in real time. The virtual and real information complement each other, increasing the amount of information and understanding that assemblers perceive in the real scene. The augmented scene can be displayed through devices such as monitors, projectors, or head-mounted displays.

[0023] Because AR technology has the ability to simultaneously display information from both the real and virtual worlds, it can seamlessly overlay computer-generated virtual assembly information (such as 3D models, assembly animations, or text prompts) onto the assembly site. Therefore, this application addresses the problems of low assembly efficiency and long assembly cycles in traditional solutions. With the aim of assisting in the assembly of mechanical products, it proposes a mechanical production assembly navigation technology. This technology utilizes AR technology to integrate the real assembly site with virtual assembly information, reducing the cognitive and comprehension burden on assembly personnel, guiding their assembly work, reducing errors, and improving assembly efficiency and quality. The details are described below.

[0024] Please see Figure 1 , Figure 1This is a schematic diagram of an embodiment of the assembly guidance system provided in this application. The assembly guidance system 10 includes an assembly guidance device 11 and a data acquisition device 12. The assembly guidance device 11 is connected to the data acquisition device 12. The assembly guidance device 11 is used to receive scene data acquired by the data acquisition device 12 and process the data acquisition device 12 to obtain virtual assembly information. The virtual assembly information includes assembly animation and text information. The specific structure and function of the assembly guidance device 11 will be described in detail below.

[0025] In one embodiment, such as Figure 2 As shown, the assembly guidance system 10 also includes a workbench 13, and the mechanical equipment 14 includes parts 14a-14e, which are placed on the workbench 13. The assembly guidance equipment 11 includes a host device 111 and a display device 112. The data input terminal of the assembly guidance system 10 is a data acquisition device 12 containing depth information. The data acquisition device 12 can be a camera, which includes a time-of-flight (TOF) depth sensor, a high-definition RGB camera, and a seven-microphone circular array.

[0026] During the assembly process, a camera is used to photograph the target assembly scene, and depth and color information are obtained through a software development kit (SDK). The data processing and related algorithm implementation are completed by the host device 111. When the system is running, the assembler points the camera at the target assembly scene, and the enhanced image is displayed through the display device 112. The assembler can clearly see the parts that need to be assembled next and their assembly pose information (including position and orientation) on the display device 112. Moreover, if the assembler makes an installation error during the operation, the system will issue a warning and prompt the correct installation method.

[0027] Understandably, since AR glasses or handheld mobile devices cannot free the assembly workers' hands, display device 112 is chosen to display the screen. By displaying the positional relationship through display device 112, the installation efficiency of assembly workers can be improved.

[0028] This embodiment proposes an assembly navigation scheme for mechanical equipment. It achieves the matching and display of virtual assembly information with real parts without the need to add labels to the parts, making it convenient for assembly personnel to install each part according to the virtual assembly information and preventing installation errors. In addition, during the assembly process, the operation sequence and operation results can be recorded and interpreted, and alarm prompts can be given for position and sequence errors, thereby improving the correctness of the installation.

[0029] Please see Figure 3 , Figure 3This is a flowchart illustrating an embodiment of the part assembly guidance method provided in this application. The method is applied to an assembly guidance device, which guides the assembly of parts in a mechanical device. The mechanical device includes at least two parts. The method includes:

[0030] S31: Obtain the preset installation operation information of the parts, the first source point cloud data, and the scene data under the current assembly scene.

[0031] Retrieve pre-set installation operation information, which is used to identify the installation sequence and position of each part of the mechanical equipment; for example, ... Figure 2 As shown, the installation sequence can be set to install parts 14a to 14e sequentially.

[0032] In addition, it is necessary to acquire the first source point cloud data corresponding to each part. This first source point cloud data is the point cloud data corresponding to the part, which is the complete point cloud data of the part, that is, the point cloud data of the part from 360 degrees. The first source point cloud data is obtained by scanning the part. Specifically, the first source point cloud data is acquired as follows: the part is mounted on a scanning turntable, the software parameters and capture quantity (i.e., the number of images captured) of the processing software are set, the part is scanned using a 3D laser scanner, and the part is rotated 360 degrees to acquire the image information of the part. The acquired image information is then transmitted to the processing software, which processes the acquired 2D image data (i.e., image information) into 3D point cloud data (i.e., the first source point cloud data). These point cloud data in space are then connected to construct a complete curved surface, which is the process of forming the 3D model of the part.

[0033] S32: Process the scene data to obtain the first target point cloud data.

[0034] Scene data is data obtained by the acquisition device from capturing the current assembly scene. Scene data includes depth information and color information. Preprocessing methods in related technologies can be used to process the scene data, such as filtering or enhancement. Then, the preprocessed scene data is filtered to obtain the first target point cloud data, which includes the real-time point cloud data of the parts in the target assembly scene.

[0035] In a specific embodiment, in order to obtain point cloud data, the intrinsic parameter matrix of the camera can be obtained using an Application Programming Interface (API). The intrinsic parameter matrix includes: 1) focal length f, which represents the distance from the camera's focal point to the mapping plane; 2) adjacent pixel points ax and ay, which respectively represent how many units a pixel occupies in the horizontal and vertical directions of the image captured by the camera; 3) the coordinates (u0, v0) of the origin of the image coordinate system in the pixel coordinate system, which represents the difference in the number of horizontal and vertical pixels between the center pixel coordinates and the origin pixel coordinates of the image.

[0036] Using the following formula, based on the depth camera's internal matrix, depth information can be converted into point cloud data in real time, thus reconstructing the point cloud data of the currently captured object (including parts for assembly personnel warning):

[0037]

[0038] Among them, a x a y u0 and v0 are the internal parameters of the depth camera; the coordinates P'(Xc, Yc, Zc) of the point cloud data in 3D space can be calculated from the coordinates P(u, v) of the pixels in the depth image.

[0039] S33: Register the first target point cloud data with the first source point cloud data corresponding to the part to be assembled to obtain the real-time pose information of the part to be assembled.

[0040] The part to be assembled is one of the parts in the target assembly scene. It is the part that is currently being installed in the target assembly scene. The name and other relevant information of the part to be installed (i.e. the part to be assembled) can be generated by using the preset installation operation information, such as the assembly pose information. The first target point cloud data and the first source point cloud data corresponding to the part to be assembled (denoted as the current source point cloud data to be matched) are registered by using the registration method. The real-time pose information and the name of the part to be assembled can be obtained.

[0041] For example, suppose the mechanical equipment includes three parts A1 to A3. Part A1 has already been installed, so the part to be assembled is now part A2. The source point cloud data to be matched is the first source point cloud data corresponding to part A2. The first source point cloud data includes the real-time point cloud data of parts A1 to A3. By matching the current source point cloud data to be matched with the first target point cloud data, it can be determined which point cloud data in the first target point cloud data belong to the real-time point cloud data of part A2. After determining the real-time point cloud data of part A2, the real-time pose information of part A2 is calculated, that is, the position and orientation of part A2 are calculated.

[0042] S34: Based on the real-time pose information of the part to be assembled and the preset installation operation information, if it is determined that the part to be assembled meets the preset installation conditions, the next assembly information is displayed to guide the assembly of the next part to be assembled.

[0043] Based on the real-time pose information and preset installation operation information of the part to be assembled, it is determined whether the part meets the preset installation conditions. If the part meets the preset installation conditions, it indicates that the installation of the part is correct, that is, its installation position and sequence are correct. At this time, the relevant information of the next part to be assembled (i.e., the next assembly information) can be obtained from the preset installation operation information and displayed or broadcast to guide the assembly of the next part. The next assembly information includes the name of the next part to be assembled and its assembly pose information. For example, such as Figure 2 As shown, the part to be assembled is part 14b. When it is determined that the installation of part 14b meets the requirements, the information of part 14c is displayed on the display device.

[0044] In one implementation, a human-computer interaction (HCI) scheme can be adopted so that during the guided assembly process, assemblers can quickly interact with the assembly guidance system through gestures or voice, thereby realizing system auxiliary functions such as process guidance and assembly information query. Assemblers can choose the appropriate interaction method according to the actual target assembly scenario, thereby improving the system's versatility. Specifically, the purpose of HCI is to enable the system to understand the instructions issued by the assemblers and execute the corresponding actions according to the assemblers' wishes. Depending on the interaction object, HCI technology is mainly divided into two types: interaction between assemblers and virtual objects, and control of the workflow by assemblers. Interaction between assemblers and virtual objects involves operations such as creating, moving, rotating, or scaling virtual objects. Control of the workflow by assemblers mainly involves control commands such as starting, pausing, selecting information, and moving to the previous or next step in the assembly task, with the goal of enabling assemblers to quickly obtain the required virtual assembly information.

[0045] Furthermore, since gesture interaction does not completely free the assembly personnel's hands, it causes some interference to the continuity of assembly. Voice interaction is not suitable for use in noisy assembly environments. Therefore, this embodiment adopts a combination of gesture interaction and voice interaction, which are supported simultaneously. The gesture interaction and voice interaction are described in detail below.

[0046] (1) Voice interaction

[0047] This embodiment utilizes voice development tools (such as the Microsoft Voice SDK) combined with the Microsoft Foundation Class Library (MFC) Dialog box class to develop a voice interaction system for mechanical assembly guidance. This aims to improve the efficiency of human-computer interaction during actual assembly guidance and enhance the learning experience for assembly personnel. Specifically, the voice interaction function mainly includes a dictionary management module, a voice recognition module, and a voice interaction service module. After the assembly personnel issue a voice command, the audio device receives it. The voice recognition module is then invoked by the voice interaction service module to recognize the received voice information and execute the corresponding function based on the recognition result.

[0048] For example, the voice commands included in this embodiment are mainly: a) "Turn on voice broadcast" and "Turn off voice broadcast". These two voice commands can broadcast relevant assembly information of the current assembly process to help assembly personnel; b) "Previous step" and "Next step". These two voice commands are mainly used to help assembly personnel during the assembly guidance process; c) "Start recording" and "Stop recording". These two voice commands support real-time recording and saving of the mechanical assembly process to provide resource support for later improvement of the assembly process.

[0049] (2) Gesture Interaction

[0050] Gesture recognition technology primarily achieves different interactive effects through varying hand movements of assembly workers. When assembly workers are in a noisy work environment, gesture interaction can replace voice interaction. Specifically, sensors in the camera provide skeletal joint information, accurately detecting hand joints. Therefore, the sensors can track the human skeleton, extract joint coordinates, and perform vector calculations to determine the direction the human is pointing, enabling gesture recognition for human-robot interaction and providing assembly workers with virtual assembly information such as "previous step" and "next step."

[0051] Understandably, if assemblers guide the installation of parts through voice or gesture interaction, the assembly guidance equipment needs to respond promptly in order to improve the assembly speed.

[0052] This embodiment provides a mechanical production assembly navigation method that realizes the matching and display of virtual assembly information with real parts, making it convenient for assembly personnel to install according to the prompts and reducing the probability of installation errors. Because it adopts the method of registering point cloud data, compared with the contour matching scheme, it can identify parts and solve the pose of parts in working scenes with low light intensity, cluttered background and few texture features, realize the combination of virtual and real, and can be applied to real production assembly environment.

[0053] Please see Figure 4, Figure 4 This is a flowchart illustrating another embodiment of the part assembly guidance method provided in this application. The method is applied to an assembly guidance device, which guides the assembly of mechanical equipment, which includes at least two parts. The method includes:

[0054] S41: Obtain the preset installation operation information of the parts, the first source point cloud data, and the scene data under the current assembly scene.

[0055] S41 is the same as S31 in the above embodiment, and will not be described again here.

[0056] S42: The scene data is filtered using a direct-pass filtering method to obtain the first target point cloud data.

[0057] The acquired scene data may contain invalid background information and noise. Noise can negatively impact the effectiveness of subsequent target recognition, localization, and tracking. Therefore, a direct-pass filtering method is used to filter the scene data and generate first target point cloud data. This first target point cloud data includes the point cloud data of parts in the target assembly scene. Specifically, a dimension and its value range can be specified. The scene data is then traversed sequentially, and it is determined whether the value of the scene data in the specified dimension is within the corresponding value range. Points with values ​​outside the value range are deleted. The remaining points after the traversal constitute the filtered point cloud data (i.e., the first target point cloud data). This method is suitable for eliminating invalid operational background.

[0058] Furthermore, the core of similarity matching between the first target point cloud data and the first source point cloud data acquired by the 3D laser scanner lies in solving the optimal transformation relationship between the two. During the target recognition process, the 3D model of the part acquired by the 3D laser scanner and the size of the part in the target assembly scene are completely consistent. By using similarity matching between the first source point cloud data and the reconstructed first target point cloud data, the parts appearing in the environment can be identified, and the posture information of the parts in the real environment can be obtained.

[0059] Since the first source point cloud data and the first target point cloud data come from different devices, their point cloud densities (i.e., the density of the point cloud) are also different. It is difficult to obtain an accurate transformation matrix when registering two point cloud data with different densities, which will cause some interference to the robustness of target recognition. Therefore, this embodiment adopts an adaptive filtering method to adaptively adjust the density of the point cloud data, which will be described in detail below.

[0060] S43: Adaptive filtering is used to filter the first target point cloud data and the first source point cloud data respectively to obtain the second target point cloud data and the second source point cloud data.

[0061] The difference between the density of the second target point cloud data and the density of the second source point cloud data is less than a preset density difference. For example, the density of the second target point cloud data is approximately the same as the density of the second source point cloud data. The following detailed explanation uses the adaptive filtering method as an example of voxel filtering, which includes the following steps:

[0062] 1) Based on the point cloud data to be processed, establish a bounding box.

[0063] The point cloud data to be processed includes either the first target point cloud data or the first source point cloud data; a bounding box encloses the point cloud data to be processed, the boundary of the bounding box is larger than the boundary of the point cloud data to be processed, the volume of the bounding box is V, and the length, width, and height of the bounding box are l = x. max -x min w=y max -y min h = z max -z min x max x min These represent the maximum and minimum values ​​in the x-direction of the point cloud data to be processed, and the y-direction values ​​are respectively. max y min These represent the maximum and minimum values ​​in the y-direction and z-direction of the point cloud data to be processed, respectively. max z min These represent the maximum and minimum values ​​in the z-direction of the point cloud data to be processed, respectively.

[0064] Furthermore, the bounding box comprises multiple voxels, which are the smallest units of digital data segmentation in three-dimensional space. The voxel filtering method involves dividing the input point cloud data (i.e., the point cloud data to be processed) into voxels in a reasonable manner, and then sampling the point cloud data within each voxel to reduce the amount of point cloud data while preserving the shape characteristics of the point cloud.

[0065] In one embodiment, a three-dimensional voxel grid is created from the point cloud data to be processed. The voxel grid is equivalent to a collection of tiny three-dimensional cubes in space. Then, within each voxel (i.e., a three-dimensional cube), the centroid of all points in the voxel is used to approximate the other points in the voxel, so that all points in the voxel are represented by a centroid point. After performing voxel filtering on all point cloud data, the filtered point cloud data can be obtained.

[0066] Furthermore, the voxel size has a significant impact on the sampling rate of point cloud data. If the voxel is too large, the features of the sampled point cloud data will not be obvious; if the voxel is too small, the point cloud data contained in the voxel will be reduced, and the sampling effect will be poor. Therefore, the voxel size setting needs to be calculated based on the preset sampling rate and preset sampling density, which will be described in detail below.

[0067] 2) Based on the preset sampling rate and preset point cloud density, the bounding box is divided to obtain multiple voxels.

[0068] The preset sampling rate and preset point cloud density are both set by the assembler. The specific operation of dividing the bounding box using the preset sampling rate and preset point cloud density is as follows:

[0069] A) Set the current size of the voxel to the preset initial size.

[0070] The default initial size is the size of the voxel set in advance based on experience or application needs. The length and width of the voxel are set to be the same. The current size is the length or width of the voxel.

[0071] B) Based on the preset sampling rate and preset point cloud density, update the current size to obtain the voxel size.

[0072] Updating the current size involves the following steps:

[0073] B1) Update the current size based on the current size and the preset sampling rate.

[0074] Calculate the ratio of the preset value to the preset sampling rate to obtain the first value; perform a square root operation on the first value to obtain the second value; and multiply the second value by the current size to determine the current size.

[0075] For example, if the preset value is 1, then assume the current size is updated using the following formula:

[0076]

[0077] The relationship between the preset sampling rate and the number of point cloud data points per voxel is shown below:

[0078]

[0079] The point cloud density is calculated as follows:

[0080]

[0081] From the above formulas (1) to (3), we can obtain the following formula:

[0082]

[0083] The formula for calculating the number of voxels is as follows:

[0084]

[0085] From formulas (5) and (6), we can obtain the following formula:

[0086]

[0087] Where s is the point cloud density, m is the number of point cloud data in each voxel, L is the current size, L' is the updated current size, and p is the preset sampling rate.

[0088] B2) Filter multiple voxels corresponding to the current size to obtain filtered data; obtain the number of point cloud data in the filtered data to obtain the current point cloud quantity; calculate the current point cloud density based on the current point cloud quantity and the volume of the bounding box.

[0089] Calculate the ratio of the current point cloud quantity to the bounding box volume to obtain the current point cloud density; specifically, perform voxel filtering according to the new voxel side length calculated by formula (6), calculate the current filtered point cloud quantity (i.e., the current point cloud quantity) N', and then the current point cloud density s' is as follows:

[0090]

[0091] B3) Update the current size based on the current point cloud density and the preset point cloud density, and return the step of updating the current size based on the current size and the preset sampling rate, until the difference between the current point cloud density and the preset point cloud density falls within the preset difference range.

[0092] Determine if the current point cloud density is less than the preset point cloud density; if the current point cloud density is less than the preset point cloud density, then the difference between the current point cloud density and the preset step size is determined as the current point cloud density; if the current point cloud density is greater than or equal to the preset point cloud density, then the sum of the current point cloud density and the preset step size is determined as the current point cloud density, where the preset step size is the change factor for each iteration.

[0093] For example, let the preset sampling density be denoted as u. Compare the size of s' with u. If s' < u, then L' = L' - β; if s' > u, then L' = L' + β. Through multiple iterative calculations, a suitable voxel size can be finally determined to divide the bounding box. Here, β is the preset step size, and the specific value of β can be set according to experience or application needs, for example:

[0094] C) Calculate the number of voxels based on the voxel size and the volume of the bounding box.

[0095] Calculate the ratio of the bounding box volume to the cube of the voxel size to obtain the number of voxels.

[0096] 3) Filter the point cloud data within each voxel.

[0097] The filtering method used for the point cloud data within each voxel is the same as that in related technologies, and will not be repeated here.

[0098] In this embodiment, the first source point cloud data and the first target point cloud data are filtered using the above-described filtering scheme, resulting in approximately the same point cloud density.

[0099] S44: Register the second target point cloud data with the second source point cloud data corresponding to the part to be assembled to obtain the real-time pose information of the part to be assembled.

[0100] The preset installation operation information includes the installation sequence of parts and the assembly pose information of parts; the filtered source point cloud data (i.e., the second source point cloud data) and the filtered target point cloud data (i.e., the second target point cloud data) are registered to identify parts in the environment, and the rotation matrix and translation matrix between the second source point cloud data and the second target point cloud data corresponding to the part to be assembled are calculated to determine the position of the virtual assembly information to be superimposed in the real world.

[0101] Furthermore, since the second target point cloud data inevitably contains multiple objects, it is necessary to segment the second target point cloud data before registration. In this embodiment, a segmentation algorithm based on Euclidean clustering is used to determine whether the point cloud data should be clustered into one class by using the distance between adjacent point clouds as the criterion; then the second source point cloud data and the segmented point cloud data are registered.

[0102] Traditional point cloud registration algorithms have high requirements for the positional relationship between point clouds. If the initial spatial positions of two point clouds differ significantly, local optima will emerge, leading to registration failure. Therefore, this embodiment employs a combination of initial registration and fine registration to match multiple local point cloud data sets. Specifically, a sample consistency initial point cloud registration algorithm is used to coarsely register multiple point cloud data sets in different coordinate systems, adjusting their spatial positions to provide a better initial registration position for fine registration. After coarse registration, the Iterative Closest Point (ICP) algorithm is used for fine registration, where the ICP algorithm registers the pose transformation relationship between two adjacent frames.

[0103] In one embodiment, point cloud matching based solely on depth information can generate real-time poses. However, further matching using RGB information optimizes the obtained real-time poses, making them more accurate. Since the depth camera provides both color and depth information, this embodiment combines RGB and depth information to obtain the real-time pose information of the part to be assembled. That is, the pose is generated based on depth information matching, and then optimized using RGB information, enabling real-time acquisition of the relative positional relationship between moving parts and the fixed camera in the target assembly scene. This method improves the robustness of the 3D tracking registration process. It utilizes the camera's API interface to acquire 3D feature points containing color information in real time. Feature extraction algorithms (e.g., Fast Point Feature Histograms (FPFH)) are used to detect key points of color information in the 3D scene, obtaining corresponding key point descriptor vectors. The nearest neighbor algorithm and the principle of maximizing the inner product of vectors are used to match and optimize the key points. Based on this, Random Sample Consensus is utilized... The Consensus (RANSAC) algorithm removes incorrectly matched keypoint pairs to obtain a keypoint cloud with a high registration rate. Finally, the ICP registration method is used to register the keypoint pairs and obtain the transformation matrix. Furthermore, to improve matching efficiency and reduce computational complexity, a k-dimensional tree (kd) nearest neighbor search algorithm is used to accelerate the search for nearest neighbors, ultimately obtaining the rotation matrix and translation vector.

[0104] Since the size and dimensions of each part in the target assembly scenario are fixed, this embodiment matches the size of the 3D model of the part obtained by the 3D laser scanner with the point cloud data after each clustering. The length, width, height and volume of the model are set as the standard confidence level, and the point cloud data after the clustering with the highest confidence level is used as the real-time point cloud data corresponding to the part to be assembled.

[0105] S45: Based on the real-time pose information of the part to be assembled and the preset installation operation information, determine whether the part to be assembled meets the preset installation conditions.

[0106] The current part to be assembled is inspected to obtain its real-time pose information and installation sequence. It is then determined whether the current pose information is the same as the pose information corresponding to the current part in the preset installation operation information (referred to as the current reference pose information) and whether the installation sequence is the same as the installation sequence corresponding to the current part in the preset installation operation information (referred to as the current reference installation sequence). If the current pose information is the same as the current reference pose information and the installation sequence is the same as the current reference installation sequence, then the preset installation conditions are met. The current reference pose information includes the reference installation position and the reference installation direction.

[0107] In one embodiment, the parts are tracked to obtain target tracking and recognition results. These results are used to distinguish the part to be assembled (i.e., the currently assembled part) from the other parts in the current state. By tracking the currently assembled part, the spatial positional relationship between the currently assembled part and the depth camera is calculated in real time. When the currently assembled part moves from its initial position to its corresponding reference installation position, a corresponding voice prompt indicating correct installation is given. Specifically, as the currently assembled part moves, to ensure a good virtual-real integration effect, the transformation matrix between the currently assembled part and the camera is continuously calculated, i.e., the six-degree-of-freedom pose of the currently assembled part is tracked accurately in real time. The computer acquires the real-time pose information and accurately superimposes the virtual guidance information onto the currently assembled part in the real scene.

[0108] This embodiment proposes an environment perception method based on depth camera visual information. It uses a point cloud registration method that integrates depth and color information for tracking and registration, and applies it to an assembly guidance system. It can identify the parts to be assembled and solve the real-time pose information of the parts to be assembled in working scenes with low light intensity, cluttered backgrounds and few texture features, thus achieving a combination of virtual and real information. It can be applied to real production assembly environments.

[0109] S46: If the current part to be assembled meets the preset installation conditions, the next assembly information is displayed to guide the assembly of the next part to be assembled.

[0110] S46 is the same as S34 in the above embodiment, and will not be described again here.

[0111] S47: If the part to be assembled does not meet the preset installation conditions, an alarm message will be generated.

[0112] If it is determined that the current part to be assembled does not meet the preset installation conditions, it indicates that the installation of the current part to be assembled does not meet the requirements. This may be because the installation sequence of the current part to be assembled is incorrect or the installation position of the current part to be assembled is incorrect. At this time, an alarm message can be generated to remind the assembly personnel to adjust the current part to be assembled to meet the installation requirements.

[0113] In one implementation, such as Figure 5 As shown, an operation error prevention module can be set up, which includes a position error judgment module and a sequence error judgment module. The position error judgment module is used to determine whether the part in a certain installation step is correctly assembled based on the real-time perceived pose information (including the position and direction of the installation). The error judgment module is used to determine whether the assembler has selected the wrong assembly object based on the shape of the part (e.g., a 3D model) during the assembly process. If it is determined that the requirements are not met, a prompt and alarm will be issued.

[0114] S48: Render the real-time point cloud data of the part to be assembled, obtain the rendering result, and display the rendering result.

[0115] like Figure 5 As shown, an information visualization module can be set up to visualize virtual assembly information. Specifically, the information visualization module includes a rendering module, which can render the real-time point cloud data of the parts to be assembled. The development environment for the rendering module can be Unity3D, which is widely used in the development of industrial scene visualization. It supports the development of external extension plugins and leverages its compatibility to achieve multi-platform compatibility of the virtual scene through environment attribute configuration for different platforms (such as Android). The AR assembly scene module (containing the executable program of the AR assembly scene) is placed on the client and published on the device (such as a monitor, head-mounted display, or tablet that supports AR development). It is mainly used for assembly guidance, that is, to overlay the virtual scene with the physical scene, and observe the virtual projection of the assembly animation while observing the assembly object, thereby obtaining more assembly information.

[0116] Furthermore, the scanned 3D model is converted into the corresponding Unity format, placed at the baseline installation position, and a virtual scene is constructed using Unity. The model of the part with the real proportions is rendered, and corresponding materials are added to give it better lighting effects, making the process of combining the virtual and real more realistic. Based on the real information of the part, the user interface (UI) is designed in world coordinates and local coordinates to realize the real-time expression of assembly information.

[0117] In one implementation, such as Figure 5 As shown, a 3D model, virtual human demonstration animation, and text description are designed based on the assembly process to achieve real-time guidance of the assembly process, improve assembly efficiency, and reduce assembly error rate and assembly accidents. The 3D model is used to express the 3D structure of the parts and can provide richer visual perception effects. The text prompts are used to express various auxiliary assembly information, including the name of the parts, precautions, and a detailed description of the assembly steps. The demonstration animation is used to express the assembly relationship of the parts and to demonstrate the assembly process, thereby improving the visualization effect.

[0118] In another specific embodiment, the assembly pose information includes position and orientation. Based on the position and orientation of the part to be assembled, an indicator matching the part to be assembled is added to the rendering result, that is, to indicate the correct installation position of the part to be assembled during the assembly process.

[0119] Furthermore, after identifying the parts that need to be assembled in a certain step of the assembly process, the position and orientation of the parts are used to generate 3D indicator arrows that point to the correct installation position; in addition, information that needs attention in the current step can also be displayed.

[0120] In another specific embodiment, viewpoint information of the target object can be obtained. The target object can be a person, and the viewpoint information is the viewpoint from which the human eye looks. The orientation of the rendering result is adjusted based on the viewpoint information so that the adjusted rendering result matches the viewpoint information.

[0121] Furthermore, a viewpoint tracking module can be configured, such as... Figure 5 As shown, the viewpoint tracking module is used to track the head position of the assembler and convert it into position information in the projection screen coordinate system. Based on this position information, the projection matrix is ​​adjusted to draw the image of the corresponding viewpoint. At the same time, the orientation information of the virtual camera in the rendering scene is changed according to the viewpoint position of the assembler.

[0122] This embodiment uses a viewpoint tracking module to replace the positions of the eyes with head skeleton points. It uses the camera's API interface function to capture the head skeleton points of the assembly personnel, adjusts the projection matrix according to the position of the head skeleton points in the projection screen coordinate system, and renders the image of the corresponding viewpoint. This achieves adaptive adjustment of the displayed image according to the assembly personnel's viewpoint, and can display images from different viewing angles.

[0123] In another specific embodiment, such as Figure 2 As shown, there is a virtual character 15 in the scene, who plays the role of an expert, providing technical assembly guidance, promoting effective assembly training, improving assembly efficiency during the assembly process, and reducing the error rate. Compared with the traditional method of guiding by drawings, the assembly efficiency has been improved.

[0124] In other specific embodiments, assemblers can also perceive the assembly guidance process by wearing AR glasses that include a depth camera. Therefore, there is no need to place a depth camera and a display in the target assembly scene; the assembly process can be displayed directly by relying on the glasses.

[0125] This embodiment employs a point cloud registration method that integrates depth and color information for tracking and registration. This method can be applied to assembly guidance systems and exhibits good robustness in mechanical assembly environments with low light intensity and lack of surface texture. Furthermore, during the assembly process, the spatial positional relationship between the part to be installed and the depth camera can be calculated in real time. When the part to be installed finally moves from its initial position to the reference installation position, it indicates that the installation is correct. In addition, alarms can be issued for positional errors and sequence errors, allowing assemblers to adjust the parts in a timely manner and improve the accuracy of installation.

[0126] Please see Figure 6 , Figure 6 This is a schematic diagram of an embodiment of the assembly guidance device provided in this application. The assembly guidance device 60 includes a memory 61 and a processor 62 connected to each other. The memory 61 is used to store a computer program. When the computer program is executed by the processor 62, it is used to implement the part assembly guidance method in the above embodiment.

[0127] Please see Figure 7 , Figure 7 This is a schematic diagram of an embodiment of a computer-readable storage medium provided in this application. The computer-readable storage medium 70 is used to store a computer program 71. When the computer program 71 is executed by a processor, it is used to implement the part assembly guidance method in the above embodiment.

[0128] The computer-readable storage medium 70 can be any medium capable of storing program code, such as a server, USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0129] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0131] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0132] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0133] The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for guiding the assembly of parts, used to guide the assembly of parts in mechanical equipment, said mechanical equipment comprising at least two parts, characterized in that, The method includes: Acquire the preset installation operation information of the part, the first source point cloud data, and the scene data under the current assembly scenario, wherein the first source point cloud data is the point cloud data corresponding to the part; The scene data is processed to obtain first target point cloud data, wherein the first target point cloud data includes real-time point cloud data of parts in the target assembly scene; The first target point cloud data is registered with the first source point cloud data corresponding to the part to be assembled to obtain the real-time pose information of the part to be assembled. The part to be assembled is the part currently being installed in the target assembly scene. Based on the real-time pose information of the part to be assembled and the preset installation operation information, if it is determined that the part to be assembled meets the preset installation conditions, the next assembly information is displayed to guide the assembly of the next part to be assembled. The next assembly information includes the name of the next part to be assembled and the assembly pose information of the next part to be assembled. The registration of the first target point cloud data with the first source point cloud data corresponding to the part to be assembled includes: The first source point cloud data was registered using a combination of initial registration and fine registration, and the registration results were optimized using depth and color information. The step of processing the scene data to obtain the first target point cloud data includes: The scene data is filtered using a direct-pass filtering method to obtain the first target point cloud data; Before the step of registering the first target point cloud data with the first source point cloud data corresponding to the part to be assembled, the following steps are included: An adaptive filtering method is used to filter the first target point cloud data and the first source point cloud data respectively to obtain the second target point cloud data and the second source point cloud data. The difference between the density of the second target point cloud data and the density of the second source point cloud data is less than a preset density difference. The adaptive filtering method includes: Based on the point cloud data to be processed, a bounding box is established, wherein the point cloud data to be processed includes the first target point cloud data or the first source point cloud data. Based on a preset sampling rate and a preset point cloud density, the bounding box is divided to obtain multiple voxels; Filter the point cloud data within each voxel; The step of dividing the bounding box into multiple voxels based on a preset sampling rate and a preset point cloud density includes: Set the current size of the voxel to a preset initial size; Based on the preset sampling rate and the preset point cloud density, the current size is updated to obtain the voxel size; The number of voxels is calculated based on the voxel size and the volume of the bounding box; The step of updating the current size based on the preset sampling rate and the preset point cloud density to obtain the voxel size includes: The current size is updated based on the current size and the preset sampling rate; Filter the multiple voxels corresponding to the current size to obtain filtered data; Obtain the number of point cloud data in the filtered data to get the current number of point cloud data; The current point cloud density is calculated based on the current number of point clouds and the volume of the bounding box. Based on the current point cloud density and the preset point cloud density, the current size is updated, and the step of updating the current size based on the current size and the preset sampling rate is returned until the difference between the current point cloud density and the preset point cloud density falls within the preset difference range.

2. The method for guiding the assembly of parts according to claim 1, characterized in that, The method further includes: If, based on the real-time pose information of the part to be assembled and the preset installation operation information, it is determined that the part to be assembled does not meet the preset installation conditions, an alarm message is generated.

3. The method for guiding the assembly of parts according to claim 1, characterized in that, The preset installation operation information includes the installation sequence of the parts and the assembly posture information of the parts. Before the step of displaying the next assembly information, the following steps are included: When the real-time pose information of the part to be assembled is the same as the assembly pose information of the part to be assembled in the preset installation operation information, and the installation order of the part to be assembled is the same as the installation order of the part to be assembled in the preset installation operation information, it is determined that the preset installation conditions are met.

4. The method for guiding the assembly of parts according to claim 1, characterized in that, The step of calculating the current point cloud density based on the current number of point clouds and the volume of the bounding box includes: The current point cloud density is obtained by calculating the ratio of the current point cloud quantity to the bounding box volume.

5. The method for guiding the assembly of parts according to claim 1, characterized in that, The step of updating the current size based on the current point cloud density and the preset point cloud density includes: Determine whether the current point cloud density is less than the preset point cloud density; If so, the difference between the current point cloud density and the preset step size is determined as the current point cloud density; If not, the sum of the current point cloud density and the preset step size is determined as the current point cloud density.

6. The method for guiding the assembly of parts according to claim 1, characterized in that, The method further includes: The real-time point cloud data of the part to be assembled is rendered to obtain and display the rendering result.

7. The method for guiding the assembly of parts according to claim 6, characterized in that, The method further includes: Obtain the viewpoint information of the target object, and adjust the orientation of the rendering result based on the viewpoint information so that the adjusted rendering result matches the viewpoint information.

8. The method for guiding the assembly of parts according to claim 7, characterized in that, The assembly pose information includes position and orientation, and the method further includes: Based on the position and the direction, an indicator that matches the next part to be assembled is added to the rendering result.

9. An assembly guiding device, characterized in that, It includes an interconnected memory and a processor, wherein the memory is used to store a computer program, which, when executed by the processor, is used to implement the guided method for assembling parts according to any one of claims 1-8.

10. An assembly guidance system, characterized in that, It includes an assembly guidance device and a data acquisition device. The assembly guidance device is connected to the data acquisition device and is used to receive scene data collected by the data acquisition device and process the data acquisition device to obtain virtual assembly information. The assembly guidance device is the assembly guidance device as described in claim 9.

11. A computer-readable storage medium for storing a computer program, characterized in that, When executed by a processor, the computer program is used to implement the method for guiding the assembly of parts according to any one of claims 1-8.

Images