A multi-robot target detection method for aircraft panel assembly

Abstract

The present invention provides a multi-robot target detection method for aircraft panel assembly, comprising the following steps: constructing a multi-source dataset, distributing the multi-source dataset to multiple local robots so that each robot's dataset is non-independent and identically distributed; each local robot uses YOLOv8 as a target detection model, and performs global model training using a federated learning algorithm; in an aircraft panel assembly scenario, multiple robots combine the target detection model with a pose algorithm to perform object detection, outputting the object category, location, and confidence level; when at least two robots have consistent detection results for the same object, the detection result is confirmed as incremental learning data, and the global target detection model is continuously updated using a federated incremental learning algorithm. This method improves the generalization and accuracy of target detection, protects data privacy and security, and ensures high-precision and robust target detection performance during aircraft panel assembly.

Description

Technical Field

[0001] The present invention relates to computer vision and distributed machine learning technologies, and in particular to a multi-robot target detection method for aircraft panel assembly. Background Art

[0002] With the rapid development of intelligent manufacturing and industrial automation, robots are increasingly being used across various industrial sectors. In particular, robots are being widely used to improve production efficiency and assembly precision in complex manufacturing processes such as aircraft panel assembly. To achieve efficient automated assembly, object detection technology is crucial, helping robots accurately identify and locate various objects required for assembly, such as long hengs, bulkheads, and corner pieces, ensuring a smooth assembly process.

[0003] However, existing datasets for aircraft panel assembly object detection face the dual challenges of high confidentiality and insufficient data. The aviation manufacturing industry places extremely high demands on data confidentiality, making relevant data difficult to obtain and share. Furthermore, the limited number and diversity of existing datasets make it difficult to meet the requirements of deep learning models for large amounts of high-quality training data, hindering the performance improvement of object detection models.

[0004] In the specific application scenario of aircraft panel assembly, existing technologies still face challenges with high storage requirements and strict privacy protection requirements. High-precision object detection models typically require the storage of large amounts of parameters and training data, but industrial site storage resources are limited, making efficient management and access difficult. Furthermore, centralized data processing methods are prone to sensitive information leakage and cannot meet the strict data privacy and security requirements of aviation manufacturers.

[0005] Furthermore, existing robotic systems lack the ability to continuously evolve and learn after deployment, preventing them from optimizing and updating themselves based on the ever-changing field environment and newly acquired data. This makes it difficult for robots to adapt and improve their performance when faced with new assembly tasks or environmental changes, limiting their potential and flexibility in intelligent manufacturing.

[0006] In summary, there is an urgent need for a multi-robot target detection method that can effectively solve the problems of data confidentiality, data shortage, large storage requirements, privacy protection, and continuous learning capabilities to improve the automation and intelligence level of aircraft panel assembly. Summary of the Invention

[0007] The present invention aims to address the existing problems of high confidentiality of target detection data, insufficient data, large storage requirements, strict privacy protection, and lack of continuous learning ability of robots during aircraft panel assembly. A multi-robot target detection method for aircraft panel assembly is provided, specifically as follows:

[0008] Step 1: Build a multi-source dataset

[0009] The present invention first constructs a multi-source dataset, including:

[0010] 3D physical object dataset: SolidWorks was used to design and assemble 3D models of the long heng, bulkhead, and corner piece used in the aircraft panel assembly, ensuring the correct shape and hole placement. These components were then 3D printed and spray-coated with orange metallic powder to recreate the authentic texture. Finally, the 3D physical object dataset was constructed by capturing images from multiple angles and under various lighting conditions, and annotating the images using the X-AnyLabeling tool to generate annotation boxes.

[0011] Virtual dataset: The 3D models of the designed long heng, partition frame, and corner piece were imported into Blender software. Then, operations such as adding materials, randomly placing models, and randomly setting lighting and camera positions were performed in the software. The virtual scene was then rendered and the corresponding annotation boxes were automatically generated to obtain the virtual dataset.

[0012] Web crawler dataset: A dataset of common objects in aircraft panel assembly, such as rivets, screws, and nuts, was obtained through web crawlers. The X-AnyLabeling tool was then used to annotate the images and generate annotation boxes.

[0013] By integrating the above three datasets, a multi-source dataset was constructed.

[0014] Step 2: Dataset allocation

[0015] The multi-source dataset is distributed to multiple local robots according to a specific method to ensure that the dataset of each robot is not independent and identically distributed. The specific distribution method is as follows:

[0016] Robot 1: Phillips screws, hexagonal nuts, real blind rivets, real long hones, real bulkheads, and real corner pieces.

[0017] Robot 2: Hexagon socket screws, square nuts, crawler rivets, rendered long hon, rendered bulkhead and rendered corner pieces.

[0018] Robot 3: Self-tapping screws, wing nuts, real solid rivets, rendered long hon, rendered bulkhead, and rendered corner piece. Through this distribution method, each robot's dataset obtains data with the same labels but different categories.

[0019] Step 3: Global model training based on federated learning

[0020] Each local robot uses YOLOv8 as the local target detection model and performs global model training through the federated learning algorithm:

[0021] (1) Each local robot trains the YOLOv8 model based on its local dataset and calculates the local model parameters.

[0022] (2) Upload the model parameters of multiple local robots to the global server and calculate the global model parameters through the federated averaging algorithm.

[0023] (3) Distribute the updated global model parameters to each local robot for the next round of training until the training is completed.

[0024] Step 4: Target detection and result determination

[0025] In the aircraft panel assembly scenario, multiple robots use the trained target detection model and combine the robot and camera posture algorithms to perform target detection and output object category, location, and confidence level.

[0026] When at least two robots have the same detection results for an object, the detection results of the object are used as incremental learning data. The specific judgment method is as follows:

[0027] (1) Each local robot transforms its detected object positions (category, position, and confidence) into positions in the global coordinate system.

[0028] (2) If the detection results of two or more robots for the same object are consistent, and the overlap and confidence levels meet the threshold requirements, the detection results are considered consistent.

[0029] Step 5: Model update based on federated incremental learning

[0030] When at least two robots have consistent detection results for an object, and the overlap and confidence levels meet the threshold requirements, the detection results of the object are used as incremental learning data. The specific process is as follows:

[0031] (1) Each local robot uses its incremental learning dataset to train the YOLOv8 model, and introduces a regularization term in the local loss function to limit the deviation between the local model parameters and the global model parameters. i The local model loss function of each robot L ( W i ) is:

[0032] L ( W i )= L ( W i , D i )+λ( W i -W ) 2 / 2;

[0033] in, W i Indicates the i The local model of the robot, W is the current global model parameter, L ( W i , D i ) indicates the i The loss function of the robot YOLOv8, D i Indicates the i A dataset of robots, λ is a regularization hyperparameter;

[0034] (2) The global server performs weighted averaging on the model parameters uploaded by multiple local robots and updates the global model parameters.

[0035] (3) Distribute the updated global model parameters to each local robot for the next round of training until the federated incremental learning process is completed.

[0036] Compared with the prior art, the beneficial effects are as follows:

[0037] This invention successfully overcomes the challenges of high data confidentiality, insufficient data, and high storage requirements in existing technologies by constructing and rationally distributing multi-source datasets, combining the YOLOv8 object detection model with a federated learning algorithm. Specifically, by utilizing 3D printing and virtual rendering technologies, rich and diverse training data is generated, significantly improving the training effectiveness and generalization capabilities of the object detection model. Furthermore, the use of a federated learning algorithm enables distributed model training, effectively protecting data privacy and security and avoiding the potential information leakage risks associated with centralized data processing.

[0038] Furthermore, the introduction of a federated incremental learning mechanism enables the robotic system to continuously learn and self-optimize. By ensuring consistency across multiple robots, the accuracy and reliability of incremental learning data are ensured, thereby continuously improving the precision and robustness of the global object detection model. This not only enhances the robot's recognition capabilities and efficiency in complex assembly scenarios, but also strengthens the system's adaptability to environmental changes and new tasks, significantly advancing the automation and intelligence of aircraft panel assembly.

[0039] In summary, the multi-robot target detection method and system provided by the present invention can build an efficient and secure target detection model under the conditions of high confidentiality and insufficient data, and realize the intelligent evolution of the system through continuous incremental learning. It has significant technical advantages and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the implementation of the present invention or the existing technical solutions, the following is a brief introduction to the drawings required for the embodiments or the description of the existing technology. In the drawings:

[0041] Figure 1 Schematic diagram of the overall flow of the multi-robot target detection method of the present invention.

[0042] Figure 2 This is a schematic diagram of the modeling and assembly process using SolidWorks in the construction of a 3D physical data set.

[0043] Figure 3 Schematic diagram of 3D printing and spraying orange metal powder in the construction of a 3D physical data set.

[0044] Figure 4 Images taken and annotated for the construction of a 3D object dataset.

[0045] Figure 5 Schematic diagram of generating a dataset for Blender rendering in Virtual Dataset Generation.

[0046] Figure 6 Schematic diagram of the data set distribution.

[0047] Figure 7 Assign a quantity to the dataset.

[0048] Figure 8 Schematic diagram of the division of training set and test set.

[0049] Figure 9 Schematic diagram of federated learning.

[0050] Figure 10 The changes in accuracy, precision, and recall during the training process. DETAILED DESCRIPTION

[0051] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts are within the scope of protection of the present invention.

[0052] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0053] like Figure 1 As shown, the present invention discloses a multi-robot target detection method for aircraft panel assembly, comprising the following steps:

[0054] S101: Design and create a 3D physical dataset, use Blender to render and generate a virtual dataset, and obtain other common object datasets through web crawlers to build a multi-source dataset;

[0055] S102: Distribute the multi-source dataset to multiple local robots according to a certain method, so that the dataset of each robot is not independent and identically distributed;

[0056] S103: Each local robot uses YOLOv8 as the local object detection model, and a global model training is performed on multiple local robots through the federated learning algorithm;

[0057] S104: In the aircraft panel assembly scenario, multiple robots use the trained object detection model and the robot and camera pose algorithms to perform object detection and output object category, location, and confidence level.

[0058] S105: When the detection results of at least two robots for a certain object are consistent, the detection results of the object are used as incremental learning data, and the global object detection model is continuously updated through the federated incremental learning algorithm.

[0059] The method for constructing the data set in step S101 is as follows:

[0060] 1. Construction of three-dimensional physical data set.

[0061] SolidWorks is used to perform detailed 3D modeling of key objects such as long heng, bulkhead and corner piece in aircraft wall panel assembly, and complete the virtual assembly of parts to ensure the correctness of hole alignment and structure, such as Figure 2 Subsequently, these parts were printed using 3D printing technology, and orange metal powder was sprayed on their surfaces to restore the real texture, as shown in the figure. Figure 3 Finally, by shooting at multiple angles and in multiple lighting conditions and using the X-AnyLabeling tool to annotate the target detection frame on the image, a 3D object dataset with annotated frames is generated, as shown in Figure 4 shown.

[0062] 2. Virtual dataset generation.

[0063] Import the designed 3D models of the long heng, partition frame and corner piece into Blender software, and then render and generate a virtual scene by adding materials, randomly placing models, and randomly setting lights and camera positions in the software. At the same time, the corresponding annotation boxes are automatically generated to form a virtual data set, such as Figure 5 shown.

[0064] 3. Web crawler to obtain the dataset.

[0065] We use web crawler technology to crawl image datasets of common objects in aircraft panel assembly, such as rivets, screws, and nuts. We then use the X-AnyLabeling tool to annotate the images and generate a dataset of common objects with annotated boxes.

[0066] In the data distribution method in step S102, this embodiment takes a total of 3 robots as an example, and the data distribution and quantity of each robot are as follows: Figure 6 and Figure 7 As shown below:

[0067] Robot 1: Contains Phillips screws, hexagonal nuts, blind rivets, physical long hon, physical bulkhead, and physical corner pieces. The data volume of each type of object is about 200;

[0068] Robot 2: Contains hexagon socket screws, square nuts, crawler rivets, rendered long hon, rendered bulkhead, and rendered corner pieces. The rivet data volume exceeds 300, and the other types of samples range from 100 to 250.

[0069] Robot 3: Contains self-tapping screws, wing nuts, solid rivets, rendered long hon, rendered bulkheads and rendered corner pieces. The number of samples of each type of object ranges from 150 to 250.

[0070] By using the above distribution method, we ensure that each robot's dataset obtains data with the same labels but different categories, thus achieving non-independent and identically distributed data. Next, we randomly extract 80% of the data as the local training set for each client, and combine the remaining 20% ​​of the data from each dataset as the test set for each client, as shown in the following example: Figure 8 shown.

[0071] The federated incremental learning algorithm in step S103 is as follows: Figure 9As shown in the figure, the specific steps include the following: First, each local robot uses the YOLOv8 model to perform object detection training based on its local dataset and calculates the corresponding local model parameters. Next, each local robot uploads its trained model parameters to the global server. The server uses the federated averaging algorithm to perform a weighted average of the model parameters of multiple local robots to calculate the global model parameters. Subsequently, the updated global model parameters are distributed to each local robot for the next round of training. This process is repeated until the global model is fully trained.

[0072] The performance changes of each client during training are as follows: Figure 10 As shown in the figure, after the federated learning model uses the federated averaging algorithm for global aggregation, all indicators show a steady and gradually increasing trend, ultimately demonstrating good convergence. Compared with the model without federated averaging, the federated averaging model maintains higher indicator values ​​throughout the training process and significantly outperforms the baseline model without aggregation after convergence, demonstrating the advantages of federated learning for non-IID data.

[0073] The target detection process in step S104 includes:

[0074] The robots use the trained global object detection model to capture images of the assembly scene through cameras, and then perform object detection in combination with their own pose algorithms, outputting the object's category, location, and confidence level. During the pose conversion process, each local robot converts the detected detection box into a position in the global coordinate system based on the pose information of its camera, ensuring that robots from different positions and perspectives can uniformly compare the same object. The mathematical expression for pose conversion is as follows:

[0075] P global = R · P local + T ;(1)

[0076] in, P global Represents the global coordinate position after transformation, P local Indicates the location of the local detection box, R is the rotation matrix, T is the translation variable.

[0077] The incremental learning process in step S105 includes:

[0078] Through the consistency judgment algorithm, when two or more robots have consistent detection results for the same object, and the overlap and confidence of the detection results meet the preset threshold requirements, the detection results of the object are deemed consistent. Specifically, if the overlap and confidence of the detection results are greater than the respective given thresholds, the detection results of the object are deemed consistent. Subsequently, the consistent detection results are used as incremental learning data for use by each robot. Each local robot uses the incremental learning dataset to train the YOLOv8 model, and introduces a regularization term in the model's loss function to limit the deviation between the local model parameters and the global model parameters. The improved i The local model loss function of each robot L ( W i ) is:

[0079] L ( W i )= L ( W i , D i )+λ( W i - W ) 2 / 2; (2)

[0080] in, W i Indicates the i The local model of the robot, W is the current global model parameter, L ( W i , D i ) indicates the i The loss function of the robot YOLOv8, D i Indicates the i A dataset of robots, λ is a regularization hyperparameter;

[0081] The local robot uploads the updated model parameters to the global server. The server updates the global model parameters through a weighted average algorithm and distributes them to each local robot until the federated incremental learning process is completed.

[0082] The method of the present invention improves the generalization and accuracy of target detection by constructing multi-source datasets and performing non-independent and identically distributed (NIID) distribution. Federated learning is used to ensure data privacy and security, enabling efficient model training under multi-robot collaboration. Furthermore, the federated incremental learning mechanism enables the global model to dynamically adapt to changes in the assembly environment, ensuring high-precision and robust target detection performance during the complex and dynamic aircraft panel assembly process. This method is suitable for high-precision target detection in complex assembly environments, exhibits good scalability and adaptability, and significantly improves the automation and intelligence level of the assembly process.

Claims

1. A multi-robot target detection method for aircraft panel assembly, characterized in that: The following steps are involved: (1) Design and produce a 3D physical dataset, use Blender to render and generate a virtual dataset, and obtain other common object datasets through web crawlers to construct a multi-source dataset; (2) Distributing the multi-source datasets to multiple local robots so that the dataset of each robot is not independent and identically distributed; (3) Each local robot uses YOLOv8 as the local target detection model, and a global model is trained on multiple local robots through a federated learning algorithm; (4) In the aircraft panel assembly scenario, multiple robots use the trained target detection model and the pose algorithm of the robot and camera to perform target detection and output the object category, location and confidence level; (5) When the detection results of at least two robots for a certain object are consistent, the detection results of the object are used as incremental learning data, and the global object detection model is continuously updated through the federated incremental learning algorithm; The federated incremental learning algorithm is performed through the following steps: (1) Each local robot trains the YOLOv8 model based on its local data and calculates the local model parameters; (2) Upload the model parameters of multiple local robots to the global server and calculate the global model parameters through the federated averaging algorithm; (3) Distribute the updated global model parameters to each local robot for the next round of training until the training is completed; The method for determining whether the detection results are consistent comprises the following steps: (1) The object location detected by each local robot through the YOLOv8 model includes object category, location, and confidence; (2) Each local robot converts the detection frame it detects into a position in the global coordinate system based on the pose information of its camera, ensuring that robots from different positions and different perspectives can uniformly compare the same object; (3) If the detection results of two or more robots for the same object are consistent, and the overlap and confidence meet the threshold requirements, the detection results of the object are considered consistent, and the detection results will be used as the dataset for incremental learning of each robot for further training of the global model; The federated incremental learning includes the following steps: (1) Each local robot uses its incremental learning dataset to train the YOLOv8 model, and introduces a regularization term into the model’s loss function to limit the deviation between the local model parameters and the global model parameters. The improved local model loss function L(w i ) is: Among them, w i represents the local model of the i-th robot, w is the current global model parameter, represents the loss function of the i-th robot YOLOv8, represents the dataset of the i-th robot, and λ is the regularization hyperparameter; (2) On the global server side, after receiving the model parameters uploaded by multiple local robots, a weighted average is performed to update the global model parameters; (3) Distribute the updated global model parameters to each local robot for the next round of training until the federated incremental learning process is completed.

2. A multi-robot target detection method for aircraft panel assembly according to claim 1, characterized in that: The 3D physical object dataset was created using SolidWorks to design 3D models of the long heng, bulkhead, and corner piece used in the aircraft panel assembly. The models were then assembled to ensure the correct design of the shapes and hole positions. These components were then printed using 3D printing technology and sprayed with orange metal powder to restore their authentic texture. Finally, the images were photographed from multiple angles and under various lighting conditions, and the X-AnyLabeling tool was used to annotate the images and generate annotation boxes.

3. The multi-robot target detection method for aircraft panel assembly according to claim 2, characterized in that: The virtual dataset is generated by importing the three-dimensional models of the designed long heng, partition frame and corner piece into Blender software, and using the operations of adding materials, randomly placing models, randomly setting lights and camera positions in the software to render and generate a virtual scene, and automatically generate corresponding annotation boxes.

4. The multi-robot target detection method for aircraft panel assembly according to claim 1, characterized in that: Other common object datasets obtained by the web crawler include rivets, screws, and nuts, which are common objects in aircraft panel assembly. The X-AnyLabeling tool is then used to annotate the images and generate annotation boxes.

5. The multi-robot target detection method for aircraft panel assembly according to claim 1, characterized in that: The multi-source data set is distributed to at least three local robots according to a certain method, and the specific distribution method includes: Robot 1: Contains Phillips screws, hexagonal nuts, real blind rivets, real long honing, real bulkheads and real corner pieces; Robot 2: Contains hexagon socket screws, square nuts, crawler rivets, rendering long hon, rendering bulkhead and rendering corner pieces; Robot 3: Contains self-tapping screws, wing nuts, real solid rivets, rendering long hon, rendering bulkhead and rendering corner pieces; The above allocation method allows each robot's dataset to obtain data with the same label but different categories.

Images