Multi-robot target detection method for aircraft panel assembly

Abstract

The invention provides a multi-robot target detection method for aircraft panel assembly, which comprises the following steps: constructing a multi-source data set, distributing the multi-source data set to a plurality of local robots, enabling the data set of each robot to be a non-independent identically-distributed data set, and enabling each local robot to adopt YOLOv8 as a target detection model, performing global model training through a federal learning algorithm, performing object detection by a plurality of robots in combination with a target detection model and a pose algorithm in an aircraft panel assembly scene, outputting object types, positions and confidence coefficients, and when detection results of at least two robots on the same object are consistent, determining that the detection results are used as incremental learning data, and the global target detection model is continuously updated through a federal incremental learning algorithm. According to the method, the generalization ability and accuracy of target detection are improved, data privacy and safety are guaranteed, and it is ensured that high-precision and robust target detection performance is kept in the aircraft panel assembly process.

Description

Technical Field

[0001] The invention relates to computer vision and distributed machine learning technology, and in particular to a multi-robot target detection method for aircraft panel assembly. Background Art

[0002] With the rapid development of intelligent manufacturing and industrial automation, robots are increasingly used in various industrial fields, especially in complex manufacturing processes such as aircraft panel assembly, where robots are widely used to improve production efficiency and assembly accuracy. In order to achieve efficient automated assembly, target detection technology becomes the key to help robots accurately identify and locate various objects required for assembly, such as long heng, bulkheads and corner pieces, thereby ensuring the smooth progress of the assembly process.

[0003] However, existing aircraft panel assembly target detection datasets face the dual challenges of high confidentiality and insufficient data. The aviation manufacturing industry has extremely high requirements for data confidentiality, which makes it difficult to obtain and share relevant data. In addition, the number and diversity of existing datasets are limited, which makes it difficult to meet the needs of deep learning models for large amounts of high-quality training data, restricting the performance improvement of target detection models.

[0004] In the specific application scenario of aircraft panel assembly, existing technologies still have problems with large storage requirements and high privacy protection requirements. High-precision target detection models usually need to store a large amount of parameters and training data, but the storage resources at industrial sites are limited and difficult to manage and call efficiently. In addition, centralized data processing methods are prone to sensitive information leakage and cannot meet the strict requirements of aviation manufacturing companies for data privacy and security.

[0005] In addition, existing robot systems lack the ability to continuously evolve and learn after deployment, and are unable to optimize and update themselves according to the changing environment and newly acquired data on site. This makes it difficult for robots to adapt and improve their performance when faced with new assembly tasks or environmental changes, limiting their application potential and flexibility in intelligent manufacturing.

[0006] In summary, there is an urgent need for a multi-robot target detection method that can effectively solve the problems of data confidentiality, data insufficiency, large storage requirements, privacy protection, and continuous learning capabilities to improve the level of automation and intelligence in aircraft panel assembly. Summary of the invention

[0007] The present invention aims to solve the problems of high confidentiality of target detection data, insufficient data, large storage requirements, strict privacy protection and lack of continuous learning ability of robots in the existing aircraft panel assembly process, and provides a multi-robot target detection method for aircraft panel assembly, which is as follows: Step 1: Build a multi-source dataset The present invention first constructs a multi-source data set, including: 3D physical data set: SolidWorks was used to design the 3D models of the long heng, bulkhead and corner piece in the aircraft wall panel assembly, and the assembly was completed to ensure the correctness of the shape and hole design. After that, these parts were printed using 3D printing technology, and orange metal powder was sprayed on the surface to restore the real texture. Finally, the 3D physical data set was constructed by shooting at multiple angles and multiple lighting conditions, and using the X-AnyLabeling tool to annotate the images and generate annotation boxes.

[0008] Virtual data set: The three-dimensional models of the designed long heng, partition frame and corner piece are imported into Blender software, and operations such as adding materials, randomly placing models, and randomly setting lighting and camera positions are performed in this software, and then a virtual scene is rendered and generated, and the corresponding annotation box is automatically generated, thereby obtaining a virtual data set.

[0009] Web crawler dataset: A dataset of common objects in aircraft panel assembly, such as rivets, screws, and nuts, is obtained through web crawlers. The X-AnyLabeling tool is then used to annotate the images and generate annotation boxes.

[0010] By integrating the above three datasets, a multi-source dataset was constructed.

[0011] Step 2: Dataset allocation The multi-source datasets are distributed to multiple local robots in a specific way to ensure that the datasets of each robot are not independent and identically distributed datasets. The specific distribution method is as follows: Robot 1: Phillips screws, hexagonal nuts, real blind rivets, real long hones, real bulkheads and real corner pieces.

[0012] Robot 2: Hexagon socket screws, square nuts, crawler rivets, rendering long hon, rendering bulkhead and rendering corner pieces.

[0013] Robot 3: self-tapping screws, wing nuts, real solid rivets, rendered long hon, rendered bulkhead, and rendered corner piece. With this distribution, each robot’s dataset gets data with the same labels but different categories.

[0014] Step 3: Global model training based on federated learning Each local robot uses YOLOv8 as the local target detection model and performs global model training through the federated learning algorithm: (1) Each local robot trains the YOLOv8 model based on its local dataset and calculates the local model parameters.

[0015] (2) Upload the model parameters of multiple local robots to the global server and calculate the global model parameters through the federated averaging algorithm.

[0016] (3) Distribute the updated global model parameters to each local robot for the next round of training until the training is completed.

[0017] Step 4: Target detection and result determination In the aircraft panel assembly scenario, multiple robots use the trained target detection model and combine the robot and camera posture algorithm to perform target detection and output object category, location and confidence.

[0018] When at least two robots have the same detection results for an object, the detection results of the object are used as incremental learning data. The specific determination method is as follows: (1) Each local robot transforms the position of the object it detects (category, position, and confidence) into a position in the global coordinate system.

[0019] (2) If the detection results of two or more robots for the same object are consistent, and the overlap and confidence levels meet the threshold requirements, the detection results are considered consistent.

[0020] Step 5: Model update based on federated incremental learning When at least two robots have consistent detection results for an object, and the overlap and confidence meet the threshold requirements, the detection results of the object are used as incremental learning data. The specific process is as follows: (1) Each local robot uses its incremental learning dataset to train the YOLOv8 model and introduces a regularization term in the local loss function to limit the deviation between the local model parameters and the global model parameters. i The local model loss function of each robot L ( W i ) is: L ( W i )= L ( W i , D i )+λ( W i - W ) 2 / 2; in, W i Indicates i The local model of the robot, W is the current global model parameter, L (W i , D i ) indicates the i The loss function of YOLOv8 for each robot, D i Indicates i The dataset is a robot dataset, and λ is a regularization hyperparameter; (2) The global server performs weighted averaging on the model parameters uploaded by multiple local robots and updates the global model parameters.

[0021] (3) Distribute the updated global model parameters to each local robot for the next round of training until the federated incremental learning process is completed.

[0022] Compared with the prior art, the beneficial effects are as follows: The present invention successfully overcomes the problems of high data confidentiality, insufficient data and large storage requirements in the prior art by constructing and reasonably allocating multi-source data sets, combining the YOLOv8 target detection model and the federated learning algorithm. Specifically, by using 3D printing and virtual rendering technology, rich and diverse training data are generated, which greatly improves the training effect and generalization ability of the target detection model. At the same time, the federated learning algorithm is used to realize distributed model training, effectively protecting the privacy and security of data and avoiding the risk of information leakage that may be caused by centralized data processing.

[0023] In addition, the introduction of the federated incremental learning mechanism enables the robot system to have the ability to continuously learn and self-optimize. The accuracy and reliability of incremental learning data are ensured through consistency judgment among multiple robots, thereby continuously improving the accuracy and robustness of the global target detection model. This not only improves the robot's recognition ability and work efficiency in complex assembly scenarios, but also enhances the system's adaptability to environmental changes and new tasks, significantly improving the automation and intelligence level of aircraft panel assembly.

[0024] In summary, the multi-robot target detection method and system provided by the present invention can build an efficient and secure target detection model under the conditions of high confidentiality and insufficient data, and realize the intelligent evolution of the system through continuous incremental learning. It has significant technical advantages and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the implementation of the present invention or the existing technical solutions, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. In the drawings: Figure 1 It is a schematic diagram of the overall flow of the multi-robot target detection method of the present invention.

[0026] Figure 2 This is a schematic diagram of modeling and assembly using SolidWorks in the construction of a 3D physical data set.

[0027] Figure 3 Schematic diagram of 3D printing and spraying orange metal powder in the construction of a 3D physical data set.

[0028] Figure 4 Images taken and annotated for the construction of a 3D object dataset.

[0029] Figure 5 Diagram of generating a dataset for Blender rendering in Virtual Dataset Generation.

[0030] Figure 6 Schematic diagram of the data set allocation.

[0031] Figure 7 Assign quantities to the dataset.

[0032] Figure 8 This is an illustration of the division of training set and test set.

[0033] Fig. 9 Schematic diagram of federated learning.

[0034] Fig.10 It is the changes of accuracy, precision and recall during the training process. DETAILED DESCRIPTION

[0035] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0037] like Figure 1 As shown, the present invention discloses a multi-robot target detection method for aircraft panel assembly, comprising the following steps: S101: Design and create 3D physical datasets, use Blender to render and generate virtual datasets, and obtain other common object datasets through web crawlers to build multi-source datasets; S102: Distribute the multi-source data sets to multiple local robots in a certain method, so that the data set of each robot is a non-independent and identically distributed data set; S103: Each local robot uses YOLOv8 as a local target detection model, and performs global model training on multiple local robots through a federated learning algorithm; S104: In the aircraft panel assembly scenario, multiple robots use the trained target detection model and the pose algorithm of the robot and the camera to perform target detection and output the object category, position and confidence level; S105: When the detection results of at least two robots for a certain object are consistent, the detection results of the object are used as incremental learning data, and the global target detection model is continuously updated through the federated incremental learning algorithm.

[0038] The method for constructing the data set in step S101 is as follows: 1. Construction of three-dimensional physical data set.

[0039] SolidWorks is used to perform detailed 3D modeling of key objects such as long heng, bulkhead and corner piece in aircraft wall panel assembly, and complete the virtual assembly of parts to ensure the alignment of holes and the correctness of structure, such as Figure 2 Then, these parts were printed using 3D printing technology, and orange metal powder was sprayed on their surfaces to restore the real texture, as shown in the figure below. Figure 3 Finally, by shooting at multiple angles and in multiple lighting conditions, and using the X-AnyLabeling tool to annotate the target detection frame on the image, a 3D physical object dataset with annotated frames is generated, as shown in Figure 4 shown.

[0040] 2. Virtual dataset generation.

[0041] Import the designed 3D models of the long heng, partition frame and corner piece into Blender software, and render and generate a virtual scene by adding materials, randomly placing models, randomly setting lights and camera positions, and automatically generating corresponding annotation boxes to form a virtual data set, such as Figure 5 shown.

[0042] 3. Web crawler to obtain data set.

[0043] Web crawler technology is used to crawl image datasets of common objects in aircraft panel assembly, such as rivets, screws, and nuts. The X-AnyLabeling tool is then used to annotate the images and generate a dataset of common objects with annotation boxes.

[0044] In the data distribution method in step S102, the present embodiment takes a total of 3 robots as an example, and the data distribution and quantity of each robot are as follows: Figure 6 and Figure 7 As shown below: Robot 1: Contains cross screws, hexagonal nuts, blind rivets, real long hon, real bulkhead and real corner piece, and the data volume of each type of object is about 200; Robot 2: Contains hexagon socket screws, square nuts, crawler rivets, rendered long hon, rendered bulkheads, and rendered corner pieces. The rivet data volume exceeds 300, and the other types of samples are between 100 and 250. Robot 3: Contains self-tapping screws, wing nuts, solid rivets, rendered long hon, rendered bulkheads and rendered corner pieces. The number of samples of each type of object ranges from 150 to 250.

[0045] Through the above allocation method, it is ensured that the data set of each robot obtains data with the same label but different categories, so as to achieve non-independent and identically distributed data. Next, 80% of the data is randomly selected as the local training set of each client, and the remaining 20% ​​of the data of each data set is combined as the test set of each client, such as Figure 8 shown.

[0046] The federated incremental learning algorithm in step S103 is as follows: Fig. 9 As shown in the figure, the specific steps include: first, each local robot uses the YOLOv8 model to perform target detection training based on its local data set and calculates the corresponding local model parameters; then, each local robot uploads its trained model parameters to the global server, and the server uses the federated average algorithm to perform weighted average on the model parameters of multiple local robots to calculate the global model parameters; then, the updated global model parameters are distributed to each local robot for the next round of training. This process will be repeated until the global model is trained.

[0047] The performance changes of each client during training are as follows: Fig.10 As shown in the figure. After the federated learning model uses the federated average algorithm for global aggregation, all indicators show a steady and gradually rising trend, and finally show good convergence. Compared with the model without the federated average, the federated average model maintains higher indicator values ​​throughout the training process, and significantly leads the baseline model without aggregation after convergence, showing the advantages of federated learning under non-independent and identically distributed data.

[0048] The target detection process in step S104 includes: The robot uses the trained global target detection model to capture images in the assembly scene through the camera, combines their own pose algorithms to perform target detection, and outputs the category, position, and confidence of the object. During the pose conversion process, each local robot converts the detected detection frame into a position in the global coordinate system based on the pose information of its camera, ensuring that robots from different positions and perspectives can uniformly compare the same object. The mathematical expression of pose conversion is as follows: P global = R · P local + T ; (1) in, P global represents the global coordinate position after transformation, P local Indicates the location of the local detection box, R is the rotation matrix, T is the translation variable.

[0049] The incremental learning process in step S105 includes: Through the consistency judgment algorithm, when two or more robots have consistent detection results for the same object, and the overlap and confidence of the detection results meet the preset threshold requirements, the detection results of the object are deemed consistent. Specifically, if the overlap and confidence of the detection results are greater than the respective given thresholds, the detection results of the object are deemed consistent. Subsequently, the consistent detection results are used as incremental learning data for each robot to use. Each local robot uses the incremental learning dataset to train the YOLOv8 model, and introduces a regularization term in the model's loss function to limit the deviation between the local model parameters and the global model parameters. The improved i The local model loss function of each robot L ( W i ) is: L ( W i )= L ( W i , D i )+λ( W i - W ) 2 / 2; (2) in, W i Indicates i The local model of the robot, Wis the current global model parameter, L ( W i , D i ) indicates the i The loss function of YOLOv8 for each robot, D i Indicates i The dataset is a robot dataset, and λ is a regularization hyperparameter; The local robot uploads the updated model parameters to the global server. The server updates the global model parameters through a weighted average algorithm and distributes them to each local robot until the federated incremental learning process is completed.

[0050] The method of the present invention improves the generalization ability and accuracy of target detection by constructing multi-source data sets and performing non-independent and identically distributed distribution. Federated learning is used to ensure data privacy and security, and efficient model training under multi-robot collaboration is achieved. At the same time, the federated incremental learning mechanism enables the global model to dynamically adapt to changes in the assembly environment, ensuring high-precision and robust target detection performance in the complex and dynamic aircraft panel assembly process. The method is suitable for high-precision target detection in complex assembly environments, has good scalability and adaptability, and significantly improves the automation and intelligence level of the assembly process.

Claims

1. A multi-robot target detection method for aircraft panel assembly, characterized in that: The following steps are involved: (1) Design and produce a 3D physical dataset, use Blender to render and generate a virtual dataset, and use web crawlers to obtain other common object datasets to construct a multi-source dataset. (2) Distribute the above multi-source datasets to multiple local robots, so that the dataset of each robot is not independent and identically distributed; (3) Each local robot uses YOLOv8 as the local target detection model, and a global model is trained for multiple local robots through a federated learning algorithm; (4) In the aircraft panel assembly scenario, multiple robots use the trained target detection model and the pose algorithm of the robot and camera to perform target detection and output the object category, location and confidence level. (5) When the detection results of at least two robots for an object are consistent, the detection results of the object are used as incremental learning data, and the global object detection model is continuously updated through the federated incremental learning algorithm.

2. A multi-robot target detection method for aircraft panel assembly according to claim 1, characterized in that: The three-dimensional physical data set uses SolidWorks to design the three-dimensional models of the long heng, bulkhead and corner piece in the aircraft wall panel assembly, and completes the assembly to ensure the design correctness of the shape and hole position. Then, these parts are printed using 3D printing technology, and orange metal powder is sprayed on the surface to restore their real texture. Finally, they are photographed from multiple angles and multiple lighting conditions, and the X-AnyLabeling tool is used to annotate the images to generate annotation boxes.

3. A multi-robot target detection method for aircraft panel assembly according to claim 2, characterized in that: The virtual data set is generated by importing the three-dimensional models of the designed long heng, partition frame and corner piece into Blender software, and using the operations of adding materials, randomly placing models, randomly setting lights and camera positions in the software to render and generate a virtual scene, and automatically generate a corresponding annotation box.

4. The multi-robot target detection method for aircraft panel assembly according to claim 1, characterized in that: Other common object datasets obtained by the web crawler include rivets, screws and nuts, which are common objects in aircraft panel assembly. The X-AnyLabeling tool is then used to annotate the images and generate annotation boxes.

5. The multi-robot target detection method for aircraft panel assembly according to claim 1 is characterized in that: The multi-source data set is distributed to at least three local robots according to a certain method, and the specific distribution method includes: Robot 1: includes cross screws, hexagonal nuts, real blind rivets, real long hon, real bulkhead and real corner pieces; Robot 2: includes hexagon socket screws, square nuts, crawler rivets, rendering long hon, rendering bulkhead and rendering corner pieces; Robot 3: includes self-tapping screws, wing nuts, real solid rivets, rendering long hon, rendering bulkhead and rendering corner pieces; Through the above allocation method, the dataset of each robot obtains data with the same label but different categories.

6. The multi-robot target detection method for aircraft panel assembly according to claim 1, characterized in that: The federated incremental learning algorithm is performed through the following steps: (1) Each local robot trains the YOLOv8 model based on its local data and calculates the local model parameters; (2) Upload the model parameters of multiple local robots to the global server and calculate the global model parameters through the federated averaging algorithm; (3) Distribute the updated global model parameters to each local robot for the next round of training until the training is completed.

7. The multi-robot target detection method for aircraft panel assembly according to claim 1, characterized in that: The method for determining whether the detection results are consistent comprises the following steps: (1) The object location detected by each local robot through the YOLOv8 model includes object category, location, and confidence; (2) Each local robot converts the detection box it detects into a position in the global coordinate system based on the pose information of its camera, ensuring that robots from different positions and perspectives can uniformly compare the same object; (3) If the detection results of two or more robots for the same object are consistent, and the overlap and confidence meet the threshold requirements, the detection results of the object are deemed consistent, and the detection results will be used as the data set for incremental learning of each robot for further training of the global model.

8. The multi-robot target detection method for aircraft panel assembly according to claim 1, characterized in that: The federated incremental learning includes the following steps: (1) Each local robot uses its incremental learning dataset to train the YOLOv8 model and introduces a regularization term in the model’s loss function to limit the deviation between the local model parameters and the global model parameters. i The local model loss function of each robot L ( W i ) is: L ( W i )= L ( W i , D i )+λ( W i - W ) 2 / 2; in, W i Indicates i The local model of the robot, W is the current global model parameter, L ( W i , D i ) indicates the i The loss function of YOLOv8 for each robot, D i Indicates i The dataset is a robot dataset, and λ is a regularization hyperparameter; (2) On the global server side, after receiving the model parameters uploaded by multiple local robots, the global model parameters are updated by weighted averaging. (3) Distribute the updated global model parameters to each local robot for the next round of training until the federated incremental learning process is completed.

Images