Mechanical arm path planning method based on civil aviation passenger plane appearance detection and related products

Abstract

The invention discloses a mechanical arm path planning method based on civil aviation passenger plane appearance detection and related products, and belongs to the technical field of civil aviation passenger plane appearance detection. A scanning path of a mechanical arm is simulated through a fuselage three-dimensional digital twinborn model, and the scanning path is dynamically optimized by utilizing a genetic algorithm, so that the scanning path can fully cover the fuselage surface of an appearance detection area of a civil aviation passenger plane, and the condition of missing detection caused by unreasonable path planning is avoided; the feedback value is calculated and compared with the preset feedback threshold value range, the detection effect can be evaluated in real time, the movement speed and light source parameters of the mechanical arm are dynamically adjusted according to the evaluation result, and the detection process is more accurate and flexible; according to the method, the extension trend of the to-be-detected damaged target is predicted by using the long-short-term memory neural network, and the development direction of the damage can be pre-judged in advance, so that a more efficient scanning path is planned, unnecessary detection areas are reduced, and the detection efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of civil airliner appearance detection, and particularly to a robotic arm path planning method and related products based on civil airliner appearance detection. Background Art

[0002] As a core link to ensure flight safety, the technological development of civil airliner appearance detection is crucial for the air transportation industry. Currently, civil airliner appearance detection mainly adopts two methods: traditional manual detection and partial automated detection. Traditional manual detection mainly relies on inspectors using simple tools to conduct a one-by-one close inspection of the aircraft fuselage. This method has significant drawbacks: firstly, the detection efficiency is extremely low. It takes a large amount of time to detect a single civil airliner, seriously restricting the flight turnaround efficiency; secondly, the detection accuracy highly depends on the experience and skill level of the inspectors. There are differences in the judgment criteria among different inspectors, resulting in frequent missed detections and false detections, and it is difficult to effectively ensure flight safety.

[0003] With the continuous development of technology, some automated detection systems have gradually been applied to the field of civil airliner appearance detection. These systems usually use fixed cameras or simple mobile devices to collect images of the aircraft fuselage, and then use image processing technology to analyze the collected images to identify whether there are damages on the fuselage. However, in practical applications, these automated detection systems have exposed a series of obvious defects: the civil airliner fuselage has a complex curved surface structure, and the existing fixed path planning detection systems have limitations in design and are difficult to comprehensively cover these curved surface areas. During the detection process, it is easy to miss some key areas, resulting in potential safety hazards not being discovered in time, posing a serious threat to flight safety; moreover, the detection path and parameters cannot be adjusted in time, resulting in the subsequent detection still using inefficient or inaccurate detection methods, making the overall detection efficiency low and difficult to meet the requirements of high-efficiency, accuracy, and comprehensive coverage for civil airliner appearance detection.

[0004] Therefore, how to achieve high-efficiency, accuracy, and comprehensive coverage of civil airliner appearance detection has become an urgent technical problem for those skilled in the art to solve. Summary of the Invention

[0005] The purpose of the present invention is to provide a robotic arm path planning method and related products based on civil airliner appearance detection to overcome the problems of incomplete coverage and low efficiency of existing detection methods.

[0006] The present invention solves the above technical problems through the following technical solutions: A robotic arm path planning method based on civil airliner appearance detection, comprising the following steps: Obtain the to-be-inspected damage targets in the appearance inspection area of a civil airliner, construct a three-dimensional digital twin model of the civil airliner's fuselage, predict the expansion trend of the to-be-inspected damage targets based on a long short-term memory neural network, and combine the prediction results with the three-dimensional digital twin model of the fuselage to plan the scanning path of the robotic arm; Simulate the scanning path of the robotic arm using the three-dimensional digital twin model of the fuselage, and determine whether the scanning path conflicts with the movement trajectory of obstacles or whether the scanning path does not completely cover the fuselage surface in the appearance inspection area of the civil airliner. If the determination result is yes, dynamically optimize the scanning path through a genetic algorithm until the scanning path does not conflict with the movement trajectory of obstacles and completely covers the fuselage surface in the appearance inspection area of the civil airliner; Detect the appearance inspection area of the civil airliner according to the scanning path, obtain the detection feedback data, calculate the feedback value. When the feedback value is within the preset feedback threshold range, optimize the movement speed and light source parameters of the robotic arm; when the feedback value is lower than the lower limit of the preset feedback threshold range, increase the number of robotic arms for multi-robotic arm collaborative scanning; when the feedback value exceeds the upper limit of the preset feedback threshold range, maintain the current detection state.

[0007] A further improvement of the present invention lies in that: the obtaining of the to-be-inspected damage targets in the appearance inspection area of the civil airliner specifically includes: Collect the scene information data of the appearance inspection area of the civil airliner in real time, and calculate the scene complexity index; Determine whether the scene complexity index is greater than or equal to the preset scene complexity index threshold. If the determination is yes, deploy an adaptive light source array in the appearance inspection area of the civil airliner, and adjust the light source parameters until the scene complexity index is less than the preset scene complexity index threshold; Obtain the fuselage surface data of the appearance inspection area of the civil airliner, generate an RGB-D-thermal image through a fusion algorithm; input the RGB-D-thermal image into a trained model to obtain an image of the area where damage may exist, and locate the to-be-inspected damage targets through a density clustering algorithm.

[0008] A further improvement of the present invention lies in that: the generating of the RGB-D-thermal image through the fusion algorithm is specifically:

[0009] Wherein, is the pixel value at the coordinate (x, y) of the RGB-D-thermal image generated through the fusion algorithm; is the visible light image pixel value; is the lidar depth data; is the infrared thermal imaging data; are the normalization weights of the RGB, depth, and thermal imaging data respectively and 。

[0010] A further improvement of the present invention lies in that: the scene information data includes light intensity, surface complexity, and obstacle density; The specific scene complexity index is:

[0011] And, , ; Wherein, is the scene complexity index; is the time; is the feature vector at time t; is the attenuation function; is the light intensity; is the surface complexity; is the obstacle density; is the weight matrix; is the weight matrix for convolution operation; α and β are weight coefficients and satisfy 0 < α < 1, 0 < β < 1, and α + β = 1; is the constant term bias vector; is the convolution operation.

[0012] A further improvement of the present invention lies in that: the construction of the three-dimensional digital twin model of the civil airliner fuselage specifically includes: Based on the digital twin concept, construct a three-dimensional digital twin model of the civil airliner fuselage; obtain the movement trajectory of the obstacle, and continuously update the changes of the damage target to be inspected and the movement trajectory of the obstacle into the three-dimensional digital twin model of the fuselage.

[0013] A further improvement of the present invention lies in that: the specific feedback value is:

[0014] Wherein, is the feedback value; is the weight coefficient, The values of are all between 0 and 1 and ; T is the detection period; is the scanning speed of the robotic arm at time t; is the detection accuracy at time t; is the false detection rate at time t; is; is the attenuation coefficient, The value of is between 0 and 1.

[0015] A further improvement of the present invention lies in that: the preset feedback threshold range is specifically ; Wherein, is the lower limit of the preset feedback threshold range, specifically:

[0016] is the upper limit of the preset feedback threshold range, specifically:

[0017] Among them, , where i is the order of the detected feedback value, and k is the total number of feedback values detected within the detection period; is the average value of the detected feedback values; is the th detected feedback value.

[0018] The present invention also provides a robotic arm path planning system based on the appearance detection of civil airliners, including: The first module is used to obtain the damage targets to be detected in the appearance detection area of the civil airliner, construct a three-dimensional digital twin model of the fuselage of the civil airliner, predict the expansion trend of the damage targets to be detected based on the long short-term memory neural network, and combine the prediction results and the three-dimensional digital twin model of the fuselage to plan the scanning path of the robotic arm; The second module is used to simulate the scanning path of the robotic arm using the three-dimensional digital twin model of the fuselage, determine whether the scanning path conflicts with the movement trajectory of the obstacle or the scanning path does not completely cover the fuselage surface of the appearance detection area of the civil airliner. If the judgment result is yes, the scanning path is dynamically optimized through the genetic algorithm until the scanning path does not conflict with the movement trajectory of the obstacle and completely covers the fuselage surface of the appearance detection area of the civil airliner; The third module is used to detect the appearance detection area of the civil airliner according to the scanning path, obtain the detection feedback data, calculate the feedback value, optimize the movement speed and light source parameters of the robotic arm when the feedback value is within the preset feedback threshold range; when the feedback value is lower than the lower limit of the preset feedback threshold range, increase the number of robotic arms for multi-robotic arm collaborative scanning; when the feedback value exceeds the upper limit of the preset feedback threshold range, maintain the current detection state.

[0019] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the robotic arm path planning method based on the appearance detection of civil airliners as described above are implemented.

[0020] The present invention also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the robotic arm path planning method based on the appearance detection of civil airliners as described above are implemented.

[0021] Compared with the prior art, the positive and progressive effects of the present invention are as follows: The robotic arm path planning method based on civil aviation airliner appearance detection provided by the present invention simulates the scanning path of the robotic arm through a three-dimensional digital twin model of the fuselage, and dynamically optimizes the scanning path using a genetic algorithm, ensuring that the scanning path can fully cover the fuselage surface of the civil aviation airliner appearance detection area, and avoiding missed detection situations caused by unreasonable path planning; by calculating the feedback value and comparing it with the preset feedback threshold range, the detection effect can be evaluated in real time, and the movement speed and light source parameters of the robotic arm can be dynamically adjusted according to the evaluation results, making the detection process more accurate and flexible; using a long short-term memory neural network to predict the expansion trend of the damage target to be detected can anticipate the development direction of the damage in advance, so as to plan a more efficient scanning path, which helps to reduce unnecessary detection areas and improve the detection efficiency.

[0022] Furthermore, this method can dynamically evaluate the complexity of the detection environment by collecting the scene information data of the civil aviation airliner appearance detection area in real time and calculating the scene complexity index, providing a basis for subsequent light source adjustment and detection strategy formulation; when the scene complexity index is greater than or equal to the preset threshold, an adaptive light source array is deployed in the detection area, and the light source parameters are adjusted until the scene complexity index is lower than the threshold, effectively solving the detection difficulties caused by complex environmental light or insufficient illumination, and improving the clarity and accuracy of the detection images. Description of the Drawings

[0023] The accompanying drawings in the specification are used to provide a further understanding of the present invention, and constitute a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention, and do not constitute an improper limitation to the present invention.

[0024] Figure 1 It is a flow schematic diagram of a robotic arm path planning method based on civil aviation airliner appearance detection of the present invention; Figure 2 It is a flow schematic diagram of Embodiment 1 of the present invention. Detailed Embodiments

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0026] In the description of the present invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their combinations.

[0027] It should also be understood that the terms used in the specification of the present invention are merely for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include the plural forms.

[0028] It should be understood that although the terms first, second, third, etc. may be used in the embodiments of the present invention to describe preset ranges, etc., these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0029] Depending on the context, the word "if" as used herein can be interpreted as "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detected (stated condition or event)" can be interpreted as "when determined" or "in response to determining" or "when detected (stated condition or event)" or "in response to detecting (stated condition or event)".

[0030] The following further describes the present invention in detail with reference to the accompanying drawings and specific embodiments, which is an explanation of the present invention rather than a limitation.

[0031] See Figure 1 , a robotic arm path planning method based on civil airliner appearance detection, comprising the following steps: Obtain the damage targets to be inspected in the civil airliner appearance detection area, construct a three-dimensional digital twin model of the civil airliner fuselage, predict the expansion trend of the damage targets to be inspected based on a long short-term memory neural network, and combine the prediction results with the three-dimensional digital twin model of the fuselage to plan the scanning path of the robotic arm; Use the three-dimensional digital twin model of the fuselage to simulate the scanning path of the robotic arm, and determine whether the scanning path conflicts with the movement trajectory of the obstacle or the scanning path does not completely cover the fuselage surface of the civil airliner appearance detection area. If the judgment result is yes, dynamically optimize the scanning path through a genetic algorithm until the scanning path does not conflict with the movement trajectory of the obstacle and completely covers the fuselage surface of the civil airliner appearance detection area; Detect the appearance detection area of a civil airliner according to the scanning path, obtain the detection feedback data, calculate the feedback value, and optimize the movement speed and light source parameters of the robotic arm when the feedback value is within the preset feedback threshold range; when the feedback value is lower than the lower limit of the preset feedback threshold range, add the number of robotic arms and perform multi-robotic arm collaborative scanning; when the feedback value exceeds the upper limit of the preset feedback threshold range, maintain the current detection state.

[0032] The robotic arm path planning method based on the appearance detection of civil airliners provided by the present invention simulates the scanning path of the robotic arm through a three-dimensional digital twin model of the fuselage, and dynamically optimizes the scanning path using a genetic algorithm, ensuring that the scanning path can fully cover the fuselage surface of the appearance detection area of the civil airliner and avoiding missed detections caused by unreasonable path planning; by calculating the feedback value and comparing it with the preset feedback threshold range, the detection effect can be evaluated in real time, and the movement speed and light source parameters of the robotic arm can be dynamically adjusted according to the evaluation results, making the detection process more accurate and flexible; using a long short-term memory neural network to predict the expansion trend of the damage target to be detected can anticipate the development direction of the damage in advance, thereby planning a more efficient scanning path, which helps to reduce unnecessary detection areas and improve the detection efficiency.

[0033] Specifically, the damage target to be detected in the appearance detection area of the civil airliner is obtained as follows: Real-time collect the scene information data of the appearance detection area of the civil airliner and calculate the scene complexity index; Judge whether the scene complexity index is greater than or equal to the preset scene complexity index threshold. If the judgment is yes, deploy an adaptive light source array in the appearance detection area of the civil airliner and adjust the light source parameters until the scene complexity index is less than the preset scene complexity index threshold; Obtain the fuselage surface data of the appearance detection area of the civil airliner, generate an RGB-D-thermal image through a fusion algorithm; input the RGB-D-thermal image into the trained model to obtain an image of the area where damage may exist, and locate the damage target to be detected through a density clustering algorithm.

[0034] This method can dynamically evaluate the complexity of the detection environment by real-time collecting the scene information data of the appearance detection area of the civil airliner and calculating the scene complexity index, providing a basis for subsequent light source adjustment and detection strategy formulation; when the scene complexity index is greater than or equal to the preset threshold, deploy an adaptive light source array in the detection area and adjust the light source parameters until the scene complexity index is lower than the threshold, effectively solving the detection difficulty problem caused by complex environmental light or insufficient illumination, and improving the clarity and accuracy of the detection image.

[0035] Specifically, the generation of the RGB-D-thermal image through the fusion algorithm is specifically as follows:

[0036] Among them, is the pixel value of the RGB-D-thermal image generated through the fusion algorithm at the coordinate (x, y); is the pixel value of the visible light image; is the lidar depth data; is the infrared thermal imaging data; are the normalization weights of the RGB, depth, and thermal imaging data respectively, and .

[0037] Specifically, the scene information data includes light intensity, surface complexity, and obstacle density; The specific scene complexity index is:

[0038] And, , ; Among them, is the scene complexity index; is the time; is the feature vector at time t; is the attenuation function; is the light intensity; is the surface complexity; is the obstacle density; is the weight matrix; is the weight matrix of the convolution operation; α, β are weight coefficients and satisfy 0 < α < 1, 0 < β < 1, α + β = 1; is the constant term bias vector; is the convolution operation.

[0039] Specifically, the construction of the three-dimensional digital twin model of the civil airliner fuselage specifically includes: Based on the digital twin concept, construct the three-dimensional digital twin model of the civil airliner fuselage; obtain the movement trajectory of the obstacle, and continuously update the changes of the damage target to be inspected and the movement trajectory of the obstacle into the three-dimensional digital twin model of the fuselage.

[0040] Specifically, the feedback value is specifically:

[0041] Among them, is the feedback value; is the weight coefficient, The values of all are between 0 and 1 and ; T is the detection period; is the scanning speed of the robotic arm at time t; is the detection accuracy rate at time t; is the false detection rate at time t; is; is the attenuation coefficient, and its value ranges from 0 to 1.

[0042] Specifically, the specific range of the preset feedback threshold is ; Among them, is the lower limit of the preset feedback threshold range, specifically:

[0043] is the upper limit of the preset feedback threshold range, specifically:

[0044] Among them, , i is the order of the detection feedback value, and k is the total number of feedback values detected within the detection period; is the average value of the detection feedback values; is the th detection feedback value.

[0045] The present invention also provides a robotic arm path planning system for civil aviation airliner appearance detection, including: The first module is used to obtain the to-be-detected damage targets in the civil aviation airliner appearance detection area, construct a three-dimensional digital twin model of the civil aviation airliner fuselage, predict the expansion trend of the to-be-detected damage targets based on a long short-term memory neural network, and combine the prediction results and the three-dimensional digital twin model of the fuselage to plan the scanning path of the robotic arm; The second module is used to simulate the scanning path of the robotic arm using the three-dimensional digital twin model of the fuselage, determine whether the scanning path conflicts with the movement trajectory of the obstacle or the scanning path does not completely cover the fuselage surface of the civil aviation airliner appearance detection area. If the determination result is yes, the scanning path is dynamically optimized through a genetic algorithm until the scanning path does not conflict with the movement trajectory of the obstacle and completely covers the fuselage surface of the civil aviation airliner appearance detection area; The third module is used to detect the civil aviation airliner appearance detection area according to the scanning path, obtain detection feedback data, calculate the feedback value, optimize the movement speed and light source parameters of the robotic arm when the feedback value is within the preset feedback threshold range; add the number of robotic arms for multi-robotic arm collaborative scanning when the feedback value is lower than the lower limit of the preset feedback threshold range; and maintain the current detection state when the feedback value exceeds the upper limit of the preset feedback threshold range.

[0046] Embodiment 1 See Figure 2 , a robotic arm path planning method based on the appearance detection of civil airliners, comprising the following steps: Step 1: After analyzing the acquisition scenario in the acquisition area, generate a scenario complexity from the acquisition scenario data. If the scenario complexity exceeds the expectation, deploy an adaptive light source in the acquisition area; the specific content of the said Step 1 is as follows: Step 101: When it is necessary to detect the appearance of a civil airliner, determine the detection area and analyze the detection scenario in the current detection cycle in the detection area. Among them, using devices such as a light sensor, a 3D lidar, and a curvature sensor, etc., collect the light intensity L(t), the fuselage surface complexity S(t), and the obstacle density O(t) of the detection area at time t in real time, and summarize the obtained detection scenario data to generate a detection scenario data set; Step 102: Generate a scenario complexity from the data in the detection scenario data set , and evaluate the complexity of the current detection environment according to the scenario complexity, in the following way:

[0047] Wherein: is the feature vector at time t, , including the light intensity L(t), the fuselage surface complexity S(t), and the obstacle density O(t), is the weight matrix of the feature vector, is the weight matrix of the convolution operation; is the attenuation function, α and β are weight coefficients, satisfying 0 < α < 1, 0 < β < 1, α + β = 1, b is the constant term bias vector, and * is the convolution operation.

[0048] Pre-set a complexity threshold according to historical data and the complexity of the grasping scenario, and compare the calculated with the pre-set threshold. If exceeds the pre-set threshold, deploy an adaptive light source array in the detection area, and optimize the lighting conditions in the detection area by automatically adjusting parameters such as the angle and intensity of the light source to ensure uniform and clear imaging, providing a good basis for subsequent image acquisition and damage identification.

[0049] Step 2: After automatically controlling the light source, collect the fuselage surface data in the detection area, fuse these data to generate an RGB-D-thermal image, and distinguish the background surface on the fuselage from the target to be inspected through clustering analysis; the specific content of the said Step 2 is as follows: Step 201: Use a high-resolution camera, an infrared sensor, and a lidar to synchronously collect data on the surface of a civil airliner fuselage. During the collection process, perform spatio-temporal alignment on the acquired visible light images, infrared thermal imaging data, and lidar distance data to ensure the consistency of different modality data in terms of time and space; Step 202: Through a fusion algorithm, fuse these data to generate an RGB-D-thermal image in the following way:

[0050] where, are the normalized weights of RGB, depth, and thermal imaging data respectively and satisfy ; fully combines the advantages of different modality data and provides richer information for damage identification; Step 203: Use the pre-trained YOLOv7 algorithm to process the generated RGB-D-thermal image, and adopt clustering algorithms such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise) to perform clustering analysis on the pixel points in the image. According to the density distribution characteristics of the pixel points, distinguish the background surface from the targets to be inspected (such as cracks, dents, corrosion areas, etc.) to achieve precise positioning and identification of the damage area.

[0051] Step 3: Construct a three-dimensional digital twin model of the fuselage. During the detection process, detect the changes of the targets to be inspected and the movement trajectories of obstacles through sensors and update them to the digital twin model in real time; the specific content of Step 3 is as follows: Step 301: Based on digital twin technology, construct an accurate three-dimensional digital twin model of the fuselage according to the design drawings, actual dimensions of the civil airliner, and the fuselage status data collected in real time through sensors; Step 302: During the detection process, continuously monitor the changes in the damage location and the movement trajectories of obstacles (such as antennas, sensors, etc.) through sensors, and update this information to the digital twin model in real time, so that the model can always reflect the real state of the fuselage; Step 4: Use an LSTM neural network to predict the expansion trend of the damage area, and combine the prediction results with the geometric structure information of the fuselage in the digital twin model to plan the initial scanning path of the robotic arm and optimize the movement path through a genetic algorithm; the specific content of Step 4 is as follows: Step 401: Plan the initial scanning path of the robotic arm according to the detection of the damage area by the LSTM neural network and the digital twin model; Use a long short-term memory neural network (LSTM) to predict the expansion trend of the damaged area. The LSTM neural network has strong time series processing capabilities and can learn the changing patterns of the damaged area over time. Combine the prediction results with the geometric structure information of the fuselage in the digital twin model to plan the initial scanning path of the robotic arm, ensuring that the path can cover the areas where damage may exist and improving the comprehensiveness of detection.

[0052] Step 402: During the simulation verification process, check whether there are conflicts between the path and obstacles, and whether the coverage of the fuselage surface is comprehensive. Optimize the robotic arm movement path according to the simulation situation; Input the planned initial scanning path into the digital twin model for simulation verification. By simulating the movement process of the robotic arm in the model, check whether there are conflicts between the path and obstacles, and whether the coverage of the fuselage surface is comprehensive; if there are conflicts or incomplete coverage in the path, use the genetic algorithm to dynamically optimize the path; the genetic algorithm adjusts parameters such as the order of path nodes, the movement angles and speeds of the robotic arm by simulating selection, crossover, and mutation operations in the natural evolution process, and gradually searches for the optimal scanning path to ensure the efficiency and safety of the detection work.

[0053] Step 5: After collecting the detection feedback data during the detection process, construct the acquisition feedback value and feedback threshold from the collected feedback data, and adopt corresponding coordination processing strategies based on the relationship between the acquisition feedback value and the feedback threshold; the specific content of the said Step 5 is as follows: Step 501: After the end of the current detection cycle, collect the detection feedback results during the detection process, obtain the scanning speed of the robotic arm and the detection accuracy rate, etc. Under dimensionless conditions, construct the detection feedback value from the detection feedback data in the following way:

[0054] Among them, is the scanning speed of the robotic arm at time t, is the detection accuracy rate at time t, is the false detection rate at time t, is the weight coefficient, and the values are all between 0 and 1 and , , The value is between 0 and 1, is the detection cycle; Construct the feedback threshold from a series of consecutive detection feedback values in the following way:

[0055] Among them, , where \(i\) is the order of the detected feedback value and \(k\) is the total number of feedback values detected within the detection period; is the average value of the detected feedback values; is the th detected feedback value; Step 502, after the detection period ends, compare with a preset threshold value, and adopt different adjustment strategies according to the comparison results, specifically as follows: If exceeds the preset threshold value, it indicates that the current detection process has a good effect, and the current detection parameters and the motion state of the robotic arm are maintained; If is within the preset threshold range, it indicates that there is still room for improvement in the detection effect. Adjust the motion speed of the robotic arm and the intensity of the light source through an optimization algorithm to further improve the detection efficiency and accuracy; If is lower than the preset threshold value, it means that the current detection effect is not good. At this time, increase the number of robotic arms and start the multi-robot collaborative scanning algorithm. During the multi-robot collaboration process, reasonably allocate the detection tasks of each robotic arm and coordinate their motion paths to avoid collisions between robotic arms and ensure that the detection work can be completed efficiently and accurately.

[0056] Through scene complexity analysis and the deployment of an adaptive light source array, this method can effectively cope with complex lighting conditions and ensure clear and accurate images can be obtained in different lighting environments. The multi-modal image acquisition and fusion technology combines the advantages of multiple sensors, further improving the detection ability of the fuselage curved surface structure, achieving stable and efficient detection in the full scene, greatly improving the consistency and accuracy of imaging, and reducing missed detections and false detections caused by lighting and curved surface structures; With the help of the digital twin model and the LSTM neural network, the expansion trend of the damaged area can be accurately predicted, and a scanning path that comprehensively covers the fuselage surface can be planned. At the same time, during the detection process, by updating the model in real time and optimizing the path through the genetic algorithm, the robotic arm can dynamically avoid obstacles, achieve efficient and safe detection operations, avoid detection blind spots and equipment damage risks caused by obstacles, and improve the comprehensiveness and reliability of detection; Based on the detected feedback value The dynamic adjustment strategy enables the detection system to optimize the detection parameters and path planning in real time according to the actual detection effect. This real-time feedback optimization mechanism effectively reduces redundant scans and improves the detection efficiency. At the same time, by continuously optimizing the detection process, the accuracy of detection is further improved, ensuring that tiny damages and defects can be detected in a timely manner. This method supports the collaborative work of multiple robotic arms. When facing large-scale detection tasks, it can give full play to the advantages of multiple robotic arms through reasonable task allocation and path coordination, improve the detection efficiency, and shorten the detection time. The multi-robot collaborative operation mode ensures the timeliness of the appearance detection of civil airliners, meets the requirements of civil aviation operations for fast and efficient detection, reduces the time for the aircraft to be grounded due to detection, and improves the normal operation rate of flights.

[0057] Embodiment 2 A robotic arm path planning system for the appearance detection of civil airliners, comprising: Environmental perception unit: Integrating various devices such as light sensors, 3D lidars, and curvature sensors, it can collect environmental data such as light intensity, fuselage curvature, and obstacle distribution in the detection area in real time, and transmit this data to the subsequent unit to provide basic information support for the entire detection system; Image processing unit: Receives the data collected by the environmental perception unit and the image data collected by the multi-modal sensor. This unit includes a multi-modal image fusion processing module, a damage detection module based on the YOLOv7 algorithm, and a clustering algorithm module for distinguishing the background surface from the target to be detected, realizes the processing and analysis of images, extracts damage information, and provides a basis for path planning and detection decision-making; Path planning unit: Responsible for constructing a digital twin model of the fuselage and planning the robotic arm path according to the damage prediction result and model information. This unit includes a digital twin model construction and update module, an LSTM neural network prediction module, and a path planning and optimization module using a genetic algorithm to ensure that the planned path can cover the areas where damages may exist, avoid obstacles, and at the same time ensure the optimality of the path; Collaborative control unit: In the scenario of multiple robotic arms working, it undertakes the important responsibilities of task allocation and motion coordination. This unit includes a task allocation module and a motion coordination module. The task allocation module divides the entire detection task into multiple subtasks according to the positions, load capacities of each robotic arm, and the characteristics of the detection area, and assigns them to different robotic arms; the motion coordination module realizes path coordination by establishing a communication mechanism between robotic arms, exchanging information such as their positions, motion states, and detection results in real time, and avoiding collisions between robotic arms; Feedback optimization unit: Calculate the detection feedback value , and based on the comparison result with the threshold, send an adjustment instruction to other units. This unit includes a feedback value calculation module and an adjustment instruction sending module. The feedback value calculation module accurately calculates according to the defined formula . The adjustment instruction sending module controls the adjustment of parameters such as the number of robotic arms, movement speed, and light source intensity according to the comparison result, realizing the dynamic optimization of the detection process.

[0058] This system calculates the scene complexity by collecting data such as light intensity, fuselage curvature, and obstacle distribution. When the threshold is exceeded, the light is optimized; multi-modal images are fused and damages are recognized; a digital twin model is constructed and combined with an LSTM neural network to plan and optimize the path to avoid obstacles; the detection feedback value is used to adjust strategies, such as optimizing the speed and increasing the collaborative operation of robotic arms. This invention can adapt to complex lighting and curved surface structures, efficiently avoid obstacles, optimize detection in real time, improve the detection efficiency and accuracy, and ensure the flight safety of civil airliners.

[0059] Based on the same inventive concept, an embodiment of this application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps of the robotic arm path planning method based on the appearance detection of civil airliners. Specifically, the computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may include RAM (Random Access Memory) and / or cache, etc. The non-volatile memory may include ROM (Read Only Memory), hard disk, flash memory, optical disc, magnetic disk, etc.

[0060] Based on the same inventive concept, an embodiment of this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the robotic arm path planning method based on the appearance detection of civil airliners. Among them, the memory may contain internal memory, such as high-speed random access memory, and may also include non-volatile memory, such as at least one disk memory, etc.; the processor, network interface, and memory are interconnected through an internal bus. The internal bus may be an Industry Standard Architecture bus, a Peripheral Component Interconnect standard bus, an Extended Industry Standard Architecture bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory is used to store programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include internal memory and non-volatile memory and provide instructions and data to the processor.

[0061] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method or a computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM (Compact Disc Read-Only Memory), optical memory, etc.) that contain computer-usable program code.

[0062] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer device or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0063] These computer program instructions can also be stored in a computer-readable memory that can direct a computer device or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implement the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0064] These computer program instructions can also be loaded onto a computer device or other programmable data processing devices, so that a series of operation steps are executed on the computer device or other programmable devices to generate a process implemented by the computer device. Thus, the instructions executed on the computer device or other programmable devices provide steps for implementing the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 or multiple blocks.

[0065] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present invention.

[0066] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and its equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A robotic arm path planning method based on the appearance detection of civil airliners, characterized in that, Including the following steps: Obtain the damage targets to be inspected in the appearance inspection area of a civil airliner, construct a three-dimensional digital twin model of the civil airliner's fuselage, predict the expansion trend of the damage targets to be inspected based on a long short-term memory neural network, and combine the prediction results with the three-dimensional digital twin model of the fuselage to plan the scanning path of the robotic arm; Use the three-dimensional digital twin model of the fuselage to simulate the scanning path of the robotic arm, and determine whether the scanning path conflicts with the movement trajectory of the obstacle or the scanning path does not completely cover the fuselage surface in the appearance inspection area of the civil airliner. If the judgment result is yes, dynamically optimize the scanning path through a genetic algorithm until the scanning path does not conflict with the movement trajectory of the obstacle and completely covers the fuselage surface in the appearance inspection area of the civil airliner; Inspect the appearance inspection area of the civil airliner according to the scanning path, obtain inspection feedback data, calculate the feedback value, and optimize the movement speed and light source parameters of the robotic arm when the feedback value is within the preset feedback threshold range; When the feedback value is lower than the lower limit of the preset feedback threshold range, increase the number of robotic arms and perform collaborative scanning with multiple robotic arms; When the feedback value exceeds the upper limit of the preset feedback threshold range, maintain the current inspection state.

2. The robotic arm path planning method based on the appearance detection of civil airliners according to claim 1, wherein The obtaining of the damage targets to be inspected in the appearance inspection area of the civil airliner specifically includes: Collect the scene information data in the appearance inspection area of the civil airliner in real time and calculate the scene complexity index; Judge whether the scene complexity index is greater than or equal to the preset scene complexity index threshold. If the judgment is yes, deploy an adaptive light source array in the appearance inspection area of the civil airliner and adjust the light source parameters until the scene complexity index is less than the preset scene complexity index threshold; Obtain the fuselage surface data in the appearance inspection area of the civil airliner, generate an RGB-D-thermal image through a fusion algorithm; input the RGB-D-thermal image into the trained model to obtain an image of the area where damage may exist, and locate the damage targets to be inspected through a density clustering algorithm.

3. A robotic arm path planning method based on the appearance detection of civil airliners according to claim 2, characterized in that, The generating of the RGB-D-thermal image through the fusion algorithm is specifically: in, The pixel value of the RGB-D-thermal image at the coordinate (x, y) is generated by the fusion algorithm; is the pixel value of the visible light image; is the lidar depth data; It is infrared thermal imaging data; are the normalized weights of RGB, depth, and thermal imaging data respectively. .

4. A robotic arm path planning method based on the appearance detection of civil airliners according to claim 2, characterized in that, The scene information data includes light intensity, surface complexity, and obstacle density; The scene complexity index is specifically: And, , ; Among them, is the scene complexity index; is the moment; is the feature vector at moment t; is the attenuation function; is the illumination intensity; is the surface complexity; is the obstacle density; is the weight matrix; is the weight matrix of the convolution operation; α, β are weight coefficients and satisfy 0 < α < 1, 0 < β < 1, α + β = 1; is the constant term bias vector; is the convolution operation.

5. A robotic arm path planning method based on the appearance detection of civil airliners according to claim 1, characterized in that, The constructing of the three-dimensional digital twin model of the civil airliner's fuselage specifically includes: Based on the digital twin concept, construct a three-dimensional digital twin model of the civil airliner's fuselage; obtain the movement trajectory of the obstacle, and continuously update the changes of the damage targets to be inspected and the movement trajectory of the obstacle into the three-dimensional digital twin model of the fuselage.

6. The robotic arm path planning method based on the appearance detection of civil airliners according to claim 1, characterized in that, The feedback value is specifically: Among them, is the feedback value; is the weight coefficient, The values of are all between 0 and 1 and ; T is the detection period; is the scanning speed of the robotic arm at time t; is the detection accuracy at time t; is the false detection rate at time t; is; is the attenuation coefficient, The value of is between 0 and 1.

7. A robotic arm path planning method based on the appearance detection of civil airliners according to claim 5, characterized in that, The specific preset feedback threshold range is ; in, The lower limit of the preset feedback threshold range is: is the upper limit of the preset feedback threshold range, specifically: Among them, , where i is the order of the detection feedback value, and k is the total number of feedback values detected within the detection period; is the average value of the detected feedback values; is the th detected feedback value.

8. A robotic arm path planning system based on civil aircraft appearance inspection, characterized in that: Including: The first module is used to obtain the damage targets to be inspected in the appearance inspection area of the civil airliner, construct a three-dimensional digital twin model of the civil airliner's fuselage, predict the expansion trend of the damage targets to be inspected based on a long short-term memory neural network, and combine the prediction results with the three-dimensional digital twin model of the fuselage to plan the scanning path of the robotic arm; The second module is used to simulate the scanning path of the robotic arm using the 3D digital twin model of the fuselage to determine whether the scanning path conflicts with the motion trajectory of obstacles or does not completely cover the fuselage surface in the civil aircraft appearance inspection area. If the judgment result is yes, the scanning path is dynamically optimized using a genetic algorithm until the scanning path does not conflict with the motion trajectory of obstacles and completely covers the fuselage surface in the civil aircraft appearance inspection area; The third module is used to inspect the exterior inspection area of the civil aircraft according to the scanning path, obtain inspection feedback data, calculate the feedback value, and optimize the movement speed and light source parameters of the robotic arm when the feedback value is within a preset feedback threshold range; When the feedback value is lower than the lower limit of the preset feedback threshold range, the number of robotic arms is increased to perform multi-robotic arm collaborative scanning; When the feedback value exceeds the upper limit of the preset feedback threshold range, the current detection state is maintained.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the robot arm path planning method based on civil aircraft appearance inspection as described in any one of claims 1 to 7 are implemented.

10. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the robot arm path planning method based on civil aircraft appearance inspection as described in any one of claims 1 to 7 are implemented.

Images