Bimodal three-dimensional target detection method and device for small parts of aero-engine

Abstract

The invention provides a dual-mode three-dimensional target detection method and device for small parts of an aero-engine, and relates to the technical field of three-dimensional vision. The method comprises the following steps: scanning the surface of the aero-engine by adopting a high-precision three-dimensional scanner to obtain training detection data; constructing an aero-engine bimodal database according to the training detection data; building a to-be-trained three-dimensional target detection model based on the Transform model structure and the multi-layer perception mechanism; calling an aero-engine bimodal database, and training the to-be-trained three-dimensional target detection model to obtain a three-dimensional target detection model; adopting a high-precision scanner to scan the surface of the aero-engine to be detected to obtain target detection data; and inputting the target detection data into the three-dimensional target detection model to obtain a small part target detection result. The bimodal three-dimensional target detection method is an accurate and efficient bimodal three-dimensional target detection method for small parts of an aero-engine.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional vision technology, and in particular to a dual-mode three-dimensional target detection method and device for small aircraft engine parts. Background Art

[0002] Aircraft engines are an indispensable core component in the modern aviation industry. Assembly is an important step in the production process of aircraft engines. Its process is complex, involving a wide range of disciplines, and has a great impact on manufacturing efficiency, quality, and cost. However, there are inevitably some defects and problems in the assembly process. After the assembly is completed, the engine must be fully assembled and inspected. Traditional manual inspection is inefficient and prone to missed inspections and misdetections. In the digital inspection process, for the three-dimensional positioning problem of small parts in complex aircraft engine scenes, images can provide rich texture information but lack three-dimensional position information, while three-dimensional point clouds can provide accurate three-dimensional position information but lack texture information. Through the fusion of image modality and point cloud modality information, the complementarity of the two modal information can be achieved. Therefore, dual-modal fusion has become a path worthy of attention in the field of industrial target detection.

[0003] The PETR network was proposed at the 2022 European Conference on Computer Vision. The network is based on the Transformer structure of multi-view perception and uses multi-view images to encode spatial positions to perceive three-dimensional targets within the camera's field of view. Multi-view images provide texture information of the target at different perspectives, which is conducive to the three-dimensional recognition of small parts. However, this method only uses single-modal image information, and three-dimensional detection relies on randomly generated Transformer query points in space. When dealing with the problem of positioning small parts in complex aero-engine scenes, it faces the problem of low accuracy, and there is still a lot of room for improvement in detection performance. Therefore, adding three-dimensional point cloud information containing precise spatial positions is an effective way to improve detection performance.

[0004] In the existing technology, China lacks an accurate and efficient dual-modal three-dimensional target detection method for small aircraft engine parts. Summary of the invention

[0005] In order to solve the technical problem that the single-mode detection method in the prior art has reduced accuracy when dealing with small target detection problems in complex scenes, the embodiment of the present invention provides a dual-mode three-dimensional target detection method and device for small aircraft engine parts. The technical solution is as follows:

[0006] On the one hand, a dual-mode three-dimensional target detection method for small aircraft engine parts is provided, which is implemented by a dual-mode three-dimensional target detection device, and the method includes:

[0007] Using a high-precision three-dimensional scanner to scan the surface of the aircraft engine to obtain training test data; constructing an aircraft engine dual-modal database based on the training test data;

[0008] Build a 3D object detection model to be trained based on the Transformer model structure and multi-layer perceptron;

[0009] Calling the aircraft engine dual-modal database to train the three-dimensional target detection model to be trained to obtain a three-dimensional target detection model;

[0010] A high-precision scanner is used to scan the surface of the aircraft engine to be inspected to obtain target detection data; the target detection data is input into the three-dimensional target detection model to obtain small parts target detection results.

[0011] On the other hand, a dual-mode 3D target detection device for small aircraft engine parts is provided, and the device is applied to a dual-mode 3D target detection method for small aircraft engine parts. The device comprises:

[0012] A database construction module is used to scan the surface of the aircraft engine using a high-precision three-dimensional scanner to obtain training test data; and to construct an aircraft engine dual-modal database based on the training test data;

[0013] Model building module, used to build a 3D object detection model to be trained based on the Transformer model structure and multi-layer perceptron;

[0014] A model training module, used to call the aircraft engine dual-modal database, train the three-dimensional target detection model to be trained, and obtain a three-dimensional target detection model;

[0015] The target detection module is used to use a high-precision scanner to scan the surface of the aircraft engine to be detected to obtain target detection data; the target detection data is input into the three-dimensional target detection model to obtain small parts target detection results.

[0016] On the other hand, a dual-modal three-dimensional target detection device is provided, which includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned dual-modal three-dimensional target detection methods for small aircraft engine parts is implemented.

[0017] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned dual-modal three-dimensional target detection methods for small aircraft engine parts.

[0018] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0019] The present invention proposes a dual-modal three-dimensional target detection method for small parts of aircraft engines. The method is based on the surface image and three-dimensional point cloud of the aircraft engine, and makes full use of the dual-modal information to make up for the problem of low positioning accuracy of small parts caused by insufficient single-modal information representation; the image data can fully represent the color, state and other texture information of the target; the three-dimensional point cloud data can accurately reflect the real spatial position and geometric shape of the object. The dual-modal feature extraction module is used to extract more significant high-dimensional dual-modal information; the densified feature fusion module is used to realize the feature-level fusion of dual-modal heterogeneous information; the Transformer decoder with normalized spatial position coding is used to predict the spatial position and size of the target. The present invention is suitable for the application conditions of complex aviation industry parts assembly defect detection, can effectively improve the three-dimensional positioning effect of parts, and provide a prerequisite for the downstream parts assembly morphological defect judgment, which is of great significance. The present invention is an accurate and efficient dual-modal three-dimensional target detection method for small parts of aircraft engines. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 This is a flow chart of a dual-mode three-dimensional target detection method for small aircraft engine parts provided by an embodiment of the present invention;

[0022] Figure 2 It is a schematic diagram of dual-modal data of a local body of an aircraft engine provided by an embodiment of the present invention;

[0023] Figure 3 It is a schematic diagram of three-dimensional target true value annotation provided by an embodiment of the present invention;

[0024] Figure 4 is a schematic diagram of a three-dimensional target detection model structure provided by an embodiment of the present invention;

[0025] Figure 5 is a schematic diagram of a feature fusion process provided by an embodiment of the present invention;

[0026] Figure 6 is a schematic diagram of a channel self-attention mechanism provided by an embodiment of the present invention;

[0027] Figure 7 is a schematic diagram of the structure of a detection result prediction module provided by an embodiment of the present invention;

[0028] Figure 8 This is a block diagram of a dual-mode three-dimensional target detection device for small aircraft engine parts provided by an embodiment of the present invention;

[0029] Fig. 9 It is a structural schematic diagram of a dual-modal three-dimensional target detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0031] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0032] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0033] In the embodiments of the present invention, sometimes the subscripts such as W 1 It may be written in non-subscript form such as W1. When the difference is not emphasized, the meaning is the same.

[0034] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0035] The embodiment of the present invention provides a dual-mode 3D target detection method for small aircraft engine parts. The method can be implemented by a dual-mode 3D target detection device, which can be a terminal or a server. Figure 1 The flowchart of the dual-modal three-dimensional target detection method for small aircraft engine parts is shown in the figure. The processing flow of the method may include the following steps:

[0036] S1. Use a high-precision three-dimensional scanner to scan the surface of the aircraft engine to obtain training test data; build an aircraft engine bimodal database based on the training test data.

[0037] Among them, the training detection data includes three-dimensional point cloud data, image data, scanner camera pose and camera internal parameter data;

[0038] 3D point cloud data is used to express the geometric shape information of the aircraft engine surface;

[0039] Image data is used to express the texture information of the aircraft engine surface.

[0040] In a feasible implementation, a high-precision scanner is used to scan the surface of the aircraft engine. The scanning principle is binocular three-dimensional reconstruction, which can simultaneously capture the geometric shape information and texture information of the aircraft engine surface.

[0041] The binocular images, local 3D point clouds and camera poses obtained during each scan are exported and used together with the camera intrinsic parameters to construct an aircraft engine dual-modal database. Each sample contains binocular images and point clouds such as Figure 2 shown.

[0042] The present invention uses a three-dimensional scanning system GOM ATOS Q to obtain dual-mode complete aircraft engine data; provides three-dimensional target true values ​​used in the training process through manual annotation; Figure 3 The annotated true value box is shown; during training, the training set and validation set are divided in a ratio of 7:1.

[0043] S2. Build a 3D target detection model to be trained based on the Transformer model structure and multi-layer perceptron.

[0044] Among them, the three-dimensional object detection model includes a bimodal feature extraction module, a dense bimodal feature fusion module and a detection result prediction module based on the Transformer decoder.

[0045] In a feasible implementation, the structure of the three-dimensional target detection model is shown in Figure 4. The present invention uses two convolutional neural networks as the three-dimensional point cloud modality branch and the image modality branch of the dual-modality feature extraction module.

[0046] The densified bimodal feature fusion module is used to fuse 3D point cloud features and image features. The 3D point cloud is projected into the image through the scanner camera posture and internal parameters to obtain the correspondence between points and pixels. The point cloud features after downsampling are obtained by mean filtering the point cloud feature map and integrating the point features falling in the same pixel. There is a single-shot correspondence between the point cloud features and the image features, that is, each point feature has a corresponding pixel feature, but each pixel feature does not necessarily have a corresponding point feature. The downsampling point cloud features are densified to obtain a densified point cloud feature map of the same size as the image feature map, that is, a bijective correspondence is formed between the point cloud features and the image features.

[0047] S3. Call the aircraft engine bimodal database to train the three-dimensional target detection model to be trained, and obtain the three-dimensional target detection model.

[0048] In a feasible implementation, the present invention uses a loss function L to supervise the training process. The training process is as follows:

[0049] (1);

[0050] in, and Represent the true value of the category and the predicted result respectively; and Represent the true value box and the predicted box respectively; and Represent the weight of the category prediction task and the weight of the bounding box prediction task respectively; and They represent the Focal loss function and L1 loss function for supervised category prediction tasks respectively.

[0051] S4. Use a high-precision scanner to scan the surface of the aircraft engine to be inspected and obtain target detection data; input the target detection data into the three-dimensional target detection model to obtain the target detection results of small parts.

[0052] Optionally, the target detection data is input into a three-dimensional target detection model to obtain small parts target detection results, including:

[0053] According to the target detection data, the target three-dimensional point cloud data, the target image data, the current scanner camera posture and the current camera internal parameter data are obtained;

[0054] Perform high-dimensional feature extraction on the target three-dimensional point cloud data and the target image data to obtain a target point cloud feature map and a target image feature map;

[0055] Based on the channel attention mechanism, according to the current scanner camera posture and the current camera internal parameter data, the target point cloud feature map and the target image feature map are feature fused to obtain the second fused feature map;

[0056] Performing position encoding on the second fused feature map to obtain a third fused feature map;

[0057] Based on the Transformer decoder and the multi-layer perceptron, the target detection result is predicted according to the third fusion feature map to obtain the small parts target detection result.

[0058] In a feasible implementation, two convolutional neural networks are used to extract high-dimensional features from images and three-dimensional point clouds respectively. The features from the image and the features from the three-dimensional point cloud have the same dimension. The neural networks used to extract image features include but are not limited to Residual Network (ResNet), An Energy and GPU-Computation Efficient Backbone Network for Real-Time Object Detection (VoVNet), and Feature Pyramid Networks (FPN); the neural networks used to extract three-dimensional point cloud features include but are not limited to PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation (PointNet) and Deep Graph Convolutional Neural Network (DGCNN).

[0059] Optionally, high-dimensional feature extraction is performed on the target three-dimensional point cloud data and the target image data to obtain a target point cloud feature map and a target image feature map, including:

[0060] Based on the deep graph convolutional neural network, feature extraction is performed based on the target 3D point cloud data to obtain the target point cloud feature map;

[0061] Based on the lightweight feature extraction backbone network and the feature pyramid network, feature extraction is performed according to the target image data to obtain the target image feature map.

[0062] In a feasible implementation, two convolutional neural networks are used to extract high-dimensional features from images and three-dimensional point clouds respectively.

[0063] VoVNet is combined with the feature pyramid to analyze the input image Features of different scales are extracted and integrated to obtain 256-dimensional image modality features for each image in the binocular image. ( , Represent the image width and height respectively), the image feature extraction process is as follows (2):

[0064] (2);

[0065] in, Represents an image feature extraction operation.

[0066] DGCNN is used to input point cloud Extract high-dimensional features to obtain 256-dimensional point cloud modal features ( represents the number of points contained in the point cloud), the point cloud feature extraction process is as follows (3):

[0067] (3)

[0068] in, Represents a point cloud feature extraction operation.

[0069] Optionally, based on the channel attention mechanism, according to the current scanner camera posture and the current camera internal parameter data, the target point cloud feature map and the target image feature map are feature fused to obtain a second fused feature map, including:

[0070] Based on the target image feature map, under the guidance of the current scanner camera posture and the current camera internal parameters, the target point cloud feature map is densified to obtain a target dense point cloud feature map; the size of the target dense point cloud feature map is consistent with the size of the target image feature map;

[0071] Cascading and splicing the target dense point cloud feature map and the target image feature map to obtain the first fusion feature;

[0072] Based on the channel self-attention mechanism, the channel features of the first fusion feature are integrated to obtain the second fusion feature.

[0073] In a feasible implementation manner, the process of obtaining the first fusion feature is as follows: Figure 5 As shown. The sparse point cloud feature map is densified to make it have the same characteristics as the image. Figure 1 For point clouds, their projections in images are sparse, so they need to be densified to align the point cloud feature vector and the image feature vector.

[0074] The densification process can be carried out in the form of zero filling, nearest neighbor interpolation, linear interpolation, etc. The densification process is as follows (4):

[0075] (4);

[0076] Among them, Densify means densification.

[0077] The densification process in the present invention adopts the form of zero filling: for each pixel in the image If a point projection falls in it, the feature vectors of these points are subjected to the maximum pooling operation to remove redundancy as the new point cloud feature vector , the steps are as follows (5):

[0078] (5);

[0079] Among them, Maxpool represents the maximum pooling operation.

[0080] If no point projection falls within it, a zero vector is added to fill the gap. The steps are as follows:

[0081] (6);

[0082] The bimodal features are concatenated together through feature concatenation to obtain a 512-dimensional fusion feature. , the steps are as follows (7):

[0083] (7);

[0084] in, Represents a feature cascade.

[0085] The channel self-attention mechanism used to obtain the second fusion feature based on the first fusion feature is as follows: Figure 6 As shown. The feature channels of the spliced ​​features are integrated through the channel self-attention mechanism to learn the correlation between the bimodal features and obtain the final fusion feature. , the steps are as follows (8):

[0086] (8);

[0087] Among them, CA represents the channel self-attention mechanism.

[0088] The final dense point cloud feature map With image feature map Have the same size and characteristic dimensions.

[0089] Optionally, position encoding is performed on the second fused feature map to obtain a third fused feature map, including:

[0090] Performing two-dimensional sinusoidal position encoding on the second fused feature map to obtain an encoded fused feature map;

[0091] The encoded fused feature map is normalized into three-dimensional position encoding to obtain a third fused feature map.

[0092] In a feasible implementation, two-dimensional sinusoidal position encoding and normalized three-dimensional spatial position encoding are performed on each feature vector of the dense second fusion feature map.

[0093] The 2D sinusoidal position encoding is the same as that used by the Detection Transformer (DETR) network; the normalized 3D position encoding is used in the multi-view Figure 3 Based on the 3D implicit position encoding used by the Position Embedding Transformation for Multi-View 3D Object Detection (PETR) network for 3D target detection, a rigid transformation correction is performed on the coordinate system transformation relationship in the data enhancement process, making the 3D encoding position generation more accurate.

[0094] Fusion features after normalized spatial position encoding The definition is as follows (9):

[0095] (9);

[0096] Among them, SinePE represents two-dimensional sinusoidal position encoding, and NormPE represents normalized three-dimensional spatial position encoding.

[0097] In a feasible implementation manner, the detection result prediction module structure based on the Transformer decoder constructed by the present invention is as follows: Figure 7 As shown. The Transformer decoder decodes the fused feature map with normalized spatial position encoding to obtain the position and size information of small parts. The output part consists of two multilayer perceptrons (MLP) responsible for predicting the bounding box information (including center coordinates, length, width, height) and category confidence respectively. The steps are as follows (10) and (11):

[0098] (10);

[0099] (11);

[0100] The fused features are decoded by the Transformer decoder with normalized spatial position encoding, and the bounding box information and category confidence are output respectively through two MLP branches; the normalized 3D spatial position encoding first generates a series of camera cone points in space through camera parameters and image size , calculate its normalized position coordinates , the calculation process is defined as follows (12):

[0101] (12);

[0102] in, represents the coordinate infimum of the detection space after data enhancement, Represents the coordinate supremum of the detection space after data enhancement. Perform the same 3D position encoding as PETR and add the encoding result to the fusion feature. Finally, output the bounding box information and category score through the Transformer decoder and MLP.

[0103] In a feasible implementation manner, in order to evaluate the performance of the method of the present invention, a test is conducted on real aircraft engine bimodal data, and the quantitative results are shown in Table 1 (result table of three-dimensional target detection task of small aircraft engine parts).

[0104] Table 1 Among them, the evaluation indicators include Average Precision (AP), Mean Average Translation Error for 3D object detection (mATE3D), Mean Average Scale Error for 3D object detection (mASE3D), and part recall rate Recall.

[0105] The average precision represents the evaluation result that comprehensively considers the precision and recall rate. It is a commonly used performance evaluation indicator in the field of target detection. It reflects the area under the PR curve to a certain extent. The higher the average precision, the better the model performance. The present invention adopts the 40-point interpolated average precision (40-Point Interpolated Average Precision) calculation method to obtain the average precision index as shown in 1, which has achieved the research goal (mAP≥0.80). Among them, mAP refers to the average value of AP calculated by taking the intersection over Union (IoU) as the correct threshold for bounding box prediction every 0.05 from 0.50 to 0.95. AP 50 and AP 75 They refer to the AP when the intersection-over-union ratio threshold is 0.50 and 0.75 respectively.

[0106] The three-dimensional average translation error mATE3D is set by imitating the meanAverage Translation Error (mATE) indicator in the nuScenes dataset. It reflects the offset of the predicted box relative to the true value box. The biggest difference between the two is that the former calculates the mean of the center offset distance between the true value box and the predicted box in the actual three-dimensional space, while the latter only calculates the mean of the two-dimensional offset distance from the bird's-eye view. Let the center coordinate of the i-th true value box be y c,i , the center coordinate of the prediction box that intersects with it and has a confidence level ≥ 0.50 is x c,i , a total of N recall The three-dimensional average translation error mATE3D is calculated by the following formula (13):

[0107] (13);

[0108] The 3D average scale error reflects the size error of the predicted box relative to the true value box. Similar to the 3D average translation error, let the length, width, and height of the i-th true value box be y s,i , the length, width and height of the prediction box that intersects with it and has a confidence level ≥ 0.50 are x s,i , a total of N recall The three-dimensional average scale error mASE3D is calculated by the following formula (14):

[0109] (14);

[0110] The recall rate reflects the problem of whether the parts are fully checked, which is of greater concern in the industrial manufacturing field. The higher the recall rate, the less the detection model misses the part target. Suppose there are N true value frames in the verification set. gt Of these, N intersect with the prediction box with confidence ≥ 0.50. recall Then Recall is calculated by the following formula (15):

[0111] (15);

[0112] It can be seen from the quantization results that the proposed method is superior to the baseline method in position embedding transformation multi-view Figure 3 3D target detection has better detection effect and better positioning and size prediction accuracy in the 3D target detection of small aircraft engine parts.

[0113] The present invention proposes a dual-modal three-dimensional target detection method for small parts of aircraft engines. The method is based on the surface image and three-dimensional point cloud of the aircraft engine, and makes full use of the dual-modal information to make up for the problem of low positioning accuracy of small parts caused by insufficient single-modal information representation; the image data can fully represent the color, state and other texture information of the target; the three-dimensional point cloud data can accurately reflect the real spatial position and geometric shape of the object. The dual-modal feature extraction module is used to extract more significant high-dimensional dual-modal information; the densified feature fusion module is used to realize the feature-level fusion of dual-modal heterogeneous information; the Transformer decoder with normalized spatial position coding is used to predict the spatial position and size of the target. The present invention is suitable for the application conditions of complex aviation industry parts assembly defect detection, can effectively improve the three-dimensional positioning effect of parts, and provide a prerequisite for the downstream parts assembly morphological defect judgment, which is of great significance. The present invention is an accurate and efficient dual-modal three-dimensional target detection method for small parts of aircraft engines.

[0114] Figure 8 The present invention is a block diagram of a dual-mode 3D target detection device for small aircraft engine parts according to an exemplary embodiment. The device is used in a dual-mode 3D target detection method for small aircraft engine parts. Figure 8 The device includes a database construction module 810, a model construction module 820, a model training module 830 and a target detection module 840. Among them:

[0115] The database construction module 810 is used to scan the surface of the aircraft engine using a high-precision three-dimensional scanner to obtain training test data; and to construct an aircraft engine dual-modal database based on the training test data;

[0116] A model building module 820 is used to build a 3D object detection model to be trained based on a Transformer model structure and a multi-layer perceptron;

[0117] The model training module 830 is used to call the aircraft engine dual-modal database to train the three-dimensional target detection model to be trained to obtain the three-dimensional target detection model;

[0118] The target detection module 840 is used to use a high-precision scanner to scan the surface of the aircraft engine to be inspected to obtain target detection data; the target detection data is input into the three-dimensional target detection model to obtain small parts target detection results.

[0119] Among them, the training detection data includes three-dimensional point cloud data, image data, scanner camera pose and camera internal parameter data;

[0120] 3D point cloud data is used to express the geometric shape information of the aircraft engine surface;

[0121] Image data is used to express the texture information of the aircraft engine surface.

[0122] Among them, the three-dimensional object detection model includes a bimodal feature extraction module, a dense bimodal feature fusion module and a detection result prediction module based on the Transformer decoder.

[0123] Optionally, the target detection module 840 is further configured to:

[0124] According to the target detection data, the target three-dimensional point cloud data, the target image data, the current scanner camera posture and the current camera internal parameter data are obtained;

[0125] Perform high-dimensional feature extraction on the target three-dimensional point cloud data and the target image data to obtain a target point cloud feature map and a target image feature map;

[0126] Based on the channel attention mechanism, according to the current scanner camera posture and the current camera internal parameter data, the target point cloud feature map and the target image feature map are feature fused to obtain the second fused feature map;

[0127] Performing position encoding on the second fused feature map to obtain a third fused feature map;

[0128] Based on the Transformer decoder and the multi-layer perceptron, the target detection result is predicted according to the third fusion feature map to obtain the small parts target detection result.

[0129] Optionally, the target detection module 840 is further configured to:

[0130] Based on the deep graph convolutional neural network, feature extraction is performed based on the target 3D point cloud data to obtain the target point cloud feature map;

[0131] Based on the lightweight feature extraction backbone network and the feature pyramid network, feature extraction is performed according to the target image data to obtain the target image feature map.

[0132] Optionally, the target detection module 840 is further configured to:

[0133] Based on the target image feature map, under the guidance of the current scanner camera posture and the current camera internal parameters, the target point cloud feature map is densified to obtain a target dense point cloud feature map; the size of the target dense point cloud feature map is consistent with the size of the target image feature map;

[0134] Cascading and splicing the target dense point cloud feature map and the target image feature map to obtain the first fusion feature;

[0135] Based on the channel self-attention mechanism, the channel features of the first fusion feature are integrated to obtain the second fusion feature.

[0136] Optionally, the target detection module 840 is further configured to:

[0137] Performing two-dimensional sinusoidal position encoding on the second fused feature map to obtain an encoded fused feature map;

[0138] The encoded fused feature map is normalized into three-dimensional position encoding to obtain a third fused feature map.

[0139] The present invention proposes a dual-modal three-dimensional target detection method for small parts of aircraft engines. The method is based on the surface image and three-dimensional point cloud of the aircraft engine, and makes full use of the dual-modal information to make up for the problem of low positioning accuracy of small parts caused by insufficient single-modal information representation; the image data can fully represent the color, state and other texture information of the target; the three-dimensional point cloud data can accurately reflect the real spatial position and geometric shape of the object. The dual-modal feature extraction module is used to extract more significant high-dimensional dual-modal information; the densified feature fusion module is used to realize the feature-level fusion of dual-modal heterogeneous information; the Transformer decoder with normalized spatial position coding is used to predict the spatial position and size of the target. The present invention is suitable for the application conditions of complex aviation industry parts assembly defect detection, can effectively improve the three-dimensional positioning effect of parts, and provide a prerequisite for the downstream parts assembly morphological defect judgment, which is of great significance. The present invention is an accurate and efficient dual-modal three-dimensional target detection method for small parts of aircraft engines.

[0140] Fig. 9 is a schematic diagram of the structure of a dual-mode three-dimensional target detection device provided by an embodiment of the present invention, such as Fig. 9 As shown, the dual-modal three-dimensional target detection device may include the above Figure 8 The dual-modal 3D target detection device for small aircraft engine parts is shown. Optionally, the dual-modal 3D target detection device 510 may include a first processor 2001 .

[0141] Optionally, the dual-modal three-dimensional object detection device 510 may further include a memory 2002 and a transceiver 2003 .

[0142] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0143] Combine the following Fig. 9 The components of the dual-modal three-dimensional object detection device 510 are specifically introduced:

[0144] The first processor 2001 is the control center of the dual-modal three-dimensional target detection device 510, and may be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0145] Optionally, the first processor 2001 may perform various functions of the dual-modal three-dimensional object detection device 510 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0146] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Fig. 9 CPU0 and CPU1 are shown in FIG.

[0147] In a specific implementation, as an embodiment, the dual-modal 3D object detection device 510 may also include multiple processors, such as Fig. 9 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0148] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0149] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Fig. 9 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0150] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0151] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Fig. 9 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0152] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and communicate with the first processor 2001 through the interface circuit ( Fig. 9 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0153] It should be noted that Fig. 9 The structure of the dual-modal three-dimensional target detection device 510 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0154] In addition, the technical effects of the dual-modal three-dimensional target detection device 510 can refer to the technical effects of the dual-modal three-dimensional target detection method for small aircraft engine parts described in the above method embodiment, and will not be repeated here.

[0155] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0156] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0157] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0158] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0159] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0160] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0161] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0162] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0163] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0164] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0166] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0167] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A dual-modal three-dimensional target detection method for small aircraft engine parts, characterized in that: The method comprises: Using a high-precision three-dimensional scanner to scan the surface of the aircraft engine to obtain training test data; constructing an aircraft engine dual-modal database based on the training test data; Build a 3D object detection model to be trained based on the Transformer model structure and multi-layer perceptron; Calling the aircraft engine dual-modal database to train the three-dimensional target detection model to be trained to obtain a three-dimensional target detection model; A high-precision scanner is used to scan the surface of the aircraft engine to be inspected to obtain target detection data; the target detection data is input into the three-dimensional target detection model to obtain small parts target detection results.

2. The dual-modal three-dimensional target detection method for small aircraft engine parts according to claim 1 is characterized in that: The training detection data includes three-dimensional point cloud data, image data, scanner camera posture and camera internal parameter data; The three-dimensional point cloud data is used to express the geometric shape information of the aircraft engine surface; The image data is used to express texture information of the aircraft engine surface.

3. The dual-modal three-dimensional target detection method for small aircraft engine parts according to claim 1 is characterized in that: The three-dimensional object detection model includes a bimodal feature extraction module, a densified bimodal feature fusion module and a detection result prediction module based on a Transformer decoder.

4. The dual-modal three-dimensional target detection method for small aircraft engine parts according to claim 1, characterized in that: The step of inputting the target detection data into the three-dimensional target detection model to obtain a small parts target detection result includes: According to the target detection data, obtain target three-dimensional point cloud data, target image data, current scanner camera posture and current camera internal parameter data; Performing high-dimensional feature extraction on the target three-dimensional point cloud data and the target image data to obtain a target point cloud feature map and a target image feature map; Based on the channel attention mechanism, according to the current scanner camera posture and the current camera internal parameter data, the target point cloud feature map and the target image feature map are feature-fused to obtain a second fused feature map; Performing position encoding on the second fused feature map to obtain a third fused feature map; Based on the Transformer decoder and the multi-layer perceptron, the target detection result is predicted according to the third fusion feature map to obtain the small part target detection result.

5. The dual-modal three-dimensional target detection method for small aircraft engine parts according to claim 4 is characterized in that: The step of extracting high-dimensional features from the target three-dimensional point cloud data and the target image data to obtain a target point cloud feature map and a target image feature map includes: Based on a deep graph convolutional neural network, feature extraction is performed according to the target three-dimensional point cloud data to obtain a target point cloud feature map; Based on a lightweight feature extraction backbone network and a feature pyramid network, feature extraction is performed according to the target image data to obtain a target image feature map.

6. The dual-modal three-dimensional target detection method for small aircraft engine parts according to claim 4 is characterized in that: The method of performing feature fusion on the target point cloud feature map and the target image feature map based on the channel attention mechanism according to the current scanner camera posture and the current camera internal parameter data to obtain a second fused feature map includes: Based on the target image feature map, under the guidance of the current scanner camera posture and the internal parameters of the current camera, the target point cloud feature map is densified to obtain a target dense point cloud feature map; the size of the target dense point cloud feature map is consistent with the size of the target image feature map; Cascading and splicing the target dense point cloud feature map and the target image feature map to obtain a first fusion feature; Based on the channel self-attention mechanism, the channel features of the first fusion feature are integrated to obtain the second fusion feature.

7. The dual-modal three-dimensional target detection method for small aircraft engine parts according to claim 4 is characterized in that: The performing position encoding on the second fused feature map to obtain a third fused feature map includes: Performing two-dimensional sinusoidal position encoding on the second fused feature map to obtain an encoded fused feature map; Normalized three-dimensional position encoding is performed on the encoded fused feature map to obtain a third fused feature map.

8. A dual-modal 3D target detection device for small aircraft engine parts, the dual-modal 3D target detection device for small aircraft engine parts is used to implement the dual-modal 3D target detection method for small aircraft engine parts according to any one of claims 1 to 7, characterized in that: The device comprises: A database construction module is used to scan the surface of the aircraft engine using a high-precision three-dimensional scanner to obtain training test data; and to construct an aircraft engine dual-modal database based on the training test data; Model building module, used to build a 3D object detection model to be trained based on the Transformer model structure and multi-layer perceptron; A model training module, used to call the aircraft engine dual-modal database, train the three-dimensional target detection model to be trained, and obtain a three-dimensional target detection model; The target detection module is used to use a high-precision scanner to scan the surface of the aircraft engine to be detected to obtain target detection data; the target detection data is input into the three-dimensional target detection model to obtain small parts target detection results.

9. A dual-modal three-dimensional target detection device, characterized in that: The dual-mode three-dimensional target detection device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images