In-situ three-dimensional reconstruction method for weak-texture narrow inner cavity of aero-engine

Abstract

The invention relates to an in-situ three-dimensional reconstruction method for a weak-texture narrow inner cavity of an aero-engine, and relates to the technical field of reverse modeling, and the method comprises the steps: obtaining a left-eye visual angle sequence image and a right-eye visual angle sequence image, collected based on a binocular borescope, of a key part in the inner cavity of the aero-engine, constructing local point cloud data of the key component according to the left eye visual angle sequence image and the right eye visual angle sequence image; determining feature matching vectors among the key components according to the left eye visual angle sequence image, and determining feature point matching pairs among the key components according to the feature matching vectors; determining an essential matrix of a left side borescope in the binocular borescope according to the feature point matching pairs, and determining pose state parameters of the left side borescope according to the essential matrix of the left side borescope; and performing in-situ three-dimensional reconstruction on the weak-texture narrow inner cavity of the aero-engine according to the pose state parameters and the local point cloud data. The accuracy of the three-dimensional reconstruction result is improved.

Description

Background Art

[0002] In the existing methods for in-situ 3D reconstruction of the inner cavity of an aircraft engine, most of them adopt the off-site 3D reconstruction approach to deal with the narrow area with weak texture in the aircraft engine. However, this method has the problems of cumbersome operation procedures and low efficiency, and also has the problem of low accuracy of reconstruction results.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The purpose of the present disclosure is to provide an in-situ three-dimensional reconstruction method for a narrow inner cavity with weak texture of an aircraft engine, thereby at least to a certain extent overcoming the problems of cumbersome operating procedures, low efficiency and low accuracy of reconstruction results caused by the limitations and defects of related technologies.

[0005] According to one aspect of the present disclosure, a method for in-situ three-dimensional reconstruction of a narrow inner cavity with weak texture of an aircraft engine is provided, comprising:

[0006] Acquire a left-eye perspective sequence image and a right-eye perspective sequence image of a key component in an inner cavity of an aircraft engine collected by a binocular borescope, and construct local point cloud data of the key component based on the left-eye perspective sequence image and the right-eye perspective sequence image;

[0007] Determine a feature matching vector between the key components according to the left-eye perspective sequence image, and determine a feature point matching pair between the key components according to the feature matching vector;

[0008] Determine the intrinsic matrix of the left borescope in the binocular borescope according to the feature point matching pair, and determine the posture state parameters of the left borescope according to the intrinsic matrix of the left borescope;

[0009] According to the relative position parameters, posture state parameters and local point cloud data, the narrow inner cavity with weak texture of the aircraft engine is reconstructed in situ in three dimensions.

[0010] In an exemplary embodiment of the present disclosure, constructing the local point cloud data of the key component according to the left-eye perspective sequence image and the right-eye perspective sequence image includes:

[0011] Calculating the inner cavity disparity map between the left-eye perspective sequence image and the right-eye perspective sequence image, and obtaining the camera intrinsic parameters, camera extrinsic parameters and the baseline distance between the binocular borescope and the inner cavity of the aircraft engine of the binocular borescope;

[0012] The inner cavity disparity map is converted according to the camera internal parameters and the baseline distance to obtain a depth map of the actual distance between the inner cavity of the aircraft engine and the binocular borescope;

[0013] A conversion relationship between a world coordinate system and a camera image coordinate system is established based on the camera extrinsic parameters, and the depth map is converted to the world coordinate system based on the conversion relationship to obtain local point cloud data of the key component.

[0014] In an exemplary embodiment of the present disclosure, determining the feature matching vectors between the key components according to the left-eye perspective sequence images includes:

[0015] Determining texture feature points of the key component according to the left-eye perspective sequence image;

[0016] Performing graph convolution processing and feature enhancement processing on the texture feature points based on a graph neural network model to obtain feature point information and visual descriptors of the texture feature points;

[0017] A feature matching vector between the key components is generated according to the feature point information and the visual descriptor.

[0018] In an exemplary embodiment of the present disclosure, determining a feature point matching pair between the key components according to the feature matching vector includes:

[0019] Calculating the inner product between the feature matching vectors to obtain a soft allocation matrix between texture feature points, and optimizing the soft allocation matrix based on a preset optimal allocation rule to obtain a matching score matrix between the key components;

[0020] The optimal solution of the matching score matrix is ​​solved using the Kuhn-Tucker necessary condition to obtain the feature point matching pairs between the key components.

[0021] In an exemplary embodiment of the present disclosure, the preset optimal allocation rule is as follows:

[0022]

[0023] Among them, the left-eye perspective sequence image includes a first left-eye perspective sequence image P1 and a second left-eye perspective sequence image P2, P1 and P2 are two adjacent left-eye perspective sequence images; P1 includes m texture feature points, and P2 includes n texture feature points; T(i,j) is a soft allocation matrix between the i-th feature point in P1 and the j-th feature point in P2.

[0024] In an exemplary embodiment of the present disclosure, the Kuhn-Tucker necessary conditions are as follows:

[0025]

[0026] Wherein, T(i,j) is the soft assignment matrix between the i-th feature point in P1 and the j-th feature point in P2, P1 includes m texture feature points, and P2 includes n texture feature points; μ k is the Lagrange multiplier.

[0027] In an exemplary embodiment of the present disclosure, the feature point matching pair includes a first texture feature point in a first left-eye perspective sequence image and a second texture feature point in a second left-eye perspective sequence image corresponding to the first texture feature;

[0028] Wherein, determining the essential matrix of the left borescope in the binocular borescope according to the feature point matching pair includes:

[0029] Normalizing the first coordinate point of the first texture feature point based on a preset first translation and scaling transformation parameter to obtain a first normalization result;

[0030] Normalizing the second coordinate point of the second texture feature point based on a preset second translation and scaling transformation parameter to obtain a second normalization result;

[0031] Constructing a first linear equation system according to the first normalization processing result, the second normalization processing result and the basic matrix to be solved;

[0032] Solving the first linear equation group to obtain the basic matrix; wherein the basic matrix encodes the affine transformation between the first left-eye perspective sequence image and the second left-eye perspective sequence image, and estimates the camera stereo motion parameters and camera stereo calibration parameters of the left borescope from the first left-eye perspective sequence image and the second left-eye perspective sequence image;

[0033] A target matching pair is determined from the feature point matching pairs according to the basic matrix, and an essential matrix of a left borescope in the binocular borescope is determined according to the target matching pair.

[0034] In an exemplary embodiment of the present disclosure, determining the intrinsic matrix of the left borescope in the binocular borescope according to the target matching pair includes:

[0035] Establishing a second linear equation group according to the target matching pair, and solving the second linear equation group using a linear algebra method to obtain an essential matrix of a left borescope in the binocular borescope;

[0036] The essential matrix describes the posture state parameters of the left-side boresight corresponding to the first left-eye perspective sequence image and the second left-eye perspective sequence image.

[0037] In an exemplary embodiment of the present disclosure, determining the posture state parameters of the left-side borescope according to the intrinsic matrix of the left-side borescope includes:

[0038] Decomposing the essential matrix of the left borescope based on a preset singular value decomposition algorithm to obtain the posture state parameters of the left borescope relative to the world coordinate system;

[0039] The posture state parameters include a rotation matrix of the left borescope relative to a world coordinate system and a translation vector of the left borescope relative to a world coordinate system.

[0040] In an exemplary embodiment of the present disclosure, based on the posture state parameters and the local point cloud data, an in-situ three-dimensional reconstruction of the narrow inner cavity of the aircraft engine with weak texture is performed, including:

[0041] The local point cloud data is aligned and registered according to the rotation matrix and translation vector in the posture state parameters to achieve in-situ three-dimensional reconstruction of the weakly textured narrow inner cavity of the aircraft engine.

[0042] An in-situ 3D reconstruction method for a weakly textured narrow inner cavity of an aircraft engine provided by an embodiment of the present disclosure comprises the following steps: on the one hand, obtaining a left-eye perspective sequence image and a right-eye perspective sequence image of key components in the inner cavity of the aircraft engine collected by a binocular borescope, and constructing local point cloud data of the key components based on the left-eye perspective sequence image and the right-eye perspective sequence image; then determining feature matching vectors between key components based on the left-eye perspective sequence image, and determining feature point matching pairs between key components based on the feature matching vectors; and then determining the essential matrix of the left borescope in the binocular borescope based on the feature point matching pairs, and determining the local point cloud data of the key components based on the local point cloud data of the key components; and finally determining the local point cloud data of the key components based on the local point cloud data of the key components. Determine the posture state parameters of the left borescope; finally, according to the relative position parameters, posture state parameters and local point cloud data, perform in-situ 3D reconstruction of the weak-textured narrow inner cavity of the aircraft engine, thereby improving the accuracy of the 3D reconstruction result and solving the problem of low accuracy of the reconstruction result in the prior art; on the other hand, since the left-eye perspective sequence images and the right-eye perspective sequence images can be collected, the weak-textured narrow inner cavity of the aircraft engine can be automatically reconstructed in situ according to the left-eye perspective sequence images and the right-eye perspective sequence images, thereby improving the efficiency of the 3D reconstruction and solving the problem of low efficiency in the prior art due to cumbersome operating procedures.

[0043] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0045] Figure 1 A flow chart schematically illustrates a method for in-situ three-dimensional reconstruction of a narrow inner cavity with weak texture of an aircraft engine according to an exemplary embodiment of the present disclosure.

[0046] Figure 2 An example diagram schematically shows a left-eye perspective sequence image and a right-eye perspective sequence image according to an example embodiment of the present disclosure.

[0047] Figure 3 A flowchart of a method for determining an essential matrix of a left borescope in a binocular borescope according to a feature point matching pair according to an exemplary embodiment of the present disclosure is schematically shown.

[0048] Figure 4 An example diagram schematically illustrates a feature matching pair obtained based on different algorithms according to an example embodiment of the present disclosure.

[0049] Figure 5 Another example diagram schematically illustrates feature matching pairs obtained based on different algorithms according to an example embodiment of the present disclosure.

[0050] Figure 6 The following diagram schematically shows an example result of in-situ three-dimensional reconstruction of a weakly textured narrow inner cavity of an aircraft engine according to an example embodiment of the present disclosure.

[0051] Figure 7 A block diagram schematically shows an in-situ three-dimensional reconstruction device for a narrow inner cavity of an aircraft engine with weak texture according to an exemplary embodiment of the present disclosure.

[0052] Figure 8 An electronic device for implementing an in-situ three-dimensional reconstruction method of a narrow inner cavity with weak texture of an aircraft engine according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0053] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0054] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0055] Since the internal structure of aircraft engines is sophisticated, including core components such as high-pressure compressors, combustion chambers, turbine discs, and guide vanes (also known as key components), these components are constantly subjected to extreme conditions of high temperature, high pressure, high-speed rotation, and complex and changeable airflow, which can easily cause problems such as wear, corrosion, and fatigue cracks, which directly threaten the performance and flight safety of the engine. Therefore, given the core position of aircraft engines as the power source of aircraft, real-time monitoring of their internal conditions and determining whether they meet flight requirements have become a key link in aviation maintenance and are crucial to ensuring flight safety. At present, the mainstream detection method relies on technicians to use industrial endoscopes to conduct visual inspections deep inside the engine. This process is not only time-consuming, but also as the detection time increases, the risk of omissions also increases, highlighting the importance of exploring auxiliary detection methods.

[0056] In this context, 3D reconstruction technology has introduced a new way for rapid inspection and maintenance of aircraft engines, and is particularly critical for establishing a digital twin model that can monitor the health status of the engine in real time. In actual application, the construction of this model is highly dependent on the 3D reconstruction technology of key components. However, these components often have complex narrow cavity structures and face challenges such as being difficult to reach by hand, poor lighting conditions, and a shortage of feature points for reference. In particular, 3D reconstruction of narrow cavities with sparse textures has become an urgent need in modern industry and a technical difficulty to overcome in the field of reverse engineering.

[0057] Existing solutions mostly use off-site 3D reconstruction to deal with small areas with weak textures in aircraft engines. This method is not only cumbersome and inefficient, but also requires the blades to be disassembled and performed in a laboratory environment, which greatly increases the complexity of blade reconstruction, limits the ability of inspectors to obtain blade structural information in real time in the field, and delays the immediate assessment of the engine status. In addition, the traditional technical route of using a monocular camera to restore a complete point cloud from a 2D image is difficult to effectively achieve accurate reconstruction of a small space when the blade surface is smooth, the texture information is insufficient, and the lighting environment inside the engine is complex.

[0058] Therefore, it is extremely urgent and of great practical value to develop a cost-effective and fast technical solution that can achieve accurate image registration through advanced algorithms even in an environment where feature points are scarce, and combine the collaborative analysis of local point clouds and image data to generate high-quality complete point clouds for small areas.

[0059] Based on this, this exemplary embodiment first provides an in-situ 3D reconstruction method for a narrow inner cavity of an aircraft engine with weak textures. This method can be run on a terminal device, server, server cluster or cloud server, etc. Of course, those skilled in the art can also run the method disclosed in this disclosure on other platforms as required, and this exemplary embodiment does not specifically limit this. Specifically, refer to Figure 1 As shown, the in-situ 3D reconstruction method of the narrow inner cavity of the aircraft engine with weak texture may include the following steps:

[0060] Step S110. Acquire a sequence of left-eye perspective images and a sequence of right-eye perspective images of key components in the inner cavity of the aircraft engine collected by the binocular borescope, and construct local point cloud data of the key components based on the sequence of left-eye perspective images and the sequence of right-eye perspective images;

[0061] Step S120. Determine the feature matching vectors between the key components according to the left-eye perspective sequence images, and determine the feature point matching pairs between the key components according to the feature matching vectors;

[0062] Step S130. Determine the intrinsic matrix of the left borescope in the binocular borescope according to the feature point matching pair, and determine the posture state parameters of the left borescope according to the intrinsic matrix of the left borescope;

[0063] Step S140. Perform in-situ three-dimensional reconstruction of the narrow inner cavity with weak texture of the aircraft engine according to the relative position parameters, posture state parameters and local point cloud data.

[0064] In the above-mentioned in-situ three-dimensional reconstruction method of a narrow inner cavity of an aircraft engine with weak texture, on the one hand, a left-eye perspective sequence image and a right-eye perspective sequence image of the key components in the inner cavity of the aircraft engine collected by a binocular boresight are obtained, and local point cloud data of the key components are constructed based on the left-eye perspective sequence image and the right-eye perspective sequence image; then, the feature matching vectors between the key components are determined based on the left-eye perspective sequence image, and the feature point matching pairs between the key components are determined based on the feature matching vectors; then, the essential matrix of the left boresight in the binocular boresight is determined based on the feature point matching pairs, and the left boresight is determined based on the essential matrix of the left boresight. The posture state parameters of the detector; finally, according to the relative position parameters, posture state parameters and local point cloud data, the weak-textured narrow inner cavity of the aircraft engine is reconstructed in situ 3D, thereby improving the accuracy of the 3D reconstruction result and solving the problem of low accuracy of the reconstruction result in the prior art; on the other hand, since the left-eye perspective sequence images and the right-eye perspective sequence images can be collected, the weak-textured narrow inner cavity of the aircraft engine can be automatically reconstructed in situ according to the left-eye perspective sequence images and the right-eye perspective sequence images, thereby improving the efficiency of the 3D reconstruction and solving the problem of low efficiency in the prior art due to cumbersome operating procedures.

[0065] Hereinafter, the in-situ three-dimensional reconstruction method of a narrow inner cavity with weak texture of an aircraft engine described in an exemplary embodiment of the present disclosure will be explained and illustrated in detail with reference to the accompanying drawings.

[0066] First, the technical implementation principle of the example embodiment of the present disclosure is explained and illustrated. Specifically, the in-situ three-dimensional reconstruction method of the narrow inner cavity of the weak texture of the aircraft engine recorded in the example embodiment of the present disclosure first calibrates the binocular borescope and collects data, obtains the parallax of the single frame and the corresponding left and right images through binocular stereo vision, and reconstructs to generate a local point cloud based on the parallax principle; secondly, selects a series of 2D images collected by the camera on the left side of the borescope as a new data set, obtains the feature matching vector based on GNN, and then uses the known feature matching vector to obtain the matching pairs between the left sequence diagrams of the key components; then, obtains the essential matrix of the left borescope through the left sequence diagram and the matching pairs on it, and performs SVD decomposition on it to obtain the left borescope posture, that is, the rotation matrix and translation vector ; Finally, a series of parameters obtained through the left sequence diagram are mapped back to the binocular stereo vision, and the point cloud obtained in combination with binocular imaging is rotated and translated to the target point cloud for registration; the in-situ three-dimensional reconstruction method of the narrow inner cavity of the aircraft engine with weak texture recorded in the exemplary embodiment of the present disclosure can overcome the challenges of insufficient light and scarce feature points without disassembling engine components, and realize high-precision and high-efficiency reconstruction of the narrow cavity with smooth blade surface, weak texture information, and complex internal light of the engine through advanced image registration algorithm and comprehensive point cloud image data analysis; this technological advancement not only greatly shortens the detection cycle and improves the real-time monitoring capability, but also simplifies the maintenance process and reduces costs.

[0067] Secondly, Figure 1 The in-situ 3D reconstruction method of the narrow inner cavity of the aircraft engine with weak texture shown in the figure is further explained and illustrated. Specifically:

[0068] In step S110, a left-eye perspective sequence image and a right-eye perspective sequence image of a key component in an inner cavity of an aircraft engine acquired based on a binocular borescope are obtained, and local point cloud data of the key component are constructed based on the left-eye perspective sequence image and the right-eye perspective sequence image.

[0069] In this example embodiment, first, a sequence of left-eye perspective images and a sequence of right-eye perspective images of key components in the inner cavity of an aircraft engine acquired based on a binocular borescope are acquired; wherein the inner cavity of an aircraft engine recorded here refers to an inner cavity of an aircraft engine having the characteristics of a smooth blade surface, weak texture information, and complex light; the key components recorded here may include but are not limited to core components such as a high-pressure compressor, a combustion chamber, a turbine disc, and a guide; at the same time, after acquiring a sequence of left-eye perspective images and a sequence of right-eye perspective images of key components in the inner cavity of an aircraft engine, Before that, it is also necessary to calibrate the binocular borescope to obtain the camera parameters of the binocular borescope, including the camera intrinsic parameters (focal length and pixel size of the camera) and the camera extrinsic parameters (position and rotation direction of the camera), etc.; and on this basis, establish the conversion relationship between the world coordinate system and the camera image coordinate system; then, use the binocular borescope to collect multi-view images of the key components of the inner cavity of the aircraft engine and construct a first data set; at the same time, the left-eye perspective sequence image and the right-eye perspective sequence image can be directly obtained from the first data set; at the same time, the obtained left-eye perspective sequence image can refer to Figure 2 As shown in (a) of the figure, the obtained right eye perspective sequence image can be referred to Figure 2 As shown in (b) in .

[0070] Secondly, constructing the local point cloud data of the key component according to the left-eye perspective sequence image and the right-eye perspective sequence image; specifically, it can be achieved in the following way: first, calculating the inner cavity disparity map between the left-eye perspective sequence image and the right-eye perspective sequence image, and obtaining the camera intrinsic parameters, camera extrinsic parameters and the baseline distance between the binocular boresight and the inner cavity of the aircraft engine; secondly, converting the inner cavity disparity map according to the camera intrinsic parameters and the baseline distance to obtain a depth map of the actual distance between the inner cavity of the aircraft engine and the binocular boresight; then, establishing a conversion relationship between the world coordinate system and the camera image coordinate system based on the camera extrinsic parameters, and converting the depth map to the world coordinate system based on the conversion relationship to obtain the local point cloud data of the key component. That is, in the actual application process, the disparity of a single frame and the corresponding left and right eye perspective sequence images can be obtained through binocular stereo vision, and then reconstructed based on the disparity principle to obtain local point cloud data; at the same time, since the endoscope (i.e., borescope) is binocular, it is necessary to first obtain a single frame image to calculate the disparity map for subsequent steps. Furthermore, since there is a disparity between the corresponding feature points in the left and right eye perspective sequence images of the inner cavity, the pixel position difference of the inner cavity of the aircraft engine in the left and right images can be reflected by matching the features on the collected left and right sequence images and calculating the disparity map; further, relying on the camera intrinsic parameters and baseline distance of the binocular borescope, the inner cavity disparity map is converted into a depth map that provides the actual distance from the inner cavity to the camera; then, it is also necessary to rotate and translate the camera coordinate system by using the transformation matrix of the camera extrinsic parameters of the binocular borescope, and finally obtain the multi-frame inner cavity partial point cloud coordinates under the binocular perspective in the world coordinate system, that is, the local point cloud data.

[0071] In step S120, feature matching vectors between the key components are determined according to the left-eye perspective sequence images, and feature point matching pairs between the key components are determined according to the feature matching vectors.

[0072] In this example embodiment, first, the feature matching vectors between the key components are determined based on the left-eye perspective sequence images; specifically, this can be achieved in the following manner: first, the texture feature points of the key components are determined based on the left-eye perspective sequence images; secondly, the texture feature points are subjected to graph convolution processing and feature enhancement processing based on a graph neural network model to obtain feature point information and visual descriptors of the texture feature points; then, based on the feature point information and visual descriptors, the feature matching vectors between the key components are generated. Specifically, in the process of practical application, after obtaining the local point cloud data of the inner cavity through binocular vision, accurate feature point matching between images and obtaining the camera pose of each frame of data become urgent problems to be solved; and the inner cavity of an aircraft engine has a limited camera pose angle due to its narrow space, and the small number of feature points in the data set makes it difficult to find the correspondence between pixels or regions between multiple images, resulting in the inability of existing three-dimensional reconstruction algorithms to provide stable and reliable reconstruction results in the research environment; therefore, in order to improve the accuracy of matching between inner cavity images to improve the accuracy of inner cavity information fusion under multiple perspectives, firstly, a series of 2D images collected by the left camera of the borescope can be selected as the second data set (that is, the left eye perspective sequence image is extracted from the first data set as the second data set); then, the left eye perspective sequence image is obtained from the second data, and the texture feature points of the key components are extracted from the left sequence image of the inner cavity (left eye perspective sequence image) collected by the binocular borescope based on the feature point extraction model, and the attention GNN (Graph Neural Network) is used to obtain and enhance the performance of the feature matching vector, which contains the feature point information and visual descriptors of the extracted texture feature points.

[0073] Secondly, the feature point matching pairs between the key components are determined according to the feature matching vectors; specifically, this can be achieved in the following manner: the inner product between the feature matching vectors is calculated to obtain a soft allocation matrix between texture feature points, and the soft allocation matrix is ​​optimized based on a preset optimal allocation rule to obtain a matching score matrix between the key components; the optimal solution of the matching score matrix is ​​solved using the Kuhn-Tucker necessary condition to obtain the feature point matching pairs between the key components; wherein the optimal allocation rule preset here is shown in the following formula (1); the Kuhn-Tucker necessary condition recorded here can be shown in the following formula (2):

[0074] min-T(1,j),-T(2,j),...,-T(m,j)(j=1,2,...,n)

[0075]

[0076] The left-eye perspective image sequence includes a first left-eye perspective image sequence P1 and a second left-eye perspective image sequence P2, P1 and P2 are two adjacent left-eye perspective image sequences; P1 includes m texture feature points, and P2 includes n texture feature points; T(i, j) is a soft allocation matrix between the i-th feature point in P1 and the j-th feature point in P2; μ k is the Lagrange multiplier.

[0077] The following will further explain and illustrate the specific generation process of feature point matching pairs. Specifically, in the actual application process, first, the feature matching vector known in the previous step is used to calculate the inner product to obtain the matching score matrix, and the matching score matrix is ​​used as input to solve the optimal solution using the Kuhn-Tucker necessary condition to obtain the feature matching pairs between the left sequence diagrams of the key components; wherein, the matching score matrix calculation process is specifically as follows:

[0078] First, two left-side sequence images of the borescope are selected as the first left-eye perspective sequence image P1 and the second left-eye perspective sequence image P2. The jth feature contained in the second left-eye perspective sequence image P2 is recorded as k j =(x j ,y j ,c j ); where (x j ,y j ) is the coordinate position of the texture feature point, c j is the confidence of the feature point; at the same time, the i-th image P i Contains a i Dimensional features description; where k is the feature position and d is the descriptor (i.e., visual descriptor). At the same time, the optimal allocation rule for the feature points of images P1 and P2 is given in this example embodiment: Assume that T is a soft allocation matrix, T(m,n) is the matching degree between the mth feature point on the P1 image and the nth feature point on the P2 image; solve the constrained optimization problem about the allocation rule, and the specific constraint formula can be shown as the formula (1) recorded above; and, since the features of the narrow inner cavity are not obvious, the relative position relationship between the feature points and their visual appearance are used to enhance the representation of the feature points; under this premise, this example embodiment uses a hybrid self-attention mechanism and a cross-attention mechanism that simulates humans browsing back and forth to match key points to improve the matching accuracy; specifically, since the i-th image P i Contains a i The k-dimensional features can be expressed by the formula, so MLP can be used to transform the k-th feature point of the low dimension j =(x j ,y j ,c j ) Information Communication Descriptorj Establish initial coupling; the specific coupling process is shown in the following formula (3):

[0079]

[0080] Secondly, the feature information is iterated based on the hybrid attention mechanism to obtain the coupling relationship of the λth layer in the GNN algorithm; the specific coupling process can be shown in the following formula (4):

[0081]

[0082] in, is the a of the λth layer on the i-th image after the hybrid attention mechanism i Dimension feature information.

[0083] Furthermore, the feature matching vector of the 2D image of the inner cavity is obtained by feature fusion as the input in the matching process, which is used to calculate the matching score of the key points between the two images in the left sequence data set; i The feature matching vector output by the GNN network is recorded as (i) f j ; At the same time, the specific graph convolution process can be shown as follows:

[0084]

[0085] At the same time, by calculating a matching probability for each pair of feature points and constructing a soft assignment matrix T between the left sequence graphs, rather than performing a hard one-to-one matching, a more flexible and robust matching strategy is provided; specifically, assuming that the first two graphs of this set of left graphs are P1 and P2, and the feature vector is (1) f and (2) f, the inner product can be used to quantify the properties of the direction and size similarity of the two vectors and calculate the matching degree between the feature points in P1 and P2. The specific calculation process of the inner product can be shown in the following formula (6):

[0086] T(m,n)=< (1) f m · (2) f n >; Formula (6)

[0087] Then, by judging the size of the inner product between the two eigenvectors, the largest value is selected to match the two feature points into pairs, which are the matching pairs between the left sequence diagrams of the key components.

[0088] Furthermore, the exemplary embodiment of the present disclosure uses the Kuhn-Tucker necessary condition to solve the optimization problem with constraints for the optimal solution of T(m,n); the Lagrange multiplier μ is introduced in the process of solving the optimal matching.k (k=1,2,...,m+n+m·n), and regard each row of the matrix as the objective function t i (i=1,2,...,m) to search for the optimal solution m times. In addition, the constraints need to be described in the form of ≥ and add the Lagrange multiplier μ k Related complementary relaxation conditions; wherein the specific constraint conditions can be shown as the formula (2) recorded above.

[0089] In step S130, the intrinsic matrix of the left borescope in the binocular borescope is determined according to the feature point matching pair, and the posture state parameters of the left borescope are determined according to the intrinsic matrix of the left borescope.

[0090] In this example embodiment, first, the essential matrix of the left borescope in the binocular borescope is determined according to the feature point matching pair; wherein the feature point matching pair recorded here includes the first texture feature point in the first left-eye perspective sequence image and the second texture feature point in the second left-eye perspective sequence image corresponding to the first texture feature; under this premise, reference Figure 3 As shown, the specific calculation process of the essential matrix can include the following steps:

[0091] Step S310, performing normalization processing on the first coordinate point of the first texture feature point based on a preset first translation and scaling transformation parameter to obtain a first normalization processing result;

[0092] Step S320, performing normalization processing on the second coordinate point of the second texture feature point based on a preset second translation and scaling transformation parameter to obtain a second normalization processing result;

[0093] Step S330, constructing a first linear equation system according to the first normalization processing result, the second normalization processing result and the basic matrix to be solved;

[0094] Step S340, solving the first linear equation group to obtain the basic matrix; wherein the basic matrix encodes the affine transformation between the first left-eye perspective sequence image and the second left-eye perspective sequence image, and estimates the camera stereo motion parameters and camera stereo calibration parameters of the left borescope from the first left-eye perspective sequence image and the second left-eye perspective sequence image;

[0095] Step S350: determining a target matching pair from the feature point matching pairs according to the basic matrix, and determining an intrinsic matrix of a left borescope in the binocular borescope according to the target matching pair.

[0096] Among them, determining the essential matrix of the left boresight in the binocular boresight according to the target matching pair can be achieved in the following way: establishing a second linear equation group according to the target matching pair, and using linear algebra methods to solve the second linear equation group to obtain the essential matrix of the left boresight in the binocular boresight; wherein the essential matrix describes the posture state parameters of the left boresight corresponding to the first left-eye perspective sequence image and the second left-eye perspective sequence image.

[0097] The following will further explain and illustrate the specific calculation process of the essential matrix. Specifically, the feature point matching pairs calculated by the aforementioned steps can restore the three-dimensional structure of the inner cavity of the aircraft engine and the position and orientation of the camera and other pose states (that is, determine the pose state parameters of the left borescope). At the same time, in the process of determining the pose state parameters, the eight-point method can be used to solve the basic matrix F, which encodes the affine transformation relationship between the two images, and estimates the camera motion and stereo calibration from the two 2D images; at the same time, in the actual application process, in order to eliminate the influence of the eight feature points of the inner cavity image on the reconstruction result and achieve unbiased estimation, the eight-point method can be normalized; for example, assuming that the coordinate points of the left sequence diagram P1 and P2 of the inner cavity of the aircraft engine are the first coordinate point (x1, y1, 1) and the second coordinate point (x2, y2, 1); apply a translation and scaling transformation Φ to each image, the size of Φ is related to the mean and variance, and after normalization The linear equation of the matrix is The first linear equation system obtained can be expressed as the following formula (7):

[0098]

[0099] In the above formula, Φ1 is the translation and scaling transformation applied to each image coordinate point P1 (i.e., the first translation and scaling transformation parameter recorded above), and Φ2 is the translation and scaling transformation applied to each image coordinate point P2 (i.e., the second translation and scaling transformation parameter recorded above); further, the RANSAC algorithm is used to iteratively select the smallest point set containing eight points (i.e., the target matching pair), evaluate the accuracy of the model, and thus identify the inliers and outliers; specifically, Matrix inverse normalization; wherein, the specific implementation process of inverse normalization can be shown in the following formula (8):

[0100]

[0101] Furthermore, a linear equation is established for each pair of target matching pairs, and the linear equations corresponding to all points are combined to obtain a second linear equation group. After obtaining the second linear equation group, the linear algebra method can be used to solve the linear equation group about the matching points to obtain the essential matrix E. The specific construction process of the linear equation is similar to the above formula (7), and no further description is given here. The second linear equation group is solved to obtain the essential matrix. Moreover, since the essential matrix E describes the relative position and posture of the cameras corresponding to P1 and P2, it contains the relative rotation matrix R and the relative translation vector t. Therefore, the essential matrix can be expressed as follows by the following formula (9):

[0102] E=[t] × R; Formula (9)

[0103] Where, [t] × A skew-symmetric matrix describing the translation vector t.

[0104] Secondly, after obtaining the essential matrix, the posture state parameters of the left-side borescope can be determined according to the essential matrix of the left-side borescope; specifically, this can be achieved in the following way: based on the preset singular value decomposition algorithm, the essential matrix of the left-side borescope is decomposed to obtain the posture state parameters of the left-side borescope relative to the world coordinate system; wherein, the posture state parameters include the rotation matrix of the left-side borescope relative to the world coordinate system and the translation vector of the left-side borescope relative to the world coordinate system. That is, in the process of actual application, the essential matrix can be decomposed by the SVD (Singular Value Decomposition) algorithm, and then the R and t matrices of the left-side borescope can be restored from the essential matrix; wherein, the decomposition matrix E can be expressed as E=MΠN T The rotation matrix is ​​given by the formula R = MΥΝ Τ / ΜΥ T N T The translation vector is obtained by the formula t=±m3, where m3 is the third column of the matrix M; where M and N are both orthogonal matrices, Π is a diagonal matrix, and γ is a special antisymmetric matrix, which can be specifically shown in the following formula (10):

[0105]

[0106] In step S140, the weakly textured narrow inner cavity of the aircraft engine is reconstructed in situ in three dimensions according to the posture state parameters and the local point cloud data.

[0107] Specifically, the specific implementation process of in-situ 3D reconstruction of the narrow inner cavity of the aircraft engine with weak texture can be achieved by the following method: aligning and registering the local point cloud data according to the rotation matrix and translation vector in the posture state parameters to achieve in-situ 3D reconstruction of the narrow inner cavity of the aircraft engine with weak texture. Specifically, in the actual application process, the left sequence image data set P i (i.e., the second data set) is mapped back to the binocular vision system, that is, a 3D reconstruction is obtained; wherein, the local point cloud data restored by the binocular stereo vision can be recorded as C i , the local point cloud data can be converted into the target point cloud coordinate system by solving R and t to realize the alignment and registration of the local point cloud of the inner cavity, thereby realizing the process of three-dimensional reconstruction; wherein, the specific implementation method of the registration and alignment process can be shown in the following formula (11):

[0108] C j =RC i +t (i≠j); Formula (11)

[0109] It should be noted here that, in the process of three-dimensional reconstruction, first, the feature point pairs between the 2D images of the inner cavity obtained by the matching algorithm in the collected 2D left sequence data set (that is, the second data set) (that is, the feature point matching pairs recorded above) can be mapped back to the binocular vision system, that is, a preliminary three-dimensional reconstruction image can be obtained, and then the local point cloud data restored by the binocular vision system is converted into the target point cloud coordinate system through R and t transformation for alignment and registration, so as to achieve the process of optimizing the preliminary three-dimensional reconstruction image, and finally realize the in-situ three-dimensional reconstruction of the weakly textured narrow inner cavity of the aircraft engine, thereby achieving the purpose of improving the accuracy of the three-dimensional reconstruction results.

[0110] Figure 4 as well as Figure 5 Figure 1 is a schematic diagram of some results under different matching algorithms; Figure 4 as well as Figure 5 In the example diagrams shown, (a) corresponds to the SIFT algorithm, (b) corresponds to the ORB algorithm, (c) corresponds to the SURF algorithm, and (d) corresponds to the algorithm described in the example embodiment of the present disclosure; OUR represents the algorithm disclosed in the example embodiment; further, it can be seen from the image results that the present embodiment has a better effect on generating left sequence image matching pairs, and the algorithm proposed in the present embodiment has more matching pairs and more accurate results. The number of matching pairs of some experiments of different algorithms under a confidence level of 75% for a weak texture left sequence image is shown in Table 1;

[0111] Table 1 Number of matching pairs of some experiments of different algorithms under 75% confidence level for weak texture left sequence images

[0112]

[0113] As can be seen from Table 1, in the case of weak texture, SIFT and ORB have extremely poor effects, which makes it almost impossible to complete image matching in low light and with few feature points, so that it is difficult to provide assistance for subsequent 3D reconstruction. The effect of SURF is slightly more significant than the above two algorithms, but due to the existence of mismatched pairs in the matching, the accuracy of subsequent reconstruction cannot be met. The method proposed in the exemplary embodiment of the present disclosure has a high degree of recognition of feature points in weak texture areas, and the number and accuracy of matching pairs are much higher than the above solutions, providing an accurate matching basis for subsequent reconstruction.

[0114] Figure 6 (a) is a three-dimensional reconstruction image obtained by the prior art. Figure 6 (b) is a three-dimensional reconstruction image obtained based on the method described in the exemplary embodiment of the present disclosure; Figure 6 It can be concluded that the three-dimensional reconstruction image obtained by the exemplary embodiment of the present disclosure can effectively solve the problem of poor reconstruction effect of weak-texture inner cavity.

[0115] At this point, the in-situ three-dimensional reconstruction method of the narrow inner cavity of an aircraft engine with weak textures recorded in the exemplary embodiments of the present disclosure has been fully realized. Based on the above-mentioned contents, it can be known that the in-situ three-dimensional reconstruction method of the narrow inner cavity of the aircraft engine with weak texture recorded in the exemplary embodiment of the present disclosure can solve the problems of low detection efficiency, limited accuracy and inability to evaluate the engine status in real time in the prior art; at the same time, the present disclosure proposes an innovative in-situ three-dimensional reconstruction technology for the detection problem of the narrow inner cavity with weak texture inside the aircraft engine, which can overcome the challenges of insufficient light and scarcity of feature points without disassembling the engine components, and realize high-precision and high-efficiency reconstruction of the narrow cavity with the characteristics of smooth blade surface, weak texture information, and complex internal light of the engine through advanced image registration algorithm and comprehensive point cloud image data analysis; this technological advancement not only greatly shortens the detection cycle and improves the real-time monitoring capability, but also simplifies the maintenance process and reduces the cost, while maintaining a high degree of operational flexibility and accuracy in complex environments, and solves the problem that the traditional inner cavity reconstruction scheme cannot form a three-dimensional model due to the limitation of the camera motion range and the poor matching effect of feature points, and provides a set of fast, accurate and cost-effective solutions for aircraft engine maintenance, which is an important technological breakthrough to ensure flight safety and improve aviation operation and maintenance efficiency.

[0116] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0117] The exemplary embodiment of the present disclosure also provides an in-situ three-dimensional reconstruction device for a narrow inner cavity of an aircraft engine with weak texture. Figure 7 As shown, the in-situ 3D reconstruction device for a narrow inner cavity of an aircraft engine with weak texture may include a local point cloud data calculation module 710, a feature point matching pair determination module 720, a posture state parameter determination module 730 and a 3D reconstruction module 740. Among them:

[0118] The local point cloud data calculation module 710 can be used to obtain a left-eye perspective sequence image and a right-eye perspective sequence image of a key component in the inner cavity of an aircraft engine collected by a binocular borescope, and construct local point cloud data of the key component based on the left-eye perspective sequence image and the right-eye perspective sequence image;

[0119] The feature point matching pair determination module 720 may be used to determine the feature matching vectors between the key components according to the left-eye perspective sequence images, and determine the feature point matching pairs between the key components according to the feature matching vectors;

[0120] A posture state parameter determination module 730 may be used to determine an intrinsic matrix of a left borescope in the binocular borescope according to the feature point matching pair, and determine a posture state parameter of the left borescope according to the intrinsic matrix of the left borescope;

[0121] The three-dimensional reconstruction module 740 can be used to perform in-situ three-dimensional reconstruction of the narrow inner cavity with weak texture of the aircraft engine according to the relative position parameters, posture state parameters and local point cloud data.

[0122] In an exemplary embodiment of the present disclosure, local point cloud data of the key component is constructed according to the left-eye perspective sequence image and the right-eye perspective sequence image, including: calculating an inner cavity disparity map between the left-eye perspective sequence image and the right-eye perspective sequence image, and obtaining camera intrinsic parameters, camera extrinsic parameters and a baseline distance between the binocular boresight and the inner cavity of the aircraft engine of the binocular boresight; converting the inner cavity disparity map according to the camera intrinsic parameters and the baseline distance to obtain a depth map of the actual distance between the inner cavity of the aircraft engine and the binocular boresight; establishing a conversion relationship between a world coordinate system and a camera image coordinate system based on the camera extrinsic parameters, and converting the depth map to the world coordinate system based on the conversion relationship to obtain local point cloud data of the key component.

[0123] In an exemplary embodiment of the present disclosure, a feature matching vector between the key components is determined based on the left-eye perspective sequence image, including: determining texture feature points of the key components based on the left-eye perspective sequence image; performing graph convolution processing and feature enhancement processing on the texture feature points based on a graph neural network model to obtain feature point information and visual descriptors of the texture feature points; and generating a feature matching vector between the key components based on the feature point information and the visual descriptor.

[0124] In an exemplary embodiment of the present disclosure, the feature point matching pairs between the key components are determined based on the feature matching vectors, including: calculating the inner product between the feature matching vectors to obtain a soft assignment matrix between texture feature points, and optimizing the soft assignment matrix based on a preset optimal assignment rule to obtain a matching score matrix between the key components; and using the Kuhn-Tucker necessary condition to solve the optimal solution of the matching score matrix to obtain the feature point matching pairs between the key components.

[0125] In an exemplary embodiment of the present disclosure, the preset optimal allocation rule is as follows:

[0126]

[0127] Among them, the left-eye perspective sequence image includes a first left-eye perspective sequence image P1 and a second left-eye perspective sequence image P2, P1 and P2 are two adjacent left-eye perspective sequence images; P1 includes m texture feature points, and P2 includes n texture feature points; T(i,j) is a soft allocation matrix between the i-th feature point in P1 and the j-th feature point in P2.

[0128] In an exemplary embodiment of the present disclosure, the Kuhn-Tucker necessary conditions are as follows:

[0129]

[0130] Wherein, T(i,j) is the soft assignment matrix between the i-th feature point in P1 and the j-th feature point in P2, P1 includes m texture feature points, and P2 includes n texture feature points; μ k is the Lagrange multiplier.

[0131] In an exemplary embodiment of the present disclosure, the feature point matching pair includes a first texture feature point in a first left-eye perspective sequence image and a second texture feature point in a second left-eye perspective sequence image corresponding to the first texture feature; wherein determining the intrinsic matrix of the left-side boresight in the binocular boresight according to the feature point matching pair includes: normalizing the first coordinate point of the first texture feature point based on a preset first translation and scaling transformation parameter to obtain a first normalization processing result; normalizing the second coordinate point of the second texture feature point based on a preset second translation and scaling transformation parameter to obtain a second normalization processing result; A first group of linear equations is constructed according to the first normalization processing result, the second normalization processing result and the basic matrix to be solved; the first group of linear equations is solved to obtain the basic matrix; wherein the basic matrix encodes the affine transformation between the first left-eye perspective sequence image and the second left-eye perspective sequence image, and estimates the camera stereo motion parameters and camera stereo calibration parameters of the left boresight from the first left-eye perspective sequence image and the second left-eye perspective sequence image; a target matching pair is determined from the feature point matching pair according to the basic matrix, and the essential matrix of the left boresight in the binocular boresight is determined according to the target matching pair.

[0132] In an exemplary embodiment of the present disclosure, the intrinsic matrix of the left boresight in the binocular boresight is determined according to the target matching pair, including: establishing a second linear equation group according to the target matching pair, and using linear algebra methods to solve the second linear equation group to obtain the intrinsic matrix of the left boresight in the binocular boresight; wherein the intrinsic matrix describes the posture state parameters of the left boresight corresponding to the first left-eye perspective sequence image and the second left-eye perspective sequence image.

[0133] In an exemplary embodiment of the present disclosure, the posture state parameters of the left-side borescope are determined according to the intrinsic matrix of the left-side borescope, including: decomposing the intrinsic matrix of the left-side borescope based on a preset singular value decomposition algorithm to obtain the posture state parameters of the left-side borescope relative to the world coordinate system; wherein the posture state parameters include the rotation matrix of the left-side borescope relative to the world coordinate system and the translation vector of the left-side borescope relative to the world coordinate system.

[0134] In an exemplary embodiment of the present disclosure, the weakly textured narrow inner cavity of the aircraft engine is reconstructed in situ three-dimensionally based on the posture state parameters and the local point cloud data, including: aligning and registering the local point cloud data according to the rotation matrix and translation vector in the posture state parameters to achieve in-situ three-dimensional reconstruction of the weakly textured narrow inner cavity of the aircraft engine.

[0135] The specific details of each module in the above-mentioned in-situ three-dimensional reconstruction device for a narrow inner cavity of an aircraft engine with weak texture have been described in detail in the corresponding in-situ three-dimensional reconstruction method for a narrow inner cavity of an aircraft engine with weak texture, so they will not be repeated here.

[0136] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of a module or unit described above can be further divided into multiple modules or units to be concretized. In addition, although the various steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that these steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired result. In addition or alternatively, some steps can be omitted, multiple steps can be combined into one step for execution, and / or one step can be decomposed into multiple steps for execution, etc.

[0137] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0138] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0139] Refer to the following Figure 8 The electronic device 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0140] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include, but are not limited to: the at least one processing unit 810, the at least one storage unit 820, a bus 830 connecting different system components (including the storage unit 820 and the processing unit 810), and a display unit 840.

[0141] The storage unit stores program codes, which can be executed by the processing unit 810, so that the processing unit 810 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 810 can perform the following steps: Figure 1 Step S110 shown in: obtaining a left-eye perspective sequence image and a right-eye perspective sequence image of the key components in the inner cavity of the aircraft engine collected based on the binocular boresight, and constructing local point cloud data of the key components based on the left-eye perspective sequence image and the right-eye perspective sequence image; step S120: determining the feature matching vectors between the key components based on the left-eye perspective sequence image, and determining the feature point matching pairs between the key components based on the feature matching vectors; step S130: determining the intrinsic matrix of the left boresight in the binocular boresight based on the feature point matching pairs, and determining the posture state parameters of the left boresight based on the intrinsic matrix of the left boresight; step S140: performing in-situ three-dimensional reconstruction of the weakly textured narrow inner cavity of the aircraft engine based on the relative position parameters, posture state parameters and local point cloud data.

[0142] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 8201 and / or a cache storage unit 8202 , and may further include a read-only storage unit (ROM) 8203 .

[0143] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0144] Bus 830 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0145] The electronic device 800 may also communicate with one or more external devices 900 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 880. As shown, the network adapter 880 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0146] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0147] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0148] According to the program product for implementing the above method in the embodiment of the present disclosure, it can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0149] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0150] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0151] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0152] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0153] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0154] Other embodiments of the present disclosure will be readily apparent to those skilled in the art after considering the specification and practicing the inventions invented herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not invented by the present disclosure. The specification and examples are to be considered merely exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. An in-situ 3D reconstruction method for a narrow inner cavity of an aircraft engine with weak texture, characterized in that: include: Acquire a left-eye perspective sequence image and a right-eye perspective sequence image of a key component in an inner cavity of an aircraft engine collected by a binocular borescope, and construct local point cloud data of the key component based on the left-eye perspective sequence image and the right-eye perspective sequence image; Determine a feature matching vector between the key components according to the left-eye perspective sequence image, and determine a feature point matching pair between the key components according to the feature matching vector; Determine an intrinsic matrix of a left borescope in the binocular borescope according to the feature point matching pair, and determine a posture state parameter of the left borescope according to the intrinsic matrix of the left borescope; According to the posture state parameters and the local point cloud data, the narrow inner cavity with weak texture of the aircraft engine is reconstructed in situ in three dimensions.

2. The in-situ 3D reconstruction method of a narrow inner cavity of an aircraft engine with weak texture according to claim 1 is characterized in that: Constructing local point cloud data of the key component according to the left-eye perspective sequence images and the right-eye perspective sequence images, including: Calculating the inner cavity disparity map between the left-eye perspective sequence image and the right-eye perspective sequence image, and obtaining the camera intrinsic parameters, camera extrinsic parameters and the baseline distance between the binocular borescope and the inner cavity of the aircraft engine of the binocular borescope; The inner cavity disparity map is converted according to the camera internal parameters and the baseline distance to obtain a depth map of the actual distance between the inner cavity of the aircraft engine and the binocular borescope; A conversion relationship between a world coordinate system and a camera image coordinate system is established based on the camera extrinsic parameters, and the depth map is converted to the world coordinate system based on the conversion relationship to obtain local point cloud data of the key component.

3. The in-situ 3D reconstruction method of a narrow inner cavity of an aircraft engine with weak texture according to claim 1 is characterized in that: Determining the feature matching vectors between the key components according to the left-eye perspective sequence images includes: Determining texture feature points of the key component according to the left-eye perspective sequence image; Performing graph convolution processing and feature enhancement processing on the texture feature points based on a graph neural network model to obtain feature point information and visual descriptors of the texture feature points; A feature matching vector between the key components is generated according to the feature point information and the visual descriptor.

4. The in-situ 3D reconstruction method of a narrow inner cavity of an aircraft engine with weak texture according to claim 1 is characterized in that: Determining a feature point matching pair between the key components according to the feature matching vector includes: Calculating the inner product between the feature matching vectors to obtain a soft allocation matrix between texture feature points, and optimizing the soft allocation matrix based on a preset optimal allocation rule to obtain a matching score matrix between the key components; The optimal solution of the matching score matrix is ​​solved using the Kuhn-Tucker necessary condition to obtain the feature point matching pairs between the key components.

5. The in-situ 3D reconstruction method of a narrow inner cavity of an aircraft engine with weak texture according to claim 4 is characterized in that: The preset optimal allocation rule is as follows: Among them, the left-eye perspective sequence image includes a first left-eye perspective sequence image P1 and a second left-eye perspective sequence image P2, P1 and P2 are two adjacent left-eye perspective sequence images; P1 includes m texture feature points, and P2 includes n texture feature points; T(i,j) is a soft allocation matrix between the i-th feature point in P1 and the j-th feature point in P2.

6. The in-situ 3D reconstruction method of a narrow inner cavity of an aircraft engine with weak texture according to claim 5 is characterized in that: The Kuhn-Tucker necessary conditions are as follows: Wherein, T(i,j) is the soft assignment matrix between the i-th feature point in P1 and the j-th feature point in P2, P1 includes m texture feature points, and P2 includes n texture feature points; μ k is the Lagrange multiplier.

7. The in-situ 3D reconstruction method of a narrow inner cavity of an aircraft engine with weak texture according to claim 1 is characterized in that: The feature point matching pair includes a first texture feature point in a first left-eye perspective sequence image and a second texture feature point in a second left-eye perspective sequence image corresponding to the first texture feature; Wherein, determining the essential matrix of the left borescope in the binocular borescope according to the feature point matching pair includes: Normalizing the first coordinate point of the first texture feature point based on a preset first translation and scaling transformation parameter to obtain a first normalization result; Normalizing the second coordinate point of the second texture feature point based on a preset second translation and scaling transformation parameter to obtain a second normalization result; Constructing a first linear equation system according to the first normalization processing result, the second normalization processing result and the basic matrix to be solved; Solving the first linear equation group to obtain the basic matrix; wherein the basic matrix encodes the affine transformation between the first left-eye perspective sequence image and the second left-eye perspective sequence image, and estimates the camera stereo motion parameters and camera stereo calibration parameters of the left borescope from the first left-eye perspective sequence image and the second left-eye perspective sequence image; A target matching pair is determined from the feature point matching pairs according to the basic matrix, and an essential matrix of a left borescope in the binocular borescope is determined according to the target matching pair.

8. The in-situ 3D reconstruction method of a narrow inner cavity of an aircraft engine with weak texture according to claim 7 is characterized in that: Determining the essential matrix of the left borescope in the binocular borescope according to the target matching pair includes: Establishing a second linear equation group according to the target matching pair, and solving the second linear equation group using a linear algebra method to obtain an essential matrix of a left borescope in the binocular borescope; The essential matrix describes the posture state parameters of the left-side boresight corresponding to the first left-eye perspective sequence image and the second left-eye perspective sequence image.

9. The in-situ 3D reconstruction method of a narrow inner cavity of an aircraft engine with weak texture according to claim 1, characterized in that: Determining the posture state parameters of the left-side borescope according to the essential matrix of the left-side borescope includes: Decomposing the essential matrix of the left borescope based on a preset singular value decomposition algorithm to obtain the posture state parameters of the left borescope relative to the world coordinate system; The posture state parameters include a rotation matrix of the left borescope relative to a world coordinate system and a translation vector of the left borescope relative to a world coordinate system.

10. The in-situ 3D reconstruction method of a narrow inner cavity of an aircraft engine with weak texture according to claim 1, characterized in that: According to the posture state parameters and the local point cloud data, an in-situ three-dimensional reconstruction of the narrow inner cavity of the aircraft engine with weak texture is performed, including: The local point cloud data is aligned and registered according to the rotation matrix and translation vector in the posture state parameters to achieve in-situ three-dimensional reconstruction of the weakly textured narrow inner cavity of the aircraft engine.

Images