Engine surface image information acquisition system

Abstract

The invention relates to the technical field of image acquisition, image processing and video monitoring processing, and provides an engine surface image information acquisition system, which comprises an industrial camera used for acquiring a high-resolution engine surface image; the precise motion platform is used for precisely carrying and guiding the industrial camera to move along three coordinate axes (X, Y and Z) which are perpendicular to one another so as to position and cover a target area on the surface of the engine; and the image information comparison device is used for mutually matching and mapping the feature points in the reference image sequence and the to-be-detected image sequence so as to generate an overlapping region of the reference image sequence and the to-be-detected image sequence, and taking the overlapping region in the reference image sequence and the to-be-detected image sequence as acquired image information. In the technical scheme provided by the invention, after the feature points in the picture information are mutually matched, the overlapped region is used as the acquired image information, so that the accuracy is higher when the static working condition and the dynamic working condition of the engine are subsequently carried out.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image acquisition, image processing and video monitoring processing equipment, in particular to an engine surface image information acquisition system. BACKGROUND

[0002] The content of this part only provides background information related to the present application, which may not constitute prior art.

[0003] In the engine assembly off-line detection link, the sealing (leakage) of the three independent circulation loops of fuel, oil and coolant inside the engine needs to be detected. In some technical solutions, image sequences are collected under static working conditions (idle speed) and dynamic working conditions (load running) of the engine respectively, and the collected image sequences are compared to determine whether there is leakage.

[0004] When detecting the engine, the engine is usually fixed on a detection workbench, and various functional pipelines are densely arranged around it. These pipelines are external pipelines for the engine when it is working, and are used for circulating internal liquids. In order to reduce the installation cost, the installation spacing between the pipeline and the engine body is small. This compact layout significantly limits the movement of the image acquisition device: the industrial camera needs to move and adjust the posture in the narrow pipeline gap, and the range of movement is very small.

[0005] In addition, the complex curved surface structure of the engine surface makes the front shooting have a visual blind area, which requires the industrial camera to have a multi-angle flipping function. However, the existing pipeline layout seriously restricts the activity space of the camera, which is specifically manifested in:

[0006] The turning radius required for camera flipping conflicts with the limited space, and interference with the pipeline or engine body is prone to occur;

[0007] The larger turning radius leads to limited angle adjustment of the camera, making it difficult to accurately collect image information below the curved surface of the engine, affecting the integrity and accuracy of image acquisition. SUMMARY

[0008] Therefore, the purpose of the present application is to provide an engine surface image information acquisition system, which can solve the technical problems in the background art.

[0009] The purpose of the present application is achieved by the following technical solutions:

[0010] An engine surface image information acquisition system, comprising:

[0011] An industrial camera for collecting high-resolution engine surface images;

[0012] Precise motion platform: used for precise loading and guiding the industrial camera to move along three mutually perpendicular coordinate axes (X, Y, Z), realizing positioning and covering of the target area on the engine surface;

[0013] Control device for coordinating the trajectory and positioning of the precise motion platform; and:

[0014] Triggering the industrial camera to collect a group of reference image sequences under the static working condition of the engine;

[0015] Triggering the industrial camera to collect a group of to-be-inspected image sequences under the dynamic working condition of the engine;

[0016] Image information comparison device, which matches and maps the feature points in the reference image sequences and the to-be-inspected image sequences to generate the overlapping areas of the reference image sequences and the to-be-inspected image sequences, and takes the overlapping areas in the reference image sequences and the to-be-inspected image sequences as the collected image information;

[0017] Turning device, which is arranged at the moving end of the precise motion platform, and the industrial camera is installed on the turning device;

[0018] Wherein:

[0019] The turning device comprises:

[0020] The mounting plate is fixedly connected with the moving end of the precise motion platform;

[0021] The mounting plate is hingedly connected with the end of the fixed cylinder;

[0022] The fixed gear is mounted on the fixed cylinder;

[0023] The first driving motor is used for driving the fixed gear to rotate;

[0024] The circular-arc rack is connected with the mounting plate and is in meshing with the fixed gear, and the first driving motor drives the fixed gear to rotate, thereby guiding the fixed cylinder to rotate around the hinged point of the mounting plate.

[0025] This application installs an industrial camera on a precision mobile platform by setting a steering device at the mobile end. In the steering device, the arc rack and the fixed gear mesh with each other. When the first drive motor drives the fixed gear to rotate, it can guide the fixed cylinder to rotate around the hinge point. This design uses the cooperation of the two to effectively reduce the turning radius of the fixed cylinder when turning, making it difficult for the industrial camera installed on the fixed cylinder to touch objects such as pipes on both sides of the engine during rotation. It breaks through the spatial limitations and significantly improves the rotation angle of the industrial camera. It can accurately capture image information below the engine surface, ensuring the comprehensiveness and accuracy of the engine surface image acquisition, and providing high-quality image data support for subsequent engine detection, analysis and other work. After matching the feature points in the image information with each other, the corresponding overlapping area is found, and the overlapping area is used as the collected image information. Therefore, the accuracy is higher when the engine is subsequently operated in static conditions (idling) and dynamic conditions (loaded operation).

[0026] When capturing both the reference and test image sequences, errors in the movement of the precision motion platform guiding the industrial camera each time make it difficult to ensure that both captures occur at the same location. This error significantly reduces the overlap area between the reference and test image sequences, increasing image acquisition costs while also impacting image acquisition accuracy and the reliability of subsequent processing and analysis.

[0027] In some possible embodiments, the control device includes:

[0028] An initial positioner, used to locate at least one initial position;

[0029] An information storage module is used to obtain the movement control data of the precision motion platform when acquiring the reference image sequence;

[0030] The control module guides the industrial camera to move to the initial position, and then uses the motion control data to control the precision motion platform to guide the industrial camera to collect the image sequence to be inspected along the acquisition path of the reference image sequence.

[0031] This technical solution uses an initial positioner to locate the initial position, and an information storage module to obtain the motion control data of the precision motion platform when capturing the reference image sequence. The control module then guides the industrial camera from the initial position and uses the same motion control data to capture the image sequence to be inspected. In this way, the industrial camera can maintain the same position on the precision motion platform when capturing the image sequence to be inspected and the reference image sequence, effectively reducing the error between the two image sequences, expanding the area of ​​overlap, reducing image acquisition costs, and improving the accuracy and reliability of image acquisition, laying the foundation for efficient subsequent image processing and precise analysis.

[0032] When collecting engine surface images, due to the existence of numerous curved surfaces on the engine surface, there are dead angles in front shooting, and the industrial camera needs to have the ability to flip. However, the large number of pipelines on both sides of the engine restrict the movement range of the industrial camera, resulting in a large required turning radius when the industrial camera rotates, limited rotation angle, and difficulty in accurately collecting image information below the curved surface of the engine, affecting the integrity and accuracy of image collection. To solve this problem, the application provides the following technical solutions:

[0033] In the engine surface image collection operation, the complex curved surface of the engine surface causes the front shooting to have difficult-to-eliminate dead angles, and the narrow space formed by the pipelines on both sides seriously restricts the movement of the industrial camera. Even if the turning device solves the problem of limited rotation, the industrial camera still cannot achieve multi-angle and omnidirectional shooting due to space limitations when collecting images, there are collection blind areas, and it cannot meet the demand for comprehensive and accurate image collection of the engine surface, it is difficult to obtain complete and accurate image information, which greatly affects the accuracy and reliability of subsequent engine detection, fault analysis and other work based on image data.

[0034] Further, the mounting plate includes a fixed plate and a steering disc, the fixed plate and the steering disc are rotationally connected, the fixed plate is hingedly connected with the fixed cylinder, and the steering disc is fixedly connected with the moving end of the precision moving platform;

[0035] The second driving motor drives the steering disc to rotate, and the rotation axis of the steering disc is perpendicular to the rotation axis of the mounting cylinder.

[0036] On the basis of the original turning device, the application adds a steering disc and a second driving motor to the mounting cylinder, the industrial camera is installed on the steering disc, and the second driving motor drives the steering disc to rotate, and the rotation axis of the steering disc is perpendicular to the rotation axis of the mounting cylinder. This innovative design builds a space three-dimensional rotation system, so that the industrial camera can not only overcome the pipeline obstruction to perform planar rotation under the drive of the mounting cylinder, but also can realize flexible rotation in the vertical direction through the steering disc. The two work together to give the industrial camera the ability to shoot towards any point in space, completely breaking the space limitation and eliminating the collection blind area, expanding the information collection range of the industrial camera to the entire engine surface, effectively covering each curved surface and dead angle area.

[0037] When comparing and analyzing the reference image sequence and the to-be-detected image sequence, due to the uneven definition of the images, if direct comparison is performed, low-definition images will interfere with the comparison result, making it difficult to accurately identify the effective information in the images. At the same time, in a large number of images, how to quickly and accurately find the images taken at the same position and determine their overlapping areas is also a big problem. If these problems cannot be effectively solved, the accuracy and reliability of image comparison will be seriously affected, and the efficiency and quality of detection, analysis and other work based on image comparison will be reduced.

[0038] The image information comparison device comprises:

[0039] An image definition screening module screens out picture information with definition lower than a preset threshold in the reference image sequence and the image sequence to be detected based on Fourier transform;

[0040] An image pair acquisition module extracts pictures taken at the same position from the reference image sequence and the image sequence to be detected to form a picture group;

[0041] A picture feature comparison module is configured to perform similarity matching on the pictures in the picture group to generate an overlapping area.

[0042] The image information comparison device can efficiently and accurately compare images. The image definition screening module based on Fourier transform can quickly identify and screen out picture information with definition lower than a preset threshold in the reference image sequence and the image sequence to be detected, effectively eliminating the interference of low-quality images in subsequent analysis and ensuring image quality from the source. The image pair acquisition module accurately extracts pictures taken at the same position from the two image sequences to form a picture group, providing an accurate image set for subsequent comparison. The picture feature comparison module performs similarity matching on the pictures in the picture group to generate an overlapping area, which can clearly define the overlapping part of the images at the same position, greatly improving the accuracy of image comparison. The three modules work together to not only quickly locate the same position in the image sequence but also ensure the quality of the extracted images, providing reliable and accurate data support for subsequent image comparison-based applications such as product quality detection and fault diagnosis, significantly improving the efficiency and accuracy of related work.

[0043] When comparing and analyzing the reference image sequence and the image sequence to be detected, the definition of the images may vary. If a direct comparison is made, low-definition images may interfere with the comparison results, making it difficult to accurately identify the effective information in the images. Meanwhile, in a large number of images, it is also a major problem to quickly and accurately find images taken at the same position and determine their overlapping areas. If these problems cannot be effectively solved, it will seriously affect the accuracy and reliability of image comparison, and thus reduce the efficiency and quality of detection and analysis based on image comparison. Based on this, the present application provides the following technical solutions:

[0044] Further, the picture feature comparison module generates the overlapping area based on the following steps:

[0045] Step 1: Randomly extract a plurality of preliminary detection regions from the reference image, extract at least one reference feature point from each preliminary detection region, and generate a reference feature point set;

[0046] Step 2: Obtain the position and size of the preliminary detection region, enlarge the area of the preliminary detection region by n times the size of the comparison region in the image to be detected, and n is greater than 1.

[0047] Step 3: Extract all feature points in the contrast region to generate a contrast feature point set, and calculate the similarity between the feature points in the contrast feature point set and the feature points in the reference feature point set, so that each reference feature point set matches one feature point in the contrast feature point set, and a feature point mapping relationship is generated;

[0048] Step 4: Calculate the similarity between the reference image in the picture group and the image to be detected based on the cosine distance to obtain a similarity score; if the similarity score exceeds the preset threshold, the subsequent step is performed, and if the similarity score is lower than the preset threshold, there is no overlapping region in the picture group;

[0049] Step 5: Input the reference image, the image to be detected, and the feature point mapping relationship into a multi-scale feature extraction model to generate an overlapping region.

[0050] The image information comparison device can efficiently and accurately compare images. The image clarity screening module based on Fourier transform can quickly identify and screen out picture information with a clarity lower than a preset threshold in the reference image sequence and the image sequence to be detected, effectively eliminating the interference of low-quality images on subsequent analysis, and ensuring image quality from the source. The image pair acquisition module accurately extracts a picture group consisting of pictures taken at the same position from the two image sequences to provide an accurate image set for subsequent comparison. The picture feature comparison module generates an overlapping region by matching the similarity of the pictures in the picture group, which can clearly define the overlapping part of the images at the same position, greatly improving the accuracy of image comparison. The three work together to quickly locate the same position in the image sequence.

[0051] Further, the extraction method of the feature points in step 3 includes the following steps:

[0052] Step 31: Construct a Gaussian pyramid for the image to be detected I(x, y);

[0053] L(x, y, σ) = G(x, y, σ) * I(x, y);

[0054] Where I(x, y) represents the image to be detected, G(x, y, σ) represents the Gaussian kernel, and σ represents the scale factor;

[0055] Step 32: Calculate the Gaussian difference D(x, y, σ) for each pixel point in I(x, y);

[0056] D(x, y, σ) = L(x, y, kσ) - L(x, y, σ), k represents a scale multiplication coefficient;

[0057] Step 33: For each pixel point, compare the D(x, y, σ) of the surrounding 26 neighborhoods to extract extreme points;

[0058] The extreme point is a point whose Gaussian difference is less than the Gaussian difference of all the 26-neighborhood points around it, or whose Gaussian difference is greater than the Gaussian difference of all the 26-neighborhood points around it;

[0059] Step 34: screening the extreme points to generate feature points, assigning a gradient amplitude and a gradient direction to each feature point, and generating a descriptor of the feature point based on the gradient amplitude and the gradient direction;

[0060]

[0061] wherein m(x, y) represents the gradient amplitude, θ(x, y) represents the gradient direction, and L represents a Gaussian image corresponding to the scale of the key point.

[0062] In the present application, the SIFT algorithm is used to extract feature points, which can accurately describe the local features in the image by using the scale, rotation and illumination invariance of the local features of the SIFT algorithm, and then match the same feature points in the to-be-tested picture and the reference picture.

[0063] When there is a point offset between the reference image and the to-be-tested image, if there are many different features in the offset part, it will inevitably lead to an abnormal decrease in the similarity threshold, thereby screening out a lot of effective picture information. Based on this, the present application provides the following technical solutions:

[0064] Further, step 4 comprises the following steps:

[0065] Step 41: obtaining a center point of the reference image, and generating a matrix region of a preset size with the center point as the center;

[0066] Step 42: obtaining a feature point in the matrix region, matching a feature point identical to the feature point from the to-be-tested image, and generating a matrix region of a preset size in the to-be-tested image based on the matching relationship of the feature points;

[0067] Step 43: performing gray scale processing on the matrix region of the reference image and the matrix region of the to-be-tested image;

[0068] Step 44: converting the pixel points of the matrix region of the reference image into a one-dimensional vector A, A={a1, a2, a i …a n}, and converting the matrix region of the to-be-tested image into a one-dimensional vector B, B={b1, b2, b i …b n}, wherein n represents the total number of pixel points in the matrix region, i represents the index of a similar point in the matrix region, b i represents the gray scale value of the i-th pixel point in the matrix region of the to-be-tested image; and a i represents the gray scale value of the i-th pixel point in the matrix region of the reference image.

[0069] calculating the cosine similarity G of A and B;

[0070]

[0071] Step 45: converting the cosine similarity G into a similarity score, if the similarity score exceeds a preset threshold, then the subsequent step is performed, if the similarity score is lower than the preset threshold, then there is no overlapping area in the picture group.

[0072] In the technical solution provided in the application, only whether the pixel points in the central region exceed the preset threshold is detected. In practice, the variation range of the reference image and the to-be-detected image will not have a large variation, so in practice, whether the reference image has a large difference compared with the to-be-detected image can be accurately judged. In this way, whether there is a corresponding difference can be judged under relatively short computing resources.

[0073] Further, the feature extraction model comprises:

[0074] an input layer comprising a 3*3 convolutional layer, used for performing preliminary feature extraction on the input reference image and to-be-detected image;

[0075] a FEB feature network comprising at least three network layers, each network layer comprising a plurality of FEB feature extraction modules, and the number of FEB feature extraction modules in the network layer gradually increases;

[0076] a pooling layer connected to the last network layer, used for performing a pooling operation on the input features;

[0077] an output layer connected to the pooling layer and used for generating an overlapping region, the overlapping region being the region with the highest overlap probability of the reference image and the to-be-detected image.

[0078] Further, the FEB feature extraction module comprises a dynamic direction guide operator, a standard convolution, and a depth separable convolution.

[0079] The dynamic direction guide operator is used to extract dynamic sparse characteristics from the preliminary features and the feature point mapping relationship, and identify the offset direction of the reference image and the to-be-detected image; the standard convolution linearly combines the geometric features extracted by the dynamic direction guide operator and the features of other channels;

[0080] The depth separable convolution: each input channel is processed using a separate convolution kernel, maintaining channel independence, and is used to combine the information of all channels.

[0081] Further, the loss function of the multi-scale feature extraction model is:

[0082] wherein, represents a total loss function, represents a main loss function, represents a mapping classification loss of feature points, represents a cross-entropy loss, λ1, λ2, respectively, are first weight parameters, second weight parameters; represents an offset smoothing loss, represents a sparsity constraint loss; N and M represent the resolution of the image, N represents the height of the image, M represents the width of the image, i represents the row index of the pixel point, j represents the column index of the pixel point, P ij represents the probability that the pixel point belongs to the feature point, Y ij represents the probability that the pixel points match each other; represents an offset in the first direction, represents an offset in the second direction, O represents an offset tensor, k represents a feature point index, and b represents a sample index.

[0083] The beneficial effects of the present application are:

[0084] The technical scheme provided by the present application effectively overcomes the cross-condition image spatial misalignment problem caused by engine vibration and motion platform precision limitation by introducing high-precision image registration technology based on feature matching. The key advantage is that:

[0085] Significantly improve registration accuracy: use robust feature points and their matching relationship to calculate spatial transformation, realize sub-pixel level accurate alignment of static and dynamic condition images in the overlapping area.

[0086] Optimize subsequent analysis input: use the accurately registered overlapping area as effective image information input for subsequent leakage detection algorithms (such as image difference method, feature change analysis, etc.), which maximizes the elimination of background noise and artifacts caused by image misalignment.

[0087] Enhance detection reliability and sensitivity: based on the analysis of the high-precision registration overlapping area, significantly improve the detection rate of small liquid leakage traces (fuel, oil, coolant) on the engine surface, and effectively reduce the false positive rate caused by registration error. Therefore, this system can provide more accurate, reliable and sensitive diagnostic results in the sealing detection of engine static and dynamic conditions. BRIEF DESCRIPTION OF DRAWINGS

[0088] Figure 1 is a structural schematic diagram of an engine surface image information acquisition system;

[0089] Figure 2 is a three-dimensional schematic diagram of a steering device;

[0090] Figure 3A side view schematic diagram of a steering device;

[0091] Figure 4 A wind channel schematic diagram of a steering device.

[0092] Figure 5 A structure schematic diagram of a feature extraction model.

[0093] Reference signs

[0094] 1. An industrial camera;

[0095] 2. A precision motion platform; 21, X-axis; 22, Y-axis; 23, Z-axis; 24, moving end;

[0096] 3. A steering device; 31, mounting plate; 311, fixed plate; 312, steering disc; 32, fixed cylinder; 33, circular arc rack; 34, fixed gear; 321, wind channel. DETAILED DESCRIPTION

[0097] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely in combination with specific embodiments. The same reference signs in the drawings represent the same components. It should be noted that the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0098] Compared with the embodiments shown in the drawings, the feasible implementation solutions within the scope of protection of the present application can have fewer components, other components not shown in the drawings, different components, differently arranged components or differently connected components, etc. In addition, two or more components in the drawings can be implemented in a single component, or a single component shown in the drawings can be implemented as a plurality of separate components.

[0099] Unless otherwise defined, the technical terms or scientific terms used herein should be understood as the usual meanings understood by those skilled in the art to which the present application belongs. The “first”, “second” and similar words used in the specification and claims of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. Similarly, “one” or “a” and similar words do not necessarily represent a quantity limitation. “Up”, “down” and the like are only used to represent a relative positional relationship, which may change accordingly when the absolute position of the described object changes.

[0100] Embodiment 1:

[0101] Reference Figure 1The first embodiment of the present application discloses an engine surface image information acquisition system. The system is used for controlling a video device to acquire image information and process the image information. The engine surface image information acquisition system comprises: an industrial camera 1 used for acquiring high-resolution engine surface images; a precision motion platform 2 used for accurately loading and guiding the industrial camera 1 to move along three mutually perpendicular coordinate axes (X, Y, Z), so as to realize positioning and covering of a target region of an engine surface; a control device used for coordinately controlling a trajectory and positioning of the precision motion platform 2; and triggering the industrial camera 1 to acquire a group of reference image sequences under a static working condition of an engine, triggering the industrial camera 1 to acquire a group of to-be-inspected image sequences under a dynamic working condition of the engine, and an image information comparison device used for mutually matching and mapping feature points in the reference image sequences and the to-be-inspected image sequences, so as to generate an overlapping region of the reference image sequences and the to-be-inspected image sequences, and taking the overlapping region in the reference image sequences and the to-be-inspected image sequences as the acquired image information.

[0102] The industrial camera 1 is a lens capable of acquiring images and has high resolution performance. The precision motion platform 2 adopts a three-axis orthogonal structure under a Cartesian coordinate system and is composed of an X-axis 21, a Y-axis 22 and a Z-axis 23 motion module. Each axis motion module is driven by a linear motor, the linear motor has the characteristics of no transmission gap, high acceleration and fast response speed, and can realize micron-level motion accuracy. The X-axis 21, the Y-axis 22 and the Z-axis 23 all adopt a transmission structure of gear and rack cooperation or a cooperation structure of slide rail and pulley. In the present application, the transmission structure of gear and rack cooperation is adopted, so that the industrial camera 1 can be accurately moved to a predetermined position by controlling the number of rotation circles of the gear.

[0103] The engine surface image information acquisition system further comprises a steering device 3, the steering device 3 is arranged at a moving end 24 of the precision motion platform, and the industrial camera 1 is installed on the steering device 3. The moving end 24 of the precision motion platform is the final moving part. The structure of the precision motion platform 2 is a prior art, which will not be further described here.

[0104] Reference Figure 2 and Figure 3The steering device 3 comprises a mounting plate 31, a fixed cylinder 32, a circular arc rack 33, a fixed gear 34, a second driving motor and a first driving motor. The mounting plate 31 comprises a fixed plate 311 and a steering disc 312, the fixed plate 311 and the steering disc 312 are rotationally connected, the fixed plate 311 is hingedly connected with the fixed cylinder 32, and the steering disc 312 is fixedly connected with the moving end 24 of the precision moving platform. The fixed gear 34 is mounted on the fixed cylinder 32, the fixed gear 34 is rotationally arranged on the fixed cylinder 32, the first driving motor is arranged in the fixed cylinder 32 and used for driving the fixed gear 34 to rotate, and the circular arc rack 33 is fixedly connected with the mounting plate 31 at one end. The fixed cylinder 32 is provided with a gap for the circular arc rack 33 to insert. The circular arc rack 33 and the fixed gear 34 are in meshing connection, the first driving motor drives the fixed gear 34 to rotate, and then guides the fixed cylinder 32 to rotate around the hinge point of the mounting plate 31. The second driving motor drives the steering disc 312 to rotate, and the rotation axis of the steering disc 312 is perpendicular to the rotation axis of the mounting cylinder. In the present application, the first driving motor drives the fixed gear 34 to rotate, which can guide the fixed cylinder 32 to rotate up and down, and the second driving motor drives the steering disc 312 to rotate left and right.

[0105] The control device is used for coordinating the trajectory and positioning of the precision moving platform 2, and collecting a group of reference image sequences and a group of to-be-inspected image sequences. In practice, a plurality of positions are set in advance, the industrial camera 1 is moved to these positions and the angle is adjusted, and then an image is shot to form a corresponding image sequence.

[0106] Reference Figure 4 In some embodiments, the fixed cylinder 32 is provided with an air duct 321, the air duct 321 delivers cooling gas through a rubber hose to cool the industrial camera 1, so as to avoid the industrial camera 1 from stopping working due to high temperature environment. In some embodiments, the fixed cylinder is provided with an air duct 321, the air duct 321 delivers cooling gas through a rubber hose to cool the industrial camera, so as to avoid the industrial camera from stopping working due to high temperature environment.

[0107] Embodiment 2: Embodiment 2 provides a more specific image information comparison device on the basis of Embodiment 1.

[0108] The image information comparison device comprises an image sharpness screening module, an image pair acquisition module and a picture feature comparison module.

[0109] The image sharpness screening module screens out picture information with sharpness lower than a preset threshold value in the reference image sequence and the to-be-inspected image sequence based on Fourier transform. How to judge the picture sharpness based on Fourier transform is prior art and will not be described here. If the image sharpness is too low,

[0110] The image pair acquisition module extracts a picture group from the same position from the reference image sequence and the to-be-detected image sequence; and the picture feature comparison module is configured to perform similarity matching on the pictures in the picture group to generate an overlapping region.

[0111] After the overlapping region is acquired, it is necessary to determine whether it is necessary to re-shoot according to the size of the overlapping region. In practice, there is a certain overlapping part in the adjacent shooting positions, and screening the overlapping region is only a pretreatment of the collected images. The pixel points with the same coordinates in the reference image and the to-be-detected image correspond to the same position.

[0112] The picture feature comparison module generates the overlapping region based on the following steps:

[0113] Step 1: randomly extracting a plurality of preliminary detection regions from the reference image, extracting at least one reference feature point from each preliminary detection region, and generating a reference feature point set;

[0114] The selection manner of the preliminary detection region is random selection. In practice, a preliminary detection region inspection window size can be pre-set, and then the preliminary detection regions that do not overlap each other are randomly generated.

[0115] When the preliminary detection region is generated, it is necessary to make the preliminary detection region in the center of the entire reference image.

[0116] Step 2: acquiring the position and size of the preliminary detection region, enlarging the area of the preliminary detection region by n times of the to-be-detected image to generate a comparison region, and n is greater than 1; in practice, n = 2.

[0117] Step 3: extracting all feature points in the comparison region to generate a comparison feature point set, calculating the similarity between the feature points in the comparison feature point set and the feature points in the reference feature point set, so that each reference feature point set matches one feature point in the comparison feature point set, and generating a feature point mapping relationship.

[0118] One feature point is extracted in the preliminary detection region, and then one same feature point is found in the comparison region. If the two feature points are the same, it means that the two feature points correspond. The manner of calculating the similarity of the feature points will not be further described here, and the specific extraction manner of the feature points is given as follows:

[0119] The extraction manner of the feature points in step 3 includes the following steps:

[0120] Step 31: constructing a Gaussian pyramid for the to-be-detected image I(x, y);

[0121] L(x, y, σ) = G(x, y, σ) * I(x, y);

[0122] Wherein, I(x, y) represents the image to be detected, G(x, y, sigma) represents a Gaussian kernel, and sigma represents a scale factor;

[0123] Step 32: Gaussian difference D(x, y, sigma) is calculated for each pixel point in I(x, y);

[0124] D(x, y, sigma) = L(x, y, k sigma) - L(x, y, sigma), k represents a scale multiplication coefficient;

[0125] Step 33: For each pixel point, D(x, y, sigma) of the surrounding 26 neighbors is compared to extract extreme points;

[0126] The extreme point is the Gaussian difference that is less than the Gaussian difference of all pixel points in the surrounding 26 neighbors, or the Gaussian difference that is greater than the Gaussian difference of all pixel points in the surrounding 26 neighbors;

[0127] Step 34: The extreme points are screened to generate feature points, each feature point is assigned a gradient amplitude and a gradient direction, and a descriptor of the feature point is generated based on the gradient amplitude and the gradient direction;

[0128]

[0129] Wherein, m(x, y) represents a gradient amplitude, theta(x, y) represents a gradient direction, and L represents a Gaussian image corresponding to a key point scale.

[0130] Step 4: The reference image in the picture group and the image to be detected are calculated based on the cosine distance to calculate the similarity, and a similarity score is obtained; if the similarity score exceeds a preset threshold, subsequent steps are performed, and if the similarity score is lower than the preset threshold, there is no overlapping area in the picture group.

[0131] Because of the light change or other shadow shielding, the reference image and the image to be detected may have a large difference, because only the shadow and color change, no influence will be generated when the feature matching is performed. However, the accuracy of judging whether the engine leaks liquid in the subsequent comparison will be affected. Therefore, the present scheme needs to further judge whether the similarity of the reference image and the image to be detected in the picture group exceeds a preset threshold, in particular:

[0132] Step 41: The center point of the reference image is obtained, and a matrix region of a preset size is generated with the center point as the center;

[0133] Step 42: The feature points in the matrix region are obtained, the same feature points as the feature points in the image to be detected are matched from the image to be detected, and a matrix region of a preset size is generated in the image to be detected based on the matching relationship of the feature points;

[0134] Step 43: The matrix region of the reference image and the matrix region of the image to be detected are subjected to gray scale processing;

[0135] Step 44: converting the pixel points of the matrix region of the reference image into a one-dimensional vector A, A = {a1, a2, a i n Step 44: converting the pixel points of the matrix region of the reference image into a one-dimensional vector A, A = {a1, a2, a i n Step 44: converting the pixel points of the matrix region of the reference image into a one-dimensional vector A, A = {a1, a2, a i Step 44: converting the pixel points of the matrix region of the reference image into a one-dimensional vector A, A = {a1, a2, a i Step 44: converting the pixel points of the matrix region of the reference image into a one-dimensional vector A, A = {a1, a2, a

[0136] Step 44: converting the pixel points of the matrix region of the reference image into a one-dimensional vector A, A = {a1, a2, a

[0137]

[0138] Step 45: converting the cosine similarity G into a similarity score, if the similarity score exceeds the preset threshold, then the subsequent step is performed, if the similarity score is lower than the preset threshold, then there is no overlapping region in the picture group.

[0139] The similarity calculation in this application is very efficient, which is actually to determine whether the gray values of the reference image and the gray image are close, if they are close, it means that the light and shadow change is not large, which is suitable for comparative analysis. Moreover, by introducing the high-precision image registration technology based on feature matching, the cross-condition image spatial misalignment problem caused by engine vibration and motion platform precision limitation is effectively overcome.

[0140] Step 5: inputting the reference image, the to-be-detected image and the feature point mapping relationship into the multi-scale feature extraction model to generate the overlapping region.

[0141] Reference Figure 5 , embodiment 3: embodiment 3 provides a multi-scale feature extraction model based on embodiment 2;

[0142] The feature extraction model comprises:

[0143] An input layer comprising a 3*3 convolutional layer for performing preliminary feature extraction on the input reference image and to-be-detected image;

[0144] A FEB feature network comprising at least three network layers, each network layer comprising a plurality of FEB feature extraction modules, and the number of FEB feature extraction modules in the network layer gradually increases;

[0145] A pooling layer connected to the last network layer for performing pooling operation on the input features;

[0146] ​​An output layer connected to the pooling layer is used to generate an overlap region, which is a region with the highest overlap probability of the reference image for the to-be-detected image.

[0147] Specifically, the input layer is a 3*3 convolution layer capable of performing preliminary feature extraction, and a normalization factor is further included in the input layer, and an activation function is used for activation after the preliminary feature extraction is completed.

[0148] The FEB feature network includes at least three network layers, the first network layer has only two FEB feature extraction modules, and each subsequent network layer increases one FEB feature extraction module.

[0149] The preliminary features extracted by the input layer and the feature point mapping relationship are input to the FEB feature extraction module, which gradually finds similar regions in the reference image and the to-be-detected image. The specific structure of the pooling layer and the output layer is prior art, which will not be further described here.

[0150] The FEB feature extraction module includes a dynamic direction guide operator, a standard convolution, and a depth separable convolution.

[0151] The dynamic direction guide operator is used to extract dynamic sparse characteristics from the preliminary features and the feature point mapping relationship, and identify the offset direction of the reference image and the to-be-detected image; the standard convolution linearly combines the geometric features extracted by the dynamic direction guide operator with the features of other channels. The depth separable convolution: each input channel is processed using a separate convolution kernel, maintaining channel independence, and is used to merge information from all channels.

[0152] Each FEB feature extraction module dynamic direction guide operator is used to calculate the offset direction of the reference image and the to-be-detected image from one direction, so in the first network layer, only two directions are used to identify the offset direction of the reference image and the to-be-detected image. With the increase in the number of FEB feature extraction modules in the network layer, the accuracy of the considered direction in the corresponding network layer is higher, and it is easier to find the overlap region.

[0153] The training method of the multi-scale feature extraction model will not be described here, and the loss function during training is as follows:

[0154]

[0155]

[0156] wherein, Ltotal represents the total loss function, Lmain represents the main loss function, Lmap represents the mapping classification loss of the feature points, Lcross represents the cross-entropy loss, and λ1 and λ2 are the first weight parameter and the second weight parameter, respectively. denotes an offset smoothness loss, denotes a sparsity constraint loss; N and M denote the resolution of the image, N denotes the height of the image, M denotes the width of the image, i denotes the row index of the pixel point, j denotes the column index of the pixel point, P ij denotes the probability that the pixel point belongs to the feature point, Y ij denotes the probability that the pixel points match each other; denotes the offset in the first direction, denotes the offset in the second direction, O denotes the offset tensor, k denotes the feature point index, and b denotes the sample index.

[0157] In the technical scheme provided in the present application, the denotes a sparsity constraint loss, in which a two-direction offset loss function is set in the sparsity constraint loss, so that the offset direction during feature point matching can be guided in the iteration process, greatly increasing the convergence rate of the model.

[0158] The reason why the introduction of the offset direction can greatly increase the convergence rate of the model is that the image content in the to-be-tested image and the reference image is the same, and the physical size of the real world reflected by the pixel points in the image is also the same. Although the offset direction of the to-be-tested image and the reference image is random, all the feature points only have one offset direction. Therefore, in the training process, if the offset direction can be quickly found, the comparison of the two images can be quickly completed. For this reason, a dynamic direction guiding operator related to the offset vector and a corresponding loss function are introduced, so that the model can quickly find the offset direction of the reference image and the to-be-tested image.

[0159] The above is only a preferred embodiment of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An engine surface image information acquisition system, characterized in that: include: Industrial cameras for capturing high-resolution images of engine surfaces; Precision motion platform: used to accurately carry and guide the industrial camera to move along three mutually perpendicular coordinate axes (X, Y, Z) to achieve positioning and coverage of the target area on the engine surface; A control device for coordinating and controlling the trajectory and positioning of the precision motion platform; and: Trigger the industrial camera to capture a set of reference image sequences under the static working condition of the engine; Trigger the industrial camera to capture a set of image sequences to be inspected under dynamic engine conditions; An image information comparison device matches and maps the feature points in the reference image sequence and the image sequence to be tested to generate an overlapping area between the reference image sequence and the image sequence to be tested, and uses the overlapping area between the reference image sequence and the image sequence to be tested as the collected image information; Steering device: The steering device is arranged at the mobile end of the precision mobile platform, and the industrial camera is installed on the steering device; in: The steering device includes: A mounting plate, fixedly connected to the moving end of the precision moving platform; A fixing cylinder, wherein the mounting plate is hinged to an end portion of the fixing cylinder; A fixed gear, mounted on the fixed cylinder; a first driving motor, configured to drive the fixed gear to rotate; The arc rack is connected to the mounting plate and meshes with the fixed gear. The first drive motor drives the fixed gear to rotate and then guides the fixed cylinder to rotate around the hinge point with the mounting plate.

2. The engine surface image information acquisition system according to claim 1, characterized in that: The control device includes: An initial positioner, used to locate at least one initial position; An information storage module is used to obtain the movement control data of the precision motion platform when acquiring the reference image sequence; The control module guides the industrial camera to move to the initial position, and then uses the motion control data to control the precision motion platform to guide the industrial camera to collect the image sequence to be inspected along the acquisition path of the reference image sequence.

3. The engine surface image information acquisition system according to claim 1, characterized in that: A steering wheel and a second drive motor are also provided on the mounting cylinder; The industrial camera is mounted on the steering wheel. The second driving motor drives the steering wheel to rotate. The rotation axis of the steering wheel and the rotation axis of the mounting cylinder are perpendicular to each other.

4. The engine surface image information acquisition system according to claim 1, characterized in that: The image information comparison device includes: An image clarity screening module, which uses Fourier transform to screen out image information with clarity below a preset threshold in the reference image sequence and the image sequence to be tested; The image pair acquisition module extracts images taken at the same position from the reference image sequence and the image sequence to be tested to form an image group; The image feature comparison module is used to perform similarity matching on images in the image group to generate overlapping areas.

5. The engine surface image information acquisition system according to claim 4, characterized in that: The image feature comparison module generates overlapping regions based on the following steps: Step 1: Randomly select several preliminary detection areas from the reference image, extract at least one reference feature point from each preliminary detection area, and generate a reference feature point set; Step 2: Obtain the position and size of the preliminary detection area, and enlarge the area of ​​the preliminary detection area by n times the area of ​​the image to be tested to generate a contrast area, where n is greater than 1; Step 3: Extract all feature points in the comparison area to generate a comparison feature point set, calculate the similarity between the feature points in the comparison feature point set and the feature points in the reference feature point set, so that each reference feature point set matches one feature point in the comparison feature point set, and generate a feature point mapping relationship; Step 4: Calculate the similarity between the reference image and the image to be tested in the image group based on the cosine distance to obtain a similarity score; if the similarity score exceeds the preset threshold, proceed to the subsequent steps; if the similarity score is lower than the preset threshold, there is no overlapping area in the image group; Step 5: Input the reference image, the image to be inspected, and the feature point mapping relationship into the multi-scale feature extraction model to generate the overlapping area.

6. The engine surface image information acquisition system according to claim 5, characterized in that: The feature point extraction method in step 3 includes the following steps: Step 31: construct a Gaussian pyramid for the image to be detected I(x, y); L(x,y,σ)=G(x,y,σ)*I(x,y); Where I(x, y) represents the image to be detected, G(x, y, σ) represents the Gaussian kernel, and σ represents the scale factor; Step 32: Calculate the Gaussian difference D(x, y, σ) for each pixel in I(x, y); D(x,y,σ)=L(x,y,kσ)-L(x,y,σ), where k represents the scale multiplication factor. Step 33: For each pixel, compare D(x, y, σ) with the surrounding 26 neighborhoods to extract the extreme points; The extreme point is the point where the Gaussian difference is smaller than the Gaussian difference of all pixels in the surrounding 26 neighborhoods, or the Gaussian difference is greater than the Gaussian difference of all pixels in the surrounding 26 neighborhoods; Step 34: Screen the extreme points to generate feature points, assign a gradient magnitude and a gradient direction to each feature point, and generate a descriptor of the feature point based on the gradient magnitude and the gradient direction; Among them, m(x, y) represents the gradient amplitude, θ(x, y) represents the gradient direction, and L represents the Gaussian image corresponding to the key point scale.

7. The engine surface image information acquisition system according to claim 4, characterized in that: Step 4 includes the following steps: Step 41: Obtain the center point of the reference image, and generate a matrix area of ​​a preset size with the center point as the center; Step 42: Acquire a feature point in the matrix area, match a feature point identical to the feature point from the image to be tested, and generate a matrix area of ​​a preset size in the image to be tested based on the matching relationship of the feature points; Step 43: grayscale processing is performed on the matrix area of ​​the reference image and the matrix area of ​​the image to be measured; Step 44: Convert the pixel points of the matrix area of ​​the reference image into a one-dimensional vector A, where A = {a1, a2, a i …a n }, convert the matrix area of ​​the image to be tested into a one-dimensional vector B, B={b1, b2, b i …b n }, n represents the total number of pixels in the matrix area, i represents the index of the similar point in the matrix area, b i Represents the grayscale value of the i-th pixel in the matrix area of ​​the image to be tested; a i Represents the grayscale value of the i-th pixel in the matrix area of ​​the reference image; Calculate the cosine similarity G between A and B; Step 45: Convert the cosine similarity G into a similarity score. If the similarity score exceeds a preset threshold, proceed to the subsequent steps. If the similarity score is lower than the preset threshold, there is no overlapping area in the image group.

8. The engine surface image information acquisition system according to claim 5, characterized in that: Feature extraction models include: The input layer includes a 3*3 convolutional layer, which is used to perform preliminary feature extraction on the input reference image and the image to be tested; The FEB feature network includes at least three network layers, each of which includes several FEB feature extraction modules, and the number of FEB feature extraction modules in the network layer gradually increases; The pooling layer is connected to the last network layer and is used to perform pooling operations on the input features; The output layer is connected to the pooling layer to generate overlapping areas. The overlapping areas are the areas where the probability of the reference image overlapping with the image to be tested is the highest.

9. The engine surface image information acquisition system according to claim 8, characterized in that: The FEB feature extraction module includes: dynamic direction guidance operator, standard convolution and depth-separable convolution; The dynamic direction guidance operator is used to extract dynamic sparse features from the mapping relationship between preliminary features and feature points, and identify the offset direction of the reference image and the image to be tested. The standard convolution linearly combines the geometric features extracted by the dynamic direction guidance operator with the features of other channels. Depthwise separable convolution: Each input channel is processed using a separate convolution kernel to maintain channel independence and merge information from all channels.

10. The engine surface image information acquisition system according to claim 8, characterized in that: The loss function of the multi-scale feature extraction model is: in, represents the total loss function, represents the main loss function, Represents the mapping classification loss of feature points, represents the cross entropy loss, λ1 and λ2 are the first and second weight parameters respectively; represents the offset smoothing loss, Represents the sparsity constraint loss; N and M represent the resolution of the image, N represents the height of the image, M represents the width of the image, i represents the row index of the pixel, j represents the column index of the pixel, P ij Indicates the probability that a pixel belongs to a feature point, Y ij Indicates the probability that pixels match each other; Indicates the offset in the first direction, Represents the offset in the second direction, O represents the offset tensor, k represents the feature point index, and b represents the sample index.

Images