Fan casing front and rear hole position positioning method and system based on image recognition

Abstract

The application discloses a fan shell front and rear hole position positioning method and system based on image recognition, comprising the following steps: S1: collecting a fan shell image through a calibrated binocular camera, and performing stereo correction and illumination invariance enhancement processing to obtain binocular enhanced image data; S2: performing instance segmentation on the binocular enhanced image data based on an instance segmentation neural network to obtain a hole position pixel region set with front and rear semantic labels; S3: performing data-driven hole position coarse matching on the hole position pixel region set through a graph neural network, and constructing a physical constraint rule model to optimize and verify the matching result, and outputting an accurate image matching pair set of the front and rear hole positions; and S4: based on the accurate image matching pair set, constructing a cost function to solve the shell pose and obtain final world coordinates, and realizing the front and rear hole position positioning of the fan shell.

Description

Technical Field

[0001] This invention relates to the field of image data recognition technology, and in particular to a method and system for locating the front and rear holes of a wind turbine casing based on image recognition. Background Technology

[0002] As a crucial piece of industrial equipment, the precise positioning of the front and rear holes on the casing of a fan is essential for its assembly and performance assurance. Fan casing hole positioning technology has evolved from traditional methods to intelligent vision systems. Early methods relied primarily on manual measurement using calipers, templates, or mechanical clamps for coarse positioning. These methods were inefficient, inaccurate, and heavily influenced by operator experience, making them unsuitable for mass production. With the development of industrial automation, semi-automatic positioning systems based on encoders and sensors emerged, but these cannot handle complex hole patterns or environmental variations.

[0003] Currently, images acquired by traditional binocular cameras are easily affected by uneven lighting and shadow changes, leading to a decrease in segmentation and matching accuracy. Furthermore, current matching is mostly data-driven, often resulting in incorrect matching pairs. The lack of physical constraint rule model verification leads to large errors in 3D coordinate calculation. Therefore, this paper proposes a method and system for locating the front and rear holes of a wind turbine casing based on image recognition. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art and to achieve the above objectives, the present invention proposes the following technical solution:

[0005] A method for locating the front and rear holes of a wind turbine casing based on image recognition includes:

[0006] S1: Acquire images of the wind turbine casing using a calibrated binocular camera, and perform stereo correction and illumination invariance enhancement processing to obtain binocular enhanced image data;

[0007] S2: Perform instance segmentation on the binocular enhanced image data based on the instance segmentation neural network to obtain a set of pixel regions with front and back semantic labels for the aperture positions;

[0008] S3: Perform coarse matching of hole positions using a graph neural network on the set of pixel regions at the hole positions, and build a physical constraint rule model to optimize and verify the matching results, outputting a set of precise image matching pairs for the hole positions before and after.

[0009] S4: Based on the set of accurate image matching pairs, a cost function is constructed to solve the shell pose and obtain the final world coordinates, thereby realizing the positioning of the front and rear holes of the wind turbine shell.

[0010] The process of stereo calibration is as follows:

[0011] Image of the left side of the fan casing And the right image Obtain the left perspective transformation matrix of the left image. The right perspective transformation matrix of the right image :

[0012] ,in, It is the intrinsic parameter matrix of the left camera after epipolar parallel constraint. It is the left camera rotation matrix after epipolar parallel constraint. for The inverse matrix;

[0013] ,in, It is the intrinsic parameter matrix of the right camera after epipolar parallel constraint. It is the rotation matrix of the right camera after the epipolar parallel constraint. for The inverse matrix;

[0014] Left perspective transformation matrix based on the left image The right perspective transformation matrix of the right image Perspective transformations were performed on the left and right images respectively to complete the stereo correction:

[0015] Perspective transformation of the left image: ;

[0016] Perspective transformation of the right image: ;

[0017] Where T represents transpose, such that the left and right images after stereo correction satisfy the constraint of parallel epipolar lines. These are the pixel coordinates in the original image. This is the representation of homogeneous coordinates;

[0018] The stereo-corrected left and right images , Represented as corrected image data , .

[0019] The process of enhancing illumination invariance is as follows:

[0020] The Retinex algorithm is used to correct the image data. The left and right images in the image are decomposed into reflection components. And the illumination component L, i.e. Then from the stereo-corrected image Remove light component The reflection component is obtained. The images are then processed using Retinex, with Retinex results calculated at all scales and weighted averaged. All image data are then integrated to obtain the final binocular enhanced image data. .

[0021] The process of obtaining the set of pixel regions at the aperture is as follows:

[0022] A MaskR-CNN neural network model for instance segmentation is pre-defined, and labeled image data of wind turbine casing holes are prepared for training. Each hole instance is labeled at the pixel level, and semantic labels before and after each hole are also added. The semantic tag representing the front hole. Semantic tags indicating the back hole;

[0023] The parameters of the instance segmentation neural network model are adjusted using the backpropagation algorithm, and the binocular augmented image data is used during the inference phase. The input is fed into a trained instance segmentation neural network model to generate a mask for each hole, i.e., the pixel region of the hole, resulting in... and ,in, This indicates the number of pixel regions representing holes in the left image. This indicates the number of pixel regions representing holes in the right image. Indicates the first image in the left image Each hole is a pixel area. Indicates the first image in the right image Each hole is a pixel area. and Each sub-data item includes preceding and following semantic tags. and , Indicates the front hole, Representing the rear hole, ultimately, we obtain a set of pixel regions with front and rear semantic labels for the hole positions. .

[0024] The coarse matching process for hole positions is as follows:

[0025] Based on pixel region set The relationship between pixel regions at the aperture is constructed into a bipartite graph structure. ,in, The nodes include the pixel region nodes of the hole positions in the left image. and the hole location pixel region node in the right image bipartite graph structure The edge in This indicates the potential matching relationship between the left and right hole pixel regions;

[0026] Using graph neural networks to analyze bipartite graphs The output node representation is processed, and then coarse matching of the hole positions is performed for each hole position pixel region node in the left image. Find the pixel region node with the highest similarity to the hole in the right image, and use it as the initial matching pair to obtain the coarse matching result set. g represents the number of matching results, and each m represents a matching pair. Each matching pair contains the pixel coordinates of the hole position in the left image and the pixel coordinates of the hole position in the right image.

[0027] The process of constructing a physical constraint rule model to optimize and verify the matching results is as follows:

[0028] Construct a physical constraint rule model, including the rule architecture and constraint conditions;

[0029] The constraints in the physical constraint rule model are determined, including that the line connecting the centers of the front and rear holes in three-dimensional space is parallel to a certain axis of the wind turbine casing, and the distance between the front and rear holes is... Within the range;

[0030] A rule architecture for a physical constraint rule model is constructed, and the precise 3D coordinates of the left and right images are calculated based on this rule architecture. and ,based on and Get the vector between two points Based on vectors Obtain the direction vector relative to the axis of the fan casing The included angle And set an angle threshold. ,when At that time, it is considered to be within the permissible angle range, and if If the match is valid, it is retained; otherwise, it is discarded. The coarse matching result set is then processed using the physical constraint rule model. After optimizing and validating all matching pairs, a set of accurate image matching pairs is obtained. .

[0031] The rule architecture of the physical constraint rule model is represented as follows:

[0032] ;

[0033] in, This indicates the center coordinates of the aperture in the pixel region of the aperture in the left image, where f represents the camera focal length. Z represents the principal point coordinates of the left camera, and Z represents the depth of the aperture in three-dimensional space. This indicates the horizontal position of the center coordinates of the hole location within the pixel region of the hole location in the right image. Indicates the coordinates of the principal point of the right camera. The horizontal position of the hole is represented by Z, which indicates the depth of the hole in three-dimensional space. Indicates the optical center of the right camera Relative to the optical center of the left camera Translation component in the X direction.

[0034] The process of obtaining the final world coordinates is as follows:

[0035] A set of reference hole positions is determined, and the corresponding local coordinates are: ;

[0036] Rotate the fan casing matrix Using three-dimensional vectors This indicates that the set is based on precise image matching. Precise three-dimensional coordinates of the front and rear holes With local coordinates Satisfying the relationship Then define the cost function. ,right Find the derivative and set it to zero, then iteratively solve for the corresponding... and Thus, the optimal rotation matrix of the wind turbine casing is obtained. And the optimal wind turbine casing translation vector Using the calculated and Combined with the known coordinates of the hole position in the local coordinate system Substitute back into the formula Obtain new formula The final world coordinates of each front and rear hole position in the world coordinate system can be directly calculated. .

[0037] The process of constructing the cost function solution is as follows:

[0038] The rotation matrix of the wind turbine casing is represented by a three-dimensional vector. It means, that is ,in, It is a three-dimensional vector The antisymmetric matrix is ​​then used to construct the cost function: .

[0039] A wind turbine casing front and rear hole positioning system based on image recognition includes:

[0040] Image acquisition module: Acquires images of the wind turbine casing using a calibrated binocular camera, and performs stereo correction and illumination invariance enhancement processing to obtain binocular enhanced image data;

[0041] Image segmentation module: Based on the instance segmentation neural network, the module performs instance segmentation on the binocular enhanced image data to obtain a set of pixel regions with front and back semantic labels for aperture positions;

[0042] Image matching module: Performs coarse matching of hole positions using graph neural network on the set of pixel regions, and builds a physical constraint rule model to optimize and verify the matching results, outputting a set of accurate image matching pairs for the holes before and after the matching.

[0043] Hole Position Positioning Module: Based on a set of precise image matching pairs, a cost function is constructed to calculate the shell pose and obtain the final world coordinates, thereby realizing the positioning of the front and rear holes of the wind turbine shell.

[0044] The present invention has the following beneficial effects:

[0045] In this invention, firstly, by using stereo correction and illumination invariance enhancement, the Retinex algorithm is used to decompose the reflection component and illumination component, eliminating environmental interference and ensuring image data quality. Combined with an instance segmentation neural network, semantic labels are assigned before and after the hole position, achieving pixel-level accurate segmentation. This ensures that the positioning remains stable even in complex environments, and the accuracy is significantly improved compared to traditional methods.

[0046] Secondly, a graph neural network is used for data-driven coarse matching, and a physical constraint rule model is introduced to automatically optimize the matching results, eliminate incorrect matching pairs, reduce manual intervention, improve matching accuracy, reduce false negative and false positive rates, and ensure the reliability of the accurate image matching pair set. The entire process, from image acquisition to pose calculation, is fully automated, avoiding the labor costs of manual measurement. The optimal wind turbine casing rotation matrix is ​​solved iteratively through a cost function. Translation vector Quickly output final world coordinates This significantly reduces positioning time. Attached Figure Description

[0047] Figure 1 This diagram illustrates the steps of the image recognition-based method and system for locating the front and rear holes of a wind turbine casing, as proposed in this invention.

[0048] Figure 2 This is a system block diagram of the image recognition-based method and system for locating the front and rear holes of a wind turbine casing proposed in this invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Example 1: As Figure 1 As shown, the image recognition-based method for locating the front and rear holes of a wind turbine casing proposed in this invention includes:

[0051] S1: Acquire images of the wind turbine casing using a calibrated binocular camera, and perform stereo correction and illumination invariance enhancement processing to obtain binocular enhanced image data;

[0052] Images of the fan casing are acquired using a calibrated binocular camera. , ,in, The left image is shown. The image on the right is a stereoscopic correction image of the acquired wind turbine casing.

[0053] Furthermore, the purpose of stereo correction is to ensure that corresponding points in the two images captured by the binocular camera are located on the same horizontal scan line, thereby simplifying the subsequent stereo matching process;

[0054] Based on the intrinsic and extrinsic parameters of the binocular camera, calculate the transformation matrix required for stereo correction, including the left perspective transformation matrix of the left image. And the right perspective transformation matrix of the right camera ;

[0055] Let the intrinsic parameter matrix of the left camera be... The extrinsic parameter matrix is The intrinsic parameter matrix of the right camera is The extrinsic parameter matrix is Then the rotation matrix R and translation vector t between the two cameras satisfy the following: ,in, It is the extrinsic translation vector of the left camera, describing the position of the left camera in the world coordinate system. It is the extrinsic translation vector of the right camera, describing the position of the right camera in the world coordinate system. The rotation matrix is ​​the extrinsic parameter of the left camera. This is the rotation matrix in the extrinsic parameters of the right camera;

[0056] For the left camera, the transformation matrix for stereo correction The calculation process is as follows:

[0057] First, based on the constraint requirement of parallel polar lines, the intrinsic parameter matrix is... The intrinsic parameter matrix is Rotation matrix and rotation matrix Perform polar parallel constraints;

[0058] Calculate the left perspective transformation matrix required to transform the left image to a corrected image with parallel epipolar lines. ,Right now: ,in It is the intrinsic parameter matrix of the left camera after epipolar parallel constraint. It is the left camera rotation matrix after epipolar parallel constraint. for The inverse matrix;

[0059] Calculate the right perspective transformation matrix required to transform the right image to a corrected epipolar parallel image. ,Right now: ,in It is the intrinsic parameter matrix of the right camera after epipolar parallel constraint. It is the rotation matrix of the right camera after the epipolar parallel constraint. for The inverse matrix;

[0060] Transformation matrix based on left camera stereo correction Transformation matrix for stereo correction of the right camera Perspective transformations were performed on the left and right images respectively to complete the stereo correction:

[0061] Perspective transformation of the left image: ;

[0062] Perspective transformation of the right image: ;

[0063] Where T represents transpose, such that the left and right images after stereo correction satisfy the constraint of parallel epipolar lines. These are the pixel coordinates in the original image. This is the representation of homogeneous coordinates;

[0064] The stereo-corrected left and right images , Represented as corrected image data , ;

[0065] Then, illumination invariance enhancement processing is performed. In actual production environments, the fan casing is affected by different lighting conditions, resulting in uneven brightness and insufficient contrast in the acquired images, which affects subsequent hole location recognition. To enhance the illumination invariance of the images, the Retinex algorithm is used:

[0066] Correct image data The left and right images in the image are decomposed into reflection components. And the illumination component L, i.e. ;

[0067] Then from the stereo-corrected image Remove light component The reflection component is obtained. The images are then processed using Retinex, with Retinex results calculated at all scales and weighted averaged. All image data are then integrated to obtain the final binocular enhanced image data. ;

[0068] At this time, binocular augmented image data The image after stereo correction Based on this, the illumination component L was removed, while the reflection component was retained and highlighted. The final image achieved illumination invariance enhancement processing.

[0069] S2: Perform instance segmentation on the binocular enhanced image data based on the instance segmentation neural network to obtain a set of pixel regions with front and back semantic labels for the aperture positions;

[0070] After obtaining the preprocessed binocular enhanced image data, the instance segmentation neural network is used to segment the data into instances to obtain a set of pixel regions with front and back semantic labels for the aperture positions.

[0071] Pre-defined instance segmentation neural network model MaskR-CNN, using the binocular augmented image data obtained in step S1. The input is fed into the instance segmentation neural network model for training and inference.

[0072] During training, pre-labeled image data of the wind turbine casing holes are prepared. Each hole instance is labeled at the pixel level, and semantic labels are added before and after each hole. The semantic tag representing the front hole. The semantic tag indicates the back hole, and the front and back semantic tags are used to distinguish whether it is a front hole or a back hole;

[0073] By using the backpropagation algorithm, the parameters of the instance segmentation neural network model are adjusted so that it can accurately identify the holes in the wind turbine casing image, segment the pixel region of each hole, and predict the semantic labels before and after each hole.

[0074] During the inference phase, the binocular augmented image data The input is fed into a trained instance segmentation neural network model. The model first generates candidate regions containing hole locations using a Region Proposal Network (RPN). Then, it classifies each candidate region to determine whether it is a hole location and whether it is a front or rear hole. Simultaneously, it generates a mask for each hole location, i.e., the pixel region of the hole location, resulting in... and ,in, This indicates the number of pixel regions representing holes in the left image. This indicates the number of pixel regions representing holes in the right image. Indicates the first image in the left image Each hole is a pixel area. Indicates the first image in the right image Each hole is a pixel area. and Each sub-data item includes preceding and following semantic tags. and , Indicates the front hole. Indicates the rear hole;

[0075] Finally, a set of pixel regions with front and back semantic labels for hole positions is obtained. ;

[0076] Furthermore, for each segmented aperture pixel region, the model extracts its high-dimensional feature vector. These feature vectors contain information about the shape, texture, and other aspects of the aperture. In this way, for the left and right images in the binocular enhanced image data, sets of aperture pixel regions with front and back semantic labels are obtained respectively.

[0077] S3: Perform coarse matching of hole positions using a graph neural network on the set of pixel regions at the hole positions, and build a physical constraint rule model to optimize and verify the matching results, outputting a set of precise image matching pairs for the hole positions before and after.

[0078] Based on pixel region set It is necessary to set the pixel regions of the holes in the left and right images of the collection. , A coarse matching is performed, and then optimized and verified using a physical constraint rule model to obtain the precise three-dimensional coordinates and matching relationship of the front and rear hole positions;

[0079] The relationship between pixel regions at the aperture is constructed into a bipartite graph structure. Specifically:

[0080] Bipartite graph structure Nodes in It includes two categories, one of which is the pixel region node of the hole in the left image. Another type is the pixel region node of the hole in the right image. ;

[0081] Bipartite graph structure The edge in This indicates the potential matching relationship between the left and right hole pixel regions;

[0082] Furthermore, the potential matching relationship is directly determined by calculating the cosine similarity of the feature vectors corresponding to the pixel regions of the hole positions;

[0083] Use a graph neural network to analyze this bipartite graph. Through processing, graph neural networks use message passing mechanisms to pass information between nodes and update node representations. After several layers of message passing, the representation of each node can better reflect its matching probability with other nodes.

[0084] Then, based on the node representations output by the graph neural network, coarse matching of hole positions is performed for each hole position pixel region node in the left image. Find the pixel region of the hole in the right image with the highest similarity (calculate cosine similarity) and use it as the initial matching pair to obtain the coarse matching result set. g is the number of matching results, and each m represents a matching pair (each matching pair contains the pixel coordinates of the hole position in the left image and the pixel coordinates of the hole position in the right image).

[0085] After obtaining the coarse matching result M, it is necessary to construct a physical constraint rule model to optimize and verify it, because coarse matching may result in incorrect matching, while physical constraints can use the actual geometric relationship of the hole positions on the wind turbine casing to filter out the correct match.

[0086] The process of constructing a physical constraint rule model to optimize and verify the matching results is as follows:

[0087] The physical constraint rule model consists of a rule framework and constraint conditions, specifically:

[0088] First, determine the constraints in the physical constraint rule model, including:

[0089] The front and rear openings of the fan casing are subject to geometric constraints in physical space. These constraints include that the line connecting the centers of the front and rear openings in three-dimensional space should be parallel to a certain axis of the fan casing, and that the distance between the front and rear openings is within a certain range, denoted as... ;

[0090] Then, construct the rule architecture of the physical constraint rule model:

[0091] Calculate the projection coordinates of each hole location in the left and right images. For the hole location pixel region in the left image and the corresponding hole location pixel region in the right image, obtain the center coordinates, including the center coordinates of the hole location in the hole location pixel region of the left image. And the center coordinates of the hole position in the pixel region of the hole position in the right image. ;

[0092] Calculate the three-dimensional coordinates of different aperture positions, assuming the optical center of the left camera is... The optical center of the right camera is Then the projection of the hole position onto the left image satisfies the similar triangle relationship, that is: ;

[0093] in, Here, f is the center coordinate of the aperture position within the pixel region of the aperture position in the left image, and f is the camera focal length. Z represents the principal point coordinates of the left camera, indicating the position of the image plane center in the pixel coordinate system, and Z represents the depth of the aperture in three-dimensional space.

[0094] Specifically, the depth Z of the hole is obtained through a formula, which relates to the coordinates. And related to camera intrinsic parameters, this proportional relationship directly establishes the connection between the three-dimensional coordinates X and Y of the aperture position and the depth Z;

[0095] Similarly, the projection of the hole position onto the right image should satisfy the similar triangle relationship, that is: ;

[0096] in, This indicates the horizontal position of the center coordinates of the hole location within the pixel region of the hole location in the right image. Indicates the coordinates of the principal point of the right camera. The horizontal position of the hole is represented by Z, which indicates the depth of the hole in three-dimensional space. Indicates the optical center of the right camera Relative to the optical center of the left camera The translation component in the X direction, i.e., the positional offset of the optical centers of the two cameras in the X-axis direction;

[0097] Specifically, in Under this constraint of parallel and horizontal epipolar lines, for a point in the left image, its corresponding point in the right image must be located in the same row (i.e., the same Y coordinate). It is only necessary to find the matching point in the horizontal direction (X direction). Therefore, in the triangulation calculation of the right image, it is only necessary to use the difference in pixel coordinates (parallax) in the horizontal direction (X direction) to calculate the depth Z and X coordinate of the hole.

[0098] The final rule architecture of the physical constraint rule model is represented as follows: ;

[0099] Then, based on the rule architecture of the physical constraint rule model, the precise three-dimensional coordinates of the left and right images are calculated. For the coarse matching result set For each matching pair (each matching pair contains a left image aperture pixel coordinate and a right image aperture pixel coordinate), calculate their corresponding precise 3D coordinates. and The geometric relationship between them is verified to determine whether it conforms to the physical constraints in the physical constraint rule model.

[0100] The precise 3D coordinates of the left and right images are calculated based on the rule framework of the physical constraint rule model. and Calculate the vector between two points Then, calculate the vector. Vector of the fan casing axis The included angle The formula is: ,in, Represents the precise three-dimensional coordinates of the left and right images. and The distance between them Represents direction vector The modulus length;

[0101] Set an angle threshold ,when At that time, it is considered to be within the permissible angle range, and if If the match is valid, the pair is considered a valid match and is retained; otherwise, it is considered an incorrect match and is discarded.

[0102] The coarse matching result set is obtained through the physical constraint rule model. After optimizing and validating all matching pairs, a set of accurate image matching pairs is obtained. Each matching pair contains the precise three-dimensional coordinates of the front and rear hole positions, which serve as input for step S4.

[0103] Specifically, for the pixel region of the hole in the left image and the corresponding pixel region of the hole in the right image, since the two cameras photographed the same hole from different angles, there is a parallax in the pixel position of the hole in the left and right images. Using this parallax, as well as the intrinsic and extrinsic parameters of the binocular camera, the position of the hole in three-dimensional space is deduced to form a preliminary coarse matching pair. Then, the coarse matching pair is optimized and verified by the physical constraint rule model, and finally the accurate three-dimensional coordinates of the front and rear holes are obtained.

[0104] S4: Based on a set of precise image matching pairs, a cost function is constructed to solve the shell pose and obtain the final world coordinates, thereby realizing the positioning of the front and rear holes of the wind turbine shell;

[0105] The orientation of the wind turbine casing includes the rotation matrix of the actual wind turbine casing. Translation vector of the fan casing It can determine the accurate position of the fan casing and the front and rear holes in the world coordinate system, thereby enabling the positioning of the front and rear holes of the fan casing.

[0106] Using the set of precise image matching pairs obtained in step S3 The pose is calculated using the precise three-dimensional coordinates of the front and rear hole positions. First, a set of reference hole positions is determined, and their corresponding local coordinates are... ;

[0107] Then, based on the set of precise image matching pairs... Precise three-dimensional coordinates of the front and rear holes With local coordinates Satisfying the relationship , For the rotating matrix of the wind turbine casing, Let the translation vector of the wind turbine casing be defined, and then a cost function is defined. The goal is to find the optimal rotation matrix of the wind turbine casing. And the optimal wind turbine casing translation vector This minimizes the reprojection error.

[0108] Furthermore, reprojection error is the error in reconstructing the precise three-dimensional coordinates in the world coordinate system. The error between the point projected onto the image plane after the pose transformation to the local coordinate system and the actual detected hole position projected onto the image plane;

[0109] The rotation matrix of the wind turbine casing is represented by a three-dimensional vector. It means, that is ,in, It is a three-dimensional vector The antisymmetric matrix is ​​then used to construct the cost function:

[0110] ;

[0111] right Find the derivative and set it to zero, then iteratively solve for the corresponding... and Thus, the optimal rotation matrix of the wind turbine casing is obtained. And the optimal wind turbine casing translation vector That is, the position and orientation of the fan casing, which ultimately achieves the positioning of the front and rear holes of the fan casing;

[0112] Then use the calculated and Combined with the known coordinates of the hole position in the local coordinate system Substitute back into the formula Obtain new formula The final world coordinates of each front and rear hole position in the world coordinate system can be directly calculated. This enables the positioning of the front and rear holes on the fan casing.

[0113] For example, for a certain front hole, its local coordinates are... After the above calculations, its world coordinates are obtained. ,in, Let T be the final world coordinates corresponding to a certain front hole, where T is the transpose symbol, which converts the vector from a row vector to a column vector;

[0114] For a certain rear hole, the local coordinates are... Similarly, world coordinates can be obtained. ,in, The final world coordinates corresponding to a certain hole;

[0115] When all front and rear holes are in the final world coordinate system Once all the holes are calculated, their spatial positions in the world coordinate system are determined, thus enabling the positioning of the front and rear holes on the fan casing.

[0116] Example 2: Figure 2 As shown, the image recognition-based front and rear hole positioning system for a wind turbine casing includes:

[0117] Image acquisition module: Acquires images of the wind turbine casing using a calibrated binocular camera, and performs stereo correction and illumination invariance enhancement processing to obtain binocular enhanced image data;

[0118] Image segmentation module: Based on the instance segmentation neural network, the module performs instance segmentation on the binocular enhanced image data to obtain a set of pixel regions with front and back semantic labels for aperture positions;

[0119] Image matching module: Performs coarse matching of hole positions using graph neural network on the set of pixel regions, and builds a physical constraint rule model to optimize and verify the matching results, outputting a set of accurate image matching pairs for the holes before and after the matching.

[0120] Hole Position Positioning Module: Based on a set of precise image matching pairs, a cost function is constructed to calculate the shell pose and obtain the final world coordinates, thereby realizing the positioning of the front and rear holes of the wind turbine shell.

[0121] In the application, several formulas are calculated by removing dimensions and taking their numerical values. The formulas are established by collecting a large amount of data and simulating the most recent real situation. Some coefficients or weights in the formulas are set by those skilled in the art according to the actual situation, so they will not be elaborated here.

[0122] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0123] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for locating the front and rear holes of a fan casing based on image recognition, characterized in that, include: S1: Acquire images of the wind turbine casing using a calibrated binocular camera, and perform stereo correction and illumination invariance enhancement processing to obtain binocular enhanced image data; S2: Perform instance segmentation on the binocular enhanced image data based on the instance segmentation neural network to obtain a set of pixel regions with front and back semantic labels for the aperture positions; S3: Perform coarse matching of hole positions using a graph neural network on the set of pixel regions at the hole positions, and build a physical constraint rule model to optimize and verify the matching results, outputting a set of precise image matching pairs for the hole positions before and after. The process of constructing a physical constraint rule model to optimize and verify the matching results is as follows: Construct a physical constraint rule model, including the rule architecture and constraint conditions; The constraints in the physical constraint rule model are determined, including that the line connecting the centers of the front and rear holes in three-dimensional space is parallel to a certain axis of the wind turbine casing, and that the distance between the front and rear holes is within a certain range. Within the range; A rule architecture for a physical constraint rule model is constructed, and the precise 3D coordinates of the left and right images are calculated based on this rule architecture. and ,based on and Get the vector between two points Based on vectors Obtain the direction vector relative to the axis of the fan casing The included angle And set an angle threshold. ,when At that time, it is considered to be within the permissible angle range, and if If the match is valid, it is retained; otherwise, it is discarded. The coarse matching result set is then processed using the physical constraint rule model. After optimizing and validating all matching pairs, we obtain the set of accurate image matching pairs. ; S4: Based on the set of accurate image matching pairs, a cost function is constructed to solve the shell pose and obtain the final world coordinates, thereby realizing the positioning of the front and rear holes of the wind turbine shell.

2. The method for locating the front and rear holes of a fan casing based on image recognition according to claim 1, characterized in that, The process of stereo calibration is as follows: Image of the left side of the fan casing And the right image Obtain the left perspective transformation matrix of the left image. The right perspective transformation matrix of the right image : ,in, It is the intrinsic parameter matrix of the left camera after epipolar parallel constraint. It is the left camera rotation matrix after epipolar parallel constraint. for The inverse matrix, This represents the intrinsic parameter matrix of the left camera; ,in, It is the intrinsic parameter matrix of the right camera after epipolar parallel constraint. It is the rotation matrix of the right camera after the epipolar parallel constraint. for The inverse matrix, This represents the intrinsic parameter matrix of the right camera; Left perspective transformation matrix based on the left image The right perspective transformation matrix of the right image Perspective transformations were performed on the left and right images respectively to complete the stereo correction: Perspective transformation of the left image: ; Perspective transformation of the right image: ; Where T represents transpose, such that the stereo-corrected left and right images satisfy the constraint of parallel epipolar lines. () represents the pixel coordinates in the original image. This is the representation of homogeneous coordinates; The stereo-corrected left and right images , Represented as corrected image data , .

3. The method for locating the front and rear holes of a fan casing based on image recognition according to claim 2, characterized in that, The process of enhancing illumination invariance is as follows: The Retinex algorithm is used to correct the image data. The left and right images in the image are decomposed into reflection components. And the illumination component L, i.e. Then from the stereo-corrected image Remove light component The reflection component is obtained. The images are then processed using Retinex, with Retinex results calculated at all scales and weighted averaged. All image data are then integrated to obtain the final binocular enhanced image data. .

4. The method for locating the front and rear holes of a fan casing based on image recognition according to claim 1, characterized in that, The process of obtaining the set of pixel regions at the aperture is as follows: A MaskR-CNN neural network model for instance segmentation is pre-defined, and labeled image data of wind turbine casing holes are prepared for training. Each hole instance is labeled at the pixel level, and semantic labels before and after each hole are also added. The semantic tag representing the front hole. Semantic tags indicating the back hole; The parameters of the instance segmentation neural network model are adjusted using the backpropagation algorithm, and the binocular augmented image data is used during the inference phase. The input is fed into a trained instance segmentation neural network model to generate a mask for each hole, i.e., the pixel region of the hole, resulting in... and ,in, This indicates the number of pixel regions representing holes in the left image. This indicates the number of pixel regions representing holes in the right image. Indicates the first image in the left image Each hole is a pixel area. Indicates the first image in the right image Each hole is a pixel area. Indicates the semantic tags before and after. Indicates the front hole, Representing the rear hole, ultimately, we obtain a set of pixel regions with front and rear semantic labels for the hole positions. .

5. The method for locating the front and rear holes of a fan casing based on image recognition according to claim 4, characterized in that, The coarse matching process for hole positions is as follows: Based on pixel region set The relationship between pixel regions at the aperture is constructed into a bipartite graph structure. ,in, Representing the structure of a bipartite graph The node, The nodes include the pixel region nodes of the hole positions in the left image. and the hole location pixel region node in the right image bipartite graph structure The edge in This indicates the potential matching relationship between the left and right hole pixel regions; Using graph neural networks to analyze bipartite graphs The output node representation is processed, and then coarse matching of the hole positions is performed for each hole position pixel region node in the left image. Find the pixel region node with the highest similarity to the hole in the right image, and use it as the initial matching pair to obtain the coarse matching result set. g represents the number of matching results, and each m represents a matching pair. Each matching pair contains the pixel coordinates of the hole position in the left image and the pixel coordinates of the hole position in the right image.

6. The method for locating the front and rear holes of a fan casing based on image recognition according to claim 1, characterized in that, The rule architecture of the physical constraint rule model is represented as follows: ； in, This indicates the center coordinates of the aperture in the pixel region of the aperture in the left image, where f represents the camera focal length. Z represents the principal point coordinates of the left camera, and Z represents the depth of the aperture in three-dimensional space. This indicates the horizontal position of the center coordinates of the hole location within the pixel region of the hole location in the right image. Indicates the coordinates of the principal point of the right camera. Horizontal position, Indicates the optical center of the right camera Relative to the optical center of the left camera Translation component in the X direction.

7. The method for locating the front and rear holes of a fan casing based on image recognition according to claim 1, characterized in that, The process of obtaining the final world coordinates is as follows: Determine the local coordinates corresponding to the reference hole position. Based on precise image matching sets Precise three-dimensional coordinates of the front and rear holes With local coordinates Satisfying the relationship ,in, For the rotating matrix of the wind turbine casing, Let this be the translation vector of the fan casing; The rotation matrix of the wind turbine casing is represented by a three-dimensional vector. Represent this, and then define the cost function. ,right Find the derivative and set it to zero, then iteratively solve for the corresponding... and Thus, the optimal rotation matrix of the wind turbine casing is obtained. And the optimal wind turbine casing translation vector Using the calculated and Combined with the known coordinates of the hole position in the local coordinate system Substitute back into the formula Obtain new formula The final world coordinates of each front and rear hole position in the world coordinate system can be directly calculated. .

8. The method for locating the front and rear holes of a fan casing based on image recognition according to claim 7, characterized in that, The process of constructing the solution to the cost function is as follows: The rotation matrix of the wind turbine casing is represented by a three-dimensional vector. Indicates, that is ,in, It is a three-dimensional vector The antisymmetric matrix is ​​then used to construct the cost function: .

9. A fan casing front and rear hole positioning system based on image recognition, implemented according to any one of claims 1-8, characterized in that, Image acquisition module: Acquires images of the wind turbine casing using a calibrated binocular camera, and performs stereo correction and illumination invariance enhancement processing to obtain binocular enhanced image data; Image segmentation module: Based on the instance segmentation neural network, the module performs instance segmentation on the binocular enhanced image data to obtain a set of pixel regions with front and back semantic labels for aperture positions; Image matching module: Performs coarse matching of hole positions using graph neural network on the set of pixel regions, and builds a physical constraint rule model to optimize and verify the matching results, outputting a set of accurate image matching pairs for the holes before and after the matching. Hole Position Positioning Module: Based on a set of precise image matching pairs, a cost function is constructed to calculate the shell pose and obtain the final world coordinates, thereby realizing the positioning of the front and rear holes of the wind turbine shell.

Images