Panoramic image splicing method for nuclear power fuel assembly

By employing deep learning-based image classification and template matching techniques, the challenge of stitching weakly textured images in nuclear power fuel assemblies has been solved, achieving efficient and accurate panoramic image stitching and supporting the visual detection and deformation parameter calculation of fuel assemblies.

CN119006273APending Publication Date: 2024-11-22CHINA NUCLEAR POWER OPERATION TECH CORP +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310555143.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-15
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Existing panoramic video image stitching methods are difficult to use accurately in nuclear power fuel assemblies because the fuel rods have weak textures, resulting in poor detection and early warning effects.

Method used

A deep learning-based image classification model is used to identify image frames with grids. Image offsets are calculated and rotation corrections are performed through template matching. Key frames are then selected for panoramic image stitching.

Benefits of technology

This improved the accuracy and efficiency of panoramic image stitching for fuel assemblies, ensuring the accuracy of fuel assembly detection and early warning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006273A_ABST
    Figure CN119006273A_ABST
Patent Text Reader

Abstract

The invention provides a panoramic image stitching method for a nuclear power fuel assembly, and the method comprises the following steps: 1, extracting image frames of a collected video, making a classification data set, and carrying out the training learning through a deep learning-based image classification model; 2, identifying an image frame with a grillwork; 3, selecting a key frame; 4, the inclination angle of the fuel assembly is rotationally corrected; 5, calculating the offset of the fuel assembly moving in the vertical direction and the offset of the fuel assembly shaking in the horizontal direction in the image through template matching, and estimating the motion state of the assembly; and step 6, splicing all the key frames in sequence by using the obtained image speed offset to obtain a global spliced image. According to the method, the problem that weak feature image splicing cannot be achieved through an image registration method based on a feature operator is solved, and the moving speed of the fuel assembly is accurately obtained through grillwork classification and template matching.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of nuclear power fuel assembly image stitching, and in particular to a panoramic image stitching method for nuclear power fuel assemblies. Background Art

[0002] Nuclear power fuel assemblies are the core components of nuclear power plant reactors. They serve in a highly irradiated heavy water environment, are washed by high-temperature, high-velocity coolants, and are subjected to long-term chemical corrosion from fission products and complex alternating mechanical loads. Harsh working environments may cause deformation of fuel assemblies, resulting in serious consequences such as the inability to insert control rods, cracking of fuel rod cladding, and fracture of grids, affecting the safe and stable operation of the reactor. Regular detection and early warning of key deformation dimensions of fuel assemblies (assembly length, assembly curvature, fuel rod width, fuel rod spacing, etc.) are important measures to ensure the safe operation of reactors.

[0003] An existing method is to use a camera to record four-sided videos of the fuel assembly during the unloading and transfer process, and use image stitching technology to calculate the size parameters.

[0004] The existing video image panoramic stitching method extracts image sequences with a certain overlapping area and stitches them by finding the corresponding position of the template or feature points in the stitched image in the reference image. Since the fuel rods are the main components of the fuel assembly in the length direction, and the fuel rods have weak textures and even cannot be observed, the above method is difficult to achieve good practical application results and is difficult to guarantee in terms of functional completion and stitching accuracy. Summary of the invention

[0005] The object of the present invention is to provide a panoramic image stitching method for a nuclear power fuel assembly, which solves the problem that it is difficult to obtain a panoramic stitching image when the fuel assembly is in a moving state and has weak features along the moving direction.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A panoramic image stitching method for a nuclear power fuel assembly comprises the following steps:

[0008] Step 1: Extract image frames from the captured video, create a classification data set, and use a deep learning-based image classification model for training and learning;

[0009] Step 2: Identify the image frame with the grid;

[0010] Step 3: Select key frames;

[0011] Step 4: Rotate and correct the inclination angle of the fuel assembly;

[0012] Step 5: Use template matching to calculate the vertical displacement of the fuel assembly and the horizontal displacement of the fuel assembly in the image to estimate the motion state of the assembly;

[0013] Step 6: Using the obtained image velocity offset, stitch all key frames in sequence to obtain a global stitched image.

[0014] In step 1, image classification uses a classification neural network with an image as input and a number as output.

[0015] In step 1, the classification data set uses the images with grid and the images without grid in the collected fuel assembly video frames.

[0016] In step 2, the image frames of the captured video are used as input to the trained image classification model to identify the image frames with grids.

[0017] In step 3, the position of the fuel assembly grid in the current frame is determined according to the position of the grid image in a certain sequence, and a key frame is selected.

[0018] In step 4, the intersection points of the outer edge of the fuel assembly and the upper and lower edges of the field of view of the acquired image are calculated, and the inclination angle of the line connecting the midpoints of the upper and lower intersection points is used as the inclination angle of the fuel assembly for rotation correction.

[0019] Step 4 includes:

[0020] Step 4.1: Use edge detection method to calculate the edge of the object in the image, and then use Hough transform or Radon transform to calculate the straight line formed by the edge;

[0021] Step 4.2: Calculate the angle between the outer contour line and the image boundary, obtain four intersection points, calculate the line connecting the centers of the two points on the upper boundary of the image and the centers of the two points on the lower boundary of the image, and the inclination angle of the connecting line is the inclination angle of the fuel assembly;

[0022] Step 4.3: Use the tilt angle calculated in step 4.2 to construct a rotation matrix and transform each pixel in the image. The transformed pixel value is obtained by bilinear interpolation of the surrounding four pixels.

[0023] In step 5, template matching is used to estimate the offset speed between the frames with grid images, and the offset speed of the non-grid image is calculated by averaging the speeds of nearby frames with grids.

[0024] Step 5 includes:

[0025] Step 5.1: Cut the template in the image grid part, match it with other images, and calculate the relative offset between the images;

[0026] Step 5.2: Divide the longitudinal offset obtained by template matching by the number of frames between the two images to obtain the grid speed;

[0027] Step 5.3: The speed of the non-grid portion of the image is obtained by averaging the speeds of the adjacent grid portions;

[0028] Step 5.4: Randomly select templates for matching in the image, ignoring the longitudinal deviation, and the lateral deviation is the horizontal offset.

[0029] Step 6 includes:

[0030] Step 6.1: Using the relative deviations between all frames, the horizontal and vertical deviations can be superimposed, and the maximum value is taken as the size of the panoramic image;

[0031] Step 6.2: stack the frames into a panorama;

[0032] Step 6.3: For the overlapping parts between different frames in the panorama, weights are constructed by the distances between the pixel points and the centers of different frames to perform weighted fusion.

[0033] Compared with the prior art, the panoramic image stitching method for nuclear power fuel assemblies provided by the present invention has the following beneficial effects:

[0034] The present invention overcomes the problem that the image registration method based on feature operators cannot solve the splicing of weak feature images, and accurately obtains the moving speed of the fuel assembly through grid classification and template matching.

[0035] The present invention regards the process of pulling the fuel assembly in water as a uniform speed, uses the speed of the characteristic obvious area to estimate the global moving speed, and can obtain the relative offset of any video frame.

[0036] The present invention classifies fuel assembly images according to whether they are provided with a grid through a trained neural network, thereby improving the accuracy of grid position estimation and speed estimation.

[0037] The present invention overcomes the problem that feature-based speed estimation methods such as the optical flow method are time-consuming, and by selecting some key frames for speed estimation, the time consumed by image registration in panoramic image stitching is effectively reduced, thereby improving work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flow chart of a panoramic image stitching method for a nuclear power fuel assembly provided by the present invention;

[0040] Figure 2 Collecting grid-containing image frames for nuclear power sites;

[0041] Figure 3 Collecting grid-free image frames for nuclear power sites;

[0042] Figure 4 A schematic diagram of reference values ​​of a fuel assembly image frame sequence provided by the present invention;

[0043] Figure 5 A schematic diagram of a fuel assembly image frame sequence through a model prediction value provided by an embodiment of the present invention;

[0044] Figure 6 The key frame rotation correction flow chart provided by the present invention;

[0045] Figure 7 A schematic diagram of a fuel assembly boundary point search method provided by the present invention;

[0046] Figure 8 The image of the fuel assembly after rotation correction provided by the present invention;

[0047] Fig. 9 A flow chart of speed estimation provided by the present invention;

[0048] Fig.10 The grid template image in the key frame provided by the present invention;

[0049] Fig.11 A template matching result in another key frame provided by the present invention;

[0050] Fig.12 The image stitching flow chart provided by the present invention;

[0051] Fig.13 This is a panoramic view of the fuel assembly obtained by splicing the original images collected at the nuclear power site provided by the present invention. DETAILED DESCRIPTION

[0052] The following is further explained in detail through specific implementation methods.

[0053] like Figure 1 As shown, the present invention provides a panoramic image stitching method for nuclear power fuel assemblies.

[0054] Step 1: Image frame classification model training. Extract image frames from the captured video, create a classification dataset based on whether the image contains a grid, and use a deep learning-based image classification model to train the created dataset.

[0055] The collected fuel assembly video frames can be divided into image frames with grids and image frames without grids according to whether there are grids in them. Figure 2 and Figure 3 As shown in the figure, the image frame with grid contains a long horizontally distributed fuel assembly grid. When the fuel assembly moves along its length, the strong horizontal distribution feature of the grid in the image will also move with it; the image frame without grid does not contain a long horizontally distributed fuel assembly grid, but only the fuel rods distributed along the longitudinal direction, lacking the horizontal features that move with the fuel assembly when it moves along its length.

[0056] Image classification uses a classification neural network, with an image as input and a number as output: 0 or 1, 0 represents the current image is a gridless image, 1 represents the current image is a grid-attached image. Model classification requires model parameter training, and a training data set needs to be produced. The data set uses images with and without grids in the collected fuel assembly video frames. The training data set covers the following differences: the shapes of different fuel assembly grids are slightly different, the adjacent grids on the same surface of the same fuel assembly are different, and the grids on different surfaces are not exactly the same. The service cycle, on-site lighting and other factors cause differences in the surface state of the grid. In order to avoid the deviation of the number of positive and negative samples, the number of selected two types of images should be close to 1:1, and image enhancement operations (including rotation, shearing, and scaling) are added during the training process.

[0057] Step 2: Image frame classification. The image frames of the captured video are used as input to the trained image classification model to identify image frames with grids.

[0058] After training, the actual fuel assembly image is classified. A fuel assembly video image is imported into the model for classification. The distribution curve of the frame sequence of the fuel assembly length interval is as follows: Figure 4 and Figure 5 As shown, Figure 4 The medium-thick curve is the actual result of the fuel assembly image frame sequence. Figure 5 The middle thin curve is the result obtained by model prediction of the fuel assembly image frame sequence.

[0059] Step 3: Key frame selection. Considering the uniformity of the moving speed of the fuel assembly, the order of the continuous images with the grid reflects the position of the grid in the image, and the images with the grid in a better visual position are selected as key frames.

[0060] Through the grid classification, the grid image sequence is obtained. Since the fuel assembly moves from the bottom to the top of the camera field of view, and the moving speed is uniform, the position of the fuel assembly grid in the current frame can be roughly determined according to the position of the grid image in a certain sequence. In this way, the key frame image that meets the use requirements is selected in the image frame sequence.

[0061] Step 4: Rotation correction. Calculate the intersection of the outer edge of the fuel assembly and the upper and lower edges of the field of view of the acquired image, and tilt the line connecting the midpoints of the upper and lower intersections.

[0062] The angle is used as the inclination angle of the fuel assembly for rotation correction.

[0063] The algorithm flow of rotation correction is as follows Figure 6 As shown in Figure 1, to correct an image of a fuel assembly, it is necessary to find the tilt angle of the fuel rod of the assembly. Use line segment detection to find the line segments in the image and screen out the edge segments of the outermost fuel rods, such as Figure 7 As shown in , the intersection of the outermost line segments in the four black dashed boxes and the upper and lower boundaries of the image can determine the position of the fuel assembly. Calculate the inclination angle of the line connecting the midpoint of the two points on the upper boundary of the image and the middle of the two points on the lower boundary, and regard the inclination angle as the inclination angle of the fuel assembly for rotation correction. Calculate the angle of the line connecting the left point of the upper boundary of the image and the left point of the lower boundary, and regard the inclination angle as the inclination angle of the fuel assembly for rotation correction. The image obtained after bilinear interpolation is as follows Figure 8 shown.

[0064] Step 4 includes:

[0065] Step 4.1: Line detection: Use edge detection methods to calculate the edges of objects in the image, and then use Hough transform or Radon transform to calculate the straight lines formed by the edges.

[0066] Step 4.2: Angle calculation: calculate the angle between the outer contour line and the image boundary, obtain four intersection points, calculate the line connecting the centers of the two points on the upper boundary of the image and the two points on the lower boundary of the image, and the inclination angle of the connecting line is the inclination angle of the fuel assembly.

[0067] Step 4.3: Bilinear interpolation. Use the tilt angle calculated in step 4.2 to construct a rotation matrix and transform each pixel in the image. The transformed pixel value is obtained by bilinear interpolation of the surrounding four pixels.

[0068] Step 5: Velocity estimation: Template matching is used to estimate the migration velocity between frames with grid images, and the migration velocity of non-grid images is obtained by averaging the velocities of nearby frames with grids.

[0069] The speed estimation algorithm flow is as follows Fig. 9 As shown in , the offset of the grid position can be obtained by using the template matching method based on normalized cross-correlation, and the grid with more obvious features is selected as the template, such as Fig.10 As shown, calculate the position of the template on another key frame, such as Fig.11As shown in the figure, the horizontal and vertical offsets of the fuel assembly in the two frames of images can be calculated based on the template matching, and the vertical moving speed of the fuel assembly can be calculated based on the frame number difference between the two frames of images.

[0070] Step 5 specifically includes:

[0071] Step 5.1: Template matching, extract the template from the image frame, match it with other images, and calculate the relative offset between the images.

[0072] Step 5.2: Estimate the grid velocity. Divide the longitudinal offset obtained by template matching by the number of frames between the two images to obtain the grid velocity (pixels / frame).

[0073] Step 5.3: Estimate the non-grid velocity. The velocity of the non-grid portion of the image is obtained by averaging the adjacent grid velocities.

[0074] Step 5.4: Estimate the horizontal offset. Take any template from the image for matching and ignore the vertical deviation. The horizontal deviation is the horizontal offset.

[0075] Step 6: Image stitching: Using the obtained image velocity offset, stitch all key frames in sequence, and obtain the global stitching image through weighted fusion.

[0076] The algorithm flow of image stitching is as follows Fig.12 As shown, according to the offset between the images of each key frame, the size of the stitched panorama is calculated, and then the image is superimposed on the panorama, and a uniform panoramic stitched image is obtained by weighted fusion of the images (according to the method commonly used in image fusion algorithms, the weight is constructed by the distance between the pixel point and the center of different frames), as shown in Fig.13 shown.

[0077] Step 6 includes:

[0078] Step 6.1: Calculate the size of the panoramic image. Using the relative deviations between all frames, the horizontal and vertical deviations can be superimposed, and the maximum value is taken as the size of the panoramic image.

[0079] Step 6.2: Image superposition, stacking the frames into a panorama.

[0080] Step 6.3: Weighted image fusion. For the overlapping parts between different frames in the panorama, weights are constructed by the distances between the pixel points and the centers of different frames to perform weighted fusion.

[0081] The present invention decomposes a fuel assembly video captured by a camera into single-frame images, classifies the video images frame by frame through a pre-trained classifier through an image classification method based on deep learning, performs binary judgment on whether the image contains a grid, and then finds an image sequence in which the grid exists according to the appearance and disappearance of the grid; extracts frames in which the grid is at a specific position from the image sequence as key frames; calculates the inclination angle of the fuel assembly in the key frames, completes the rotation correction, pitch correction and tilt correction of the image frames through bilinear interpolation, and obtains the corrected posture of the spliced ​​frames during the movement of the fuel assembly; calculates the moving distance and speed between key frames through template matching, and then estimates the moving speed of the fuel assembly, and calculates the horizontal offset between any frames through template matching; selects spliced ​​frames according to a fixed sampling interval, calculates the size of the panoramic image through the offset of the spliced ​​image, obtains the complete posture relationship between the spliced ​​frames based on the posture and offset information of the spliced ​​image, and obtains the panoramic image of the fuel assembly using image fusion technology.

[0082] The present invention realizes the splicing of panoramic images of fuel assemblies under weak feature conditions, and improves the splicing accuracy and efficiency, providing a basis for the visual detection of fuel assemblies and the calculation of various deformation parameters.

[0083] The present invention classifies fuel assembly images according to whether they have grids through a trained neural network, thereby improving the accuracy of grid position estimation and speed estimation.

[0084] The present invention regards the process of pulling the fuel assembly in water as a uniform speed, and uses the speed of the characteristic obvious area to estimate the global moving speed, so as to obtain the relative offset of any video frame.

[0085] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A panoramic image stitching method for nuclear power fuel assemblies, characterized in that: The following steps are involved: Step 1: Extract image frames from the captured video, create a classification data set, and use a deep learning-based image classification model for training and learning; Step 2: Identify the image frame with the grid; Step 3: Select key frames; Step 4: Rotate and correct the inclination angle of the fuel assembly; Step 5: Use template matching to calculate the vertical displacement of the fuel assembly and the horizontal displacement of the fuel assembly in the image to estimate the motion state of the assembly; Step 6: Using the obtained image velocity offset, stitch all key frames in sequence to obtain a global stitched image.

2. The panoramic image stitching method for nuclear power fuel assembly according to claim 1, characterized in that: In step 1, image classification uses a classification neural network with an image as input and a number as output.

3. The panoramic image stitching method for nuclear power fuel assembly according to claim 1, characterized in that: In step 1, the classification data set uses the images with grid and the images without grid in the collected fuel assembly video frames.

4. The panoramic image stitching method for nuclear power fuel assembly according to claim 1, characterized in that: In step 2, the image frames of the captured video are used as input to the trained image classification model to identify the image frames with grids.

5. The panoramic image stitching method for nuclear power fuel assembly according to claim 1, characterized in that: In step 3, the position of the fuel assembly grid in the current frame is determined according to the position of the grid image in a certain sequence, and a key frame is selected.

6. The panoramic image stitching method for nuclear power fuel assembly according to claim 1, characterized in that: In step 4, the intersection points of the outer edge of the fuel assembly and the upper and lower edges of the field of view of the acquired image are calculated, and the inclination angle of the line connecting the midpoints of the upper and lower intersection points is used as the inclination angle of the fuel assembly for rotation correction.

7. The panoramic image stitching method for nuclear power fuel assembly according to claim 6, characterized in that: Step 4 includes: Step 4.1: Use edge detection method to calculate the edge of the object in the image, and then use Hough transform or Radon transform to calculate the straight line formed by the edge; Step 4.2: Calculate the angle between the outer contour line and the image boundary, obtain four intersection points, calculate the line connecting the centers of the two points on the upper boundary of the image and the centers of the two points on the lower boundary of the image, and the inclination angle of the connecting line is the inclination angle of the fuel assembly; Step 4.3: Use the tilt angle calculated in step 4.2 to construct a rotation matrix and transform each pixel in the image. The transformed pixel value is obtained by bilinear interpolation of the surrounding four pixels.

8. The panoramic image stitching method for nuclear power fuel assembly according to claim 1, characterized in that: In step 5, template matching is used to estimate the offset speed between the frames with grid images, and the offset speed of the non-grid image is calculated by averaging the speeds of nearby frames with grids.

9. The panoramic image stitching method for nuclear power fuel assembly according to claim 8, characterized in that: Step 5 includes: Step 5.1: Cut the template in the image grid part, match it with other images, and calculate the relative offset between the images; Step 5.2: Divide the longitudinal offset obtained by template matching by the number of frames between the two images to obtain the grid speed; Step 5.3: The speed of the non-grid portion of the image is obtained by averaging the speeds of the adjacent grid portions; Step 5.4: Randomly select templates for matching in the image, ignoring the longitudinal deviation, and the lateral deviation is the horizontal offset.

10. The panoramic image stitching method for nuclear power fuel assembly according to claim 1, characterized in that: Step 6 includes: Step 6.1: Using the relative deviations between all frames, the horizontal and vertical deviations can be superimposed, and the maximum value is taken as the size of the panoramic image; Step 6.2: stack the frames into a panorama; Step 6.3: For the overlapping parts between different frames in the panorama, weights are constructed by the distances between the pixel points and the centers of different frames to perform weighted fusion.