Rail vehicle image registration method, device and electronic equipment

Abstract

This invention provides a method, apparatus, and electronic device for image registration of rail vehicles. The method includes: inputting an image to be detected into a first detection model; detecting Regions of Interest (ROIs) including key components in the image using the first detection model to obtain a first coordinate set corresponding to the ROIs; obtaining a first transformation matrix based on the first coordinate set and a first reference coordinate set; performing a first registration based on the first transformation matrix; inputting the first-registered image into a second detection model; detecting key components in the image using the first detection model to obtain a second coordinate set; obtaining a second transformation matrix based on the second coordinate set and a second reference coordinate set; obtaining an offset reference vector based on the second transformation matrix; determining offset vectors corresponding to at least one key component based on the offset reference vector and a preset tolerance; and performing a second registration on the first-registered image based on the offset vectors. This invention improves image registration performance and increases detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of rail transit equipment manufacturing technology, and provides a method, apparatus and electronic device for image registration of rail vehicles. Background Technology

[0002] Train inspection robots are intelligent mobile robot systems specifically designed for the automated inspection, testing, and monitoring of railway trains (including high-speed trains, bullet trains, subway trains, and regular trains). When a train inspection robot repeatedly inspects the same component of a train, pose shifts can occur during the real-time acquisition of the image to be inspected due to navigation issues and deviations in stopping position. For example, the robot may shift along the vehicle's direction or rotate along the X, Y, or Z axes. This pose difference leads to discrepancies between the acquired image and the template image. Directly inputting the acquired image into the inspection model may reduce inspection accuracy and affect overall inspection precision. Summary of the Invention

[0003] This invention provides a method, apparatus, and electronic device for registering images of rail vehicles, which uses deep learning to perform two registrations on the images to be detected in order to improve detection accuracy.

[0004] This invention proposes a method for image registration of rail vehicles, comprising: inputting an image to be detected into a pre-trained first detection model; detecting regions of interest (ROIs) including key components in the image to be detected using the first detection model to obtain a first coordinate set corresponding to the ROIs; the first detection model is a deep neural network; obtaining a first transformation matrix based on the first coordinate set and a first reference coordinate set corresponding to the ROIs pre-labeled in a template image; performing a first registration on the image to be detected based on the first transformation matrix to obtain a first-registered image; inputting the first-registered image into a pre-trained second detection model; detecting key components in the image to be detected using the first detection model to obtain a second coordinate set corresponding to the key components; the second detection model is a deep neural network; obtaining a second transformation matrix based on the second coordinate set and a second reference coordinate set corresponding to the key components pre-labeled in the template image; obtaining an offset reference vector based on the second transformation matrix; determining offset vectors corresponding to at least one key component based on the offset reference vector and a preset tolerance; and performing a second registration on the first-registered image based on the offset vectors to obtain a second-registered image.

[0005] According to one embodiment of the present invention, the region of interest is a rectangular frame; the first coordinate set includes the x-coordinate and y-coordinate of the center point of the region of interest, the width of the region of interest, and the height of the region of interest; or, the first coordinate set includes the coordinates of the corner points corresponding to the region of interest.

[0006] According to one embodiment of the present invention, obtaining a first transformation matrix based on a first coordinate set and a first reference coordinate set corresponding to a pre-marked region of interest in a template image includes: determining at least three pairs of non-collinear points according to the first coordinate set and the first reference coordinate set corresponding to the pre-marked region of interest in the template image; constructing a first two-dimensional affine transformation model corresponding to at least three pairs of non-collinear points; the first two-dimensional affine transformation model includes multiple first parameters representing at least one of translation, rotation, scaling, and shearing transformations; and solving for the multiple first parameters of the first two-dimensional affine transformation model using the least squares method or singular value decomposition method to obtain the first transformation matrix.

[0007] According to one embodiment of the present invention, the second coordinate set includes the x-coordinate and y-coordinate of the center of the key component, the width of the outer rectangle of the key component, and the height of the region of interest; or, the second coordinate set includes the coordinates of the corner points corresponding to the outer rectangle of the key component.

[0008] According to one embodiment of the present invention, a second transformation matrix is ​​obtained based on a second coordinate set and a second reference coordinate set corresponding to key components pre-marked in a template diagram. This includes: determining at least three pairs of non-collinear points based on the second coordinate set and the second reference coordinate set corresponding to key components pre-marked in the template diagram; constructing a second two-dimensional affine transformation model corresponding to at least three pairs of non-collinear points; the second two-dimensional affine transformation model includes multiple second parameters representing rigid motion or affine deformation; and solving for the multiple second parameters of the second two-dimensional affine transformation model using the least squares method or singular value decomposition method to obtain the second transformation matrix.

[0009] According to one embodiment of the present invention, obtaining a second transformation matrix based on a second coordinate set and a second reference coordinate set corresponding to key components pre-annotated in a template image includes: determining at least three pairs of non-collinear points based on the second coordinate set and the second reference coordinate set corresponding to key components pre-annotated in the template image; and calling the cv2.estimateAffine2D or cv2.getAffineTransform function in an open-source computer vision library, inputting at least three pairs of non-collinear points into the cv2.estimateAffine2D or cv2.getAffineTransform function to obtain the returned second transformation matrix.

[0010] According to one embodiment of the present invention, obtaining an offset reference vector based on a second transformation matrix includes: extracting multiple translation components corresponding to multiple key components from the second transformation matrix; and determining the offset reference vector based on the multiple translation components.

[0011] According to one embodiment of the present invention, determining the offset vector corresponding to at least one key component based on the offset reference vector and a preset tolerance includes: determining the difference between the translation component corresponding to a key component and the offset reference vector, comparing the difference with the preset tolerance, and determining, based on the comparison result, to select either the offset reference vector or the translation component as the offset vector corresponding to a key component.

[0012] This invention also provides a rail vehicle image registration device, comprising: a first detection module, configured to input an image to be detected into a pre-trained first detection model, detect regions of interest (ROIs) including key components in the image to be detected using the first detection model, and obtain a first coordinate set corresponding to the ROIs; a primary registration module, configured to obtain a first transformation matrix based on the first coordinate set and a first reference coordinate set corresponding to the ROIs pre-labeled in a template image; and perform primary registration on the image to be detected based on the first transformation matrix to obtain a primary registered image; a second detection module, configured to input the primary registered image into a pre-trained second detection model, detect key components in the image to be detected using the first detection model, and obtain a second coordinate set corresponding to the key components; a second transformation module, configured to obtain a second transformation matrix based on the second coordinate set and a second reference coordinate set corresponding to the key components pre-labeled in the template image; an acquisition module, configured to obtain an offset reference vector based on the second transformation matrix; and a secondary registration module, configured to determine offset vectors corresponding to at least one key component based on the offset reference vector and a preset tolerance; and perform secondary registration on the primary registered image based on the offset vectors to obtain a secondary registered image.

[0013] The present invention also provides an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the rail vehicle image registration method as described above.

[0014] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the rail vehicle image registration method as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the rail vehicle image registration method as described above.

[0016] This invention provides a method, apparatus, and electronic device for image registration of rail vehicles. The method involves detecting regions of interest (ROIs) including key components in an image to be detected using a first detection model to obtain a first coordinate set. A first transformation matrix is ​​obtained based on the first coordinate set and a pre-annotated first reference coordinate set in a template image. First registration (coarse registration) is performed on the image to be detected based on the first transformation matrix. Then, the first detection model detects key components in the image to be detected to obtain a second coordinate set. A second transformation matrix is ​​obtained based on the second coordinate set and a pre-annotated second reference coordinate set in the template image. An offset reference vector is obtained based on the second transformation matrix. Offset vectors corresponding to each key component are obtained based on the offset reference vector and a preset tolerance. Second registration (fine registration) is performed on the first-registered image using the offset vectors. Thus, by using deep learning to perform two registrations on the image to be detected, a high-quality image with high registration accuracy with the template image can be obtained, improving the accuracy of the detection model in detecting the image to be detected. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the image registration method for rail vehicles provided by the present invention.

[0019] Figure 2 This is a flowchart illustrating an embodiment of the image registration method for rail vehicles provided by the present invention.

[0020] Figure 3 This is a schematic diagram of the software module structure of the image registration device for rail vehicles provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0022] Figure label: 301. First detection module; 302. Primary registration module; 303. Second detection module; 304. Second transformation module; 305. Acquisition module; 306. Secondary registration module; 410. Processor; 420. Communication interface; 430. Memory; 440. Communication bus. Detailed Implementation

[0023] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0024] Train inspection robots are intelligent equipment integrating mobile robot technology, multi-sensor fusion, artificial intelligence, and the Internet of Things (IoT). They are specifically designed for the daily inspection, condition monitoring, and predictive maintenance of rail transit vehicles. The train inspection robot inspects various components of the train, collecting images of key components as inspection images. A detection model (or detection algorithm) then performs inspections on these images. The core task of the detection model is to automatically and quantitatively identify and measure anomalies in the visual state of key components. For example, it detects whether key components have surface defects (cracks, peeling, corrosion, dents, etc.), structural integrity (whether components are missing, deformed / bent, or have foreign objects attached), and connection status (whether bolts are loose, whether there are welding defects, and whether seals are aged / damaged).

[0025] The registration methods in related technologies use traditional point feature recognition for image registration, or some solutions in other technical fields use deep learning methods for image calibration. On the one hand, the registration effect of traditional point feature recognition cannot meet the expected requirements; on the other hand, due to the different objects of interest and application scenarios, the image registration methods in other technical fields, when applied to the field of rail transit equipment manufacturing technology to register images collected by inspection robots, have poor registration effects and are difficult to achieve the expected performance goals.

[0026] Specifically, in train vehicle inspection applications, inspection robots need to navigate to the vicinity of the vehicle to collect images. In terms of image acquisition, they are more susceptible to the effects of dust, lighting conditions, and other factors, resulting in differences in image grayscale and shape. Compared to other fields such as remote sensing, the imaging conditions are worse. Furthermore, the viewing angle can cause image distortion along the z-axis. Therefore, some traditional image registration methods using point features and template matching struggle to find corresponding effective feature points on the inspection image, requiring improvement in registration accuracy. In addition, some registration schemes, after rigid image transformation, struggle to simultaneously register the offsets of various objects in the image caused by angular rotation, resulting in poor registration performance.

[0027] In view of this, embodiments of the present invention propose a method, apparatus, and electronic device for image registration of rail vehicles. After identifying the region of interest using a deep learning method, coarse registration is performed based on the region of interest. Then, key component level detection is performed. Based on the relative pose relationship between the template image and the key components in the image to be detected, secondary registration (fine registration) is performed on the image to be detected. Secondary registration can accurately register subtle offsets caused by angular rotation and other reasons, improve the image registration effect, and thus improve the detection accuracy.

[0028] This method can be applied to the inspection of trains by inspection robots in the field of rail transit equipment manufacturing, and it can also be applied to other application scenarios that require image registration, such as remote sensing and medical imaging.

[0029] The image registration method, apparatus, and electronic equipment for rail vehicles proposed in embodiments of the present invention will be described below with reference to the accompanying drawings. The image registration method for rail vehicles proposed in embodiments of the present invention, as follows... Figure 1 As shown, the method includes: Step 101: Input the image to be detected into the pre-trained first detection model, and use the first detection model to detect the Region of Interest (ROI) in the image to be detected, which includes key components, and obtain the first coordinate set corresponding to the ROI.

[0030] The first detection model is a deep neural network.

[0031] Image sensors mounted on the inspection robot can acquire images of the train's exterior to be inspected (referred to as inspection images). The inspection images are new images acquired in real time during the inspection process; they are images that need to be processed, registered, and ultimately used as input to the inspection model to perform the inspection task.

[0032] The first detection model, also known as the key area detection model, can be implemented using one or more combinations of the following models: YOLOv8 (You Only Look Once version 8) model: YOLOv8 provides models with varying complexity from nanoscale (n) to ultra-large scale (x), which can be flexibly selected according to the computing power of the deployed device. It has high detection accuracy and fast speed. For example, for detecting targets such as positioning axle boxes, using a medium scale (such as YOLOv8m) can achieve excellent results.

[0033] The two-stage model combines Faster R-CNN (Faster Region-Based Convolutional Neural Network) with ResNet-50 / 101 (Residual Network with 50 / 101 layers). Faster R-CNN can detect key regions.

[0034] Deformable DETR (Deformable Detection Transformer): A Transformer-based detector that can better model global context relationships. It may exhibit stronger robustness for locating key regions with complex backgrounds and partial occlusion.

[0035] The first coordinate set can be the coordinates of the center point of the region of interest (including X-axis and Y-axis coordinates) and the height and width of the bounding box of the region of interest (key region). The first coordinate set can also include the coordinates of at least two corner points of the bounding box of the region of interest. For example, it could be the coordinates of the top-left corner and the bottom-right corner; or the coordinates of the bottom-left corner and the top-right corner. In other words, the first coordinate set includes the x-coordinate and y-coordinate of the center point of the region of interest, the width of the region of interest, and the height of the region of interest; or, the first coordinate set includes the coordinates of the corner points corresponding to the region of interest.

[0036] Step 102: Based on the first coordinate set and the first reference coordinate set corresponding to the pre-labeled region of interest in the template image, obtain the first transformation matrix; based on the first transformation matrix, perform a registration on the image to be detected to obtain a first registered image.

[0037] A template image refers to a standard reference image, which can be understood as a standard image of a key train component acquired under ideal conditions (precise positioning and correct angle of the inspection robot). Subsequent newly captured images (images to be inspected) must be aligned with the template image. The template image can come from historical data or initial calibration and contains the correct position and shape of the key components.

[0038] The first transformation matrix is ​​obtained based on the first coordinate set and the first reference coordinate set. Specifically, it can be achieved as follows: Based on the first coordinate set and the first reference coordinate set, at least three pairs of non-collinear points are determined; based on the at least three pairs of non-collinear points, a corresponding first two-dimensional affine transformation model is constructed, and multiple first parameters of the first two-dimensional affine transformation model are solved by the least squares method to obtain the first two-dimensional affine transformation model, which is the first transformation matrix.

[0039] Step 103: Input the first registered image into the pre-trained second detection model, and use the first detection model to detect key components in the image to be detected, and obtain the second coordinate set corresponding to the key components.

[0040] The second detection model is a deep neural network. Specifically, the second detection model can be implemented based on one or more combinations of the following models: YOLOv11 (You Only Look Once version 11): This model can significantly improve the recall rate and localization accuracy of small local targets such as key components.

[0041] The Single Shot MultiBox Detector (SSD) based on Recursive Feature Pyramid and Switchable Atrous Convolution (SFP & SAC) demonstrates that the recursive feature pyramid can better integrate multi-scale features and enhance the semantic information of small targets such as bolts; the switchable atrous convolution can adaptively adjust the receptive field to better capture the context of small targets such as bolts.

[0042] The Mask Region-based CNN with Vision Transformer backbone (Mask R-CNN with ViT) is suitable for solving component occlusion problems in dense scenes. If the components to be inspected in the inspection area of ​​a train exterior are very dense and severely occluded, requiring not only bounding boxes but also precise component outlines (instance segmentation) for more refined analysis (such as whether the hexagonal head of a bolt is intact), Mask R-CNN can solve such problems. This embodiment of the invention combines Vision Transformer as the backbone network, which can better utilize global information of the region of interest to distinguish instances where key components are stuck together.

[0043] Corresponding to the first coordinate set, the second coordinate set can be the coordinates (including X-axis and Y-axis coordinates) of the center point of the region of interest (ROI) pre-labeled in the template diagram, and the height and width of the ROI's bounding box. The second coordinate set may also include the coordinates of at least two corner points of the ROI's bounding box, for example, the coordinates of the top-left corner and the bottom-right corner; or the coordinates of the bottom-left corner and the top-right corner. In other words, the second coordinate set includes the x-coordinate and y-coordinate of the center of the key component, the width of the key component's circumscribed rectangle, and the height of the ROI; or, the second coordinate set includes the coordinates of the corner points corresponding to the circumscribed rectangle of the key component.

[0044] Step 104: Based on the second coordinate set and the second reference coordinate set corresponding to the key components pre-marked in the template diagram, obtain the second transformation matrix.

[0045] Based on the second coordinate set and the second reference coordinate set, the second transformation matrix is ​​obtained, which can be achieved in the following way: Method 1: Based on the second coordinate set and the second reference coordinate set, determine at least three pairs of non-collinear points. Construct a corresponding second two-dimensional affine transformation model based on these three pairs of non-collinear points. Solve for multiple second parameters of the second two-dimensional affine transformation model using the least squares method. The resulting second two-dimensional affine transformation model is the second transformation matrix. These multiple second parameters represent subtle rigid motions or affine deformations.

[0046] Affine transformation refers to the change in shape of a graphic or image caused by affine transformation. Affine transformation is a specific type of geometric transformation that preserves the "straightness" and "parallelism" of the graphic. Straightness means that a straight line remains a straight line after the transformation and will not become a curve. Parallelism means that parallel lines remain parallel after the transformation. In other words, affine transformation is the new shape obtained by uniformly stretching, compressing, rotating, or translating a mesh, where every line in the mesh remains a straight line, and previously parallel lines remain parallel.

[0047] Method 2: Based on the second coordinate set and the second reference coordinate set, determine at least three pairs of non-collinear points. By calling a function and inputting these three pairs of non-collinear points, the second transformation matrix is ​​obtained from the returned result. For example, the `cv2.estimateAffine2D` or `cv2.getAffineTransform` functions from open-source computer vision libraries can be called. By inputting at least three pairs of non-collinear points into `cv2.estimateAffine2D` or `cv2.getAffineTransform`, multiple second parameters are returned, leading to the second transformation matrix.

[0048] Step 105: Obtain the offset reference vector based on the second transformation matrix.

[0049] Specifically, in some embodiments, multiple translation components corresponding to multiple key components are extracted from the second transformation matrix, and then the offset reference vector is determined based on the multiple translation components through voting or cluster analysis.

[0050] Step 106: Based on the offset reference vector and the preset tolerance, determine the offset vector corresponding to at least one key component; according to the offset vector, perform secondary registration on the primary registration image to obtain the secondary registration image.

[0051] Based on the offset reference vector and the preset tolerance, determine the offset vector corresponding to at least one key component. Specifically, determine the difference between the translation component corresponding to a key component and the offset reference vector, compare the difference with the preset tolerance, and determine the offset reference vector or translation component as the offset vector corresponding to a key component based on the comparison result.

[0052] Figure 1 The illustrated embodiment utilizes a deep learning model (or deep neural network) to detect and pre-annotate key regions in the image to be detected. During the registration process, key regions are identified for coarse registration. Then, the deep model extracts features of key components in the image, and fine registration is performed using the spatial correspondence between these features and the corresponding key components in the template image. Thus, through two registration processes, a high-quality image to be detected with high accuracy in registering with the template image can be obtained, improving the registration effect of the image to be detected and enhancing the accuracy of the detection model in detecting the image.

[0053] The following are some specific examples.

[0054] like Figure 2 As shown, in one specific embodiment, the rail vehicle image registration method may include the following process: Step 201: Obtain a template image with key region annotation data and a first training dataset, and train the first detection model using the first training dataset.

[0055] Obtain a template image of the train's exterior with labeled data. The template image includes pre-labeled key areas, such as the area where the bogie axle boxes are located. When acquiring the template image, the image acquisition parameters (such as the shooting angle) can be adjusted to ensure that the key areas are prominent, their area relative to the entire image exceeds a preset threshold, and they are centered in the image.

[0056] Multiple template images with key region annotations at corresponding locations are used as the training dataset to train the first detection model, resulting in the trained first detection model.

[0057] Step 202: Call the key region detection model (i.e., the first detection model) to detect (locate) the region of interest (ROI) (or key region) in the image to be detected, and obtain the first coordinate set corresponding to the ROI.

[0058] For example, a critical area is an area that includes the critical component to be inspected. For instance, the inspection purpose of a train inspection robot is to collect images including critical components, and then use a pre-trained model to identify whether the critical components have surface defects, structural incompleteness, loose bolts, etc. The critical area should include at least one critical component to be identified.

[0059] In this embodiment, the first coordinate set is a four-point coordinate set, for example, the first coordinate set is: [Xcenter] DETArea-key Ycenter DETArea-key Width DETArea-key Height DETArea-key Among them, Xcenter DETArea-key The X-axis coordinate and Y-axis coordinate of the center point of the identified key region bounding box are represented by the key region bounding box. DETArea-key Width represents the Y-axis coordinate of the center point of the identified key region bounding box. DETArea-key Height represents the width of the bounding box of the identified key region. DETArea-key This indicates the height of the bounding box of the identified key region.

[0060] Step 203: Obtain the key region annotation data in the template image corresponding to the image to be detected, and obtain the first reference coordinate set.

[0061] For example, the first reference coordinate set is: [Xcenter] TEMArea-key Ycenter TEMArea-key Width TEMArea-key Height TEMArea-key ], where center TEMArea-key The X-axis coordinate of the center point of the bounding box of the key area marked in the template drawing; Y-axis coordinate of the center point. TEMArea-key Width represents the Y-axis coordinate of the center point of the bounding box of the key area marked in the template drawing. TEMArea-key Height indicates the width of the bounding box of the key area marked in the template drawing. TEMArea-key This indicates the height of the bounding box of the key area marked in the template drawing.

[0062] Step 204: Calculate the first transformation matrix M1 using the first coordinate set and the first reference coordinate set, perform a transformation registration, and obtain a first-registered image.

[0063] A transformation registration is performed in step 204, which can correct large offsets.

[0064] The first transformation matrix M1 is calculated using the first coordinate set and the first reference coordinate set, specifically in the following manner: The first coordinate set is derived from the center-width-height representation [Xcenter] DETArea-key Ycenter DETArea-key Width DETArea-key Height DETArea-key Convert the coordinates of the four corner points to (x1, y1), (x2, y2), (x3, y3), and (x4, y4). Similarly, convert the first reference coordinate set to the coordinates of the four corner points.

[0065] Suppose there exists a first transformation matrix M1 (usually an affine transformation matrix, 2×3 in size) that, after applying this transformation to the first coordinate set, aligns it as closely as possible with the first reference coordinate set. In this embodiment, the least squares method or Singular Value Decomposition (SVD) can be used to solve a system of linear equations to obtain M1.

[0066] At least three pairs of non-collinear points are needed to uniquely determine an affine transformation. First, at least three pairs of non-collinear points are determined from the first reference coordinate set and the first coordinate set. Based on these three pairs of non-collinear points, the corresponding first transformation matrix is ​​constructed. For example, the mathematical expression of the constructed first transformation matrix is: Where (x, y) are points in the first coordinate set, (x', y') are transformed points, and [a, b, c, d, e, f] are the six parameters of M1. This is an overdetermined system of equations, and the optimal M1 can be solved using singular value decomposition (SVD) or least squares method, so that the sum of squared errors of all corresponding points after transformation is minimized.

[0067] Based on the above exemplary description, it can be seen that in some embodiments, the specific method for obtaining the first transformation matrix may be: determining at least 3 pairs of non-collinear points according to the first coordinate set and the first reference coordinate set corresponding to the region of interest pre-marked in the template diagram, constructing a first two-dimensional affine transformation model corresponding to at least 3 pairs of non-collinear points, and solving multiple first parameters of the first two-dimensional affine transformation model using the least squares method or singular value decomposition method to obtain the first transformation matrix.

[0068] In other embodiments, after determining at least 3 pairs of non-collinear points, function tools provided by image libraries such as OpenCV, such as cv2.getAffineTransform (requires 3 pairs of points) or cv2.estimateAffinePartial2D / cv2.estimateAffine2D, can be called. The coordinates of at least 3 pairs of non-collinear points are input, and the corresponding function tool returns the above six parameters to obtain the first transformation matrix.

[0069] The first transformation matrix M1 can be applied to the entire image to be detected, performing rotation, translation, scaling, and shearing transformations on it. This corrects the large-scale offsets and rotations caused by inconsistencies in navigation and parking, making the key areas in the image to be detected roughly aligned with the template image, thus achieving primary or coarse registration.

[0070] Step 205: Obtain a template image with annotation data of key components and a second training dataset, and train the second detection model using the second training dataset.

[0071] Critical components are those to be inspected, such as bolts and latches. The labeling data for critical components includes the coordinates of each critical component's specific location.

[0072] Multiple images with corresponding locations and annotation data of key components are used as the second training dataset to train the second detection model, resulting in the trained second detection model.

[0073] Step 206: Use the second detection model to detect (locate) each key component and obtain the second coordinate set of each key component.

[0074] For example, the second coordinate set of a key component is: [Xcenter] DETPart-key Ycenter DETPart-key Width DETPart-key Height DETPart-key ].

[0075] Step 207: Obtain the key component annotation data of the template image corresponding to the image to be detected, and obtain the second reference coordinate set.

[0076] For example, the second reference coordinate set is: [Xcenter] TEMPart-key Ycenter TEMPart-key Width TEMPart-key Height TEMPart-key ].

[0077] Step 208: Calculate the second transformation matrix (affine transformation matrix M2) based on the second coordinate set and the second reference coordinate set.

[0078] Specifically, in some embodiments, the second transformation matrix can be obtained using either method one or method two as follows: Method 1: Based on the second coordinate set and the second reference coordinate set corresponding to the key components pre-marked in the template diagram, at least three pairs of non-collinear points are identified. A second two-dimensional affine transformation model corresponding to at least three pairs of non-collinear points is constructed. This model includes multiple second parameters representing rigid motion or affine deformation. The multiple second parameters of the second two-dimensional affine transformation model are solved using the least squares method or singular value decomposition method to obtain the second transformation matrix. For details on Method 1, please refer to the method for obtaining the first transformation matrix in step 204 above.

[0079] Method 2: First, based on the second coordinate set and the second reference coordinate set corresponding to the key components pre-marked in the template image, determine at least 3 pairs of non-collinear points. Then, call the cv2.estimateAffine2D or cv2.getAffineTransform function in the open-source computer vision library, input the at least 3 pairs of non-collinear points into the cv2.estimateAffine2D or cv2.getAffineTransform function, and obtain the returned second transformation matrix.

[0080] Step 209: Obtain the offset reference vector by voting based on the second transformation matrix (affine transformation matrix M2).

[0081] Specifically, multiple translation components corresponding to multiple key components are extracted from the second transformation matrix, and then the offset reference vector is determined based on the multiple translation components.

[0082] For example, first extract the translation component: For the affine transformation matrix M2i corresponding to the i-th critical component (a × 3 matrix in the form of [[a, b, c], [d, e, f]]), its translation components are directly given by the matrix element (c, f). The vector Vi = (ci, fi) of the i-th critical component intuitively represents the horizontal displacement (ci) and vertical displacement (fi) required to correct the i-th critical component from its current position to the standard position in the template diagram.

[0083] Next, the voting space is constructed: the two-dimensional translation vector space (c, f) is discretized to form a regular grid (voting box). Each grid cell (or interval) represents a possible range of offset vectors. The granularity of the grid (i.e., the range of pixels covered by each cell) is a hyperparameter that needs to be set according to the positioning accuracy and noise level (e.g., each cell covers a range of ±2 pixels).

[0084] Based on the above voting space, perform voting statistics: traverse the offset vectors Vi of all N key components, determine the grid cell into which each Vi falls; increment the count value of the grid cell into which it falls by 1.

[0085] Determine the offset reference: After the statistics are completed, the grid cell with the highest count value represents the "most voted offset". The center coordinates of the cell with the most votes (or the average of all offset vectors therein) are selected as the offset reference vector V. base = (c base , f base ).

[0086] Step 210: Determine the offset vector of each key component based on the offset reference vector and tolerance, and perform secondary registration.

[0087] The offset vector includes the offset direction and the offset amount.

[0088] Specifically, the difference between the translation component and the offset reference vector corresponding to a key component is determined, and this difference is compared with a preset tolerance. Based on the comparison result, the offset reference vector or translation component is selected as the offset vector corresponding to the key component. For example, the deviation between the offset vector Vi of each key component and the offset reference vector is calculated: ΔVi = Vi - V base = (dx i - dx base dy i - dy base If |ΔVi| (e.g., Euclidean distance) is less than the tolerance T, then the offset of the critical component is considered to be consistent with the common reference and belongs to the "normal deviation". The reference offset V is then directly adopted. base Perform secondary registration on the component (i.e., move its position by V). base If |ΔVi| is greater than or equal to the tolerance T, the component is considered an "outlier." Possible causes include: severely inaccurate detection frame, abnormal component condition (e.g., detachment), or severe occlusion. In this case, there are two strategies: 1. Use individual offset: Determine that the offset vector Vi of the critical component better reflects its true condition and use its own offset vector Vi for registration. 2. Still use baseline offset: Treat the critical component as noise and still use V... base Register them and mark them as "required for verification" in subsequent inspections.

[0089] This invention proposes a method for image registration of rail transit vehicles. It utilizes deep learning to identify key regions and performs primary registration with these key regions of a template image, addressing issues of large offsets and large angular variations. The method uses deep learning to identify objects within the image, calculates offset vectors between these objects and the objects marked on the template image, uses a voting mechanism to find the offset benchmark, and confirms the offset of each component within a tolerance range, thus achieving secondary registration.

[0090] This invention employs deep learning for vehicle key area localization, specifically a method of coarse registration after training key area recognition. Compared to traditional methods, this approach offers advantages such as good generalization, high robustness, accuracy, and efficiency, especially in scenarios involving angle changes, lighting variations, and dust coverage. Addressing the issue that rigid image transformations in a single registration process cannot accurately register all objects within the image, a secondary registration method is adopted. This method extracts the objects to be inspected from within the image and then performs secondary registration, utilizing the spatial relationships of the objects to be inspected to transform them one by one, resulting in more accurate detection results.

[0091] This invention also provides a rail vehicle image registration device, which can be integrated into an inspection robot, for example, into the robot's intelligent agent. Figure 3 As shown, the device may include the following modules: The first detection module 301 is used to input the image to be detected into a pre-trained first detection model, and to detect the region of interest (ROI) in the image to be detected, which includes key components, and to obtain the first coordinate set corresponding to the region of interest.

[0092] The primary registration module 302 is used to obtain a first transformation matrix based on a first coordinate set and a first reference coordinate set corresponding to a pre-labeled region of interest in the template image; and to perform primary registration on the image to be detected based on the first transformation matrix to obtain a primary registered image.

[0093] The second detection module 303 is used to input the first registered image into the pre-trained second detection model, detect key components in the image to be detected through the first detection model, and obtain the second coordinate set corresponding to the key components.

[0094] The second transformation module 304 is used to obtain the second transformation matrix based on the second coordinate set and the second reference coordinate set corresponding to the key components pre-marked in the template diagram.

[0095] The acquisition module 305 is used to obtain the offset reference vector based on the second transformation matrix.

[0096] The secondary registration module 306 is used to determine the offset vector corresponding to at least one key component based on the offset reference vector and the preset tolerance; and to perform secondary registration on the primary registration image according to the offset vector to obtain the secondary registration image.

[0097] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440. The processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus. The processor 410 can call logical instructions stored in the memory 430 to execute the following methods: The image to be detected is input into a pre-trained first detection model. The first detection model detects Regions of Interest (ROIs) in the image, including key components, and obtains a first coordinate set corresponding to the ROIs. The first detection model is a deep neural network. Based on the first coordinate set and a first reference coordinate set corresponding to the ROIs pre-labeled in the template image, a first transformation matrix is ​​obtained. The image to be detected is registered once based on the first transformation matrix to obtain a first-registered image. The first-registered image is input into a pre-trained second detection model. The first detection model detects key components in the image, and obtains a second coordinate set corresponding to the key components. The second detection model is a deep neural network. Based on the second coordinate set and a second reference coordinate set corresponding to the key components pre-labeled in the template image, a second transformation matrix is ​​obtained. An offset reference vector is obtained according to the second transformation matrix. Based on the offset reference vector and a preset tolerance, an offset vector corresponding to at least one key component is determined. The first-registered image is then registered twice according to the offset vector to obtain a second-registered image.

[0098] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] This invention discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can perform the methods provided in the above-described method embodiments, such as including: The image to be detected is input into a pre-trained first detection model. The first detection model detects Regions of Interest (ROIs) in the image, including key components, and obtains a first coordinate set corresponding to the ROIs. The first detection model is a deep neural network. Based on the first coordinate set and a first reference coordinate set corresponding to the ROIs pre-labeled in the template image, a first transformation matrix is ​​obtained. The image to be detected is registered once based on the first transformation matrix to obtain a first-registered image. The first-registered image is input into a pre-trained second detection model. The first detection model detects key components in the image, and obtains a second coordinate set corresponding to the key components. The second detection model is a deep neural network. Based on the second coordinate set and a second reference coordinate set corresponding to the key components pre-labeled in the template image, a second transformation matrix is ​​obtained. An offset reference vector is obtained according to the second transformation matrix. Based on the offset reference vector and a preset tolerance, an offset vector corresponding to at least one key component is determined. The first-registered image is then registered twice according to the offset vector to obtain a second-registered image.

[0100] On the other hand, embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the transmission methods provided in the above embodiments, including, for example: The image to be detected is input into a pre-trained first detection model. The first detection model detects Regions of Interest (ROIs) in the image, including key components, and obtains a first coordinate set corresponding to the ROIs. The first detection model is a deep neural network. Based on the first coordinate set and a first reference coordinate set corresponding to the ROIs pre-labeled in the template image, a first transformation matrix is ​​obtained. The image to be detected is registered once based on the first transformation matrix to obtain a first-registered image. The first-registered image is input into a pre-trained second detection model. The first detection model detects key components in the image, and obtains a second coordinate set corresponding to the key components. The second detection model is a deep neural network. Based on the second coordinate set and a second reference coordinate set corresponding to the key components pre-labeled in the template image, a second transformation matrix is ​​obtained. An offset reference vector is obtained according to the second transformation matrix. Based on the offset reference vector and a preset tolerance, an offset vector corresponding to at least one key component is determined. The first-registered image is then registered twice according to the offset vector to obtain a second-registered image.

[0101] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. Such computer software products can be stored in computer-readable storage media, such as ROM / RAM, magnetic disks, optical disks, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for image registration of rail vehicles, characterized in that, The method includes: The image to be detected is input into a pre-trained first detection model, and the first detection model detects the regions of interest (ROIs) in the image to be detected that include key components, thereby obtaining a first set of coordinates corresponding to the regions of interest; the first detection model is a deep neural network. Based on the first coordinate set and the first reference coordinate set corresponding to the region of interest pre-labeled in the template image, a first transformation matrix is ​​obtained; based on the first transformation matrix, the image to be detected is registered once to obtain a first registered image; The first-registered image is input into a pre-trained second detection model, and the first detection model detects key components in the image to be detected, thereby obtaining a second coordinate set corresponding to the key components; the second detection model is a deep neural network. Based on the second coordinate set and the second reference coordinate set corresponding to the key components pre-marked in the template diagram, a second transformation matrix is ​​obtained; Based on the second transformation matrix, the offset reference vector is obtained; Based on the offset reference vector and the preset tolerance, offset vectors corresponding to at least one key component are determined; according to the offset vectors, the primary registration image is re-registered to obtain a secondary registration image.

2. The method according to claim 1, characterized in that, The region of interest is a rectangular frame; The first coordinate set includes the x-coordinate and y-coordinate of the center point of the region of interest, the width of the region of interest, and the height of the region of interest; Alternatively, the first coordinate set may include the corner coordinates corresponding to the region of interest.

3. The method according to claim 1, characterized in that, Based on the first coordinate set and the first reference coordinate set corresponding to the region of interest pre-labeled in the template image, a first transformation matrix is ​​obtained, including: Based on the first coordinate set and the first reference coordinate set corresponding to the region of interest pre-marked in the template image, at least three pairs of non-collinear points are determined; a first two-dimensional affine transformation model corresponding to the at least three pairs of non-collinear points is constructed; the first two-dimensional affine transformation model includes multiple first parameters representing at least one of the transformation methods of translation, rotation, scaling, and shearing. The first parameters of the first two-dimensional affine transformation model are solved by the least squares method or the singular value decomposition method to obtain the first transformation matrix.

4. The method according to claim 1, characterized in that, The second coordinate set includes the x-coordinate and y-coordinate of the center of the key component, the width of the bounding rectangle of the key component, and the height of the region of interest; Alternatively, the second coordinate set may include the corner coordinates corresponding to the outer rectangle of the key component.

5. The method according to any one of claims 1-4, characterized in that, Based on the second coordinate set and the second reference coordinate set corresponding to the key components pre-marked in the template diagram, a second transformation matrix is ​​obtained, including: Based on the second coordinate set and the second reference coordinate set corresponding to the key components pre-marked in the template diagram, at least three pairs of non-collinear points are determined; a second two-dimensional affine transformation model corresponding to the at least three pairs of non-collinear points is constructed; the second two-dimensional affine transformation model includes multiple second parameters representing rigid motion or affine deformation; The second two-dimensional affine transformation model is solved by using the least squares method or singular value decomposition method to obtain the second transformation matrix.

6. The method according to any one of claims 1-4, characterized in that, Based on the second coordinate set and the second reference coordinate set corresponding to the key components pre-marked in the template diagram, a second transformation matrix is ​​obtained, including: Based on the second coordinate set and the second reference coordinate set corresponding to the key components pre-marked in the template diagram, at least 3 pairs of non-collinear points are determined. Call the cv2.estimateAffine2D or cv2.getAffineTransform function from the open-source computer vision library, input the at least 3 pairs of non-collinear points into the cv2.estimateAffine2D or cv2.getAffineTransform function, and obtain the returned second transformation matrix.

7. The method according to claim 1, characterized in that, Based on the second transformation matrix, the offset reference vector is obtained, including: Extract multiple translation components corresponding to multiple key components from the second transformation matrix; The offset reference vector is determined based on the multiple translation components.

8. The method according to claim 7, characterized in that, Based on the offset reference vector and the preset tolerance, determine the offset vector corresponding to at least one key component, including: The difference between the translation component corresponding to a key component and the offset reference vector is determined, the difference is compared with the preset tolerance, and the offset reference vector or the translation component is selected as the offset vector corresponding to the key component based on the comparison result.

9. A rail vehicle image registration device, characterized in that, include: The first detection module is used to input the image to be detected into a pre-trained first detection model, and use the first detection model to detect the region of interest (ROI) in the image to be detected that includes key components, and obtain the first coordinate set corresponding to the region of interest. The primary registration module is used to obtain a first transformation matrix based on the first coordinate set and the first reference coordinate set corresponding to the region of interest pre-labeled in the template image; and to perform primary registration on the image to be detected based on the first transformation matrix to obtain a primary registered image. The second detection module is used to input the first registered image into a pre-trained second detection model, and detect key components in the image to be detected through the first detection model to obtain the second coordinate set corresponding to the key components. The second transformation module is used to obtain a second transformation matrix based on the second coordinate set and the second reference coordinate set corresponding to the key components pre-marked in the template diagram; The acquisition module is used to obtain the offset reference vector based on the second transformation matrix; The secondary registration module is used to determine the offset vector corresponding to at least one key component based on the offset reference vector and the preset tolerance; and to perform secondary registration on the primary registration image according to the offset vector to obtain a secondary registration image.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the rail vehicle image registration method as described in any one of claims 1 to 8.

Images