Pantograph pose vision measurement method, device and system and medium

Abstract

The invention provides a pantograph pose vision measurement method, device and system and a medium, and the method comprises the steps: carrying out the high-definition imaging of a pantograph, carrying out the image feature point extraction of a pantograph image, solving to obtain a pantograph pose initial value, extracting the edge information of a pantograph topology result, and obtaining a pose optimal solution through a nonlinear optimization method, all-weather, all-time and multi-vehicle type pantograph automatic and intelligent detection is realized, and related detection results are timely sent to a driver so as to guide timely parking for overhaul and maintenance.

Description

Technical Field

[0001] The invention relates to a pantograph posture visual measurement method, device, system and medium, belonging to the technical field of photoelectric detection technology. Background Art

[0002] As an important means of railway transportation, high-speed rail has the characteristics of long mileage, high safety requirements, harsh operating environment, and complex maintenance. Among them, online inspection of the pantograph operating status is a key link to ensure the safe and efficient operation of the train. For a long time, the posture parameters of the pantograph during high-speed operation are important parameters for evaluating the operating stability of the pantograph and the quality of the line. Traditional measurement methods can only measure acceleration and displacement, and cannot measure accurate pantograph posture parameters, which seriously restricts the accuracy and efficiency of current locomotive roof foreign body detection.

[0003] At present, with the rapid development of technologies such as photoelectric detection and machine vision, they are widely used in rail transit safety detection due to their non-contact, high-precision, and large-range advantages, effectively improving detection efficiency and accuracy. However, due to the high roof of the locomotive, the fast running speed of the train, and the complex operating environment, traditional photoelectric measurement devices cannot meet the detection requirements of the pantograph posture when the train is running at high speed. Therefore, realizing high-precision dynamic visual detection of the roof pantograph posture in complex environments is of great significance to ensuring the quality of locomotive safety detection and improving detection efficiency. Summary of the invention

[0004] The purpose of the present invention is to provide a pantograph posture visual measurement method, device, system and medium, which extracts image feature points of the pantograph image to obtain the initial value of the pantograph posture, and extracts edge point information of the pantograph topology result that remains unchanged, and obtains the optimal posture solution through a nonlinear optimization method, thereby realizing automatic and intelligent detection of pantographs of all-weather, all-day and multi-vehicle models, so as to guide inspection and maintenance.

[0005] In order to achieve the above objectives / solve the above technical problems, the present invention is implemented by adopting the following technical solutions.

[0006] In one aspect, the present invention provides a pantograph posture visual measurement method, comprising:

[0007] Acquire pantograph image data in real time and perform preprocessing;

[0008] Input the preprocessed real-time pantograph image data into the pre-trained Yolov8 network model to extract the image coordinates of the pantograph key points in the real-time pantograph image data;

[0009] The pantograph key point image coordinates are combined with the pre-acquired pantograph key point three-dimensional coordinates to calculate the current pantograph posture initial value through the RPnP method;

[0010] Based on the initial value of the pantograph posture, the objective function is established by minimizing the back-projection error of the pre-acquired pantograph head edge point information, and the optimal solution of the pantograph posture is obtained through nonlinear optimization.

[0011] Furthermore, the method for pre-acquiring the three-dimensional coordinates of the pantograph key points specifically includes:

[0012] A three-dimensional coordinate system is established with the center of gravity of the design model of the pantograph carbon slide plate and the bottom bracket assembly as the coordinate origin, and the three-dimensional coordinates of the key points of the pantograph are obtained. The key points of the pantograph include: the end of the pantograph head and the connection between the carbon slide plate and the bottom bracket.

[0013] Furthermore, the method for extracting the image coordinates of the pantograph key points in the real-time pantograph image data by the pre-trained Yolov8 network model specifically includes:

[0014] The features of the pantograph image are extracted through the backbone network of the Yolov8 network model, and then the features are fused through the neck network. Finally, a feature map is generated in the head network, and the image coordinates of the pantograph key points are extracted based on the feature map.

[0015] Furthermore, the pantograph key point image coordinates are combined with the pre-acquired pantograph key point three-dimensional coordinates through the RPnP method to calculate the current pantograph posture initial value, which specifically includes:

[0016] Set the objective function:

[0017] ;

[0018] in: is the current optimal solution for the pantograph posture, is the position of the pantograph in the camera coordinate system, is the image coordinate of the i-th pantograph key point, is the camera’s intrinsic parameter matrix, is the three-dimensional coordinate of the key point of the i-th pantograph, is the total number of pantograph key points;

[0019] When the sum of squares of the reprojection errors of all pantograph key points is minimized, the initial value of the pantograph posture is obtained through the objective function. , including the rotation vector and translation vector of the initial value of the pantograph posture.

[0020] Furthermore, the pantograph head edge points specifically include: the upper edges of the pantograph heads on both sides of the pantograph, which are projected onto the camera image coordinates as the pantograph head edge point image coordinates, which can be used for subsequent optimization to solve the optimal solution for the pantograph posture.

[0021] Furthermore, based on the initial value of the pantograph posture, the objective function is established by minimizing the back-projection error of the pantograph head edge point information obtained in advance, and the optimal solution of the pantograph posture is obtained by nonlinear optimization, which specifically includes:

[0022] ;

[0023] in, Indicates the back-projection operation of the pantograph edge point on the camera, is the back projection error of the edge point; is the three-dimensional coordinate of the i-th edge point; is the image coordinate of the i-th edge point; They are the rotation vector and translation vector of the initial value of the pantograph posture; is the number of edge points.

[0024] In a second aspect, the present invention provides a pantograph posture visual measurement device, comprising:

[0025] An acquisition module is used to acquire pantograph image data in real time and perform preprocessing;

[0026] An extraction module is used to input the preprocessed real-time pantograph image data into a pre-trained Yolov8 network model to extract the image coordinates of the pantograph key points in the real-time pantograph image data;

[0027] A calculation module, used to calculate the initial value of the current pantograph posture by combining the image coordinates of the pantograph key points with the pre-acquired three-dimensional coordinates of the pantograph key points through the RPnP method;

[0028] The optimization module is used to establish an objective function based on the initial value of the pantograph posture and minimize the back-projection error of the pre-acquired pantograph bow head edge point information, and obtain the optimal solution of the pantograph posture through nonlinear optimization.

[0029] In a third aspect, the present invention provides a pantograph posture visual measurement system, comprising:

[0030] Memory, for storing computer programs / instructions;

[0031] A processor is used to execute the computer program / instructions to implement the steps of the above-mentioned pantograph posture visual measurement method.

[0032] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the above-mentioned pantograph posture visual measurement method are implemented.

[0033] In a fifth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned pantograph posture visual measurement method.

[0034] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the present invention proposes an intelligent extraction method for pantograph key points based on Yolov8, through a pre-trained pantograph key point automatic extraction network, automatically extracted pantograph key point image data, combined with the three-dimensional coordinates of the pantograph key points, and the RPnP algorithm is used to obtain the initial value of the pantograph posture, and then the minimization of the pantograph key point reprojection error is used as the objective function to solve the initial value of the pantograph posture, providing a data basis for subsequent precise calculations; based on the unchanged three-dimensional data of the edge points of the pantograph bow head, the objective function is established by minimizing the edge point reprojection error, and the pantograph posture parameters are obtained by nonlinear optimization, thereby realizing the automated and intelligent detection of pantographs of all-weather, all-day, and multi-vehicle models to guide inspection and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Shown is a schematic diagram of the process of the present invention;

[0036] Figure 2 Shown is a measurement schematic diagram of the present invention. DETAILED DESCRIPTION

[0037] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0038] The term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " generally indicates that the related objects are in an "or" relationship.

[0039] Example 1

[0040] like Figure 1~2 In one embodiment, this embodiment provides a pantograph posture visual measurement method, including:

[0041] Step S1: Real-time acquisition of pantograph image data and preprocessing, specifically including:

[0042] Step S1.1: Acquire image data of the pantograph by using a high-resolution visible light camera;

[0043] Step S1.2: Use the Letterbox resize method to resize the acquired image to the size required by the model, keep the image ratio while padding the edges, make the image size uniform, and scale the pixel value of the image to between 0 and 1.

[0044] Step S2: inputting the pre-processed real-time pantograph image data into the pre-trained Yolov8 network model to extract the image coordinates of the pantograph key points in the real-time pantograph image data, specifically including:

[0045] The features of the pantograph image are extracted through the backbone network of the Yolov8 network model, and then the features are fused through the neck network. Finally, a feature map is generated in the head network, and the image coordinates of the pantograph key points are extracted according to the feature map. The image coordinates of the pantograph key points are defined as ;

[0046] Among them: YOLOv8 outputs three feature maps of scale: 80x80, 40x40 and 20x20, and the head network outputs classification and regression feature maps of 6 scales in total;

[0047] Convert the bbox branch of the head network output, and use Softmax and Conv calculations to convert the integral form into a 4-dimensional bbox format;

[0048] The category prediction branches and bbox prediction branches of three different scales are concatenated and dimensionally transformed; the shapes of the category prediction branch and bbox prediction branch are (b, 80x80+40x40+20x20, 80) and (b, 8400, 4) respectively;

[0049] The classification prediction branch performs Sigmoid calculation, while the bbox prediction branch needs to be decoded and restored to the real original image in the decoded xyxy format;

[0050] Traverse each image in the batch and use score_thr for threshold filtering. In this process, multi_label and nms_pre need to be considered to ensure that the number of detection boxes after filtering is not more than nms_pre.

[0051] The remaining detection boxes are restored to the original image scale before network output, and then NMS processing is performed to remove overlapping bounding boxes and retain the best prediction results. The final output detection box cannot exceed max_per_img to complete the reasoning process of key point detection;

[0052] Among them: The training method of the Yolov8 network model includes:

[0053] Obtain some historical pantograph image data and construct training sets and test sets, where the training set includes sample images of different weather, time of day, and light conditions. The number of training set images is no less than 10,000, and the number of test set images is no less than 2,000. Annotate the training set images according to the defined pantograph key points, and store the annotated files as offline files for training;

[0054] The training set is input into the Yolov8 network model for training. First, the training parameters are set, including the learning rate, number of training rounds, batch size, and input image size. The Adam optimizer is used for training until the training end conditions are met. The training weight file is obtained and saved as an offline file to realize the Yolov8 network model training.

[0055] Step S3: Calculate the initial value of the current pantograph posture by combining the pantograph key point image coordinates with the pre-acquired pantograph key point three-dimensional coordinates through the RPnP method, specifically including:

[0056] Step S3.1: Establish a three-dimensional coordinate system with the center of gravity of the design model of the pantograph carbon slide plate and the bottom bracket assembly as the coordinate origin, and obtain the three-dimensional coordinates of the key points of the pantograph. The key points of the pantograph include: the end of the pantograph head and the six key points at the connection between the carbon slide plate and the bottom bracket, and define the three-dimensional coordinates of the key points of the pantograph as ;

[0057] Step S3.2: Setting the objective function:

[0058] ;

[0059] in: is the current optimal solution for the pantograph posture, is the position of the pantograph in the camera coordinate system, is the image coordinate of the i-th pantograph key point, is the camera’s intrinsic parameter matrix, is the three-dimensional coordinate of the key point of the i-th pantograph, is the total number of pantograph key points;

[0060] When the sum of squares of the reprojection errors of all pantograph key points is minimized, the initial value of the pantograph posture is obtained through the objective function. , including the rotation vector and translation vector of the initial value of the pantograph posture.

[0061] Step S4: Based on the initial value of the pantograph posture, the objective function is established by minimizing the back-projection error of the pantograph head edge point information obtained in advance, and the optimal solution of the pantograph posture is obtained through nonlinear optimization, which specifically includes:

[0062] Step S4.1: Obtain the edge point of the pantograph head: the upper edges of the pantograph heads on both sides of the pantograph are taken as the edge points of the pantograph head, and the projection of the upper edges of the pantograph heads on the camera image coordinates is the image coordinates of the pantograph head edge points;

[0063] Step S4.11: Obtain pantograph images under different weather conditions as the pantograph sample set to be trained, and annotate the image edge points at the pantograph head using an annotation tool;

[0064] Step S4.12: Use a multi-level deep network to extract edge point features of images of different scales, and use a fully connected self-network to complete the prediction and output of edge point data at the pantograph head. By training the data in step S4.11 with a multi-level deep network, a weight file for extracting edge points at the pantograph head is obtained;

[0065] Step S4.13: Through the training weight file in step S4.12, edge point information is extracted from the pantograph image collected on site. Since the pantograph head is not worn by the contact network, it can be assumed that its three-dimensional structure has not changed and can be considered as a known quantity. The pantograph design model point set is ;

[0066] Step S4.2: Based on the initial value of the pantograph posture, the objective function is established by minimizing the back-projection error of the pantograph head edge point information obtained in advance, and the optimal solution of the pantograph posture is obtained through nonlinear optimization, which specifically includes:

[0067] By minimizing the error between the image edge point coordinates calculated by camera matrix projection and the detected image edge points:

[0068] ;

[0069] in, Indicates the back-projection operation of the pantograph edge point on the camera, is the back projection error of the edge point; is the three-dimensional coordinate of the i-th edge point; is the image coordinate of the i-th edge point; They are the rotation vector and translation vector of the initial value of the pantograph posture; is the number of edge points;

[0070] The specific process includes: Initialization: Select an initial camera pose estimate and ;

[0071] Establish an error model: For each edge point feature, calculate its projection on the camera imaging plane and the difference between it and the observed image point, i.e., the reprojection error;

[0072] Jacobian matrix calculation: Calculate the Jacobian matrix of the reprojection error with respect to the pose parameters (rotation and translation);

[0073] Linearization: Use Taylor expansion to linearize the error function near the current pose parameters, converting the nonlinear problem into a linear problem;

[0074] Optimization algorithm: Use the nonlinear optimization algorithm of Levenberg-Marquardt algorithm or Gauss-Newton method to iteratively update the pose parameters to minimize the objective function;

[0075] Iterative update: In each iteration, the pose parameters are updated according to the Jacobian matrix and error information;

[0076] Convergence judgment: After each iteration, check whether the change of posture parameters is less than the preset threshold. If so, it is considered to have converged and the optimization process is over;

[0077] Result verification: Verify the optimized pose parameters to ensure that the reprojection error is within an acceptable range and that the pose parameters are reasonable;

[0078] The above steps provide the basic process of solving the posture parameters based on the minimum reprojection error of the image edge point information in this embodiment. In practical applications, the steps and parameters can be adjusted according to the specific situation to achieve the best estimation effect, and the acquired posture parameters can be sent to the driver in time to guide timely parking for inspection and maintenance.

[0079] Example 2

[0080] This embodiment provides a pantograph posture visual measurement device, including:

[0081] An acquisition module is used to acquire pantograph image data in real time and perform preprocessing;

[0082] An extraction module is used to input the preprocessed real-time pantograph image data into a pre-trained Yolov8 network model to extract the image coordinates of the pantograph key points in the real-time pantograph image data;

[0083] A calculation module, used to calculate the initial value of the current pantograph posture by combining the image coordinates of the pantograph key points with the pre-acquired three-dimensional coordinates of the pantograph key points through the RPnP method;

[0084] The optimization module is used to establish an objective function based on the initial value of the pantograph posture and minimize the back-projection error of the pre-acquired pantograph bow head edge point information, and obtain the optimal solution of the pantograph posture through nonlinear optimization.

[0085] Example 3

[0086] This embodiment provides a pantograph posture visual measurement system, including:

[0087] Memory, for storing computer programs / instructions;

[0088] A processor is used to execute the computer program / instructions to implement the steps of the above-mentioned pantograph posture visual measurement method.

[0089] Example 4

[0090] This embodiment provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the above-mentioned pantograph posture visual measurement method are implemented.

[0091] Example 5

[0092] This embodiment provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned pantograph posture visual measurement method.

[0093] The present invention acquires image data through a high-resolution visible light camera. The camera has a high-speed interface and a frame rate of more than 50 Hz. It can also clearly image in high-dynamic lighting environments, including nighttime, strong light, dawn and other all-weather conditions. It can be supplemented by near-infrared lighting, which can automatically adjust the brightness of the light source according to the brightness of the light and the quality judgment result of the image collected by the camera. It has uniform and stable light output and can work continuously for a long time, meeting the requirements of outdoor all-weather and all-day imaging.

[0094] The on-board processing unit is then used to automatically detect the position of the roof pantograph, which can accurately measure and alarm any abnormal pantograph position. It has low requirements for vehicle models and environments, has a certain degree of versatility, and can be extended to the pantograph position detection field for trucks, buses, EMUs, and subway trains.

[0095] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0097] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0099] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A pantograph posture visual measurement method, characterized in that: include: Acquire pantograph image data in real time and perform preprocessing; Input the preprocessed real-time pantograph image data into the pre-trained Yolov8 network model to extract the image coordinates of the pantograph key points in the real-time pantograph image data; The pantograph key point image coordinates are combined with the pre-acquired pantograph key point three-dimensional coordinates to calculate the current pantograph posture initial value through the RPnP method; Based on the initial value of the pantograph posture, the objective function is established by minimizing the back-projection error of the pre-acquired pantograph head edge point information, and the optimal solution of the pantograph posture is obtained through nonlinear optimization.

2. The pantograph posture visual measurement method according to claim 1 is characterized in that: The method for pre-acquiring the three-dimensional coordinates of the pantograph key points specifically includes: A three-dimensional coordinate system is established with the center of gravity of the design model of the pantograph carbon slide plate and the bottom bracket assembly as the coordinate origin, and the three-dimensional coordinates of the key points of the pantograph are obtained. The key points of the pantograph include: the end of the pantograph head and the connection between the carbon slide plate and the bottom bracket.

3. The pantograph posture visual measurement method according to claim 2, characterized in that: The method for extracting the image coordinates of the pantograph key points in the real-time pantograph image data using the pre-trained Yolov8 network model specifically includes: The features of the pantograph image are extracted through the backbone network of the Yolov8 network model, and then the features are fused through the neck network. Finally, a feature map is generated in the head network, and the image coordinates of the pantograph key points are extracted based on the feature map.

4. The pantograph posture visual measurement method according to claim 1, characterized in that: The method of calculating the initial value of the current pantograph posture by combining the pantograph key point image coordinates with the pre-acquired pantograph key point three-dimensional coordinates through the RPnP method specifically includes: Set the objective function: ； in: is the current optimal solution for the pantograph posture, is the pantograph pose in the camera coordinate system, is the image coordinate of the i-th pantograph key point, is the camera’s intrinsic parameter matrix, is the three-dimensional coordinate of the key point of the i-th pantograph, is the total number of pantograph key points; When the sum of squares of the reprojection errors of all pantograph key points is minimized, the initial value of the pantograph posture is obtained through the objective function. , including: the rotation vector and translation vector of the initial value of the pantograph posture.

5. The pantograph posture visual measurement method according to claim 4, characterized in that: The pantograph head edge points specifically include: the upper edges of the pantograph heads on both sides of the pantograph.

6. The pantograph posture visual measurement method according to claim 5, characterized in that: The objective function is established based on the initial value of the pantograph posture by minimizing the back-projection error of the pantograph head edge point information obtained in advance, and the optimal solution of the pantograph posture is obtained through nonlinear optimization, which specifically includes: ； in, Indicates the back-projection operation of the pantograph edge point on the camera, is the back projection error of the edge point; is the three-dimensional coordinate of the i-th edge point; is the image coordinate of the i-th edge point; They are the rotation vector and translation vector of the initial value of the pantograph posture; is the number of edge points.

7. A pantograph posture visual measurement device, characterized in that: include: An acquisition module is used to acquire pantograph image data in real time and perform preprocessing; An extraction module is used to input the preprocessed real-time pantograph image data into a pre-trained Yolov8 network model to extract the image coordinates of the pantograph key points in the real-time pantograph image data; A calculation module, used to calculate the initial value of the current pantograph posture by combining the image coordinates of the pantograph key points with the pre-acquired three-dimensional coordinates of the pantograph key points through the RPnP method; The optimization module is used to establish an objective function based on the initial value of the pantograph posture and minimize the back-projection error of the pre-acquired pantograph bow head edge point information, and obtain the optimal solution of the pantograph posture through nonlinear optimization.

8. A pantograph posture visual measurement system, characterized in that: include: Memory, for storing computer programs / instructions; A processor, used to execute the computer program / instructions to implement the steps of the pantograph posture visual measurement method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the pantograph posture visual measurement method described in any one of claims 1-6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the pantograph posture visual measurement method described in any one of claims 1 to 6 are implemented.

Images