Image and laser fused vehicle loading state detection method

Abstract

The invention relates to the technical field of railway freight safety detection, and particularly discloses an image and laser fused vehicle loading state detection method, which comprises the steps of S1, acquiring whole vehicle point cloud data of a railway freight car and image data of each compartment; s2, dividing carriages of the railway wagon according to the point cloud data of the whole wagon to obtain point cloud data of a plurality of carriages; s3, performing data alignment and fusion according to the carriage point cloud data and the image data of each carriage to obtain fusion data of each carriage; and S4, carrying out loading abnormity detection and identification on the fused data to obtain the vehicle loading state of the railway wagon.

Description

Technical Field

[0001] The present invention relates to the technical field of railway freight safety detection, and in particular to a vehicle loading status detection method integrating images and lasers. Background Art

[0002] Railway freight inspection operations, as a key link in ensuring the safety of railway freight transportation, mainly involve the inspection of freight trains and their cargo. At present, freight inspection operations mainly rely on manual inspection, supplemented by machine inspection, to identify potential problems with the freight train body and cargo loading through manual review or automatic machine recognition of passing images. However, the current method has several shortcomings:

[0003] (1) The image data of railway freight cars lacks the three-dimensional spatial coordinate information of the car body and cargo, which may lead to inaccurate detection of hidden anomalies by manual and machine detection, thereby increasing the risk of missed detection;

[0004] (2) At present, machine inspection algorithms mainly rely on pure image analysis and recognition, and the misjudgment rate is relatively high. This not only affects the inspection efficiency, but also makes the workload of manual cargo inspection still relatively high. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a vehicle loading status detection method that integrates images and lasers. By introducing advanced vision and laser fusion technology, the accuracy and efficiency of cargo inspection operations can be improved, thereby providing a more solid guarantee for railway freight safety.

[0006] The present invention provides a vehicle loading status detection method integrating images and lasers, comprising:

[0007] Step S1, obtaining the whole vehicle point cloud data of the railway freight car and the image data of each compartment;

[0008] Step S2, dividing the carriages of the railway freight car according to the whole vehicle point cloud data to obtain a plurality of carriage point cloud data;

[0009] Step S3, aligning and fusing the point cloud data and image data of each carriage to obtain fused data of each carriage;

[0010] Step S4, detecting and identifying loading anomalies on the fused data to obtain the vehicle loading status of the railway freight car.

[0011] In a possible implementation, the whole vehicle point cloud data includes contour points of multiple cross sections of the railway freight car;

[0012] The contour points are three-dimensional coordinate points in an XYZ coordinate system, wherein the X axis is the direction of the sleeper, the Y axis is the direction along the track, and the Z axis is the upward direction perpendicular to the rail surface.

[0013] In a possible implementation, the S2 includes:

[0014] The width of the vehicle point cloud data is calculated based on the difference between the maximum and minimum values ​​in the X-axis direction of each frame of the vehicle point cloud data;

[0015] When the carriage width is less than a preset width, determining the frame corresponding to the laser data as a possible car separation point;

[0016] Cluster all possible separation points to obtain the freight car gap points of railway freight cars;

[0017] The whole vehicle point cloud data is divided according to the gap points of the truck to obtain a plurality of compartment point cloud data.

[0018] In a possible implementation, the S2 further includes:

[0019] The carriage number of each carriage is calculated according to the frame number of the freight car gap point, and the carriage number is associated with the carriage point cloud data.

[0020] In a possible implementation, the image data includes left image data, right image data and top image data of each compartment.

[0021] In a possible implementation, S3 includes:

[0022] The carriage point cloud data is screened to obtain right half point cloud data in the X-axis direction, left half point cloud data in the X-axis direction, and half point cloud data in the Z-axis direction;

[0023] Perform vehicle body feature point detection according to the right image data to obtain a plurality of first feature points;

[0024] Projecting the right half of the point cloud data in the X-axis direction to obtain plane point cloud data of YZ-axis direction components;

[0025] Performing vehicle body feature point detection on the plane point cloud data of the X-axis direction component to obtain a plurality of second feature points;

[0026] Constructing a feature mapping matrix reflecting the transformation relationship between the right image data and the plane point cloud data of the YZ axis direction components according to the first feature point and the second feature point;

[0027] Mapping the plane point cloud data onto the carriage point cloud data according to the feature mapping matrix to obtain right fusion data of the right image data and the right half point cloud data in the X-axis direction;

[0028] The left image data and the left point cloud data in the Y-axis direction are fused to obtain left fused data of the left image data and the left half point cloud data in the X-axis direction;

[0029] The top image data and the left point cloud data in the Z-axis direction are fused to obtain top fusion data of the top image data and the half-side point cloud data in the Z-axis direction;

[0030] The right fusion data, the left fusion data and the top fusion data are aggregated to obtain the fusion data of the carriage.

[0031] In a possible implementation, mapping the plane point cloud data to the carriage point cloud data according to the feature mapping matrix to obtain right fusion data of the right image data and the right half point cloud data in the X-axis direction includes:

[0032] Performing over-limit detection on the carriage point cloud data and cargo, and marking abnormal point clouds in the carriage point cloud data to obtain marked carriage point cloud data;

[0033] The plane point cloud data is mapped onto the labeled carriage point cloud data according to a feature mapping matrix to obtain the right fused data.

[0034] In a possible implementation, the number of the first feature points is related to the model and cargo of the railway freight car in the image data;

[0035] The number of the second feature points is related to the model and cargo of the railway freight car in the plane point cloud data.

[0036] In a possible implementation manner, the S1 further includes:

[0037] Performing noise removal and contrast enhancement operations on the image data of each carriage to obtain processed image data;

[0038] The data alignment and fusion are performed based on the carriage point cloud data and image data of each carriage to obtain the fused data of each carriage, including:

[0039] Data alignment and fusion are performed based on the carriage point cloud data and processed image data of each carriage to obtain the fused data of each carriage.

[0040] In a possible implementation, the S4 further includes:

[0041] When the vehicle loading state of the railway freight car is abnormal, an alarm signal is output.

[0042] The vehicle loading status detection method of the fusion image and laser provided by the present invention uses advanced data fusion technology to accurately align laser coordinates and image pixels, thereby detecting whether the cargo and vehicle body have abnormal conditions such as over-limit. In addition, it can also calculate the three-dimensional size information of the loading abnormality and display it in a fusion manner. It not only significantly improves the quality and efficiency of railway freight inspection operations, but also provides real-time decision support for railway freight inspection operators, ensuring the safety and efficiency of railway transportation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of a vehicle loading status detection method provided by an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of vehicle point cloud data provided by an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of compartments provided in an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of car point cloud data provided by an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of plane point cloud data provided by an embodiment of the present invention;

[0048] Figure 6 A schematic diagram of image data of a carriage provided by an embodiment of the present invention;

[0049] Figure 7 A schematic diagram of fusion data of a railway freight car provided by an embodiment of the present invention;

[0050] Figure 8 A schematic diagram of the fusion data detection effect of a railway freight car provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following detailed description of the embodiments of the present invention is further described in detail in conjunction with the accompanying drawings and examples. The detailed description of the following embodiments and the accompanying drawings are used to exemplarily illustrate the principles of the present invention, but cannot be used to limit the scope of the present invention, that is, the present invention is not limited to the preferred embodiments described, and the scope of the present invention is defined by the claims.

[0052] In the description of the present invention, it should be noted that, unless otherwise specified, “plurality” means two or more than two; the terms “first”, “second”, etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance; for ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0053] The present invention combines deep learning technology with image-laser point cloud fusion technology and applies it to the field of intelligent identification of railway freight car loading status, which can overcome the shortcomings of existing freight car loading status detection and identification technology and can well solve the problems of efficiency, cost, safety, etc. in freight car loading status detection. Compared with the existing technical solutions, the innovation of this patent is mainly reflected in the following two aspects:

[0054] 1) Comprehensively consider the 2D image information and 3D spatial coordinate information, accurately fuse the laser point cloud data and image data at the pixel level, thus providing more comprehensive information on the status of the truck and cargo;

[0055] 2) Make full use of laser point cloud and image features to identify abnormal loading status, thereby improving the detection effect and quantifying loading abnormalities.

[0056] Figure 1 A flow chart of a vehicle loading status detection method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the present invention provides a vehicle loading status detection method integrating images and lasers, comprising:

[0057] Step S1, obtaining the whole vehicle point cloud data of the railway freight car and the image data of each compartment;

[0058] Figure 2 A schematic diagram of vehicle point cloud data provided by an embodiment of the present invention, Figure 6 A schematic diagram of image data of a carriage provided by an embodiment of the present invention.

[0059] In one possible implementation, in order to facilitate subsequent image data recognition, noise removal and contrast enhancement operations are performed on the image data of each carriage to obtain processed image data; data alignment and fusion are performed based on the carriage point cloud data and the processed image data of each carriage to obtain fused data of each carriage.

[0060] In a possible implementation, the image data includes left image data, right image data, and top image data of each compartment.

[0061] Step S2, dividing the carriages of the railway freight car according to the whole vehicle point cloud data to obtain a plurality of carriage point cloud data;

[0062] In a possible implementation, the vehicle point cloud data includes contour points of multiple sections of a railway freight car. The contour points are three-dimensional coordinate points in an XYZ coordinate system, where the X axis is the direction of the sleeper, the Y axis is along the track, and the Z axis is vertical to the rail surface and upward.

[0063] In a possible implementation, S2 includes:

[0064] Calculate the car body width of the vehicle point cloud data based on the difference between the maximum value and the minimum value in the X-axis direction of the vehicle point cloud data for each frame.

[0065] When the car body width is less than the preset width, determine that the frame corresponding to the laser data is a possible vehicle separation point.

[0066] Perform clustering processing on all possible vehicle separation points to obtain the gap points of the railway freight car.

[0067] Divide the vehicle point cloud data according to the gap points of the freight car to obtain multiple car body point cloud data.

[0068] In a possible implementation manner, S2 further includes:

[0069] Calculate the car body numbers of each car body according to the frame numbers of the gap points of the freight car, and associate the car body numbers with the car body point cloud data.

[0070] In an example, in step 21, read the nth frame of vehicle point cloud data F[n], where n = 1,..., N, and N is the total number of laser point cloud frames.

[0071] In step 22, calculate the car body width W[n] of the nth frame of vehicle point cloud data F[n] according to the following formula:

[0072] W[n] = F[n].x_max - F[n].x_min

[0073] Where, F[n].x_max is the maximum value in the x direction of the nth frame of vehicle point cloud data, and F[n].x_min is the minimum value in the x direction of the nth frame of vehicle point cloud data.

[0074] In step 23, compare the car body width W[n] with the preset width w0;

[0075] If W[n] ≥ w0, skip the nth frame of vehicle point cloud data;

[0076] If W[n] < w0, include the nth frame of vehicle point cloud data in the possible vehicle separation points S0. The value of the preset width w0 can be set according to the width of the railway freight car, and can be set to 1 in the present invention.

[0077] Repeat step 21 - step 23 until all vehicle point cloud data is read to obtain M possible vehicle separation points S0[m], where m = 1,..., M

[0078] In step 24, perform clustering on the possible vehicle separation points S0[m], find the start and end values of each continuous subscript interval, and obtain the start and end subscripts S1[k] of the gaps between freight cars, where k = 1,..., K, and K is equal to the number of gaps between freight cars and also equal to the number of freight cars minus 1.

[0079] Step 25, calculate the average value of the start and end subscripts of the gap between each freight car as the subscript for dividing the cars between cars, complete the division of the point cloud data of the whole car, and obtain the point cloud data P[i] of each car, where i=1,…,K+1.

[0080] The effect of vehicle division is as follows Figure 3 As shown, the effect of a single carriage after segmentation is as follows Figure 4 shown. Figure 3 A schematic diagram of a compartment provided by an embodiment of the present invention, Figure 4 A schematic diagram of carriage point cloud data provided by an embodiment of the present invention.

[0081] Step S3, aligning and fusing the point cloud data and image data of each carriage to obtain fused data of each carriage;

[0082] In a possible implementation, the car point cloud data is screened to obtain the right half point cloud data in the X-axis direction, the left half point cloud data in the X-axis direction, and the half point cloud data in the Z-axis direction;

[0083] Perform vehicle body feature point detection according to the right image data to obtain a plurality of first feature points;

[0084] Project the right half of the point cloud data in the X-axis direction to obtain the plane point cloud data of the YZ-axis direction component; Figure 5 A schematic diagram of planar point cloud data provided by an embodiment of the present invention.

[0085] Performing vehicle body feature point detection on the plane point cloud data of the YZ axis direction components to obtain a plurality of second feature points;

[0086] Constructing a feature mapping matrix reflecting the transformation relationship between the right image data and the plane point cloud data of the YZ axis direction component according to the first feature point and the second feature point;

[0087] Map the plane point cloud data to the carriage point cloud data according to the feature mapping matrix to obtain the right fusion data of the right image data and the right half point cloud data in the X-axis direction;

[0088] The left image data and the left point cloud data in the Y-axis direction are fused to obtain the left fusion data of the left image data and the left half point cloud data in the X-axis direction;

[0089] The top image data and the left point cloud data in the Z-axis direction are fused to obtain top fusion data of the top image data and the half-side point cloud data in the Z-axis direction; Figure 7 A schematic diagram of fused data of a railway freight car provided by an embodiment of the present invention.

[0090] The right fusion data, the left fusion data and the top fusion data are aggregated to obtain the fusion data of the carriage.

[0091] In a possible implementation, the plane point cloud data is mapped to the car point cloud data according to the feature mapping matrix, and the right fusion data of the right image data and the right half point cloud data in the X-axis direction are obtained, including:

[0092] Perform over-limit detection on the carriage point cloud data and the cargo, and mark the abnormal point clouds in the carriage point cloud data to obtain the marked carriage point cloud data;

[0093] According to the feature mapping matrix, the plane point cloud data is mapped to the labeled carriage point cloud data to obtain the fused data on the right.

[0094] In a possible implementation, the number of first feature points is related to the model and cargo of the railway freight car in the image data; the number of second feature points is related to the model and cargo of the railway freight car in the plane point cloud data.

[0095] In one example, step S31, read the image data I[i] and the carriage point cloud data P[i] of the i-th carriage;

[0096] Step S32, performing vehicle body feature point detection on the image data I[i] to obtain C1 first feature points Corner1[c], where c=1, ..., C1, and the size of C1 is related to the vehicle type and cargo in the image data;

[0097] Step S33, filter the car point cloud data P[i], retain only the points on the right side of the car body (X>=0), and obtain the right half point cloud data P[i] in the X-axis direction r [i];

[0098] Step S34: convert the right half point cloud data P r [i] Project in the X direction to obtain the plane point cloud data P that only retains the YZ direction components p [i], such as Figure 5 As shown;

[0099] Step S35: plane point cloud data P p [i] Detect vehicle body feature points to obtain C2 second feature points Corner2[c], where c = 1, ..., C2, and the size of C2 is related to the vehicle model and cargo in the plane point cloud data;

[0100] Step S36, constructing a feature mapping matrix Establishing a transformation relationship between the first feature point and the second feature point;

[0101] Step S37, performing vehicle body and cargo over-limit detection on the carriage point cloud data P[i], and marking abnormal point clouds;

[0102] Step S38: convert the plane point cloud data Pp [i] are all mapped to the car point cloud data P[i] to obtain the fused data and corresponding recognition results, where the red points in the image represent anomalies.

[0103] The left side and top images of the vehicle compartment may be fused and recognized in a similar manner as steps S31 to S38.

[0104] Step S4, detecting and identifying loading anomalies on the fused data to obtain the vehicle loading status of the railway freight car.

[0105] In a possible implementation, step S3 obtains the fused data of the right side, left side and top of each compartment of the railway freight car, and the loading status of each compartment is obtained by comprehensive detection and identification of the fused data of the right side, left side and top of the compartment. Further, the loading status of the railway freight car can be comprehensively judged according to the loading status of all compartments of the railway freight car. Figure 8 A schematic diagram of the fusion data detection effect of a railway freight car provided in an embodiment of the present invention.

[0106] In a possible implementation, S4 further includes: when the vehicle loading state of the railway freight car is abnormal, outputting an alarm signal. The alarm signal includes the railway freight car number, the carriage number and the corresponding fusion data, so that the maintenance personnel can quickly find the maintenance point.

[0107] The vehicle loading status detection method of the fusion image and laser provided by the present invention uses advanced data fusion technology to accurately align laser coordinates and image pixels, thereby detecting whether the cargo and vehicle body have abnormal conditions such as over-limit. In addition, it can also calculate the three-dimensional size information of the loading abnormality and display it in a fusion manner. It not only significantly improves the quality and efficiency of railway freight inspection operations, but also provides real-time decision support for railway freight inspection operators, ensuring the safety and efficiency of railway transportation.

[0108] Compared with the prior art, the present invention has the following beneficial effects:

[0109] (1) Significantly improve detection accuracy: By adopting innovative laser and image data fusion technology, the present invention can achieve high-precision feature matching and data fusion for those loading anomalies such as cargo over-limit and cargo missing that are difficult to detect using image recognition technology alone. This fusion not only enhances the accuracy of detection, but also improves the system's adaptability and robustness to complex environments, ensuring comprehensive and accurate detection of the truck's loading status.

[0110] (2) Significantly improve production efficiency and quality: The present invention can quickly identify abnormal loading in railway freight cars through automatic fusion detection and recognition technology, so as to take timely measures to avoid the risk of accidents caused by excessive cargo and loose cargo. This not only improves the safety and reliability of railway transportation, but also greatly reduces the time and cost required for manual inspection, and improves the overall efficiency of cargo inspection operations.

[0111] (3) Strong environmental adaptability: The detection method of the present invention can work stably under various environmental conditions, including different lighting conditions, weather changes, etc., ensuring the reliability and consistency of the detection.

[0112] (4) Improving the standardization of freight inspection operations: The present invention provides a standardized inspection process to ensure the consistency and repeatability of each inspection, thereby improving the standardization and regularization level of railway freight inspection operations.

[0113] The technological innovation of this patent not only improves the accuracy and comprehensiveness of detection, but also improves the efficiency and quality of cargo inspection operations, bringing important technological progress and practical value to the field of railway freight car loading status detection.

[0114] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A vehicle loading status detection method integrating image and laser, characterized in that: include: Step S1, obtaining the whole vehicle point cloud data of the railway freight car and the image data of each compartment; Step S2, dividing the carriages of the railway freight car according to the whole vehicle point cloud data to obtain a plurality of carriage point cloud data; Step S3, aligning and fusing the point cloud data and image data of each carriage to obtain fused data of each carriage; Step S4, detecting and identifying loading anomalies on the fused data to obtain the vehicle loading status of the railway freight car.

2. The vehicle loading status detection method according to claim 1, characterized in that: The whole vehicle point cloud data includes contour points of multiple sections of the railway freight car; The contour points are three-dimensional coordinate points in an XYZ coordinate system, wherein the X axis is the direction of the sleeper, the Y axis is the direction along the track, and the Z axis is the upward direction perpendicular to the rail surface.

3. The vehicle loading status detection method according to claim 2, characterized in that: The S2 includes: The width of the vehicle point cloud data is calculated based on the difference between the maximum and minimum values ​​in the X-axis direction of each frame of the vehicle point cloud data; When the carriage width is less than a preset width, determining the frame corresponding to the laser data as a possible car separation point; Cluster all possible separation points to obtain the freight car gap points of railway freight cars; The whole vehicle point cloud data is divided according to the gap points of the truck to obtain a plurality of compartment point cloud data.

4. The vehicle loading status detection method according to claim 3, characterized in that: The S2 further includes: The carriage number of each carriage is calculated according to the frame number of the freight car gap point, and the carriage number is associated with the carriage point cloud data.

5. The vehicle loading status detection method according to claim 1, characterized in that: The image data includes left image data, right image data and top image data of each compartment.

6. The vehicle loading status detection method according to claim 5, characterized in that: The S3 includes: The carriage point cloud data is screened to obtain right half point cloud data in the X-axis direction, left half point cloud data in the X-axis direction, and half point cloud data in the Z-axis direction; Perform vehicle body feature point detection according to the right image data to obtain a plurality of first feature points; Projecting the right half of the point cloud data in the X-axis direction to obtain plane point cloud data of YZ-axis direction components; Performing vehicle body feature point detection on the plane point cloud data of the YZ axis direction components to obtain a plurality of second feature points; Constructing a feature mapping matrix reflecting the transformation relationship between the right image data and the plane point cloud data of the YZ axis direction components according to the first feature point and the second feature point; Mapping the plane point cloud data onto the carriage point cloud data according to the feature mapping matrix to obtain right fusion data of the right image data and the right half point cloud data in the X-axis direction; The left image data and the left point cloud data in the Y-axis direction are fused to obtain left fused data of the left image data and the left half point cloud data in the X-axis direction; The top image data and the left point cloud data in the Z-axis direction are fused to obtain top fusion data of the top image data and the half-side point cloud data in the Z-axis direction; The right fusion data, the left fusion data and the top fusion data are aggregated to obtain the fusion data of the carriage.

7. The vehicle loading status detection method according to claim 6, characterized in that: Mapping the plane point cloud data onto the carriage point cloud data according to the feature mapping matrix to obtain right fusion data of the right image data and the right half point cloud data in the X-axis direction includes: Performing over-limit detection on the carriage point cloud data and cargo, and marking abnormal point clouds in the carriage point cloud data to obtain marked carriage point cloud data; The plane point cloud data is mapped onto the labeled carriage point cloud data according to a feature mapping matrix to obtain the right fused data.

8. The vehicle loading status detection method according to claim 6, characterized in that: The number of the first feature points is related to the model and cargo of the railway freight car in the image data; The number of the second feature points is related to the model and cargo of the railway freight car in the plane point cloud data.

9. The vehicle loading status detection method according to claim 1, characterized in that: The S1 further comprises: Performing noise removal and contrast enhancement operations on the image data of each carriage to obtain processed image data; The data alignment and fusion are performed based on the carriage point cloud data and image data of each carriage to obtain the fused data of each carriage, including: Data alignment and fusion are performed based on the carriage point cloud data and processed image data of each carriage to obtain the fused data of each carriage.

10. The vehicle loading status detection method according to claim 1, characterized in that: The S4 further comprises: When the vehicle loading state of the railway freight car is abnormal, an alarm signal is output.

Images