A residual coal detection method, device and medium based on three-dimensional and two-dimensional analysis

Abstract

The application discloses a residual coal detection method and device based on three-dimensional and two-dimensional analysis, and a medium, the method comprising: scanning the carriage through a camera group pre-installed on the gantry to obtain an image data group of the carriage, the image data group comprising a three-dimensional image and a two-dimensional image; determining vehicle information, pre-processing the image data group according to the vehicle information to determine point cloud data of a single vehicle; calculating according to the point cloud data to obtain the volume of residual coal in the carriage, and determining the distribution characteristics of the residual coal according to the two-dimensional image to determine the residual coal condition in the carriage according to the volume and distribution characteristics of the residual coal. The application comprehensively uses three-dimensional image and two-dimensional image data, utilizes three-dimensional point cloud to obtain the overall form and spatial information of the carriage, and identifies the distribution of large residual coal and residual coal in recessed parts with the help of two-dimensional image, overcomes the limitation of a single data source, and can more comprehensively and accurately detect the residual coal condition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of measurement, in particular to a residual coal detection method and device based on three-dimensional and two-dimensional analysis and a medium. BACKGROUND

[0002] Due to low winter temperature in northern regions, coal often appears frozen and sticky during long-distance train transportation, forming residual coal frozen blocks, with a thickness of about 400mm, and even the whole train is frozen in severe cases. The frozen blocks have the characteristics of dispersed phase and multi-component particle collection, and are closely connected with the railway freight car body, difficult to remove, and are particularly prominent at the bottom and four walls of the coal transportation car, mostly in the form of pockets. Due to the hard texture of the residual coal frozen blocks and the close connection with the coal transportation car, unloading operation is very difficult, which greatly affects the coal cost and normal production of the majority of coal-using enterprises, and the full play of train transport capacity, causing the loss of enterprise manpower and financial resources. SUMMARY

[0003] In order to solve the above problems, the present application provides a residual coal detection method based on three-dimensional and two-dimensional analysis, which comprises: scanning the car through a camera group pre-installed on a gantry to obtain an image data group of the car, the image data group comprising three-dimensional images and two-dimensional images; determining vehicle information, pre-processing the image data group according to the vehicle information to determine the point cloud data of a single vehicle; calculating according to the point cloud data to obtain the volume of residual coal in the car, and determining the distribution characteristics of the residual coal according to the two-dimensional images, so as to determine the residual coal condition in the car according to the residual coal volume and the distribution characteristics.

[0004] In one example, the method further comprises: scanning the car through a pre-set laser radar sensor to obtain single-frame data of the car; determining the time difference between the train running speed and the single-frame data, determining the train running distance according to the time difference, and determining the three-dimensional contour image of the car according to the train running distance and the single-frame data.

[0005] In one example, the method further comprises: determining a pre-set three-dimensional sensor coordinate system and determining the origin of the three-dimensional sensor coordinate system; determining a pre-set target coordinate system and determining the origin of the target coordinate system; and converting the point cloud data through the three-dimensional sensor coordinate system to convert the point cloud data to the target coordinate system.

[0006] In one example, the image data set is preprocessed according to the vehicle information, specifically including: according to the railway wagon compartment characteristics, using point cloud convex hull processing technology to filter the invalid space three-dimensional data of the single frame slice, and only retaining the valid compartment space three-dimensional data; by continuously collecting data and combining the vehicle speed or vehicle body positioning unit, the position of the current point cloud slice on the vehicle body is calculated; when the train is divided at the train coupler, the calculation is restarted, and the data of each section is combined to form a complete three-dimensional point cloud of the current wagon; the point cloud is sorted at an equal angle resolution in the direction of train advancement to obtain a discrete series of point cloud slices corresponding to the detection target object; the point cloud between two adjacent slice surfaces is projected to obtain point cloud data, and the obtained point cloud image is three-dimensionally reconstructed to obtain a three-dimensional point cloud image of the wagon.

[0007] In one example, the method further includes: for a flat-bottom wagon type, using an integral method to directly calculate the residual coal volume, when calculating the wagon bottom height, using a statistical method combined with a filtering algorithm, removing the wagon side, traversing all the wagon bottom height information, excluding data higher than 15 cm above the wagon bottom, taking the average value of the remaining data, and applying a normal distribution method to calculate the wagon bottom height; determining the mean and standard deviation of the Gaussian function by Gaussian distribution statistical height data, taking the predetermined points as effective values, determining the points far away as outliers, adjusting the threshold to make the wagon bottom height satisfy the normal distribution, and filtering noise; sorting the point cloud slices by angle resolution and dividing them into multiple triangles or trapezoids to calculate the residual coal area; determining the distance between the slice and the next slice, calculating the residual coal volume of the current slice, and accumulating the volume of the entire train to obtain the residual coal volume of the entire train.

[0008] In one example, the method further includes: for a non-flat-bottom wagon type, using Green's formula to calculate the cross-sectional area to calculate the volume of the hollow part of the train, and dividing the closed figure into multiple triangular figures for calculation; determining the distance between the slice and the next slice, calculating the hollow volume of the current slice, and accumulating the hollow volume of the entire train to obtain the hollow volume of the entire train; according to the dictionary query of the total volume of the vehicle type, the residual coal volume is calculated.

[0009] In one example, the method further includes: according to the full load point cloud characteristics of the radar, the point cloud data is processed by horizontal and vertical slicing; for single radar modeling, taking the wagon side as the reference point, the overflow volume and the lower limit volume are calculated by slicing, and then the left and right load deviation and the front and rear load deviation information are calculated; for double radar modeling, the point cloud is processed by horizontal and vertical slicing, the volume of each small unit is calculated, and the middle line is taken as the division node to calculate the volume load deviation information of the full load coal wagon.

[0010] In one example, the method further comprises: acquiring a two-dimensional image of the car and three-dimensional point cloud data, identifying the distribution area of the large residual coal in the two-dimensional image, and analyzing the residual coal information of the missing concave part in the three-dimensional point cloud data; preprocessing the two-dimensional image of the car, including cutting the car wall based on the color difference between the car wall and the car bottom, retaining the car bottom image, and cutting the car bottom into a grid, and according to the gray value of each grid, the residual coal height value is added in proportion, wherein the color depth is proportional to the residual coal height, and the maximum depth of the concave part is adjustable parameter to adapt to the actual concave condition; the three-dimensional point cloud data is cut into a grid with the same density as the two-dimensional image grid according to the horizontal plane, and the height information of the two-dimensional image grid and the corresponding three-dimensional point cloud grid is compared; when the average height of the three-dimensional point cloud grid is lower than the height of the two-dimensional image grid, the height of the three-dimensional point cloud grid is corrected to the height of the two-dimensional image grid to include the residual coal thickness indicated by the gray scale; when the average height of the three-dimensional point cloud grid is higher than the height of the two-dimensional image grid, the average height of the three-dimensional point cloud grid is used; based on the corrected three-dimensional point cloud data, residual coal volume calculation and distribution analysis are carried out, the residual coal in the concave part is included in the total volume calculation, and is reflected on the residual coal distribution map to guide the user to process the residual coal; the two-dimensional image is used to clearly reflect the closing degree of the car side wall door and window, and the defects in the three-dimensional point cloud data are supplemented.

[0011] In one example, the method further comprises: adjusting the acquisition frequency of the line array camera line according to the train speed set by the radar; identifying and removing repeated scanning data of the radar delay by using a pre-set integrated special algorithm; and filtering the coal ash noise points of the radar by using the spatial distribution difference and time sequence difference of the car surface.

[0012] In another aspect, the application also provides a residual coal detection device based on three-dimensional and two-dimensional analysis, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the residual coal detection device based on three-dimensional and two-dimensional analysis to perform the method as described in any one of the above examples.

[0013] In another aspect, the application also provides a non-volatile computer storage medium storing computer executable instructions, which are configured to perform the method as described in any one of the above examples.

[0014] This application comprehensively utilizes both 3D and 2D image data. It leverages 3D point clouds to acquire the overall shape and spatial information of the carriage, while using 2D images to identify the distribution of large pieces of residual coal and residual coal in recessed areas. This overcomes the limitations of a single data source and enables a more comprehensive and accurate detection of residual coal conditions. In the preprocessing stage, targeted processing is applied to different data characteristics. For example, convex hull processing technology is used to filter the 3D point cloud data, preserving effective carriage space data; 2D images are segmented based on color differences to cut the carriage walls, retaining the bottom image and adding residual coal height values ​​to the grid, providing a reliable foundation for subsequent calculations. Different volume calculation methods are used for different vehicle types: the integral method is used for flat-bottomed vehicles, and Green's formula is used for non-flat-bottomed vehicles. Furthermore, off-center load calculation methods are designed for single-radar and dual-radar modeling, adapting to various practical scenarios and improving calculation accuracy. By processing 2D images and 3D point cloud data uniformly within the same system, and correcting the point cloud height by comparing corresponding grid height information, the advantages of both methods are combined, making the residual coal volume calculation closer to reality. This also enhances the reliability of carriage condition detection, providing strong protection for transportation safety. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0016] Figure 1 This is a flowchart illustrating a residual coal detection method based on three-dimensional and two-dimensional analysis in an embodiment of this application.

[0017] Figure 2 This is a schematic diagram of a residual coal detection device based on three-dimensional and two-dimensional analysis in an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0020] like Figure 1 As shown, in order to solve the above problems, this application provides a method for detecting residual coal based on three-dimensional and two-dimensional analysis, the method comprising:

[0021] S101, scan the car through a camera group pre-installed on the gantry to obtain an image data group of the car, the image data group including a three-dimensional image and a two-dimensional image.

[0022] The three-dimensional camera technology can present the volume and morphology of residual coal in the car body in three dimensions and is used as a main detection method. Meanwhile, the two-dimensional camera technology is used to directly show the residual coal state and measure the residual coal area and is used as an auxiliary method. The combination of the two can effectively improve the detection accuracy.

[0023] Although the three-dimensional camera can reflect the three-dimensional morphology of residual coal, it has certain limitations. If the residual coal completely covers the bottom of the car body, the measurement accuracy will be affected. The complex car body structure will also interfere with the calculation accuracy. In addition, rain, snow, heavy fog, and coal moisture content will also affect the measurement accuracy. In these cases, the two-dimensional image needs to be supplemented, which can not only improve the accuracy of residual coal measurement and evaluation but also make up for the shortcomings of three-dimensional measurement.

[0024] The three-dimensional camera equipment is installed on the railway gantry and uses line scanning to collect data. The data is transmitted through Ethernet. During scanning, the reflection data of the inside and bottom of the car body can be covered. The scanning direction is transverse and perpendicular to the train running direction. The scanning frequency can be adjusted within the range of 100 Hz to 600 Hz. When the train speed is 10 km / h, the scanning frequency is set to 100 Hz, which can meet the test accuracy requirements. When the scanning frequency is set to 600 Hz, it can adapt to the imaging needs of 70-80 km / h car speed. The three-dimensional camera restores the morphology of a complete car body by splicing multiple transverse scanning slices. In each slice, the angle resolution of the collected data reaches 0.25°. The three-dimensional sensor can provide the angle information and the distance to the sensor for each data collection point. After data calibration and mathematical processing, the three-dimensional morphology of the car body can be presented. With the help of point cloud algorithm, the volume of the car body can be calculated. This is the basic principle of three-dimensional camera detection of residual coal in car body.

[0025] The two-dimensional camera equipment is also installed on the gantry along the railway and also uses line scanning, which is a common means of shooting moving objects. The scanning angle and direction of the two-dimensional camera are basically synchronized with those of the three-dimensional camera. The scanning frequency of the two-dimensional camera can reach 25 kHz at most and can be adjusted according to the train running speed and the on-site environment. Unlike the three-dimensional camera, the data output of the two-dimensional camera only gives the direction position of each data collection point and does not reflect the distance from the collection point to the camera. According to the shape, light, color, and other factors of the photographed object, the two-dimensional camera data assigns different gray values to each collection point, thereby reflecting the planar image features of the photographed object.

[0026] S102, determine the vehicle information and pre-process the image data group according to the vehicle information to determine the point cloud data of a single vehicle.

[0027] The three-dimensional camera recognition at the bottom of the car, the residual coal surface shape recognition, and the detection accuracy will be affected when dealing with dead angle or bad weather conditions. Increasing the two-dimensional camera image helps to more accurately distinguish the residual coal condition in the car.

[0028] The three-dimensional camera data and the two-dimensional camera data are transmitted in real time to the detection host computer through Ethernet, and the collected data is classified and stored by the detection host computer. The detection host computer combines with the car number identifier, speedometer and other information sources to comprehensively process the collected three-dimensional camera and two-dimensional camera data. At this time, the car number and driving speed through the measuring point are known, and the data of the three-dimensional camera and the two-dimensional camera are processed in sections to obtain the two-dimensional image and three-dimensional data of each section of the car. At this time, the preliminary preparation work of detecting the residual coal in the car is completed.

[0029] S103, calculating according to the point cloud data to obtain the residual coal volume in the car, and determining the distribution characteristics of the residual coal according to the two-dimensional image, so as to determine the residual coal condition in the car according to the residual coal volume and the distribution characteristics

[0030] After obtaining the point cloud data of a single section of vehicle, the point cloud processing algorithm is used to calculate the preliminary volume of residual coal in the section of car. At the same time, by analyzing the two-dimensional image, the area and distribution characteristics of the residual coal can be obtained. Combined with other vehicle information such as vehicle type and speed, the calculation result is corrected, and finally the volume and distribution of the residual coal are determined.

[0031] In one embodiment, the train three-dimensional modeling uses a laser radar sensor, whose scanning direction is vertical direction, mainly used for scanning the cross-sectional profile of the car. The scanning frequency of the sensor is higher than 500Hz, and the single frame data not only contains the point cloud information of the train profile, but also has a time stamp. In actual application, according to the train running speed and the time difference between adjacent two frames of data, the running distance of the train in this period of time can be calculated. By synthesizing a series of continuously acquired cross-sectional profiles, the three-dimensional profile image of the car can be constructed. In order to ensure the accuracy and comprehensiveness of data acquisition, the three-dimensional sensor is installed at a height of 10 meters above the rail surface. At this height, the point cloud distribution direction perpendicular to the driving direction is the most suitable layout. During the train running, the high-frequency three-dimensional image sensor continuously acquires data. For a series of three-dimensional point cloud data collected, the point cloud data is analyzed and calculated by using the Z direction micro distance integral processing method, and finally the volume of the car interior is obtained.

[0032] In one embodiment, a three-dimensional image sensor coordinate system is used to describe the relative position between the detected object and the three-dimensional image sensor, and the coordinates are represented as [XL, YL, ZL]. The origin of this coordinate system is set as the geometric center of the three-dimensional image sensor, where the XL axis direction is horizontal forward, the YL axis direction is horizontal left, and the ZL axis direction is vertical upward, and the entire coordinate system follows the right-hand coordinate system rule. The target coordinate system is used to describe the relative position of the measured object, i.e., the car body and the center position of the steel rail, and the coordinates are represented as [X, Y, Z]. The origin of this coordinate system is located at the center of the steel rail, the X axis direction is perpendicular to the steel rail upward, the Y axis direction is horizontal right, and the Z axis direction is consistent with the direction of train operation. The point cloud data coordinates collected by the three-dimensional image sensor are based on the three-dimensional image sensor coordinate system. However, in order to meet the actual needs, we hope that the final point cloud data coordinates are based on the target coordinate system. Therefore, the coordinates of each point in the point cloud need to be converted. The coordinate conversion mainly includes two steps of rotation transformation and translation transformation. The following first takes a two-dimensional plane as an example to introduce the basic principle of unit vector coordinate conversion, and then extends to three-dimensional space. It is assumed that in the coordinate conversion process, the rotation matrix between the two coordinate systems is R, and the rotation angle is θ. Through rotation and translation operations, the coordinates of the target object point P can be converted from the three-dimensional image sensor coordinate system to the steel rail center coordinate system. The specific conversion formula is: [X, Y, Z] = R[XL, YL, ZL] + T, where [X, Y, Z] is the steel rail center coordinate system, R is the rotation matrix, [XL, YL, ZL] is the three-dimensional image sensor coordinate system, and T is the translation matrix.

[0033] In one embodiment, based on the characteristics of the railway freight car, the freight car floor is higher from the ground, which makes the three-dimensional data interface of the space around the car clear. When processing a single frame of point cloud slices, the point cloud convex hull processing technology is used to filter the three-dimensional data, which can effectively eliminate invalid data and only keep the effective three-dimensional data related to the car space. By continuously collecting point cloud data, combined with the vehicle speed information or the vehicle positioning unit, the specific position of the current slice on the vehicle body can be accurately calculated. When the train is separated at the coupler, the slice calculation process is restarted, and finally the slice data of each segment is synthesized to build a complete three-dimensional point cloud model of the current freight car. The point cloud slice operation is to sort the point cloud data in a specific direction with equal angle resolution, so as to obtain a series of discrete point cloud slices corresponding to the detection target object (i.e. the freight car). In the point cloud data image processing system based on the car coordinate system constructed in this paper, since the forward direction of the train is set as the Z-axis direction, the Z-axis is selected as the cutting direction to divide the car model point cloud into several segments. Point cloud data presents the information characteristics of an object in a discrete form, and its density has certain limits. If only a single thin plane is used to divide the point cloud, it is difficult to accurately outline the shape profile of the point cloud. Therefore, the division plane needs to be given a certain "thickness". The two adjacent slices form a point cloud segment, and the thickness between the two slices is determined by the running distance of the train, and the specific calculation method is: d = vt, t = 1 / f, wherein, v represents the running speed of the train, t is the time interval, f is the radar frequency. Based on the above principle, the point cloud data between the two slice planes is projected, and the point cloud data of this segment can be obtained. Then, the obtained point cloud image is subjected to three-dimensional reconstruction operation, and finally the complete three-dimensional point cloud image of the freight car is obtained.

[0034] In one embodiment, after obtaining the three-dimensional point cloud image of the truck, it is necessary to judge whether there is residual frozen coal in the truck according to the image, at which time the truck model is a very key parameter. After confirming the model, the system will select the corresponding algorithm according to different models. Specifically, first, evaluate whether there is frozen coal in the car through the three-dimensional image data characteristics. If the system determines that there is frozen coal in the car, it will automatically call the frozen coal volume calculation program for different models. The truck unloading detection system is suitable for open cars. For open cars, the system will classify the bottom of the car, and divide it into two categories according to the characteristics of the bottom of the car, namely flat bottom truck and non-flat bottom truck. In addition, the contour of the truck can also be reconstructed by means of point cloud, and it is compared with the length, width and height of the truck model, and the three-dimensional data is used to evaluate the situation without frozen coal in the car. In addition, the gray evaluation is carried out by means of high-definition two-dimensional image combined with AI algorithm, so as to judge whether there is frozen coal. If the gray evaluation result shows that there is frozen coal, the area of the frozen coal part in the two-dimensional image is calculated, and then the minimum resolution of the radar is combined to carry out integral calculation on the volume of the frozen coal. When calculating the cross-sectional area, the cross-sectional area of the frozen coal is calculated for the case of having frozen coal, and the cross-sectional area of the empty area is calculated for the case of empty area.

[0035] In one embodiment, the flat bottom car model mainly includes C62, C64, C70, C80, etc. When the bottom of the car is a flat bottom car, the integral method is used to directly calculate the residual coal volume. When calculating the height of the bottom of the car, considering that the bottom of the car may have uneven conditions, a statistical method is used to determine the height of the bottom of the car. The specific operation is as follows: first, use a filtering algorithm to remove the interference of the car side. Then, traverse all the bottom height information, and in this process, exclude data that is obviously higher than 15 cm above the bottom of the car. Then, take the average value of the remaining effective bottom height data. Finally, the normal distribution method is used to further calculate the height of the bottom of the car. The specific implementation of the normal distribution method is as follows: the height of the bottom of the car is statistically distributed according to the Gaussian distribution. Under normal circumstances, the shape characteristics of the Gaussian function are determined by its mean and standard deviation. According to the characteristics of Gaussian distribution, the points within a certain distance range around the symmetric center are considered as valid values, while the points far away from this range are considered as outliers. The height of the bottom of the car Kμ satisfies (Kμ-σ, Kμ+σ) approximately normal distribution, and filters noise. Cross-sectional area calculation and residual coal volume calculation: sort the point cloud slices by angle resolution, and divide them into several triangles or trapezoids, and set the residual coal area of the slice as S: S=S1+S2+S3+…+Sn. The distance between the slice and the next slice is set as d, d=vt, t=1 / f, and the residual coal volume V1 of the current slice is d*v / f. The total volume V=V1+V2+V3+…+Vn.

[0036] In one embodiment, the non-flat bottom car type mainly contains old K car, C80, etc. When the car bottom is a non-flat bottom car, the above method is not suitable. For this type of car, we use Green's formula to calculate the cross-sectional area, and then calculate the volume V0 of the hollow space in the train. According to the dictionary query, the total volume VS of the car type is calculated. Thus, the residual coal volume V = VS-V0 is calculated. Green's formula is mainly used to calculate the cross-sectional area of a complex closed figure. The calculation method is simple and easy to understand. The area formula is: S = 1 / 2i=1n[xiyi+1-yi-yi(xi+1-xi)]. The closed figure is divided into several triangular figures. The current point cloud slice cross-sectional area S = S1 + S2+S3+……+Sn. The distance between the current slice and the next slice is set to d, d = vt, t = 1 / f, and the residual coal volume V1 of the current slice is V1 = d*v / f. The volume of the whole train V = V1+V2+V3+……+Vn, and the residual coal volume V = VS-V0, where VS is the volume of the train.

[0037] In one embodiment, based on the characteristics of single radar full load point cloud, the point cloud is processed by transverse and longitudinal slicing, and then the volume of each micro unit is calculated. The middle line is taken as the dividing node to calculate the unbalanced load information of the full load coal car. It should be noted that depending on different modeling methods, the calculation method also differs: single radar modeling calculates unbalanced load based on single radar full load point cloud characteristics, taking the car side as the reference point, and calculating the overflow volume, i.e. the volume of the part that exceeds the car side, and the recess volume by slicing. On this basis, the unbalanced load in the left and right directions and the unbalanced load in the front and rear directions are further calculated. Double radar modeling can achieve 270-degree modeling and can calculate the height of the train. In the case of double radar modeling, the point cloud can be directly processed by transverse and longitudinal slicing to calculate the volume of each micro unit. Similarly, the middle line is taken as the dividing node to calculate the volume unbalanced load information of the full load coal car.

[0038] In one embodiment, when viewing the live shot carriage two-dimensional image, it is found that there are obvious large pieces of residual coal distribution in the two-dimensional image, but these residual coal is not reflected in the three-dimensional point cloud data analysis. Through the comparative analysis of two-dimensional image and three-dimensional point cloud, it is concluded that due to the concave at the bottom of the carriage, the top of the residual coal in some concave parts is lower than the carriage bottom plane, which leads to the failure to find and calculate the residual coal in these concave parts when processing three-dimensional point cloud. Therefore, two-dimensional picture is used to assist the calculation of residual coal volume, the specific method is as follows: first, cut the carriage bottom. The left and right positions of the carriage are basically fixed, and the cutting can be controlled by parameters; the front and rear positions of the carriage are different due to each image, according to the color difference between the carriage wall and the carriage bottom, cut off the carriage wall, and only keep the two-dimensional image of the carriage bottom. Then divide the carriage bottom into a grid, according to the gray value of each grid two-dimensional image, add a residual coal height value, the darker the color, the higher the residual coal height. According to the actual measurement of the concave parts of the carriage on site, the maximum depth of the concave parts is about 10 cm. According to the color depth of the two-dimensional image, the residual coal height is converted in proportion, that is, the deepest color has the maximum height, and the lightest color has zero height, that is, the maximum depth of the concave parts is reserved as an adjustable parameter that can be corrected by the user, which can be adjusted according to the actual condition of the carriage concave. After obtaining the grid height of the two-dimensional image, because the two-dimensional image and the three-dimensional point cloud are placed in the same system for unified comprehensive processing, the three-dimensional point cloud is also divided into grids with the same density based on the vehicle number and vehicle type size obtained in the vehicle number recognition system. By comparing the corresponding grids of two-dimensional and three-dimensional, when the average height of three-dimensional point cloud of each grid is lower than the grid height of two-dimensional image, the point cloud height is corrected to the grid height of two-dimensional image, and the corrected point cloud height has included the residual coal thickness indicated by the different gray values in two-dimensional image; when the average height of three-dimensional point cloud grid is higher than that of two-dimensional image, the average height of three-dimensional point cloud grid is used. The corrected three-dimensional point cloud is used for residual coal volume calculation, so that the residual coal in the concave parts can also be included in the total residual coal volume, making the residual coal quantity closer to the actual situation. It can also be reflected on the residual coal distribution map, providing guidance for users to handle residual coal. Similarly, because the two-dimensional image and the three-dimensional point cloud are placed in the same system for unified comprehensive processing, the three-dimensional point cloud reflects the roughness of the tightness of the carriage side wall door and window, which can be supplemented by the clearer image of the two-dimensional image, enhancing the reliability of transportation safety.

[0039] In an embodiment, in the scenario of railway freight car using a dumper to unload coal, the vehicle needs to go through the process of stopping, unloading, and restarting. Since the standard car length is about 13 meters, and is limited by the site environment, the area array camera cannot obtain high-resolution panoramic images at one time. Therefore, the system uses a line array camera combined with a line scanning laser radar for detection. However, the stopping of the vehicle will cause a discontinuity in the data acquisition, and a repeated scanning area will be generated when the vehicle restarts. To solve this problem, a millimeter wave radar is introduced to measure the real-time speed of the train. According to the speed change feedback by the millimeter wave radar, the line array camera acquisition frequency is dynamically adjusted: when the vehicle is moving, the acquisition is carried out, and when the vehicle is stopped, the acquisition is suspended. At the same time, the scanning of the line scanning laser radar is also suspended when the vehicle is stopped, and is restored after the vehicle restarts, and continues to build the three-dimensional model of the car body. Since there is inherent delay in the millimeter wave radar speed measurement, it may cause the system to lag in response to the speed change. Therefore, the system integrates a special algorithm for identifying and removing repeated scanning data caused by stopping / startup or speed measurement delay. The algorithm ensures that the image sequence acquired by the line array camera and the three-dimensional model built by the laser radar do not stretch or compress during the splicing or reconstruction process, thereby ensuring the geometric accuracy and accuracy of the final acquisition data.

[0040] When the railway freight car uses a dumper to unload coal, the intense unloading process will stir up a large amount of fine coal particles. These coal dusts diffuse in the air and drift to the area above the car. The coal dust particles in the air will interfere with the laser radar deployed above the car, forming noise points in the point cloud data that do not belong to the surface of the car itself. The car surface point cloud usually constitutes a continuous and smooth geometric surface, while the noise points generated by the coal dust particles exhibit discrete, isolated and chaotic spatial distribution characteristics. The spatial distribution difference is used to filter the dispersed coal dust in the air. In addition, the continuous multi-frame point cloud data is registered and aligned, and the position stability of each point in multiple frames is analyzed, and the points with rapid position change are determined as dynamic noise points, i.e. the time series difference is further used to filter the dispersed coal dust in the air. When identifying the shape of the bottom of some car bodies and residual coal surfaces, if the car bottom is not flat and the residual coal is less, the point cloud is difficult to distinguish, which will affect the detection accuracy. Increasing the two-dimensional camera picture helps to improve the residual coal condition in the car.

[0041] As shown in Figure 2 The embodiment of the present application also provides a residual coal detection device based on three-dimensional and two-dimensional analysis, which comprises:

[0042] at least one processor; and,

[0043] a memory in communication connection with the at least one processor; wherein,

[0044] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable a residual coal detection device based on three-dimensional and two-dimensional analysis to perform:

[0045] scanning the vehicle compartment through a camera group pre-installed on the gantry to obtain an image data set of the vehicle compartment, the image data set including a three-dimensional image and a two-dimensional image;

[0046] determining vehicle information, and pre-processing the image data set according to the vehicle information to determine point cloud data of a single vehicle;

[0047] calculating according to the point cloud data to obtain a residual coal volume in the vehicle compartment, determining a distribution feature of residual coal according to the two-dimensional image, and determining a residual coal condition in the vehicle compartment according to the residual coal volume and the distribution feature.

[0048] The embodiments of the present application also provide a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are configured to:

[0049] scanning the vehicle compartment through a camera group pre-installed on the gantry to obtain an image data set of the vehicle compartment, the image data set including a three-dimensional image and a two-dimensional image;

[0050] determining vehicle information, and pre-processing the image data set according to the vehicle information to determine point cloud data of a single vehicle;

[0051] calculating according to the point cloud data to obtain a residual coal volume in the vehicle compartment, determining a distribution feature of residual coal according to the two-dimensional image, and determining a residual coal condition in the vehicle compartment according to the residual coal volume and the distribution feature.

[0052] In the 1990s, it was possible to distinguish whether an improvement in a technology was a hardware improvement (e.g., an improvement in the circuit structure of a diode, transistor, switch, etc.) or a software improvement (an improvement in a method flow). However, as technology has advanced, many improvements in method flows today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into a hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A designer programs a digital system "integrated" on a PLD by himself / herself, without having to ask a chip manufacturer to design and manufacture a special integrated circuit chip. Moreover, instead of manually manufacturing an integrated circuit chip, this programming is now mostly implemented using "logic compiler" software, which is similar to a software compiler used when developing a program, and the original code before compilation must also be written in a specific programming language, which is called a hardware description language (HDL), and there are many types of HDL, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that it is only necessary to logically program a method flow using the above-mentioned hardware description languages and program it into an integrated circuit to easily obtain a hardware circuit that implements the logical method flow.

[0053] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the microprocessor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to implementing the controller in pure computer readable program code, it is also possible to implement the controller in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. to achieve the same function by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can even be considered as both a software module implementing a method and a structure within a hardware component.

[0054] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0055] For the convenience of description, the above apparatuses are described in various units respectively according to their functions. Of course, the functions of each unit can be implemented in the same or multiple software and / or hardware when implementing the present specification.

[0056] Each of the embodiments in the present application is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment mainly describes the differences from other embodiments. Especially, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0057] The device and medium provided by the embodiments of the present application are one-to-one corresponding, and therefore the device and medium also have similar beneficial technical effects to the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device and medium will not be described here again.

[0058] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. In addition, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0059] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device implemented in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in one or more flows and / or blocks.

[0060] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in one or more flows and / or blocks.

[0061] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in one or more flows and / or blocks.

[0062] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memories.

[0063] Memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, etc. in the form of a computer-readable medium, read only memory (ROM), or flash memory, etc. Memory is an example of computer-readable media.

[0064] Computer-readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media, such as modulated data signals and carrier waves.

[0065] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without further limitation, an element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0066] The above only is an embodiment of the present application, and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A residual coal detection method based on three-dimensional and two-dimensional analysis, characterized by, The method comprises the following steps: Scanning the car through a camera group pre-installed on the gantry to obtain an image data set of the car, the image data set comprising a three-dimensional image and a two-dimensional image; Determining vehicle information, and pre-processing the image data set according to the vehicle information to determine point cloud data of a single vehicle; According to the point cloud data, calculating the residual coal volume in the car and determining the distribution characteristics of the residual coal according to the two-dimensional image, so as to determine the residual coal condition in the car according to the residual coal volume and the distribution characteristics; Scanning the car through a pre-set laser radar sensor to obtain single-frame data of the car; Determining the time difference between the train running speed and the single-frame data, determining the train running distance according to the time difference, and determining the three-dimensional contour image of the car according to the train running distance and the single-frame data; Determining a pre-set three-dimensional sensor coordinate system and the origin of the three-dimensional sensor coordinate system; Determining a pre-set target coordinate system and the origin of the target coordinate system; Converting the point cloud data through the three-dimensional sensor coordinate system to convert the point cloud data into the target coordinate system; Obtaining the two-dimensional image and the three-dimensional point cloud data of the car, identifying the distribution area of the large residual coal in the two-dimensional image, and analyzing the residual coal information of the recessed part in the three-dimensional point cloud data; Pre-processing the two-dimensional image of the car, which includes cutting the car wall based on the color difference between the car wall and the car bottom, retaining the image of the car bottom, and dividing the car bottom into a grid, according to the gray value of each grid, adding residual coal height value in proportion, wherein the color depth is proportional to the residual coal height, and the maximum depth of the recessed part is adjustable to adapt to the actual recessed condition; Dividing the three-dimensional point cloud data into a grid with the same density as the two-dimensional image grid according to the horizontal plane, and comparing the height information of the two-dimensional image grid and the corresponding three-dimensional point cloud grid; When the average height of the three-dimensional point cloud grid is lower than the height of the two-dimensional image grid, the height of the three-dimensional point cloud grid is corrected to the height of the two-dimensional image grid to include the residual coal thickness indicated by the gray scale in the two-dimensional image; when the average height of the three-dimensional point cloud grid is higher than the height of the two-dimensional image grid, the average height of the three-dimensional point cloud grid is adopted; Based on the corrected three-dimensional point cloud data, residual coal volume calculation and distribution analysis are carried out, the residual coal in the recessed part is included in the total volume calculation, and is reflected on the residual coal distribution map to guide the user to handle the residual coal.

2. The method of claim 1, wherein, The pre-processing of the image data set according to the vehicle information specifically comprises: According to the characteristics of the car, the point cloud convex hull processing technology is used to filter the invalid space three-dimensional data of the single-frame slice to retain the car space three-dimensional data; By continuously collecting data and combining the vehicle speed and the vehicle body positioning unit, the position of the current point cloud slice on the vehicle body is calculated; When the train is divided at the train coupler, the calculation is restarted, and the data of each section is combined to form the complete three-dimensional point cloud of the car; The point cloud is sorted at equal angle resolution in the forward direction of the car to obtain a discrete series of point cloud slices corresponding to the detection target object; Project the point cloud between two adjacent slice surfaces to obtain point cloud data, and perform three-dimensional reconstruction on the obtained point cloud image to obtain a three-dimensional point cloud image of the car body.

3. The method of claim 1, wherein, The method further comprises: For a flat-bottomed vehicle model, the integral method is used to directly calculate the residual coal volume, when calculating the height of the vehicle bottom, the statistical method is used in combination with the filtering algorithm, all the height information of the vehicle bottom is traversed after removing the vehicle side, the data higher than 15 cm above the vehicle bottom is excluded, the average value of the remaining data is taken, and the normal distribution method is used to calculate the height of the vehicle bottom; The height data is statistically distributed by the Gaussian distribution, the mean and standard deviation of the Gaussian function are determined, the points that are far away are determined as outliers, the threshold is adjusted to make the height of the vehicle bottom satisfy the normal distribution, and the noise is filtered; The point cloud slices are sorted according to the angle resolution, and are divided into multiple triangles or trapezoids to calculate the residual coal area; The distance between the slice and the next slice is determined, the residual coal volume of the current slice is calculated, and the residual coal volume of the whole train is accumulated to obtain the residual coal volume of the whole train.

4. The method of claim 1, wherein, The method further comprises: For a non-flat-bottomed vehicle model, the Green formula is used to calculate the cross-sectional area to calculate the volume of the hollow part of the train, and the closed figure is divided into multiple triangular figures for calculation; The distance between the slice and the next slice is determined, the residual coal volume of the current slice is calculated, and the residual coal volume of the whole train is accumulated to obtain the residual coal volume of the whole train. According to the total volume corresponding to the train model queried from the dictionary, the residual coal volume is calculated.

5. The method of claim 1, wherein, The method further comprises: According to the full-load point cloud characteristics of the radar, the point cloud data is processed by horizontal and vertical slicing; For single-radar modeling, the slice method is used to calculate the overflow volume and the lower limit volume respectively based on the vehicle side as the reference point, and then the left and right load deviation and the front and rear load deviation information are calculated; For double-radar modeling, the point cloud is sliced horizontally and vertically, the volume of each unit is calculated, and the middle line is taken as the division node to calculate the volume load deviation information of the full-load car body.

6. The method of claim 1, wherein, The method further comprises: The speed of the train is measured by the pre-set radar, and the acquisition frequency of the line array camera is adjusted according to the speed of the train; The pre-set integrated special algorithm is used to identify and remove the repeated scanning data of the radar delay; The coal dust noise points of the radar are filtered by using the spatial distribution difference and time sequence difference of the car skin surface.

7. A residual coal detection device based on three-dimensional and two-dimensional analysis, characterized by, Comprise: At least one processor; And The memory is in communication connection with the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the three-dimensional and two-dimensional analysis-based residual coal detection device to perform the method as claimed in any one of claims 1-6.

8. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer executable instructions are configured to perform the method as claimed in any one of claims 1-6.

Images