Track foreign matter monitoring method, system and equipment based on heterogenous data and medium

Abstract

The invention discloses a rail foreign matter monitoring method, system and device based on heterogenous data and a medium, and the method comprises the following steps: obtaining heterogenous data including point cloud data and image data; performing rail surface searching according to the image data, and determining a target monitoring area; screening the point cloud data by using the target monitoring area to obtain a monitoring point cloud set; searching for abnormal points according to the monitoring point cloud set to obtain an abnormal point set; and projecting the abnormal point set to the target monitoring area, and monitoring foreign matters in the target monitoring area. The method can automatically identify the target monitoring area, fully utilizes the advantages of each monitoring technology, and can accurately monitor the foreign matters on the track.

Description

Technical Field

[0001] The present invention relates to the technical field of foreign object monitoring, and in particular to a track foreign object monitoring method, system, equipment and medium based on heterogeneous data. Background Art

[0002] Urban rail lines, especially ground and elevated ones, have the risk of foreign objects invading the tracks and track areas, which seriously endangers the safety of urban rail train operation. In the field of line foreign object monitoring and safety monitoring, LiDAR scanning and image and video recognition are two important mainstream monitoring methods, but the existing technology still has the following problems: On the one hand, due to the field of view problem, the collected data contains a large amount of background data (areas in the scene that do not affect driving), and the monitoring area (track, driving area) needs to be manually defined to avoid detection interference caused by changes in background data.

[0003] On the other hand, a single detection technology has limitations. Radar detection generally has a large number of false alarms and the detection results are not intuitive. Existing image detection methods generally use big data model recognition. Since the incidence of foreign object intrusion in railways is not high and most of them are simulated samples, the recognition rate for real events is often not high. At the same time, the monitoring ability for foreign objects that have not appeared will be significantly reduced. Summary of the invention

[0004] The purpose of the present invention is to provide a track foreign object monitoring method, system, device and medium based on heterogeneous data, so as to solve the problems that the data collected in the prior art has background interference and the accuracy of foreign object monitoring using a single detection technology is not high.

[0005] To solve the above technical problems, a technical solution adopted by the present invention is to provide a track foreign object monitoring method based on heterogeneous data, the method comprising the following steps: acquiring heterogeneous data, the heterogeneous data comprising point cloud data and image data; searching the track surface according to the image data to determine the target monitoring area; using the target monitoring area to filter the point cloud data to obtain a monitoring point cloud set; searching for abnormal points according to the monitoring point cloud set to obtain an abnormal point set; projecting the abnormal point set to the target monitoring area to monitor foreign objects in the target monitoring area.

[0006] In some embodiments, the image data includes a real-time original image, and the track surface search of the image data to determine the target monitoring area includes the steps of: identifying track features in the real-time original image to obtain feature recognition results; determining the target monitoring area in the real-time original image based on the feature recognition results; wherein the feature recognition results include the main track and the number of main track recognitions, when the number of main track recognitions is less than 2, the feature recognition result is inaccurate; when the number of main track recognitions is greater than or equal to 2, the convex hull of the main track is taken as the target monitoring area.

[0007] In some embodiments, the step of using the target monitoring area to filter the point cloud data and obtaining a monitoring point cloud set also includes the following steps: obtaining a mapping matrix from a point cloud coordinate system to an image coordinate system; wherein the point cloud coordinate system is the coordinate system where the point cloud data is located, and the image coordinate system is the coordinate system where the image data is located.

[0008] In some embodiments, the point cloud data is screened using the target monitoring area to obtain a monitoring point cloud set, including the steps of: using the projection matrix to project the point cloud data onto the image data to obtain a projection result; obtaining the monitoring point cloud set based on the projection result; the monitoring point cloud set includes all feature points in the point cloud data projected onto the target monitoring area.

[0009] In some embodiments, the monitoring point cloud set includes an initial point cloud set collected when the laser radar is started, and a real-time point cloud set collected every second thereafter; the abnormal point search is performed based on the monitoring point cloud set to obtain an abnormal point set, including the steps of: selecting the first real-time feature point in the real-time point cloud set, and calculating the vector moduli of the first real-time feature point and the initial feature points in the initial point cloud set respectively; judging whether the first real-time feature point is an abnormal point according to the size of each vector modulus: if the vector moduli are all greater than a first threshold, the first real-time feature point is an abnormal point; if there is a vector modulus less than or equal to the first threshold among the vector moduli, the first real-time feature point is a non-abnormal point; selecting other real-time feature points except the first real-time feature point in the real-time point cloud set in turn, repeating the above steps, and obtaining an abnormal point set composed of all abnormal points in the real-time point cloud set.

[0010] In some embodiments, the abnormal point set is projected onto the target monitoring area to monitor foreign objects in the target monitoring area, including the steps of: projecting the abnormal point set onto the target monitoring area, confirming image abnormalities, and obtaining a pixel abnormal point cloud set; wherein, the real-time pixels at the corresponding position of any abnormal point in the abnormal point set projected onto the target monitoring area are color-compared with the historical pixels at the corresponding positions in multiple historical images obtained per second during video playback, and the color differences between the real-time pixels and the historical pixels are calculated; based on the size of the color differences, it is determined whether the color differences all exceed a second threshold; if the color differences all exceed the second threshold, it is confirmed that the image data is abnormal.

[0011] In some embodiments, after confirming that the image data is abnormal, the abnormal point set is projected to the target monitoring area to monitor foreign objects in the target monitoring area, and the steps of: performing a first foreign object recognition on the image data, the first foreign object recognition refers to identifying whether there is a first foreign object of a known foreign object type in the image data; if the first foreign object is recognized, and the position of the first foreign object intersects with the target monitoring area, an alarm is issued; if the first foreign object is not recognized, or the position of the first foreign object does not intersect with the target monitoring area, the target monitoring area is subjected to a second foreign object recognition using the pixel abnormal point cloud set, the second foreign object recognition refers to identifying whether there is a second foreign object of an unknown foreign object type in the target monitoring area; wherein the pixel abnormal point cloud set is subjected to point cloud clustering to obtain a cluster set, and the cluster set can be expressed as: ; in, Indicates l Clustering; cl Indicates the l The number of cluster points in the cluster, if cl If it is less than 20, then the l The clustering is noise clustering; if cl is greater than or equal to 20, then take the l The first l reflectivity set, and the l The projection of all cluster points in the cluster on the corresponding position of the target monitoring area l Pixel color set; then calculate the first l The reflectivity set is the same as the l If the target correlation is greater than or equal to a third threshold, then the first lThe projection of all cluster points in the cluster on the corresponding position of the target monitoring area indicates that the second foreign object exists, and an alarm is issued; if the target correlation is less than a third threshold, it is determined that the second foreign object l If the second foreign object does not exist at the corresponding position of the target monitoring area when all cluster points in the cluster are projected, no alarm is issued.

[0012] The present invention also provides a track foreign object monitoring system based on heterogeneous data, the system includes an edge computing terminal, the edge computing terminal includes a data acquisition unit, a monitoring area determination unit, a point cloud data screening unit, an abnormal point search unit and a foreign object monitoring unit; the data acquisition unit is used to acquire heterogeneous data, the heterogeneous data includes point cloud data and image data; the monitoring area determination unit is used to search the track surface according to the image data and determine the target monitoring area; the point cloud data screening unit is used to filter the point cloud data using the target monitoring area to obtain a monitoring point cloud set; the abnormal point search unit is used to search for abnormal points according to the monitoring point cloud set to obtain an abnormal point set; the foreign object monitoring unit is used to project the abnormal point set to the target monitoring area to monitor foreign objects in the target monitoring area.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0014] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0015] The beneficial effects of the present invention are as follows: the present invention discloses a method, system, device and medium for monitoring foreign objects on a track based on heterogeneous data, the method comprising the following steps: obtaining heterogeneous data, the heterogeneous data comprising point cloud data and image data; searching the track surface according to the image data to determine the target monitoring area; screening the point cloud data using the target monitoring area to obtain a monitoring point cloud set; searching for abnormal points according to the monitoring point cloud set to obtain an abnormal point set; projecting the abnormal point set to the target monitoring area to monitor foreign objects in the target monitoring area. The method can automatically identify the target monitoring area, while making full use of the advantages of various monitoring technologies, and can accurately monitor foreign objects on the track. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of an embodiment of a method for monitoring foreign objects on a track based on heterogeneous data of the present invention; Figure 2This is a specific flow chart of step S2 in an embodiment of a method for monitoring foreign objects on a track based on heterogeneous data of the present invention; Figure 3 It is a specific flow chart of step S3 in an embodiment of a method for monitoring foreign objects on a track based on heterogeneous data of the present invention; Figure 4 It is a specific flow chart of step S4 in an embodiment of a method for monitoring foreign objects on a track based on heterogeneous data of the present invention; Figure 5 It is a specific flow chart of step S5 in an embodiment of a method for monitoring foreign objects on a track based on heterogeneous data of the present invention; Figure 6 It is a real-time original image used in an embodiment of a track foreign body monitoring method based on heterogeneous data of the present invention; Figure 7 2 is a schematic diagram of the result of step S21 in an embodiment of a method for monitoring foreign matter on a track based on heterogeneous data of the present invention; Figure 8 2 is a schematic diagram of the result of step S22 in an embodiment of a method for monitoring foreign matter on a track based on heterogeneous data of the present invention; Fig. 9 It is a schematic diagram of the result of step S4 in an embodiment of a method for monitoring foreign matter on a track based on heterogeneous data of the present invention; Fig.10 It is a block diagram of an embodiment of a track foreign body monitoring system based on heterogeneous data of the present invention; Fig.11 It is a block diagram of the specific composition of an edge computing terminal in an embodiment of a track foreign object monitoring system based on heterogeneous data of the present invention; Fig.12 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention; Fig.13 It is a schematic block diagram of an embodiment of a computer-readable storage medium of the present invention. DETAILED DESCRIPTION

[0017] In order to facilitate the understanding of the present invention, the present invention is described in more detail below in conjunction with the accompanying drawings and specific embodiments. Preferred embodiments of the present invention are provided in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0018] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in this specification includes any and all combinations of one or more related listed items.

[0019] like Figure 1 As shown, a track foreign body monitoring method based on heterogeneous data of the present invention is shown, comprising the following steps: Step S1: Acquire heterogeneous data, which includes point cloud data and image data.

[0020] It should be noted that the laser radar and the camera are aligned with the track through the bracket to ensure that the relative positions of the two remain unchanged. After the laser radar and the camera are started at the same time, the corresponding point cloud data and image data can be obtained respectively. Among them, the image data includes the high-definition video and / or real-time original image captured by the camera, and each frame of the high-definition video corresponds to a real-time original image.

[0021] Step S2: Search the track surface based on the image data to determine the target monitoring area.

[0022] It should be noted that track surface search refers to identifying track features in the real-time original image and obtaining feature recognition results. Track features include main track, number of main tracks, arrangement direction and shape, etc.; feature recognition results include main track and number of main track recognitions, etc. Then, the target monitoring area in the real-time original image can be determined based on the feature recognition results.

[0023] Specifically, if Figure 2 As shown, step S2 includes the following sub-steps: Step S21: Identify the track features in the real-time original image and obtain feature recognition results.

[0024] In this embodiment, the first neural network model is used to identify track features in the real-time original image to obtain feature recognition results.

[0025] Furthermore, the first neural network model adopts the BiSeNetV2 convolutional neural network semantic model. The BiSeNetV2 convolutional neural network semantic model has the advantages of high efficiency and high precision, and is particularly suitable for scenarios that require real-time processing in this application.

[0026] Step S22: Determine the target monitoring area in the real-time original image according to the feature recognition result.

[0027] Among them, when the number of main track recognition is less than 2, the feature recognition result is considered inaccurate; when the number of main track recognition is greater than or equal to 2, the convex hull of the main track in the feature recognition result is taken as the target monitoring area.

[0028] This application is generally used in dual-track scenarios, such as Figure 6 As shown, a real-time original image used in this embodiment is shown. It can be seen that the real-time original image is a double-track scene, including 4 main tracks.

[0029] Combination Figure 7 and Figure 8 , using the BiSeNetV 2 convolutional neural network semantic model, to identify Figure 6 The number of main tracks in the real-time original image shown ( Figure 7 The number of red areas in the middle), the number of main track recognition is obtained. When the number of main track recognition is less than 4, the corresponding feature recognition result is considered inaccurate; when the number of main track recognition is equal to 4, the convex hull of the main track is taken as the target monitoring area ( Figure 8 ).

[0030] In some other embodiments, other first neural network models may also be used to identify track features and achieve the desired effect.

[0031] It should be noted that the coordinate system where the point cloud data is located is the point cloud coordinate system, which is a three-dimensional coordinate system established with the laser radar's optical transceiver as the coordinate origin and the laser radar's optical axis as the z1 axis, and is a right-handed coordinate system. The coordinate system where the image data is located is the image coordinate system, which is a three-dimensional coordinate system established with the camera's focus as the coordinate origin and the depth direction corresponding to the camera's optical axis as the z2 axis. Since the relative positions of the laser radar and the camera are fixed, the mapping relationship between the point cloud coordinate system and the image coordinate system can be obtained, so that the point cloud data can be projected on the corresponding image data.

[0032] Therefore, before step S3, it is necessary to obtain the mapping matrix from the point cloud coordinate system to the image coordinate system M , so that the point cloud coordinates of any spatial point Mapped Matrix M After conversion, the corresponding image coordinates can be obtained ,Right now: ; Among them, the mapping matrix M It is a matrix with 2 rows and 3 columns, containing a total of 6 element values. Therefore, take the point cloud coordinates of at least 6 feature points in the point cloud data and the image coordinates in the corresponding image and substitute them into the above equation to solve the mapping matrix M .

[0033] In engineering practice, at least 12 feature points with a distance of more than 1 meter from each other are selected to ensure that the mapping matrix M accuracy.

[0034] Step S3: Filter the point cloud data using the target monitoring area to obtain a monitoring point cloud set.

[0035] It should be noted that the monitoring point cloud set includes all feature points in the point cloud data that are projected onto the target monitoring area.

[0036] Therefore, combined with Figure 3 , step S3 includes the following sub-steps: Step S31: Project the point cloud data onto the image data using a projection matrix to obtain a projection result.

[0037] Combination Figure 6 and Figure 8 , using the projection matrix M , projecting the collected point cloud data onto the real-time original image to obtain corresponding projection results. The projection results include projections of multiple feature points in the point cloud data onto the target monitoring area, and / or projections of multiple feature points in the point cloud data outside the target monitoring area.

[0038] Step S32: Obtain a monitoring point cloud set according to the projection result.

[0039] Combination Figure 8 , further, if the projection point after projection p In the target monitoring area, that is, , then keep the projection point p Otherwise, discard the projection point p , the feature points corresponding to the final retained projection points form a monitoring point cloud set P ,but: , C Indicates the target monitoring area.

[0040] Step S4: searching for abnormal points based on the monitoring point cloud to obtain an abnormal point set.

[0041] It should be noted that the monitoring point cloud set includes the initial point cloud set collected when the laser radar is started, and the real-time point cloud set collected every second thereafter; and the initial point cloud set includes multiple initial feature points, and the real-time point cloud set includes multiple real-time feature points. Among them, when the laser radar starts to start, there are no foreign objects in the track-shaped area, while foreign objects may appear in the track-shaped area during the subsequent point cloud data collection process. Therefore, this application uses the initial point cloud set as the comparison benchmark, and by comparing the real-time point cloud set collected later with the initial point cloud set, all real-time feature points in the real-time point cloud set are judged as abnormal, and then it can be determined which part of the track-shaped area may have foreign objects.

[0042] Specifically, if Figure 4 As shown, step S4 includes the following sub-steps: Step S41: Select the first real-time feature point in the real-time point cloud set, and calculate the vector modulus lengths of the first real-time feature point and each initial feature point in the initial point cloud set respectively.

[0043] Step S42: judging whether the first real-time feature point is an abnormal point according to the magnitude of the modulus of each vector.

[0044] Among them, if the vector moduli are all greater than the first threshold, the first real-time feature point is an abnormal point; if there is a vector modulus less than or equal to the first threshold among the vector moduli, the first real-time feature point is a non-abnormal point.

[0045] Step S43: sequentially selecting other real-time feature points except the first real-time feature point from the real-time point cloud set, repeating the above steps, and obtaining an abnormal point set consisting of all abnormal points in the real-time point cloud set.

[0046] In this embodiment, the initial point cloud set collected when the laser radar is started is recorded as , and then the real-time point cloud set collected every second is , real-time point cloud collection during traversal Each real-time feature point in , calculate real-time feature points To the initial point cloud set An initial feature point in The vector modulus of is: ; The first threshold is 0.2 (i.e., 20 cm, which is an engineering experience value). , the corresponding vector modulus If both are greater than 0.2, it is considered that the real-time feature point is an abnormal point, otherwise it is a real-time feature point is a non-abnormal point.

[0047] like Fig. 9 As shown in the figure, after completing the abnormal judgment of all real-time feature points in the real-time point cloud set, all the abnormal points obtained (blue points in the figure) are combined into an abnormal point set .

[0048] In some other embodiments, the first threshold may also adopt other values ​​to meet different accuracy requirements.

[0049] Step S5: Project the abnormal point set to the target monitoring area to monitor foreign objects in the target monitoring area.

[0050] Specifically, if Figure 5As shown, step S5 includes the following sub-steps: Step S51: Project the abnormal point set to the target monitoring area, confirm the image abnormality, and obtain a pixel abnormal point cloud set.

[0051] It should be noted that when playing back a video, each frame of the corresponding playback is a historical image, and 25 historical images can be obtained per second.

[0052] Among them, first the abnormal point set The real-time pixels at the corresponding position of any abnormal point projected on the real-time original image are compared with the historical pixels at the corresponding positions in multiple historical images obtained every second during video playback, and the corresponding color differences (the changes in the pixel values ​​at the corresponding position of the abnormal point projected on the real-time original image and the corresponding positions on the multiple historical images) are calculated; then, according to the size of each color difference, it is determined whether each color difference exceeds the second threshold; if each color difference exceeds the second threshold, the pixel change at the corresponding position of the abnormal point projected on the real-time original image is abnormal, that is, it is confirmed that the real-time original image is abnormal.

[0053] like Fig. 9 As shown, the outlier point set Project it to the target monitoring area and record the k The first position of the abnormal point corresponding to the real-time original image k Real-time pixels , and k Real-time pixels The current k Live Color ,in, , Represents anomaly point set The number of abnormal points in .

[0054] Then, call up the video playback and play it forward a seconds (corresponding to a total of 25 a frame), query a certain moment t Each abnormal point is n The corresponding position on the historical image n Historical pixels , and n Historical pixels No. n Historical Color , find the n Historical Color and k Live Color The color difference between : ; in, .

[0055] In this embodiment, a The preferred value is 5 (more than that may be due to color differences caused by changes in light and shadow). The video playback has a total of 125 frames, which corresponds to 125 historical images, which can avoid the impact of light and shadow changes.

[0056] Secondly, use color differences to judge color differences Whether there is abnormal pixel change. to are greater than the second threshold (the change in the historical pixel value compared to the current pixel value within 1 second exceeds the second threshold), then the k The pixel changes of the abnormal point projection corresponding to the real-time original image are abnormal, and the first k Outliers; otherwise, discard the k Abnormal point.

[0057] In this embodiment, the second threshold is 0.15*442, which is an engineering experience value and can accurately determine whether there is an abnormality in the real-time original image.

[0058] In some other embodiments, the second threshold may also adopt other values ​​to adapt to different color change requirements.

[0059] Furthermore, for the outlier set All the abnormal points corresponding to the abnormal pixel changes in the retained pixel are reorganized into a pixel abnormal point cloud set , which can be used in the subsequent foreign body identification process.

[0060] Step S52: According to the pixel abnormal point cloud set, foreign objects are identified in the target monitoring area.

[0061] It should be noted that if the pixel abnormal point cloud set is not empty, the second neural network model is used to perform the first foreign body recognition on the image data. The first foreign body recognition is to identify whether there is a first foreign body of a known foreign body type in the image data. If the first foreign body is recognized, and the position of the first foreign body intersects with the target monitoring area, an alarm is issued. If the first foreign body is not recognized, or the position of the first foreign body does not intersect with the target monitoring area, the pixel abnormal point cloud set is used to perform the second foreign body recognition on the target monitoring area. The second foreign body recognition refers to identifying whether there is a second foreign body of an unknown foreign body type in the target monitoring area. If the second foreign body is recognized in the target monitoring area, an alarm is issued.

[0062] In this embodiment, the second neural network model adopts the YOLOV7 convolutional neural network. The YOLOV7 convolutional neural network can be used to quickly and accurately identify whether there is a first foreign object in the real-time original image. If the first foreign object is identified in the real-time original image, and the position of the first foreign object intersects with the target monitoring area, an alarm is issued. If the first foreign object is not identified in the real-time original image, or the position of the first foreign object does not intersect with the target monitoring area, the pixel abnormal point cloud set is used to perform the second foreign object identification on the target monitoring area.

[0063] In this embodiment, a hierarchical clustering method is used to cluster pixel abnormal point cloud sets. Perform point cloud clustering, and the resulting cluster set can be expressed as: ; in, Indicates l Clustering; cl Indicates l The number of cluster points in the cluster, if cl If it is less than 20, then the l The clustering is noise clustering; if cl If it is greater than or equal to 20, then take the l The first l Reflectivity set, and l The projection of all cluster points in the cluster on the corresponding position of the target monitoring area l Pixel color set.

[0064] Further, l Reflectivity set and l The pixel color sets can be expressed as: ; ; in, Indicates l Reflectivity Set, Indicates l Pixel color set.

[0065] Furthermore, calculate the l Reflectivity Set With l Pixel Color Set The target relevance can be expressed as: ; in, T Indicates l Reflectivity Set With l Pixel Color Set The relevance of the target, Indicates l Cluster b The cluster point corresponding to b Reflectivity, Indicates l Reflectivity Set The average value of Indicates l Cluster b The cluster point is projected at the corresponding position of the target monitoring area. b Pixel color, Indicates l Pixel Color Set The average value of .

[0066] Furthermore, according to the target relevance T The size of the target is used to determine whether there is a second foreign object of unknown foreign object type in the target monitoring area. T When it is greater than or equal to the third threshold, it is considered l If the projection of all cluster points in the cluster is at the corresponding position of the target monitoring area, there is a second foreign object and an alarm is issued; if the target correlation T When it is less than the third threshold, it is considered l If there is no second foreign object at the corresponding position of the target monitoring area where all cluster points in the cluster are projected, no alarm will be issued.

[0067] In this embodiment, the third threshold is 0.8. In the natural environment without paint (coating), the reflectivity of an object is related to its material, and objects of different materials have different colors. Therefore, when t≥0.8, the third threshold can be determined. l If all cluster points in the cluster are projected at the corresponding position of the target monitoring area, there is a second foreign object and an alarm is issued. If t<0.8, it may be due to irregular changes in the point cloud caused by rain or disturbance, and no alarm is issued.

[0068] In some other embodiments, the third threshold may also adopt other values ​​to meet different accuracy requirements.

[0069] Based on the same inventive concept, Fig.10 As shown, the present invention also provides a track foreign object monitoring system based on heterogeneous data, the system includes a laser radar 101, a camera 102 and an edge computing terminal 103, the laser radar 101 and the camera 102 are used to collect track data and generate point cloud data and image data; and the laser radar 101 and the camera 102 are respectively connected to the edge computing terminal 103 through a network cable, so that the edge computing terminal 103 obtains the corresponding point cloud data and image data to monitor track foreign objects.

[0070] The laser radar 101 and the camera 102 are aligned with the track through the brackets to ensure that their relative positions remain unchanged. After the laser radar 101 and the camera 102 are started at the same time, the corresponding point cloud data and image data can be obtained respectively.

[0071] like Fig.11 As shown, the edge computing terminal 103 includes a data acquisition unit 104, a monitoring area determination unit 105, a point cloud data screening unit 106, an abnormal point finding unit 107 and a foreign object monitoring unit 108.

[0072] Among them, the data acquisition unit 104 is used to acquire heterogeneous data, and the heterogeneous data includes point cloud data and image data. The monitoring area determination unit 105 is used to search the track surface according to the image data and determine the target monitoring area. The point cloud data screening unit 106 is used to filter the point cloud data using the target monitoring area to obtain a monitoring point cloud set. The abnormal point search unit 107 is used to search for abnormal points based on the monitoring point cloud set to obtain an abnormal point set. The foreign object monitoring unit 108 is used to project the abnormal point set to the target monitoring area and monitor foreign objects in the target monitoring area.

[0073] In the present application, other technical features of the above-mentioned track foreign object monitoring system based on heterogeneous data are the same as the features disclosed in the above-mentioned method embodiment, and will not be repeated here.

[0074] Based on the same inventive concept, the present application also provides an electronic device, which includes a processor, a memory and a communication circuit, and the processor is connected to the memory and the communication circuit respectively; wherein the communication circuit is used for communication connection, the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above method.

[0075] See also Fig.12 The electronic device described in the embodiment of the present application may specifically include a processor 210 and a memory 220. The memory 220 is coupled to the processor 210.

[0076] The processor 210 is used to control the operation of the electronic device. The processor 210 may also be referred to as a CPU (Central Processing Unit). The processor 210 may be an integrated circuit chip with signal processing capabilities. The processor 210 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor 210 may also be any conventional processor, etc.

[0077] The memory 220 is used to store computer programs, and may be a RAM, a ROM, or other types of storage terminals. Specifically, the memory 220 may include one or more computer-readable storage media, which may be non-transitory or transient. The memory 220 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage terminals, flash memory storage terminals. In some embodiments, the non-transitory computer-readable storage medium in the memory 220 is used to store at least one program code.

[0078] The processor 210 is used to execute the computer program stored in the memory 220 to implement the methods described in the various method embodiments of the present application.

[0079] In some embodiments, the electronic device may further include: a peripheral terminal interface 230 and at least one peripheral terminal. The processor 210, the memory 220 and the peripheral terminal interface 230 may be connected via a bus or a signal line. Each peripheral terminal may be connected to the peripheral terminal interface 230 via a bus, a signal line or a circuit board. Specifically, the peripheral terminal includes: at least one of a radio frequency circuit 240, a display screen 250, an audio circuit 260 and a power supply 270.

[0080] The peripheral terminal interface 230 may be used to connect at least one peripheral terminal related to I / O (Input / output) to the processor 210 and the memory 220. In some embodiments, the processor 210, the memory 220, and the peripheral terminal interface 230 are integrated on the same chip or circuit board; in some other implementations, any one or two of the processor 210, the memory 220, and the peripheral terminal interface 230 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0081] The radio frequency circuit 240 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 240 communicates with the communication network and other IoT devices through electromagnetic signals, and the radio frequency circuit 240 is the communication circuit of the electronic device. The radio frequency circuit 240 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 240 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, an operator identity module card, and the like. The radio frequency circuit 240 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 240 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0082] The display screen 250 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 250 is a touch display screen, the display screen 250 also has the ability to collect touch signals on the surface or above the surface of the display screen 250. The touch signal can be input to the processor 210 as a control signal for processing. At this time, the display screen 250 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 250 can be one, set on the front panel of the electronic device; in other embodiments, the display screen 250 can be at least two, respectively set on different surfaces of the electronic device or in a folding design; in other embodiments, the display screen 250 can be a flexible display screen, set on a curved surface or a folding surface of the electronic device. Even, the display screen 250 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 250 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0083] The audio circuit 260 may include a microphone and a speaker. The microphone is used to collect sound waves from the operator and the environment, and convert the sound waves into electrical signals and input them into the processor 210 for processing, or input them into the radio frequency circuit 240 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged in different parts of the electronic device. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 210 or the radio frequency circuit 240 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 260 may also include a headphone jack.

[0084] The power supply 270 is used to power various components in the electronic device. The power supply 270 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 270 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0085] For a detailed description of the functions and execution processes of each functional module or component in the electronic device embodiment of the present application, reference can be made to the description in the above-mentioned method embodiments of the present application, which will not be repeated here.

[0086] In the several embodiments provided in the present application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the various embodiments of the electronic devices described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0087] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0088] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0089] Based on the same inventive concept, the present application also provides a computer-readable storage medium storing a computer program, which can be executed by a processor to implement the above method.

[0090] See also Fig.13 , if the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 300. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions / computer programs to enable an IoT device (which can be a personal computer, server, or network terminal, etc.) or a processor (processor) to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks or optical disks, and electronic terminals such as computers, mobile phones, laptops, tablet computers, cameras, etc. having the above-mentioned storage media.

[0091] The description of the execution process of the program data in the computer-readable storage medium can refer to the description in the above-mentioned method embodiments of the present application, and will not be repeated here.

[0092] It can be seen that the present invention discloses a method, system, device and medium for monitoring foreign objects on a track based on heterogeneous data. The method includes the following steps: obtaining heterogeneous data, which includes point cloud data and image data; searching the track surface according to the image data to determine the target monitoring area; using the target monitoring area to filter the point cloud data to obtain a monitoring point cloud set; searching for abnormal points according to the monitoring point cloud set to obtain an abnormal point set; projecting the abnormal point set to the target monitoring area to monitor foreign objects in the target monitoring area. The method can automatically identify the target monitoring area, and at the same time make full use of the advantages of various monitoring technologies to accurately monitor foreign objects on the track.

[0093] The above are only embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent structural transformations made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are included in the protection scope of the present invention.

Claims

1. A method for monitoring foreign objects on a track based on heterogeneous data, characterized in that: The method comprises the following steps: Acquiring heterogeneous data, wherein the heterogeneous data includes point cloud data and image data; Search the rail surface according to the image data to determine the target monitoring area; Using the target monitoring area to filter the point cloud data, to obtain a monitoring point cloud set; Searching for abnormal points based on the monitoring point cloud set to obtain an abnormal point set; The abnormal point set is projected to the target monitoring area to monitor foreign objects in the target monitoring area.

2. The method for monitoring foreign matter on a track according to claim 1, characterized in that: The image data includes a real-time original image, and the process of searching the track surface on the image data to determine the target monitoring area includes the following steps: Identify the track features in the real-time original image to obtain feature recognition results; Determining a target monitoring area in the real-time original image according to the feature recognition result; Wherein, the feature recognition result includes the main track and the number of main track recognitions. When the number of main track recognitions is less than 2, the feature recognition result is inaccurate. When the number of main track identifications is greater than or equal to 2, the convex hull of the main track is taken as the target monitoring area.

3. The method for monitoring foreign matter on a track according to claim 1, characterized in that: Before the point cloud data is screened by the target monitoring area to obtain a monitoring point cloud set, the step further includes: Obtain a mapping matrix from a point cloud coordinate system to an image coordinate system; wherein the point cloud coordinate system is a coordinate system where the point cloud data is located, and the image coordinate system is a coordinate system where the image data is located.

4. The method for monitoring foreign matter on a track according to claim 3, characterized in that: The method of using the target monitoring area to filter the point cloud data to obtain a monitoring point cloud set includes the following steps: Using the projection matrix, projecting the point cloud data onto the image data to obtain a projection result; The monitoring point cloud set is obtained according to the projection result; the monitoring point cloud set includes all feature points in the point cloud data projected onto the target monitoring area.

5. The method for monitoring foreign matter on a track according to claim 1, characterized in that: The monitoring point cloud set includes an initial point cloud set collected when the laser radar is started, and a real-time point cloud set collected every second thereafter; performing abnormal point search based on the monitoring point cloud set to obtain an abnormal point set includes the steps of: Selecting a first real-time feature point in the real-time point cloud set, and respectively calculating vector moduli of the first real-time feature point and each initial feature point in the initial point cloud set; According to the magnitude of each vector modulus, determine whether the first real-time feature point is an abnormal point: if each vector modulus is greater than a first threshold, the first real-time feature point is an abnormal point; if there is a vector modulus less than or equal to the first threshold among the vector moduli, the first real-time feature point is a non-abnormal point; Select other real-time feature points except the first real-time feature point from the real-time point cloud set in turn, repeat the above steps, and obtain an abnormal point set consisting of all abnormal points in the real-time point cloud set.

6. The method for monitoring foreign matter on a track according to claim 1, characterized in that: The projecting of the abnormal point set to the target monitoring area to monitor foreign objects in the target monitoring area comprises the following steps: Projecting the abnormal point set to the target monitoring area, confirming the image abnormality, and obtaining a pixel abnormal point cloud set; Among them, the real-time pixels at the corresponding position of any abnormal point in the abnormal point set projected on the target monitoring area are color-compared with the historical pixels at the corresponding positions in multiple historical images obtained every second during video playback, and the color differences between the real-time pixels and the historical pixels are calculated; according to the size of the color differences, it is determined whether the color differences all exceed the second threshold; if the color differences all exceed the second threshold, it is confirmed that the image data is abnormal.

7. The method for monitoring foreign matter on a track according to claim 6, characterized in that: After confirming that the image data is abnormal, projecting the abnormal point set to the target monitoring area to monitor foreign objects in the target monitoring area also includes the steps of: Performing first foreign body recognition on the image data, wherein the first foreign body recognition refers to recognizing whether a first foreign body of a known foreign body type exists in the image data; If the first foreign object is identified and the position of the first foreign object intersects with the target monitoring area, an alarm is issued; If the first foreign object is not identified, or the position of the first foreign object has no intersection with the target monitoring area, the pixel abnormal point cloud set is used to perform a second foreign object identification on the target monitoring area, and the second foreign object identification refers to identifying whether there is a second foreign object of an unknown foreign object type in the target monitoring area; wherein the pixel abnormal point cloud set is clustered to obtain a cluster set, and the cluster set can be expressed as: ； in, Indicates l Clustering; cl Indicates the l The number of cluster points in the cluster, if cl If it is less than 20, then the l The clustering is noise clustering; if cl is greater than or equal to 20, then take the l The first l reflectivity set, and the l The projection of all cluster points in the cluster on the corresponding position of the target monitoring area l Pixel color set; then calculate the first l The reflectivity set is the same as the l If the target correlation is greater than or equal to a third threshold, then the first l The projection of all cluster points in the cluster on the corresponding position of the target monitoring area indicates that the second foreign object exists, and an alarm is issued; if the target correlation is less than a third threshold, it is determined that the second foreign object l If the second foreign object does not exist at the corresponding position of the target monitoring area when all cluster points in the cluster are projected, no alarm is issued.

8. A track foreign body monitoring system based on heterogeneous data, characterized in that: The system includes an edge computing terminal, and the edge computing terminal includes a data acquisition unit, a monitoring area determination unit, a point cloud data screening unit, an abnormal point finding unit and a foreign body monitoring unit; The data acquisition unit is used to acquire heterogeneous data, wherein the heterogeneous data includes point cloud data and image data; The monitoring area determination unit is used to search the rail surface according to the image data to determine the target monitoring area; The point cloud data screening unit is used to screen the point cloud data using the target monitoring area to obtain a monitoring point cloud set; The abnormal point finding unit is used to find abnormal points according to the monitoring point cloud set to obtain an abnormal point set; The foreign object monitoring unit is used to project the abnormal point set to the target monitoring area to monitor foreign objects in the target monitoring area.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images