Rail detection method and apparatus, electronic device, and storage medium

Abstract

This invention relates to the field of intelligent rail transit technology, providing a track detection method, apparatus, electronic device, and storage medium. The track detection method includes: obtaining target image data representing a track region from image data acquired by a camera; obtaining target point cloud data representing the track region from point cloud data acquired by a lidar based on the camera parameter matrix and the target image data; extracting a transformation matrix from the lidar coordinate system to the track plane coordinate system based on the target point cloud data; obtaining target track data generated by projecting the target image data onto the track plane based on the camera parameter matrix and the transformation matrix; and fitting a target track equation based on the target track data. This invention's track detection scheme combines the sensing capabilities of lidar with the motion-distortion-free and pixel-dense characteristics of camera imaging, improving track detection accuracy, increasing track detection distance, and efficiently and accurately obtaining track detection results.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent rail transit, in particular to a rail detection method and device, an electronic device and a storage medium. BACKGROUND

[0002] In the technical field of intelligent rail transit, rail detection is a prerequisite for obstacle intrusion alarm and other applications. The current mainstream rail detection algorithm is based on point cloud, which extracts the rail from the point cloud by using the geometric features of the point cloud.

[0003] Due to the low frame rate, sparsity and motion distortion of point cloud, the point cloud-based rail detection algorithm usually has a short detection distance and low detection accuracy, and cannot provide data basis for long-distance obstacle intrusion alarm.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] Therefore, the present application provides a rail detection method, device, electronic device and storage medium, which can combine the sensing ability of laser radar and the characteristics of camera imaging without motion distortion and dense pixels, improve the rail detection accuracy and increase the rail detection distance, and efficiently and accurately obtain the rail detection result.

[0006] According to one aspect of the present application, a rail detection method is provided, comprising: obtaining target image data representing a rail area from image data collected by a camera; obtaining target point cloud data representing the rail area from point cloud data collected by a laser radar based on a camera parameter matrix and the target image data; extracting a conversion matrix from a coordinate system of the laser radar to a coordinate system of a rail plane according to the target point cloud data; obtaining target rail data generated by projecting the target image data to the rail plane based on the camera parameter matrix and the conversion matrix; and fitting a target rail equation according to the target rail data.

[0007] In some embodiments, before obtaining the target image data representing the rail area from the image data collected by the camera, the method further comprises: time synchronizing the camera and the laser radar.

[0008] In some embodiments, obtaining the target image data representing the rail area from the image data collected by the camera comprises: performing semantic segmentation on the image data to obtain a binary target picture, the target picture containing the target image data representing the rail area.

[0009] In some embodiments, the camera parameter matrix includes an intrinsic parameter matrix from the camera coordinate system to the pixel coordinate system and an extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system; obtaining target point cloud data representing the orbital region from point cloud data acquired by the lidar includes: projecting the point cloud data onto the target image where the target image data is located based on the camera parameter matrix; and obtaining the target point cloud data based on the point cloud data in the target image that overlaps with the target image data.

[0010] In some embodiments, the point cloud data is projected onto the target image using the following projection formula: u d =K d T ld P; where K d It is the intrinsic parameter matrix, T ld It is the extrinsic parameter matrix, P is the position coordinate of the point cloud data based on the lidar coordinate system, and u d The point cloud data refers to the position coordinates based on the pixel coordinate system.

[0011] In some embodiments, extracting the transformation matrix from the coordinate system of the lidar to the coordinate system of the orbital plane includes: using a random sampling consensus algorithm to fit the orbital plane based on the target point cloud data, and extracting the transformation matrix from the coordinate system of the lidar to the coordinate system of the orbital plane.

[0012] In some embodiments, the camera parameter matrix includes an intrinsic parameter matrix from the camera coordinate system to the pixel coordinate system and an extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system; obtaining target orbit data generated by projecting the target image data onto the orbit plane includes: obtaining a transformation matrix from the pixel coordinate system to the orbit plane coordinate system based on the camera parameter matrix and the transformation matrix; and projecting the target image data onto the orbit plane based on the transformation matrix.

[0013] In some embodiments, the transformation matrix is ​​obtained using the following transformation formula: Among them, H inv K is the transformation matrix. d It is the intrinsic parameter matrix, T ld It is the extrinsic parameter matrix, T lr The transformation matrix is ​​[0:3,(0,1,3)], which means taking the 0th, 1st, and 3rd columns of the first three rows.

[0014] In some embodiments, after projecting the target image data onto the orbital plane, the method further includes: normalizing and 2D processing the projection data generated by projecting the target image data onto the orbital plane to generate the target orbital data.

[0015] In some embodiments, fitting the target orbit equation includes: clustering the target orbit data to obtain data clusters corresponding to each orbit; performing curve fitting on each data cluster based on the least squares method; and obtaining the target orbit equation based on the curve fitting results of each data cluster.

[0016] In some embodiments, the camera includes a first camera with a short focal length and a second camera with a long focal length; in the step of obtaining target image data characterizing the track region from the image data acquired by the camera, the image data includes first image data acquired by the first camera and second image data acquired by the second camera; in the step of based on the camera parameter matrix and the target image data, the target image data is target image data corresponding to the first image data; in the step of obtaining target track data generated by projecting the target image data onto the track plane, the target image data includes target image data corresponding to the first image data and target image data corresponding to the second image data.

[0017] According to another aspect of the present invention, a track detection device is provided, comprising: an image data processing module for obtaining target image data representing a track region from image data acquired by a camera; a point cloud data processing module for obtaining target point cloud data representing the track region from point cloud data acquired by a lidar based on a camera parameter matrix and the target image data; a transformation matrix extraction module for extracting a transformation matrix from the lidar coordinate system to the track plane coordinate system based on the target point cloud data; a track data generation module for obtaining target track data generated by projecting the target image data onto the track plane based on the camera parameter matrix and the transformation matrix; and a track equation fitting module for fitting a target track equation based on the target track data.

[0018] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; a memory storing executable instructions; wherein, when the executable instructions are executed by the processor, they implement the track detection method as described in any of the above embodiments.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided for storing a program that, when executed by a processor, implements the orbit detection method as described in any of the above embodiments.

[0020] The beneficial effects of this invention compared to the prior art include at least the following:

[0021] The trajectory detection scheme of this invention combines the sensing capability of lidar with the rapid imaging, distortion-free, and dense pixel characteristics of camera. First, it obtains target image data that accurately represents the trajectory region from the image data acquired by the camera. Then, based on the target image data, it obtains target point cloud data of the trajectory region from the point cloud data acquired by lidar. Based on the target point cloud data, it accurately extracts the transformation matrix from lidar to the trajectory plane. Then, based on the transformation matrix, it projects the dense target image data onto the trajectory plane to form dense and low-distortion target trajectory data. Finally, based on the target trajectory data, it fits to obtain a high-precision, long-distance target trajectory equation.

[0022] The image and point cloud fusion track detection scheme of the present invention can effectively improve track detection accuracy, increase track detection distance, and obtain track detection results efficiently and accurately;

[0023] Furthermore, the track detection results obtained using the track detection scheme of the present invention can be further used for applications such as obstacle intrusion alarm, making long-distance obstacle intrusion alarm based on track detection possible.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0026] Figure 1 This diagram illustrates the steps of a track detection method according to an embodiment of the present invention.

[0027] Figure 2 This diagram illustrates the steps for obtaining target point cloud data in one embodiment of the present invention.

[0028] Figure 3 This diagram illustrates the steps for obtaining target orbit data in one embodiment of the present invention.

[0029] Figure 4 This diagram illustrates the steps of fitting the target trajectory equation in one embodiment of the present invention.

[0030] Figure 5 This diagram illustrates the steps of track detection using a combination of long-focus and short-focus cameras and a lidar in one embodiment of the present invention.

[0031] Figure 6 This diagram illustrates an implementation scenario of orbit detection combining long and short focal length cameras and lidar, according to an embodiment of the present invention.

[0032] Figure 7 This diagram shows a schematic of the module of the track detection device in one embodiment of the present invention;

[0033] Figure 8 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0034] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to fully and completely convey the concept of the exemplary embodiments to those skilled in the art.

[0035] The accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0036] Furthermore, the processes shown in the accompanying drawings are merely illustrative and do not necessarily include all steps. For example, some steps can be broken down, some steps can be combined or partially combined, and the actual execution order may change depending on the actual situation. The terms "first," "second," and similar terms used in the specific description do not indicate any order, quantity, or importance, but are only used to distinguish different components. It should be noted that, unless otherwise specified, embodiments of the present invention and features in different embodiments can be combined with each other.

[0037] The track detection scheme of this invention is applicable to various rail transit scenarios such as railways, subways, light rails, trams, and maglev trains.

[0038] Figure 1 The main steps of the orbit detection method in one embodiment are shown, with reference to Figure 1 As shown, in one embodiment, the orbit detection method includes:

[0039] Step S110: Obtain target image data representing the orbital region from the image data acquired by the camera.

[0040] This step utilizes the characteristics of rapid image imaging, no motion distortion, and dense pixels to obtain accurate target image data representing the orbital region from the image data acquired by the camera.

[0041] Step S120: Based on the camera parameter matrix and target image data, obtain target point cloud data representing the orbital region from the point cloud data collected by the lidar.

[0042] This step uses target image data, combined with camera parameter matrices, to color-process the point cloud data acquired by the lidar, in order to obtain target point cloud data characterizing the orbital region.

[0043] Step S130: Based on the target point cloud data, extract the transformation matrix from the coordinate system of the lidar to the coordinate system of the orbital plane.

[0044] This step, based on target point cloud data, can accurately fit the orbital plane and then extract the transformation matrix from the laser radar to the orbital plane.

[0045] Step S140: Based on the camera parameter matrix and transformation matrix, obtain the target orbit data generated by projecting the target image data onto the orbit plane.

[0046] This step, based on the transformation matrix and combined with the camera parameter matrix, projects dense target image data onto the orbital plane to form dense and low-distortion target orbital data.

[0047] Step S150: Fit the target orbit equation based on the target orbit data.

[0048] This step involves fitting high-precision, long-distance target orbit equations based on target orbit data.

[0049] Therefore, the above-mentioned image and point cloud fusion track detection scheme, combined with the sensing capabilities of LiDAR and the characteristics of rapid imaging, no motion distortion, and dense pixels of camera, can effectively improve track detection accuracy, increase track detection distance, and obtain track detection results efficiently and accurately.

[0050] Furthermore, the track detection results obtained using the above-mentioned track detection scheme can be further used for applications such as obstacle intrusion alarms, making long-distance obstacle intrusion alarms based on track detection possible.

[0051] In one embodiment, before obtaining target image data representing the orbital region from image data acquired by the camera, the method further includes: time synchronization of the camera and the lidar.

[0052] Time synchronization can be achieved using conventional synchronization methods, including software synchronization and hardware synchronization, and this invention does not impose any restrictions on these methods.

[0053] Time-synchronized cameras and lidar can acquire image data and point cloud data at a fixed frame rate (e.g., the lidar frame rate), facilitating subsequent fusion of image data and point cloud data for track detection.

[0054] Furthermore, in practical implementation scenarios, the camera and lidar can be assembled into a fixed unit to ensure that they are relatively fixed.

[0055] In one embodiment, obtaining target image data representing the orbital region from image data acquired by a camera specifically includes: performing semantic segmentation on the image data to obtain a binarized target image, wherein the target image contains target image data representing the orbital region.

[0056] Semantic segmentation of image data can be performed using a pre-trained semantic segmentation model. The specific structure and training process of the semantic segmentation model can adopt existing methods, and this invention does not limit them. For example, the semantic segmentation model can be built based on a convolutional neural network and trained on images containing track patterns.

[0057] After semantic segmentation, the image data is binarized into a target image. Pixels with a value of 1 represent the track region, and these pixels with a value of 1 are the target image data; pixels with a value of 0 represent other objects, such as the background, obstacles, etc.

[0058] In other embodiments, semantic segmentation of image data can also yield multi-valued target images, as long as target image data representing the orbital region can be extracted from them.

[0059] The camera parameter matrix includes the intrinsic parameter matrix from the camera coordinate system to the pixel coordinate system (denoted as K). d The extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system (denoted as T) ld Intrinsic parameter matrix K d and extrinsic parameter matrix T ld Before conducting track inspection, the camera should be calibrated in advance. Conventional camera calibration methods can be used, and this invention does not limit this.

[0060] Figure 2 This illustrates the main steps for obtaining target point cloud data in one embodiment, combined with... Figure 2 As shown, in one embodiment, target point cloud data characterizing the orbital region is obtained from point cloud data acquired by lidar based on camera parameter matrix and target image data, including:

[0061] Step S210: Based on the camera parameter matrix, project the point cloud data onto the target image where the target image data is located.

[0062] Specifically, based on the intrinsic parameter matrix K d and extrinsic parameter matrix Tld The following projection formula can be used to project each point in the point cloud data onto the target image:

[0063] u d =K d T ld P;

[0064] Where P is the position coordinate of a point in the point cloud data based on the lidar coordinate system; u d It refers to the position coordinates of a point in the point cloud data based on the pixel coordinate system, that is, the pixel coordinates obtained by projecting a point in the point cloud data onto the target image.

[0065] Step S220: Obtain target point cloud data based on the point cloud data in the target image that overlaps with the target image data.

[0066] Specifically, in the target image, if the projected position coordinates of a certain point are u d If the location coordinates of the point coincide with those of the target image data, then that point is identified as the target point cloud data representing the orbital region.

[0067] Taking a binary image as an example, this step can be understood as coloring the point cloud data using the target image containing the target image data. In the target image, if the projected position coordinates u of a certain point... d The pixel value of ' is 1, indicating that the point coincides with the target image data, and the point P' is colored as 1; while if the position coordinates of the projected position of a point are u d If the pixel value is 0, then that point is irrelevant to the target image data, and that point P is colored as 0. Finally, the point cloud data colored as 1 is selected to obtain the target point cloud data.

[0068] In one embodiment, the transformation matrix from the coordinate system of the lidar to the coordinate system of the orbital plane is extracted based on the target point cloud data. Specifically, this includes: using a random sampling consensus algorithm to fit the orbital plane based on the target point cloud data, and extracting the transformation matrix from the coordinate system of the lidar to the coordinate system of the orbital plane.

[0069] Random Sample Consensus (RANSAC) is an iterative method for estimating mathematical model parameters using observed data points. This embodiment utilizes RANSAC to first fit an orbital plane based on target point cloud data, and then extracts the transformation matrix from the lidar to the orbital plane based on the fitted orbital plane and the target point cloud data in the lidar's coordinate system.

[0070] Furthermore, when fitting the orbital plane using the RANSAC algorithm, since the orbital plane is actually stationary in the depth direction, only the XY coordinates of the target point cloud data can be considered, while the Z-axis coordinate in the depth direction can be ignored, in order to accelerate the fitting process and reduce the amount of data computation.

[0071] In other embodiments, other general algorithms can also be used to extract the transformation matrix from the laser radar to the orbital plane.

[0072] Figure 3 This illustrates the main steps for obtaining target orbit data in one embodiment, with reference to... Figure 3 As shown, in one embodiment, target orbit data generated by projecting target image data onto the orbit plane is obtained based on the camera parameter matrix and the transformation matrix, including:

[0073] Step S310: Based on the camera parameter matrix and the transformation matrix, obtain the transformation matrix from the pixel coordinate system to the coordinate system of the orbital plane.

[0074] As explained above, the camera parameter matrix includes the intrinsic parameter matrix K, which represents the transition from the camera coordinate system to the pixel coordinate system. d The extrinsic parameter matrix T from the lidar coordinate system to the camera coordinate system ld The transformation matrix from laser radar to the orbital plane is denoted as T. lr Therefore, the transformation matrix H from the pixel coordinate system to the orbital plane can be obtained using the following transformation formula. inv :

[0075]

[0076] Where [0:3,(0,1,3)] represents taking the 0th, 1st, and 3rd columns of the first three rows. Specifically, the obtained initial transformation matrix contains four rows and four columns, where the 4th row is redundant data, and the 2nd column corresponds to the Z-axis. As explained above when fitting the orbital plane, the Z-axis coordinates in the depth direction can be ignored, and the data in the 2nd column obtained here are all 0; therefore, taking the 0th, 1st, and 3rd columns of the first three rows of the initial transformation matrix is ​​used to obtain the effective transformation matrix H. inv .

[0077] Step S320: Based on the transformation matrix, project the target image data onto the orbital plane.

[0078] In this step, the pixel coordinates of the target image data are compared with the transformation matrix H. inv By multiplying, we can obtain the projection data of the target image data onto the orbital plane.

[0079] Furthermore, after projecting the target image data onto the orbital plane, the method further includes: step S330, normalizing and 2D processing the projection data generated by projecting the target image data onto the orbital plane to generate target orbital data.

[0080] The projection data generated by projecting the target image data onto the orbital plane is non-normalized three-dimensional projection data. To facilitate subsequent orbital equation fitting calculations and reduce the amount of data computation, this step performs normalization and two-dimensionalization processing on the projection data. Let the projection data of a point in the target image data onto the orbital plane be [x']. r ,y' r ,q' r ] T After normalization and two-dimensionalization, the target orbit data obtained is:

[0081] Through the above process of obtaining target orbit data, the target image data representing the orbit region is projected onto the orbit plane to obtain dense and low-distortion target orbit data.

[0082] Figure 4 This illustrates the main steps of fitting the target trajectory equation in one embodiment, with reference to... Figure 4 As shown, in one embodiment, fitting the target orbit equation based on the target orbit data includes:

[0083] Step S410: Cluster the target orbit data to obtain the data clusters corresponding to each orbit.

[0084] Density-based clustering algorithms, such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise), can be used to cluster the target orbital data. Each cluster represents a single orbital, thus obtaining the data cluster corresponding to each orbital.

[0085] In other embodiments, other feasible clustering algorithms may also be used to cluster the target orbit data.

[0086] Step S420: Based on the least squares method, perform curve fitting for each data cluster.

[0087] Specifically, for each data cluster, a cubic curve model can be used, and the least squares method can be used for fitting to obtain the orbit equation for each corresponding orbit efficiently and accurately.

[0088] In other embodiments, other feasible curve fitting algorithms may also be used to fit and obtain the orbital equations.

[0089] Step S430: Obtain the target trajectory equation based on the curve fitting results of each data cluster.

[0090] The target orbit equation specifically includes the orbit equations of two orbits.

[0091] Based on the track detection method described in the above embodiments, in a specific implementation scenario, the complete implementation process of track detection includes a preparation process and an online operation process. The preparation process includes: calibrating the intrinsic and extrinsic parameter matrices of the camera and training the semantic segmentation model. The online operation process includes: synchronizing the camera and LiDAR in time and forming a fixed assembly to collect image data and point cloud data at a fixed frame rate; the track detection algorithm, which fuses the image and point cloud data, continuously performs detection, constantly generating track detection results; and continuously publishing the track detection results for subsequent applications, such as allowing obstacle intrusion alarm applications to perform long-distance obstacle intrusion alarms based on the track detection results.

[0092] Further, in one embodiment, the camera used to acquire image data may include a first camera with a shorter focal length and a second camera with a longer focal length; then, in the step of obtaining target image data representing the track region from the image data acquired by the camera, the image data includes first image data acquired by the first camera and second image data acquired by the second camera; in the step of based on the camera parameter matrix and the target image data, the target image data is the target image data corresponding to the first image data; in the step of obtaining target track data generated by projecting the target image data onto the track plane, the target image data includes the target image data corresponding to the first image data and the target image data corresponding to the second image data.

[0093] The first camera with a shorter focal length (hereinafter referred to as the short-focus camera) has a focal length of around 50m, while the second camera with a longer focal length (hereinafter referred to as the long-focus camera) typically has a focal length of over 60m. The short-focus camera ensures the accuracy of the acquired image data, while the long-focus camera enables long-distance image data acquisition. By acquiring image data together with both short-focus and long-focus cameras, the accuracy of track detection can be further enhanced, and the detection distance can be increased to over 150m.

[0094] Figure 5 This illustrates the main steps of track detection in one embodiment, combining long and short focal length cameras and LiDAR, with reference to... Figure 5 As shown, orbit detection is performed by combining long and short focal length cameras and LiDAR, including:

[0095] S510 obtains target image data characterizing the orbital region from the first image data acquired by the short-focus camera and the second image data acquired by the long-focus camera.

[0096] S520 obtains target point cloud data representing the orbital region from point cloud data collected by lidar, based on the camera parameter matrix and the target image data corresponding to the first image data.

[0097] The target image data corresponding to the first image data is the target image data representing the track region extracted from the first image data acquired by the short-focus camera. This step involves coloring the point cloud data based on the target image data from the short-focus camera, which accurately obtains the target point cloud data representing the track region.

[0098] S530 extracts the transformation matrix from the coordinate system of the lidar to the coordinate system of the orbital plane based on the target point cloud data.

[0099] S540, based on the camera parameter matrix and transformation matrix, obtains target orbit data generated by projecting the target image data corresponding to the first image data and the target image data corresponding to the second image data onto the orbit plane.

[0100] The target image data corresponding to the second image data is the target image data representing the track region extracted from the second image data acquired by the telephoto camera. This step projects the target image data from both the short-focus and telephoto cameras onto the track plane, which can balance accuracy and long distance.

[0101] S550, based on the target orbit data, fits the target orbit equation.

[0102] Steps S510 to S550 in this embodiment respectively correspond to Figure 1 Steps S110 to S150 of the illustrated embodiment, Figure 1 The features and principles of track detection shown in the embodiments and described in the above embodiments can all be applied to this embodiment. Therefore, the features and principles of track detection already explained above will not be repeated.

[0103] Figure 6 This illustration shows an implementation scenario of orbit detection combining long and short focal length cameras and LiDAR. Figure 6 As shown, in a specific implementation scenario, orbit detection is performed by combining long and short focal length cameras and LiDAR, including:

[0104] The short-focus camera 610, long-focus camera 620, and LiDAR 630 are time-synchronized to acquire short-focus image data, long-focus image data, and point cloud data at a fixed frame rate. The image and point cloud fusion track detection algorithm 640 reads pre-calibrated parameters (e.g., intrinsic and extrinsic parameters of the short-focus and long-focus cameras) and a pre-trained model (e.g., a speech segmentation model) to process the short-focus image data, long-focus image data, and point cloud data, generating a target track equation. This target track equation can be further published to a higher-level system as foundational data for applications such as obstacle intrusion alarms.

[0105] The track detection schemes described above combine the sensing capabilities of lidar with the rapid, distortion-free, and densely pixelated imaging characteristics of cameras. They utilize cameras to obtain accurate target image data representing the track region, then use this target image data to color point cloud data, obtaining target point cloud data representing the track region. Based on this target point cloud data, they accurately fit the track plane and extract the transformation matrix from the lidar to the track plane. Next, based on the transformation matrix, they project the dense target image data onto the track plane, forming dense and low-distortion target track data. This data is then used to fit a high-precision, long-distance target track equation. This image and point cloud fusion track detection scheme effectively improves track detection accuracy, increases track detection distance, and efficiently and accurately obtains track detection results. Furthermore, the track detection results obtained using this scheme can be further used in applications such as obstacle intrusion alarms, making long-distance obstacle intrusion alarms based on track detection possible.

[0106] This invention also provides a track detection device that can be used to implement the track detection method described in any of the above embodiments. The features and principles of the track detection methods described in any of the above embodiments can be applied to the track detection device embodiments below. In the following track detection device embodiments, the features and principles of track detection already explained will not be repeated.

[0107] Figure 7 The main modules of the track detection device in one embodiment are shown, with reference to Figure 7 As shown, in one embodiment, the track detection device 700 includes: an image data processing module 710, used to obtain target image data representing the track region from image data acquired by a camera; a point cloud data processing module 720, used to obtain target point cloud data representing the track region from point cloud data acquired by a lidar based on the camera parameter matrix and the target image data; a transformation matrix extraction module 730, used to extract the transformation matrix from the lidar coordinate system to the track plane coordinate system based on the target point cloud data; a track data generation module 740, used to obtain target track data generated by projecting the target image data onto the track plane based on the camera parameter matrix and the transformation matrix; and a track equation fitting module 750, used to fit the target track equation based on the target track data.

[0108] Furthermore, the track detection device 700 may also include modules that implement other process steps of the above-described track detection method embodiments. The specific principles of each module can be referred to the descriptions of the above-described track detection method embodiments, and will not be repeated here.

[0109] The track detection device of this invention combines the sensing capabilities of lidar with the rapid, distortion-free, and dense pixel characteristics of camera imaging. It uses the camera to obtain target image data accurately representing the track area, then uses the target image data to color point cloud data to obtain target point cloud data representing the track area. Based on the target point cloud data, it accurately fits the track plane and extracts the transformation matrix from the lidar to the track plane. Then, based on the transformation matrix, it projects the dense target image data onto the track plane to form dense and low-distortion target track data. A high-precision, long-distance target track equation is obtained by fitting the target track data. This track detection device can also be applied to applications such as obstacle intrusion alarms, enabling long-distance obstacle intrusion alarms based on track detection.

[0110] This invention also provides an electronic device, including a processor and a memory, wherein the memory stores executable instructions, and when the executable instructions are executed by the processor, the track detection method described in any of the above embodiments is implemented.

[0111] The electronic device of this invention combines the sensing capabilities of lidar with the rapid, distortion-free, and densely pixelated imaging characteristics of a camera. It uses the camera to obtain target image data accurately representing the track area, then uses the target image data to color point cloud data to obtain target point cloud data representing the track area. Based on the target point cloud data, it accurately fits the track plane and extracts the transformation matrix from the lidar to the track plane. Then, based on the transformation matrix, it projects the dense target image data onto the track plane to form dense and low-distortion target track data. A high-precision, long-distance target track equation is obtained by fitting the target track data. This electronic device can also be applied to applications such as obstacle intrusion alarms, enabling long-distance obstacle intrusion alarms based on track detection.

[0112] Figure 8 This is a schematic diagram of the structure of the electronic device in an embodiment of the present invention. It should be understood that... Figure 8 The modules are merely shown schematically. These modules can be virtual software modules or actual hardware modules. The merging, splitting, and addition of other modules are all within the scope of protection of this invention.

[0113] like Figure 8 As shown, the electronic device 800 is presented in the form of a general-purpose computing device. The components of the electronic device 800 include, but are not limited to: at least one processing unit 810, at least one storage unit 820, a bus 830 connecting different platform components (including storage unit 820 and processing unit 810), a display unit 840, etc.

[0114] The storage unit 820 stores program code, which can be executed by the processing unit 810, causing the processing unit 810 to perform the steps of the orbit detection method described in any of the above embodiments. For example, the processing unit 810 can perform, as follows: Figure 1 The steps are shown.

[0115] Storage unit 820 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 8201 and / or cache memory 8202, and may further include a read-only memory (ROM) 8203.

[0116] The storage unit 820 may also include a program / utility 8204 having one or more program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0117] Bus 830 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0118] Electronic device 800 can also communicate with one or more external devices, such as keyboards, pointing devices, Bluetooth devices, etc. These external devices enable users to interact with electronic device 800. Electronic device 800 can also communicate with one or more other computing devices, including routers and modems. This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via network adapter 860. Network adapter 860 can communicate with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0119] This invention also provides a computer-readable storage medium for storing a program that, when executed, implements the orbit detection method described in any of the above embodiments. In some possible implementations, various aspects of this invention can also be implemented as a program product comprising program code, which, when run on a terminal device, causes the terminal device to execute the orbit detection method described in any of the above embodiments.

[0120] When executed by a processor, the storage medium of this invention combines the sensing capabilities of lidar with the rapid, distortion-free, and densely pixelated imaging characteristics of a camera. It uses the camera to obtain accurate target image data representing the track area, then uses the target image data to color point cloud data to obtain target point cloud data representing the track area. Based on the target point cloud data, it accurately fits the track plane and extracts the transformation matrix from the lidar to the track plane. Then, based on the transformation matrix, it projects the dense target image data onto the track plane to form dense and low-distortion target track data. This allows for the fitting of a high-precision, long-distance target track equation. Furthermore, the storage medium of this invention can also be applied to applications such as obstacle intrusion alarms, enabling long-range obstacle intrusion alarms based on track detection.

[0121] The storage medium may be a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the storage medium of the present invention is not limited thereto, and may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0122] The storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include, but are not limited to: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0123] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable signal medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0124] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device, for example, via the Internet using an Internet service provider.

[0125] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A track detection method, characterized in that, include: Obtain target image data representing the orbital region from the image data acquired by the camera; Based on the camera parameter matrix and the target image data, target point cloud data characterizing the orbital region is obtained from the point cloud data collected by the lidar. The camera parameter matrix includes an intrinsic parameter matrix from the camera coordinate system to the pixel coordinate system and an extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system. Obtaining target point cloud data representing the orbital region from the point cloud data acquired by the lidar includes: projecting the point cloud data onto the target image containing the target image data based on the camera parameter matrix; and obtaining the target point cloud data based on the point cloud data in the target image that overlaps with the target image data. The point cloud data is projected onto the target image using the following projection formula: ;in, It is the intrinsic parameter matrix, Here, P is the extrinsic parameter matrix, and P is the position coordinate of the point cloud data based on the lidar coordinate system. The point cloud data refers to the position coordinates based on the pixel coordinate system. Based on the target point cloud data, the transformation matrix from the coordinate system of the lidar to the coordinate system of the orbital plane is extracted, including: using a random sampling consensus algorithm to fit the orbital plane based on the target point cloud data, and extracting the transformation matrix from the coordinate system of the lidar to the coordinate system of the orbital plane; Based on the camera parameter matrix and the transformation matrix, target orbit data generated by projecting the target image data onto the orbit plane is obtained; Based on the target orbit data, fit the target orbit equation.

2. The track detection method as described in claim 1, characterized in that, Before obtaining the target image data representing the orbital region from the image data acquired by the camera, the method further includes: The camera and the lidar are synchronized in time.

3. The track detection method as described in claim 1, characterized in that, The target image data representing the orbital region is obtained from the image data acquired by the self-camera, including: The image data is semantically segmented to obtain a binarized target image, which contains the target image data representing the orbital region.

4. The track detection method as described in claim 1, characterized in that, The camera parameter matrix includes an intrinsic parameter matrix from the camera coordinate system to the pixel coordinate system and an extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system. The process of obtaining target orbit data generated by projecting the target image data onto the orbit plane includes: Based on the camera parameter matrix and the transformation matrix, obtain the transformation matrix from the pixel coordinate system to the coordinate system of the orbital plane; Based on the transformation matrix, the target image data is projected onto the orbital plane.

5. The track detection method as described in claim 4, characterized in that, The transformation matrix is ​​obtained using the following transformation formula: ； in, It is the transformation matrix, It is the intrinsic parameter matrix, It is the extrinsic parameter matrix, It is the transformation matrix, This means taking the 0th, 1st, and 3rd columns of the first three rows.

6. The track detection method as described in claim 4, characterized in that, After projecting the target image data onto the orbital plane, the method further includes: The projection data generated by projecting the target image data onto the orbital plane is normalized and converted into two dimensions to generate the target orbital data.

7. The track detection method as described in claim 1, characterized in that, The fitted target trajectory equation includes: Cluster the target orbit data to obtain data clusters corresponding to each orbit; Curve fitting is performed on each data cluster based on the least squares method; The target trajectory equation is obtained based on the curve fitting results of each data cluster.

8. The track detection method according to any one of claims 1-7, characterized in that, The camera includes a first camera with a shorter focal length and a second camera with a longer focal length; In the step of obtaining target image data representing the orbital region from the image data acquired by the camera, the image data includes first image data acquired by the first camera and second image data acquired by the second camera; In the step based on the camera parameter matrix and the target image data, the target image data is the target image data corresponding to the first image data; In the step of obtaining target orbit data generated by projecting the target image data onto the orbit plane, the target image data includes target image data corresponding to the first image data and target image data corresponding to the second image data.

9. A track detection device, characterized in that, include: The image data processing module is used to obtain target image data representing the orbital region from the image data acquired by the camera; The point cloud data processing module is used to obtain target point cloud data representing the orbital region from the point cloud data collected by the lidar based on the camera parameter matrix and the target image data. The camera parameter matrix includes an intrinsic parameter matrix from the camera coordinate system to the pixel coordinate system and an extrinsic parameter matrix from the lidar coordinate system to the camera coordinate system. The point cloud data processing module obtains target point cloud data representing the orbital region from the point cloud data collected by the lidar, including: projecting the point cloud data onto the target image where the target image data is located based on the camera parameter matrix; and obtaining the target point cloud data based on the point cloud data in the target image that overlaps with the target image data. The point cloud data is projected onto the target image using the following projection formula: ;in, It is the intrinsic parameter matrix, Here, P is the extrinsic parameter matrix, and P is the position coordinate of the point cloud data based on the lidar coordinate system. The point cloud data refers to the position coordinates based on the pixel coordinate system. The transformation matrix extraction module is used to extract the transformation matrix from the coordinate system of the lidar to the coordinate system of the orbital plane based on the target point cloud data, including: using a random sampling consensus algorithm to fit the orbital plane based on the target point cloud data, and extracting the transformation matrix from the coordinate system of the lidar to the coordinate system of the orbital plane; The orbit data generation module is used to obtain target orbit data generated by projecting the target image data onto the orbit plane based on the camera parameter matrix and the transformation matrix; The orbit equation fitting module is used to fit the target orbit equation based on the target orbit data.

10. An electronic device, characterized in that, include: processor; A memory, wherein executable instructions are stored; When the executable instructions are executed by the processor, they implement the orbit detection method as described in any one of claims 1-8.

11. A computer-readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the orbit detection method as described in any one of claims 1-8.

Images