Freight train inspection method, device and equipment and inspection robot

Abstract

The invention relates to a freight train inspection method, device and equipment and an inspection robot. The method comprises the following steps: acquiring inspection environment data corresponding to a to-be-inspected freight train of an inspection robot, wherein the inspection environment data comprises image data and point cloud data; performing path planning processing according to the image data and the point cloud data to obtain a target inspection path from the current position of the inspection robot to the target inspection site; and controlling the inspection robot to go to the target inspection site based on the target inspection path, performing inspection from the target inspection site to obtain target inspection data, and sending the target inspection data to the master console. By adopting the method, the inspection reliability can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision, in particular to a freight train inspection method, device, equipment and inspection robot. BACKGROUND

[0002] In recent years, machine vision technology has been widely used in industrial links such as automatic assembly, intelligent picking and quality detection.

[0003] The existing machine vision technology generally combines pattern recognition and deep learning algorithm to extract target features and perform spatial positioning, thereby realizing intelligent and automatic control of industrial production process. In the scene of using machine vision technology to inspect freight trains, the way to obtain inspection data still depends on manual operation or fixed path automatic guided vehicles.

[0004] However, the existing machine vision technology has the problem of low reliability in the application scene of inspecting freight trains, which has complex working environment, variable task path and irregular target distribution. SUMMARY

[0005] Therefore, it is necessary to provide a freight train inspection method, device, equipment and inspection robot capable of improving reliability to solve the above technical problems.

[0006] In a first aspect, the present application provides a freight train inspection method, comprising:

[0007] Obtaining inspection environment data corresponding to a freight train to be inspected by an inspection robot, the inspection environment data including image data and point cloud data;

[0008] Performing path planning processing according to the image data and the point cloud data to obtain a target inspection path between the current position of the inspection robot and a target inspection site;

[0009] Controlling the inspection robot to go to the target inspection site based on the target inspection path, starting inspection from the target inspection site, obtaining target inspection data, and sending the target inspection data to a general control console.

[0010] In one embodiment, the path planning processing according to the image data and the point cloud data to obtain the target inspection path between the current position of the inspection robot and the target inspection site comprises:

[0011] Using the image data and the point cloud data to obtain three-dimensional pose information corresponding to the target inspection site and a target carriage where the target inspection site is located;

[0012] Performing simultaneous localization and mapping processing based on the three-dimensional pose information, the image data and the point cloud data to obtain the target inspection path.

[0013] In one embodiment, the image data and the point cloud data are used to obtain three-dimensional pose information corresponding to a target carriage where the target inspection site is located, including:

[0014] The point cloud data is subjected to spatial clustering processing to obtain a plurality of mutually independent carriage regions;

[0015] The spatial relative distance between the inspection robot and each carriage region is calculated respectively, and the carriage region with the smallest spatial relative distance is taken as the target carriage;

[0016] The image data and the point cloud data corresponding to the target carriage are used to perform registration processing on the target carriage and a preset reference coordinate system to obtain the three-dimensional pose information.

[0017] In one embodiment, based on the three-dimensional pose information, the image data and the point cloud data, synchronous positioning and mapping processing is performed to obtain a target inspection path, including:

[0018] Based on the image data and the point cloud data, map construction processing is performed to generate a local map, and self-positioning of the inspection robot in the local map is determined through feature extraction and matching; the local map includes obstacle distribution positions;

[0019] The local map, the three-dimensional pose information and a preset preliminary inspection map are fused and updated to obtain a target inspection map;

[0020] Based on the target inspection map, dynamic obstacle avoidance path planning processing is performed to obtain the target inspection path.

[0021] In one embodiment, the target inspection site is used as a starting point for inspection to obtain target inspection data, including:

[0022] According to the task type corresponding to the inspection task, image acquisition, video recording or three-dimensional point cloud scanning operations are performed to obtain preliminary inspection data;

[0023] The preliminary inspection data is subjected to at least one of the following preprocessing to obtain target inspection data:

[0024] The preliminary inspection data obtained by performing image acquisition is subjected to image compression processing to obtain target inspection data;

[0025] The preliminary inspection data obtained by performing video recording is subjected to video segmentation and encoding processing to obtain target inspection data;

[0026] The preliminary inspection data obtained by performing three-dimensional point cloud scanning is subjected to point cloud format conversion processing to obtain target inspection data.

[0027] In one embodiment, the method further includes:

[0028] During the inspection process, an inspection robot obtains a line-of-sight vector of an image acquisition device of the inspection robot;

[0029] The point cloud data is subjected to normal estimation calculation processing to obtain a local normal vector;

[0030] The line-of-sight vector and the local normal vector are subjected to dot product calculation processing to obtain a vector included angle;

[0031] The vector included angle is used to control the inspection robot to adjust the line-of-sight direction, so that the line-of-sight vector and the local normal vector are kept perpendicular.

[0032] In a second aspect, the present application further provides a freight train inspection device, comprising:

[0033] A data acquisition module is configured to acquire inspection environment data of a freight train to be inspected by an inspection robot, wherein the inspection environment data comprises image data and point cloud data;

[0034] A path planning module is configured to perform path planning processing based on the image data and the point cloud data to obtain a target inspection path from a current position of the inspection robot to a target inspection site;

[0035] A signal transmission module is configured to control the inspection robot to go to the target inspection site based on the target inspection path, to start the inspection from the target inspection site, to obtain target inspection data, and to send the target inspection data to a general control console.

[0036] In a third aspect, the present application further provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the freight train inspection method of the first aspect when executing the computer program.

[0037] In a fourth aspect, the present application further provides an inspection robot, which is equipped with a visible light camera, a line laser radar sensor, and an electronic device of the third aspect, wherein the visible light camera and the line laser radar sensor are respectively in communication connection with the electronic device.

[0038] The visible light camera is configured to acquire image data of a freight train to be inspected by the inspection robot;

[0039] The line laser radar sensor is configured to acquire point cloud data of the freight train.

[0040] In one of the embodiments, the inspection robot is a wheeled robot.

[0041] The freight train inspection method, device, equipment and inspection robot described above identify the appearance features and spatial positions of the freight train based on image data and point cloud data collected by the sensors carried by the inspection robot, effectively distinguish tracks, carriages and surrounding obstacles, and provide accurate real-time environmental information for path planning; the target inspection path is planned according to the real-time environmental information, so that the inspection robot can safely and smoothly go to the target inspection site to start the inspection task and obtain the target inspection data. The freight train inspection method in the embodiment has high flexibility and environmental adaptability, and can stably and reliably complete the inspection task when dealing with complex situations such as track obstacles, slope changes or space limitations, thereby effectively ensuring the reliability of the freight train inspection method. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other related drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 An application environment diagram of the freight train inspection method in an embodiment;

[0044] Figure 2 A flowchart of the freight train inspection method in an embodiment;

[0045] Figure 3 A flowchart of the steps of obtaining the target inspection path in an embodiment;

[0046] Figure 4 A flowchart of the freight train inspection method in another embodiment;

[0047] Figure 5 A structural block diagram of the freight train inspection device in an embodiment;

[0048] Figure 6 An internal structure diagram of the computer device in an embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0050] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0051] The freight train inspection method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the inspection robot 102 communicates with the central control console 104 via a network. The inspection robot 102 acquires inspection environment data corresponding to the freight train to be inspected, including image data and point cloud data. Based on the image and point cloud data, path planning is performed to obtain the target inspection path from the current position of the inspection robot 102 to the target inspection location. Based on the target inspection path, the inspection robot 102 is controlled to proceed to the target inspection location, and inspection begins from there, obtaining target inspection data, which is then sent to the central control console 104. The central control console 104 can perform cloud storage, intelligent analysis, and result visualization of the target inspection data, providing reliable data support for subsequent defect identification, inspection result evaluation, and remote expert intervention.

[0052] The main control console 104 can be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0053] In one exemplary embodiment, such as Figure 2 As shown, a method for inspecting freight trains is provided, which can be applied to... Figure 1 Taking the controller or processor in the inspection robot 102 as an example, the explanation includes the following steps 202 to 206. Wherein:

[0054] Step 202: Obtain the inspection environment data corresponding to the freight train to be inspected by the inspection robot. The inspection environment data includes image data and point cloud data.

[0055] The image data was acquired by a visible light camera mounted on the inspection robot, while the point cloud data was acquired by a line lidar mounted on the inspection robot. The line lidar emits a laser beam and receives the laser beam reflected back after it illuminates the surface of an object, thereby obtaining three-dimensional information and distance information of a large number of reflection points in the surrounding environment. The collection of this information constitutes the point cloud data.

[0056] In some embodiments, before acquiring inspection environment data, the method further includes: initializing the visible light camera and the line lidar sensor, and establishing a data communication link; calibrating the spatial coordinate system of the visible light camera and the line lidar to ensure that the collected data can be uniformly mapped to the body coordinate system of the inspection robot, and then starting data acquisition, controlling the visible light camera to acquire image information, and controlling the line lidar to acquire point cloud data.

[0057] In some embodiments, the method further includes: preprocessing the raw data acquired in real time by the visible light camera and the line lidar, specifically including operations such as noise reduction, filtering, and downsampling, to obtain inspection environment data.

[0058] Step 204: Perform path planning processing based on image data and point cloud data to obtain the target inspection path from the current position of the inspection robot to the target inspection location.

[0059] The image data and point cloud data include key environmental information such as the geometry and obstacles inside the freight train carriages.

[0060] The path planning process may include: identifying and analyzing image data and point cloud data and extracting features to obtain category information of the ground, obstacles, tracks, carriages and surrounding structures in the surrounding environment; constructing an environment model that includes the spatial distribution of obstacles, the position of carriages and the boundary of the passage area; and generating an optimal path that meets the requirements of travel distance constraints, safety obstacle avoidance constraints and path smoothness based on the environment model.

[0061] Step 206: Based on the target inspection path, control the inspection robot to go to the target inspection location, start the inspection from the target inspection location, obtain the target inspection data, and send the target inspection data to the main control console.

[0062] After determining the target inspection path, the controller or processor of the inspection robot can control the chassis drive unit, inertial measurement unit, and sensors such as visible light camera and line lidar sensor of the inspection robot according to the target inspection path. Through real-time attitude calculation and path tracking algorithm, the robot’s travel speed, turning angle and acceleration and deceleration process are dynamically adjusted to ensure that the inspection robot can smoothly and safely drive to the target inspection location along the target inspection path even in complex environments.

[0063] Once the inspection begins, the system acquires real-time images, videos, or 3D data from sensors to form target inspection data, which is then transmitted back to the central control console via a 4G or 5G high-speed communication network for analysis and processing.

[0064] In the aforementioned freight train inspection method, image data and point cloud data collected by sensors mounted on the inspection robot are used to identify the appearance features and spatial location of the freight train, effectively distinguishing between tracks, carriages, and surrounding obstacles, providing accurate real-time environmental information for path planning. Based on this real-time environmental information, a safe and efficient target inspection path is planned, enabling the inspection robot to safely and smoothly reach the target inspection location to begin the inspection task and obtain the target inspection data. The freight train inspection method in this embodiment possesses high flexibility and environmental adaptability, and can reliably complete inspection tasks even when dealing with complex situations such as track obstacles, gradient changes, or limited space, thereby effectively ensuring the reliability of the freight train inspection method.

[0065] In one exemplary embodiment, based on Figure 2 The illustrated embodiments, such as Figure 3 As shown, path planning is performed based on image data and point cloud data to obtain the target inspection path from the current position of the inspection robot to the target inspection location, including:

[0066] Step 302: Using image data and point cloud data, obtain the three-dimensional pose information of the target inspection location and the target carriage where the target inspection location is located.

[0067] In one possible implementation, step 302 may further include: performing spatial clustering processing on the point cloud data to obtain multiple independent carriage areas; calculating the spatial relative distance between the inspection robot and each carriage area respectively, and taking the carriage area with the smallest spatial relative distance as the target carriage; and performing registration processing between the target carriage and a preset reference coordinate system to obtain three-dimensional pose information.

[0068] The three-dimensional pose information can include position (x, y, z) and orientation angle (roll, pitch, yaw).

[0069] In some embodiments, Euclidean clustering or semantic point cloud segmentation algorithms can be used to cluster and extract point cloud data to identify multiple independent carriage regions. Subsequently, the geometric centroid of each carriage region is calculated, or the shortest distance between points is obtained, thereby obtaining the spatial relative distance between each carriage region and the inspection robot.

[0070] The reference coordinate system in the registration process can be an existing reference model, map point cloud, or coordinate system from historical observation frames.

[0071] For example, the registration process can employ point cloud registration algorithms, such as ICP (Iterative Closest Point) or NDT (Normal Distributions Transform), to estimate the spatial rigidity transformation relationship between the target carriage and a preset reference coordinate system. The ICP algorithm iteratively finds nearest neighbor pairs between the point cloud data corresponding to the target carriage and the point cloud data based on the reference coordinate system, minimizing the sum of squared Euclidean distances between these pairs, and progressively solves for the optimal rotation matrix and displacement vector, aligning the target carriage with the reference coordinate system in three-dimensional space. The NDT algorithm divides the point cloud data corresponding to the target carriage into multiple grids, each grid using a Gaussian distribution to describe the spatial distribution characteristics of the points, and optimizes the rigid body transformation parameters by maximizing the likelihood function of the source point cloud on this Gaussian distribution.

[0072] Objective function of a point cloud registration algorithm It can be represented as:

[0073]

[0074] in, It can represent the points in the point cloud data corresponding to the target carriage. It can represent and Points in the recent point cloud data based on a reference coordinate system, where R is a preset rotation matrix and t is a preset translation variable. When When the minimum value is reached, the registration is complete, and the three-dimensional pose information can be output.

[0075] For example, the registration process can also employ geometric fitting or principal orientation analysis methods. By fitting the point cloud data corresponding to the target carriage with a cuboid or planar model, the principal axis direction and outer boundary can be estimated, thereby obtaining an approximate initial attitude estimate to improve the stability of the initial registration.

[0076] Step 304: Based on the three-dimensional pose information, image data and point cloud data, perform synchronous localization and mapping processing to obtain the target inspection path.

[0077] Among them, synchronous localization and mapping processing determines the precise distance and distribution information of obstacles in the surrounding environment based on three-dimensional pose information, image data and point cloud data, and generates a scene map for path planning and obstacle detection. Based on the scene map, a target inspection path to the target inspection location is planned.

[0078] In one possible implementation, step 304 may further include: performing map construction processing based on image data and point cloud data to generate a local map, and determining the self-localization of the inspection robot in the local map through feature extraction and matching; the local map includes the distribution location of obstacles; fusing and updating the local map, three-dimensional pose information and the preset preliminary inspection map to obtain the target inspection map; and performing dynamic obstacle avoidance path planning processing based on the target inspection map to obtain the target inspection path.

[0079] During map construction, dense point cloud mapping or raster mapping methods can be used to model the structures and obstacles in the surrounding environment, resulting in a local map that includes the location of obstacles. Fusion updating refers to merging and updating the constructed local map with the existing inspection task map based on image data and point cloud data, generating a detailed target inspection map that includes obstacle information.

[0080] In some embodiments, a target inspection map can be generated using a Bayesian probability formula for updating map grids, the expression of which is:

[0081]

[0082] in, This indicates the probability that a grid cell was occupied before the update, derived from the existing inspection mission map. Let represent the i-th grid cell in the map, which is a binary random variable. Sensor observation data representing time t can include point cloud data and image data. This represents the three-dimensional pose information of the inspection robot at time t. All sensor observation data from time 1 to t, This represents all three-dimensional pose information from time 1 to t.

[0083] In some embodiments, the target inspection map can be updated based solely on image data or solely on point cloud data.

[0084] Among them, dynamic obstacle avoidance path planning refers to the process of calling a dynamic obstacle avoidance path planning algorithm, such as a path planning algorithm based on deep reinforcement learning, based on the obstacle distribution of the target inspection map, the position of the target carriage and the three-dimensional pose information of the inspection robot, and comprehensively considering the path length, obstacle avoidance cost and motion stability, to calculate the optimal inspection path for the inspection robot to go from the current position to the target carriage as the target inspection path.

[0085] In this embodiment, the target carriage is registered with a preset reference coordinate system to obtain three-dimensional pose information, providing a basic spatial reference for obtaining the target inspection path. Image data and point cloud data are fused during the generation of the target inspection path to ensure positioning and navigation accuracy. Even in environments without external positioning signals, accurate navigation and positioning can be achieved, thereby improving the environmental adaptability and reliability of the freight train inspection method.

[0086] In one exemplary embodiment, based on Figure 2 In the embodiment shown, after the inspection robot arrives at the target inspection location along the target inspection path, it starts the inspection from the target inspection location and obtains the target inspection data, including: performing image acquisition, video recording or 3D point cloud scanning operations according to the task type corresponding to the inspection task to obtain preliminary inspection data.

[0087] Perform at least one of the following preprocessing steps on the preliminary inspection data to obtain the target inspection data:

[0088] (1) Perform image compression processing on the preliminary inspection data obtained from the image acquisition to obtain the target inspection data.

[0089] (2) Perform video segmentation encoding on the preliminary inspection data obtained from the video recording to obtain the target inspection data.

[0090] (3) Perform point cloud format conversion processing on the preliminary inspection data obtained by performing three-dimensional point cloud scanning to obtain target inspection data.

[0091] In this embodiment of the application, the collected preliminary inspection data is preprocessed to reduce data volume and improve transmission efficiency.

[0092] In some embodiments, during the process of acquiring preliminary inspection data through image acquisition, it is necessary to map the three-dimensional coordinates to the camera coordinate system using the camera intrinsic parameter matrix. The intrinsic parameter matrix can be expressed as:

[0093]

[0094] in, This indicates the camera's focal length in the horizontal direction. This indicates the camera's focal length in the vertical direction. Represents the x-coordinate of the principal point. This represents the y-coordinate of the principal point. An intrinsic parameter matrix can be used to establish a one-to-one correspondence between the image domain and the spatial domain.

[0095] In some embodiments, the method further includes: during the inspection process, adjusting the angles of the robot's leg joints in conjunction with the terrain information fed back by the lidar, so as to achieve adaptive adjustment to irregular track surfaces or obstacles, and ensure the movement stability and safety of the inspection robot.

[0096] In one possible implementation, the method further includes: during the inspection process, acquiring the gaze vector of the image acquisition device of the inspection robot; performing normal estimation calculation on the point cloud data to obtain the local normal vector; performing dot product calculation on the gaze vector and the local normal vector to obtain the vector angle; and controlling the inspection robot to adjust the gaze direction according to the vector angle so that the gaze vector and the local normal vector remain perpendicular.

[0097] The normal estimation calculation can utilize a normal estimation algorithm, treating the neighborhood around each point as a small plane, and approximating the surface orientation using the principal direction of the covariance matrix of the neighborhood point set. The covariance matrix C can be expressed as:

[0098]

[0099] Where n represents the number of point clouds, This represents the i-th point in the point cloud data. express The mean of the data corresponding to the neighborhood point set is used to perform eigenvalue decomposition on the covariance matrix C, obtaining the minimum eigenvalue. The eigenvector corresponding to the minimum eigenvalue is the surface normal vector direction of that neighborhood point set. Repeating the above process for each sampling point in the point cloud data yields a set of normal vectors, which can then be used to determine the local normal vector.

[0100] The image acquisition device can be a visible light camera or other vision device used for inspection tasks. The gaze vector can be the Z-axis direction in the camera coordinate system, representing the direction the camera is shooting; this direction is the gaze direction of the image acquisition device. The gaze vector is extracted from the extrinsic parameter matrix of the image acquisition device. The gaze vector and the local normal vector are then multiplied by a dot product. Assuming the gaze vector is v, the local normal vector is n, and the angle between the vectors is... Given by the following formula:

[0101]

[0102] When the included vector angle meets preset angle conditions, the posture rotation of the line of sight can be achieved through the joint control system of the inspection robot or the independent gimbal mechanism of the image acquisition device. This ensures that the local normal vector of the line of sight vector of the image acquisition device remains as perpendicular as possible, thereby improving the clarity and depth perception accuracy of the acquired image. For example, the angle condition can be that the included vector angle is greater than a preset angle threshold, or that it conforms to a preset angle range.

[0103] In this embodiment, during the inspection task, the angle between the line-of-sight vector and the local normal vector of the image acquisition device is continuously calculated using the cosine theorem. This allows the line-of-sight direction of the image acquisition device to be controlled in real time to be perpendicular to the surface of the target carriage. The previously acquired carriage normal vector and camera posture information ensure that the camera's line of sight remains perpendicular to the carriage surface, thereby effectively improving the clarity, edge contrast, and distinguishability of minor defects in the target inspection data, and ensuring the accuracy and robustness of subsequent detection algorithms.

[0104] In one exemplary embodiment, after collecting target inspection data, the inspection robot classifies and encapsulates different types of inspection data, such as images, videos, and point clouds, to generate a task data package with a unified format. The encapsulated data is uploaded to a remote central control console via a built-in wireless communication module. During the upload process, the data transmission status and feedback information are recorded to ensure the integrity and reliability of the uploaded data, providing support for subsequent fault detection and decision analysis.

[0105] The wireless communication module supports 4G / 5G cellular networks or Wi-Fi 6 protocol, providing low-latency, high-bandwidth real-time transmission capabilities to meet the remote communication needs of edge devices in railway scenarios. The wireless communication module can also employ encrypted communication protocols, such as TLS (Transport Layer Security) or VPN (Virtual Private Network) channels, to ensure the integrity and security of data transmission.

[0106] In one exemplary embodiment, such as Figure 4 As shown, a method for inspecting freight trains is provided, which can be applied to... Figure 1 Taking the inspection robot 102 as an example, the process includes the following steps 401 to 408. Wherein:

[0107] Step 401: Obtain the inspection environment data corresponding to the freight train to be inspected by the inspection robot. The inspection environment data includes image data and point cloud data.

[0108] Step 402: Perform spatial clustering on the point cloud data to obtain multiple independent carriage areas.

[0109] Step 403: Calculate the relative spatial distance between the inspection robot and each carriage area, and select the carriage area with the smallest relative spatial distance as the target carriage.

[0110] Step 404: Using the image data and point cloud data corresponding to the target carriage, the target carriage is registered with the preset reference coordinate system to obtain three-dimensional pose information.

[0111] Step 405: Based on image data and point cloud data, perform map construction processing to generate a local map, and determine the self-localization of the inspection robot in the local map through feature extraction and matching.

[0112] Step 406: The local map, 3D pose information and preset preliminary inspection map are fused and updated to obtain the target inspection map.

[0113] Step 407: Based on the target inspection map, perform dynamic obstacle avoidance path planning to obtain the target inspection path.

[0114] Step 408: Based on the target inspection path, control the inspection robot to go to the target inspection location, start the inspection from the target inspection location, obtain the target inspection data, and send the target inspection data to the main control console.

[0115] In one embodiment, step 408 further includes: performing image acquisition, video recording, or 3D point cloud scanning operations according to the task type corresponding to the inspection task to obtain preliminary inspection data; performing at least one of the following preprocessing operations on the preliminary inspection data to obtain target inspection data: performing image compression processing on the preliminary inspection data obtained by performing image acquisition to obtain target inspection data; performing video segmentation encoding processing on the preliminary inspection data obtained by performing video recording to obtain target inspection data; and performing point cloud format conversion processing on the preliminary inspection data obtained by performing 3D point cloud scanning to obtain target inspection data.

[0116] In one embodiment, the method further includes: during the inspection process, acquiring the gaze vector of the image acquisition device of the inspection robot; performing normal estimation calculation on the point cloud data to obtain a local normal vector; performing dot product calculation on the gaze vector and the local normal vector to obtain the vector angle; and controlling the inspection robot to adjust the gaze direction according to the vector angle so that the gaze vector and the local normal vector remain perpendicular.

[0117] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0118] Based on the same inventive concept, this application also provides a freight train inspection device for implementing the freight train inspection method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations of one or more freight train inspection device embodiments provided below can be found in the limitations of the freight train inspection method described above, and will not be repeated here.

[0119] In one exemplary embodiment, such as Figure 5 As shown, a freight train inspection device is provided, including: a data acquisition module 502, a route planning module 504, and a signal transmission module 506, wherein:

[0120] The data acquisition module 502 is used to acquire the inspection environment data corresponding to the freight train to be inspected by the inspection robot. The inspection environment data includes image data and point cloud data.

[0121] The path planning module 504 is used to perform path planning processing based on image data and point cloud data to obtain the target inspection path from the current position of the inspection robot to the target inspection location.

[0122] The signal transmission module 506 is used to control the inspection robot to go to the target inspection location based on the target inspection path, start the inspection from the target inspection location, obtain the target inspection data, and send the target inspection data to the main control console.

[0123] In one embodiment, the path planning module 504 is further configured to use image data and point cloud data to obtain the three-dimensional pose information corresponding to the target inspection location and the target carriage where the target inspection location is located; and to perform synchronous positioning and mapping processing based on the three-dimensional pose information, image data and point cloud data to obtain the target inspection path.

[0124] In one embodiment, the path planning module 504 is further used to perform spatial clustering processing on the point cloud data to obtain multiple independent carriage areas; calculate the spatial relative distance between the inspection robot and each carriage area respectively, and take the carriage area with the smallest spatial relative distance as the target carriage; use the image data and point cloud data corresponding to the target carriage to perform registration processing with the preset reference coordinate system to obtain three-dimensional pose information.

[0125] In one embodiment, the path planning module 504 is further configured to perform map construction processing based on image data and point cloud data to generate a local map, and determine the self-localization of the inspection robot in the local map through feature extraction and matching; the local map includes the distribution location of obstacles; the local map, three-dimensional pose information and preset preliminary inspection map are fused and updated to obtain the target inspection map; based on the target inspection map, dynamic obstacle avoidance path planning processing is performed to obtain the target inspection path.

[0126] In one embodiment, the signal transmission module 506 is further configured to perform image acquisition, video recording, or 3D point cloud scanning operations according to the task type corresponding to the inspection task to obtain preliminary inspection data; and to perform at least one of the following preprocessing operations on the preliminary inspection data to obtain target inspection data: performing image compression processing on the preliminary inspection data obtained by performing image acquisition to obtain target inspection data; performing video segmentation encoding processing on the preliminary inspection data obtained by performing video recording to obtain target inspection data; and performing point cloud format conversion processing on the preliminary inspection data obtained by performing 3D point cloud scanning to obtain target inspection data.

[0127] In one embodiment, the device further includes a line-of-sight adjustment module, used to acquire the line-of-sight vector of the image acquisition device of the inspection robot during the inspection process; perform normal estimation calculation on the point cloud data to obtain the local normal vector; perform dot product calculation on the line-of-sight vector and the local normal vector to obtain the vector angle; and control the inspection robot to adjust the line-of-sight direction according to the vector angle so that the direction of the line-of-sight vector and the local normal vector remain perpendicular.

[0128] Each module in the aforementioned freight train inspection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the electronic device in hardware form or independent of it, or stored in the memory of the electronic device in software form, so that the processor can call and execute the corresponding operations of each module.

[0129] In one exemplary embodiment, an electronic device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown, the electronic device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a freight train inspection method. The display unit of the electronic device forms a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the electronic device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the electronic device, or external keyboards, touchpads, or mice, etc.

[0130] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0131] In one exemplary embodiment, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0132] In one exemplary embodiment, an inspection robot is also provided. The inspection robot is equipped with a visible light camera, a line lidar sensor, and the electronic equipment described in the above embodiment. The visible light camera and the line lidar sensor are respectively communicatively connected to the electronic equipment. The visible light camera is used to collect image data corresponding to the freight train to be inspected by the inspection robot. The line lidar sensor is used to collect point cloud data corresponding to the freight train.

[0133] In this embodiment, the inspection robot 102 can be a wheeled robot, which has high mobility and environmental adaptability, and can meet the adaptability requirements of complex terrain compared with traditional wheeled or tracked mobile platforms.

[0134] In some embodiments, upon startup of the wheeled robot, its main control system, i.e., the controller or processor, loads the operating system and motion control program and completes basic hardware self-checks. Subsequently, the configured environmental perception sensors, such as visible light cameras and line LiDAR, are activated. The visible light camera begins real-time image data acquisition to reconstruct the robot's surrounding environment; the line LiDAR simultaneously acquires horizontal or vertical distance information to form point cloud data for precise detection of obstacle locations and distributions. The data acquired by these sensors covers key environmental information such as the geometry of the freight train carriages and obstacle distributions. Simultaneously with sensor data acquisition, various data types are synchronized in time and registered spatially, and unified to the robot's body coordinate system through coordinate transformation. This provides accurate and continuous environmental perception support for subsequent mapping, navigation planning, and dynamic obstacle recognition, generating a target inspection path.

[0135] The inspection robot travels along the target inspection path to the target inspection location. Through a high-precision servo drive system and a foot force / position closed-loop control mechanism, it dynamically adjusts the support height and gait rhythm of each mechanical leg to achieve active adaptation to complex terrain and posture balance, ensuring its stable movement to the designated work position and guaranteeing its motion stability and safety.

[0136] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above-described method embodiments.

[0137] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0138] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0139] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0140] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method of inspecting a freight train, characterized by, The method comprises: acquiring inspection environment data corresponding to a freight train to be inspected by an inspection robot, the inspection environment data comprising image data and point cloud data; performing path planning processing according to the image data and the point cloud data to obtain a target inspection path between a current position of the inspection robot and a target inspection location; controlling the inspection robot to go to the target inspection location based on the target inspection path, starting inspection from the target inspection location, obtaining target inspection data, and sending the target inspection data to a general control console.

2. The method of claim 1, wherein, The path planning processing according to the image data and the point cloud data to obtain a target inspection path between a current position of the inspection robot and a target inspection location comprises: obtaining three-dimensional pose information corresponding to the target inspection location and a target carriage where the target inspection location is located by using the image data and the point cloud data; performing simultaneous localization and mapping processing based on the three-dimensional pose information, the image data and the point cloud data to obtain the target inspection path.

3. The method of claim 2, wherein, The three-dimensional pose information corresponding to the target carriage where the target inspection location is located is obtained by using the image data and the point cloud data, comprising: performing spatial clustering processing on the point cloud data to obtain a plurality of mutually independent carriage areas; calculating the spatial relative distance between the inspection robot and each of the carriage areas respectively, and taking the carriage area with the smallest spatial relative distance as the target carriage; performing registration processing on the target carriage and a preset reference coordinate system by using the image data and the point cloud data corresponding to the target carriage to obtain the three-dimensional pose information.

4. The method of claim 2, wherein, The simultaneous localization and mapping processing based on the three-dimensional pose information, the image data and the point cloud data to obtain the target inspection path comprises: performing map construction processing based on the image data and the point cloud data to generate a local map, and determining self-localization of the inspection robot in the local map through feature extraction and matching; the local map comprises obstacle distribution positions; fusing and updating the local map, the three-dimensional pose information and a preset preliminary inspection map to obtain a target inspection map; performing dynamic obstacle avoidance path planning processing based on the target inspection map to obtain the target inspection path.

5. The method of claim 1, wherein, The inspection starting from the target inspection location to obtain target inspection data comprises: performing image acquisition, video recording or three-dimensional point cloud scanning operations according to a task type corresponding to an inspection task to obtain preliminary inspection data; performing at least one of the following preprocessing on the preliminary inspection data to obtain the target inspection data: performing image compression processing on the preliminary inspection data obtained by performing image acquisition to obtain the target inspection data; performing video segmentation and encoding processing on the preliminary inspection data obtained by performing video recording to obtain the target inspection data; performing point cloud format conversion processing on the preliminary inspection data obtained by performing three-dimensional point cloud scanning to obtain the target inspection data.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: acquiring a line-of-sight vector of an image acquisition device of the inspection robot during the inspection process; The point cloud data is subjected to normal estimation calculation processing to obtain a local normal vector; The line-of-sight vector and the local normal vector are subjected to dot product calculation processing to obtain a vector included angle; The vector included angle is used to control the line-of-sight direction of the inspection robot, so that the line-of-sight vector and the local normal vector are perpendicular to each other.

7. A freight train inspection device, characterized by, The device comprises: A data acquisition module is configured to acquire inspection environment data of a freight train to be inspected by an inspection robot, wherein the inspection environment data comprises image data and point cloud data; A path planning module is configured to perform path planning processing based on the image data and the point cloud data to obtain a target inspection path from a current position of the inspection robot to a target inspection location; A signal transmission module is configured to control the inspection robot to go to the target inspection location based on the target inspection path, and to perform inspection starting from the target inspection location to obtain target inspection data, and to send the target inspection data to a general control console. 8.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 6.

9. A patrol robot characterized by comprising: The inspection robot is provided with a visible light camera, a line laser radar sensor, and an electronic device as claimed in claim 8, wherein the visible light camera and the line laser radar sensor are in communication connection with the electronic device; The visible light camera is configured to collect image data of a freight train to be inspected by the inspection robot; The line laser radar sensor is configured to collect point cloud data of the freight train.

10. The patrol robot according to claim 9, wherein, The inspection robot is a wheeled robot.

Images