Immersive remote maintenance system applied to railway unattended relay station

Abstract

The invention discloses an immersive remote maintenance system applied to a railway unattended relay station, which relates to the technical field of remote maintenance and comprises a sound source point identification unit, a position output unit, a model construction unit, a sound analysis unit, an equipment joint analysis unit, a record information generation unit and a fault verification unit. According to the invention, an equipment joint analysis unit is used for further analyzing the actual position of the sound and the sound generation reason of surrounding equipment if the generation reason does not belong to normal operation in the sound recognition process due to the fact that the sound source point of the equipment possibly has interference noise; a timer is added for the situation that the actual position is too far away from the equipment and other equipment has the same generation reason, subsequent detailed sound detection is carried out, it is guaranteed that the possibility of abnormal false alarm is greatly reduced, the inspection robot is remotely controlled through the fault verification unit, equipment fault verification is facilitated, and a maintainer can make a plan in advance conveniently.

Description

Technical Field

[0001] This invention relates to the field of remote maintenance technology, specifically to an immersive remote maintenance system applied to unmanned railway relay stations. Background Technology

[0002] An immersive remote maintenance system for unmanned railway relay stations utilizes IoT technology to provide real-time remote operation and maintenance support. It uses high-definition cameras and sensors to collect equipment status data, allowing maintenance personnel to immerse themselves in the inspection process via a robot, viewing fault points and receiving auxiliary diagnostic prompts. This effectively improves fault handling efficiency and ensures the stable operation of critical railway facilities. Patent application number 202410035988.9 discloses "An inspection method and system based on an intelligent inspection terminal," relating to the field of inspection technology. The inspection method based on the intelligent inspection terminal includes: when the intelligent inspection terminal is in contact with the target equipment to be inspected... When establishing a short-range communication connection, the intelligent inspection terminal acquires the characteristic information of the target device; it generates a verification request to authenticate the inspection personnel; it takes a picture of the target device and obtains a real image of the inspected area; it compares the real image with a stereoscopic image; based on the operation process, it determines the completion progress of the inspection project; when the short-range communication connection with the target device is broken, the intelligent inspection terminal generates a data packet, which includes characteristic information, the inspection personnel's identity information, the inspection project, and the inspection record. This method ensures that the inspection process is completed in an orderly and accurate manner, and reduces the probability of missed inspections and misoperations.

[0003] The aforementioned existing technologies have solved the problem of requiring staff to be on-site and conduct relevant inspections of outdoor equipment using the naked eye and professional testing instruments. However, when the system is running, it fails to analyze abnormal sound signals emitted by the equipment, resulting in some faults not being identified in a timely manner. Furthermore, the lack of multiple inspection modes may lead to false detections and missed detections during the inspection process. The uniform inspection procedures are difficult to apply to different types of equipment and lack rationality. Summary of the Invention

[0004] The purpose of this invention is to provide an immersive remote maintenance system for unmanned railway relay stations, in order to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an immersive remote maintenance system for unmanned railway relay stations, comprising a sound analysis unit, an equipment joint analysis unit, a record information generation unit, and a fault verification unit;

[0006] The sound source identification unit determines the sound source point of each device in the designated relay station, deploys corresponding visual reference tags at all sound source point locations, acquires global positioning data, initializes local positioning data and the actual positions of the four corners of the tags, determines the estimated position of the inspection robot at the current moment and the acquired point cloud data, analyzes the relative positions of the four corners of the visual reference tags, and calculates the adjustment matrix using the actual position and relative position.

[0007] The position output unit, after acquiring the camera intrinsic parameter matrix, calculates the optimized estimated position of the current inspection robot based on the camera intrinsic parameter matrix and the adjustment matrix. It then uses the estimated position to match all points in the current point cloud data with the points in the local positioning data, thereby calculating the actual position of the current inspection robot. Finally, it constructs new local positioning data based on the actual position and the point cloud data.

[0008] The model building unit determines the cause of sound signal generation for different types of devices within a specified relay station. It then retrieves the corresponding sound signals from the device voiceprint database according to the device type and cause. After determining the feature vector and cause of the sound signals, it stores them as samples in a set. The samples are then transmitted to RNN, LSTM, and CNN models to determine the model parameters, performance coefficients, and basic weights. This process is repeated until all types of devices have corresponding RNN, LSTM, and CNN models.

[0009] Preferably, the sound source identification unit includes a tag location determination module, a point cloud data acquisition module, an image acquisition module, and a coefficient matrix construction module. The tag location determination module determines the sound source point of each device within a specified relay station and deploys corresponding visual reference tags c1, c2, ..., c at all sound source point locations. i ,...,c n ,in i represents the tag number, and O represents the tag sequence number. These represent the actual positions of the top-left, top-right, bottom-left, and bottom-right corners corresponding to the i-th visual reference label, respectively. This represents the actual x-coordinate of the top-left corner corresponding to the i-th visual reference label. This represents the actual ordinate of the top-left corner corresponding to the i-th visual reference label. After acquiring global positioning data and initializing local positioning data, the point cloud data acquisition module determines the estimated position of the inspection robot at the current moment and the acquired point cloud data, and projects the point cloud data into the local positioning data. The global positioning data includes the position information of all point cloud data and visual reference labels within the currently specified relay station, and the initialized local positioning data includes the position information of the point cloud data collected by the inspection robot at the starting position. The image acquisition module calculates the distance between the estimated position and the positions of all visual reference labels in the global positioning data. If the distance is lower than a threshold, the corresponding visual reference label image information is acquired using the camera mounted on the inspection robot; otherwise, the image acquisition operation is not performed. If the coefficient matrix construction module acquires the image information of the visual reference label, it analyzes the relative positions of the four corners of the current visual reference label based on the image information. in This represents the top-left corner relative to the x-coordinate of the i-th visual reference label. This represents the relative ordinate of the top-left corner of the i-th visual reference label, based on... and Set the corresponding adjustment matrix D, where use and Calculate the coefficient matrix in the upper left corner. in Repeat the operation until all coefficient matrices are analyzed, and then combine them to obtain the actual values ​​of each parameter in the adjustment matrix D.

[0010] Preferably, the position output unit includes an estimated position optimization module and a transformation matrix calculation module. After obtaining the camera intrinsic parameter matrix, the estimated position optimization module calculates the first transformation matrix α and the second transformation matrix β based on the camera intrinsic parameter matrix and the adjustment matrix D, where α = (α0, α1, α2). α0, α1, α2 represent the parameter values ​​in the first transformation matrix, β 11 ,β 12 ,β 13 ,β 21 ,β 22 ,β 23 ,β 31 ,β 32 ,β 33 The parameter values ​​in the second transformation matrix are represented by β, and the offset angle θ at the current time is calculated using β. The optimized estimated position L of the current inspection robot is output using α and θ, where L = (α0, α1, θ). The transformation matrix calculation module uses the estimated position to match all points in the current point cloud data with the points in the local positioning data, calculates the first transformation matrix and the second transformation matrix corresponding to the point with the minimum distance between points, and deduces the actual position of the current inspection robot based on the first transformation matrix and the second transformation matrix.

[0011] Preferably, the position output unit further includes a number reading module, a record storage module, and a data matching module. The number reading module reads the historical path set stored by the robot, determines the number of the visual reference label, and if the number is the same as the number in the historical path set, it extracts the position corresponding to the historical record number, calculates the difference between the position and the current actual position, and optimizes the actual position corresponding to all historical record numbers based on the difference. The record storage module constructs new local positioning data based on the actual position and point cloud data, and stores the original local positioning data, actual position, visual reference label number, four-corner position, and acquisition time as historical records in the historical path set. The visual reference label number is the number of the current historical record. If the data matching module does not acquire the image information of the visual reference label, it directly matches the point cloud data with the local positioning data using the unoptimized estimated position, thereby outputting the current actual position of the inspection robot.

[0012] Preferably, the model building unit includes a signal extraction module, a signal processing module, a feature vector determination module, a performance coefficient calculation module, and a weight setting module. The signal extraction module determines the j-th type of device S within a specified relay station. j The corresponding sound signal is generated by s j,1 ,s j,2 ,...,s j,mThen, where j represents the device type number and m represents the total number of causes of sound signal generation for that type of device, the corresponding sound signal is retrieved from the device voiceprint database according to the device type and cause. The signal processing module performs pre-emphasis, framing, and windowing operations on the sound signal, and then performs a Fourier transform to obtain the frequency domain energy spectrum. The feature vector determination module analyzes the frequency domain energy spectrum through a Mel filter bank to determine the energy of each frequency band. After extracting the multidimensional features of the current sound signal based on the energy of each frequency band, it uses principal component analysis to reduce the dimensionality and obtain the final feature vector of the sound signal. The performance coefficient calculation module determines the feature vectors and causes of all sound signals, stores them as samples in a set, and divides the samples in the set into a training set and a test set in a 7:3 ratio. The samples in the training set are then transmitted to the RNN, LSTM, and CNN models. After determining the model parameters, the performance coefficients γ1, γ2, and γ3 of each model are calculated using the samples in the test set. The weight setting module analyzes γ1, γ2, and γ3 to determine the basic weights ω1, ω2, and ω3 of the RNN, LSTM, and CNN models. Repeat the process until all types of devices have corresponding RNN, LSTM, and CNN models.

[0013] Preferably, the sound analysis unit includes a mode division module, a signal upload module, and a prediction result calculation module. The mode division module divides the inspection robot's working mode into two types: routine inspection and special inspection. When the inspection robot is in the routine inspection working mode, the signal upload module collects the corresponding visual reference label image information, obtains the current device type based on the label image information, acquires the real-time sound signal of the current sound source point using the microphone array on the inspection robot, and uploads both the real-time sound signal and the device type to the edge server. The prediction result calculation module retrieves the corresponding RNN model, LSTM model, and CNN model from the edge server based on the device type, determines the feature vector of the real-time sound signal, and inputs it into the three models for analysis. Based on the model prediction results and the corresponding basic weights, the module optimizes the corresponding weight coefficients, and uses a sound recognition algorithm to calculate the optimized weight coefficients and prediction results, thereby outputting the cause of the current real-time sound signal. The sound recognition algorithm is specifically as follows:

[0014]

[0015] Where F represents the final predicted output, P κ ω represents the prediction result of the κ-th model. κ C represents the basic weights of the κ-th model. κLet λ represent the prediction confidence of the κ-th model, where κ represents the model number and λ represents the adjustment coefficient.

[0016] Preferably, the device joint analysis unit includes a normal operation determination module, a distance calculation module, a candidate set generation module, a similarity determination module, and a priority coefficient comparison module. If the normal operation determination module determines that the real-time signal is generated due to normal operation, it uploads the current device number, the cause of sound generation, and the acquisition time to the edge server, and then searches for the next sound source point according to the visual reference label number order. If the distance calculation module determines that the real-time signal is not generated due to normal operation, it uses MVDR technology to calculate the actual sound source location. The distance d between the actual sound source location and the sound source point is then used to determine the location of the sound source. v If the distance exceeds the threshold, a timer is started, and a special inspection is conducted after the timer expires. When the distance between the actual sound source location and the sound source point is less than or equal to the threshold, the candidate set generation module reads visual reference label image information from other devices within five meters of the current sound source point and collects the corresponding sound signal. The sound signals are analyzed to determine the cause of each sound signal. The feature vectors of sound signals whose cause is not normal operation are combined with the device number and stored in the candidate set. The similarity determination module calculates the similarity of the sound signals u1, u2, ..., u in the candidate set. M The similarity sim1,...,sim between the feature vector and the feature vector of the sound signal v at the current sound source point t ,...,sim M Set the preset value μ, and determine u1, u2, ..., u M The distances d1, d2, ..., d between the corresponding sound source points and the current actual sound source locations are... t ,...,d M If sim exists t >μ and d t <d v If a timer is started, a special inspection will be conducted after the timer expires. Here, M represents the total number of sound signals in the candidate set, and t represents the signal sequence number. If the priority coefficient comparison module is not present, then u1,...,u... that have the same cause as the sound signal v will be selected. N According to u1,...,u N After determining the historical failure rate and distance between the corresponding device number and the actual sound source location, a coefficient analysis algorithm is used to analyze it and obtain the priority coefficient of each sound signal. If the priority coefficient of v is higher than or equal to u1,...,u NWhere N represents the total number of filtered sound signals, the current device number, the cause of the sound, the actual location information of the visual reference tag, and the acquisition time are uploaded to the edge server as recorded information; otherwise, a timer is started, and a special inspection is carried out after the timer expires.

[0017] Preferably, the recording information generation unit includes a route generation module and a special inspection module. The route generation module searches for the next visual reference label according to the label order. When the device timeout ends, the inspection robot is in a special inspection and directly generates the shortest route based on the current position and the position of the visual reference label corresponding to the device whose timeout has ended. After the inspection robot arrives according to the route, the special inspection module randomly selects five different locations within the device, reads and analyzes the sound signals at the five locations and the sound source point, determines the cause of the sound generation at different locations, takes the prediction result with the highest confidence as the actual cause of the sound source point of the current device, and generates the corresponding recording information.

[0018] Preferably, the fault verification unit includes an abnormal receiving module and a manual verification module. After the edge server receives the record information of all abnormal operation of the devices, the abnormal receiving module reports the device number, the visual reference tag position corresponding to the abnormal category, and so on. After receiving the record information, the manual verification module manually controls the inspection robot to verify the fault of the abnormal operation of the devices.

[0019] Compared with the prior art, the beneficial effects of the present invention are:

[0020] 1. This invention deploys corresponding sound source points for each device through a sound source point identification unit and a position output unit, and adds visual reference labels to the sound source points. This ensures that the inspection robot can infer its current estimated position based on the label position during movement. This design makes the final actual position more accurate. At the same time, when passing the same reference label again, the historical position information is automatically corrected. The model building unit builds corresponding RNN, LSTM and CNN models according to the device type, and sets basic weights according to the actual performance coefficients of each model to ensure that the joint analysis results are more accurate. When the sound analysis unit analyzes the cause, it needs to optimize the weight coefficients to ensure that the model with high accuracy has the largest weight and the model with low accuracy has the smallest weight. As the real-time data changes, the weight coefficients will also be dynamically adjusted to ensure that the prediction results are more in line with reality. Furthermore, the data is stored on the edge server, which speeds up the overall operation.

[0021] 2. This invention utilizes a device joint analysis unit to detect potential interference noise at the sound source of the device. During the sound recognition process, if the cause is not normal operation, the actual location of the sound and the cause of the sound from surrounding devices are further analyzed. For cases where the actual location is too far from the device and other devices have the same cause, a timer is added for subsequent detailed sound detection, greatly reducing the possibility of false alarms. The fault verification unit remotely controls the inspection robot, facilitating equipment fault verification and enabling maintenance personnel to make advance plans. Attached Figure Description

[0022] Figure 1 A schematic diagram of the overall system flow is provided for embodiments of the present invention;

[0023] Figure 2 This is an internal module block diagram of the sound source identification unit provided in an embodiment of the present invention;

[0024] Figure 3 This is an internal module block diagram of the position output unit provided in an embodiment of the present invention;

[0025] Figure 4 This is an internal module block diagram of the model building unit provided in an embodiment of the present invention;

[0026] Figure 5 This is an internal module block diagram of the device joint analysis unit provided in an embodiment of the present invention;

[0027] Figure 6 This is an internal module block diagram of the information recording generation unit provided in an embodiment of the present invention.

[0028] In the diagram: 1. Sound source identification unit; 101. Tag location determination module; 102. Point cloud data acquisition module; 103. Image acquisition module; 104. Coefficient matrix construction module; 2. Location output unit; 201. Estimated location optimization module; 202. Transformation matrix calculation module; 203. Number reading module; 204. Record storage module; 205. Data matching module; 3. Model construction unit; 301. Signal extraction module; 302. Signal processing module; 303. Feature vector determination module; 304. Performance coefficient calculation module; 30 5. Weight setting module; 4. Sound analysis unit; 401. Pattern division module; 402. Signal upload module; 403. Prediction result calculation module; 5. Equipment joint analysis unit; 501. Normal operation judgment module; 502. Distance calculation module; 503. Candidate set generation module; 504. Similarity determination module; 505. Priority coefficient comparison module; 6. Record information generation unit; 601. Route generation module; 602. Special inspection module; 7. Fault verification unit; 701. Abnormal reception module; 702. Manual verification module. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Example:

[0031] Please see Figures 1-6 The present invention provides a technical solution: an immersive remote maintenance system for unmanned railway relay stations, including a sound analysis unit 4, an equipment joint analysis unit 5, a record information generation unit 6 and a fault verification unit 7;

[0032] Sound source identification unit 1 determines the sound source point of each device in the designated relay station, deploys corresponding visual reference tags at all sound source point locations, acquires global positioning data, initializes local positioning data and the actual positions of the four corners of the tags, determines the estimated position of the inspection robot at the current moment and the acquired point cloud data, analyzes the relative positions of the four corners of the visual reference tags, and calculates the adjustment matrix using the actual position and relative position.

[0033] Position output unit 2: After obtaining the camera intrinsic parameter matrix, position output unit 2 calculates the optimized estimated position of the current inspection robot based on the camera intrinsic parameter matrix and adjustment matrix. It then uses the estimated position to match all points in the current point cloud data with the points in the local positioning data, thereby calculating the actual position of the current inspection robot. Finally, it constructs new local positioning data based on the actual position and point cloud data.

[0034] Model building unit 3 determines the cause of sound signal generation for different types of devices within a specified relay station. It then retrieves the corresponding sound signals from the device voiceprint database according to the device type and cause of generation. After determining the feature vector and cause of generation of the sound signals, it stores them as samples in a set. The samples are then transmitted to RNN, LSTM, and CNN models to determine the model parameters, performance coefficients, and basic weights. This process is repeated until all types of devices have corresponding RNN, LSTM, and CNN models.

[0035] The sound source identification unit 1 includes a tag location determination module 101, a point cloud data acquisition module 102, an image acquisition module 103, and a coefficient matrix construction module 104. The tag location determination module 101 determines the sound source point of each device within a specified relay station and deploys corresponding visual reference tags c1, c2, ..., c at all sound source point locations.i ,...,c n ,in i represents the tag number, and O represents the tag sequence number. These represent the actual positions of the top-left, top-right, bottom-left, and bottom-right corners corresponding to the i-th visual reference label, respectively. This represents the actual x-coordinate of the top-left corner corresponding to the i-th visual reference label. This represents the actual ordinate of the top-left corner corresponding to the i-th visual reference label. After acquiring global positioning data and initializing local positioning data, the point cloud data acquisition module 102 determines the estimated position of the inspection robot at the current moment and the acquired point cloud data, and projects the point cloud data into the local positioning data. The global positioning data includes all point cloud data within the currently specified relay station and the position information of the visual reference labels. The initialized local positioning data includes the position information of the point cloud data collected by the inspection robot at the starting position. The image acquisition module 103 calculates the distance between the estimated position and the positions of all visual reference labels in the global positioning data. If the value is lower than the threshold, the corresponding visual reference label image information is acquired using the camera mounted on the inspection robot; otherwise, the image acquisition operation is not performed. If the image information of the visual reference label is acquired, the coefficient matrix construction module 104 analyzes the relative positions of the four corners of the current visual reference label based on the image information. in This represents the top-left corner relative to the x-coordinate of the i-th visual reference label. This represents the relative ordinate of the top-left corner of the i-th visual reference label, based on... and Set the corresponding adjustment matrix D, where use and Calculate the coefficient matrix in the upper left corner. in Repeat the operation until all coefficient matrices are analyzed, and then combine them to obtain the actual values ​​of each parameter in the adjustment matrix D.

[0036] Position output unit 2 includes a position estimation optimization module 201 and a transformation matrix calculation module 202. After obtaining the camera intrinsic parameter matrix, the position estimation optimization module 201 calculates the first transformation matrix α and the second transformation matrix β based on the camera intrinsic parameter matrix and the adjustment matrix D, where α = (α0, α1, α2). α0, α1, α2 represent the parameter values ​​in the first transformation matrix, β 11 ,β 12 ,β 13 ,β 21 ,β 22 ,β 23 ,β31 ,β 32 ,β 33 The parameter values ​​in the second transformation matrix are represented by β, and the offset angle θ at the current time is calculated using β. The optimized estimated position L of the current inspection robot is output using α and θ, where L = (α0, α1, θ). The transformation matrix calculation module 202 uses the estimated position to match all points in the current point cloud data with the points in the local positioning data, calculates the first transformation matrix and the second transformation matrix corresponding to the point with the minimum distance between points, and deduces the actual position of the current inspection robot based on the first transformation matrix and the second transformation matrix.

[0037] The position output unit 2 also includes a number reading module 203, a record storage module 204, and a data matching module 205. The number reading module 203 reads the historical path set stored by the robot and determines the number of the visual reference label. If the number is the same as the number in the historical path set, the position corresponding to the historical record number is extracted, and the difference between the position and the current actual position is calculated. The actual position corresponding to all historical record numbers is optimized based on the difference. The record storage module 204 constructs new local positioning data based on the actual position and point cloud data. The original local positioning data, actual position, visual reference label number, four corner positions, and acquisition time are stored as historical records in the historical path set. The visual reference label number is the number of the current historical record. If the data matching module 205 does not acquire the image information of the visual reference label, it directly uses the unoptimized estimated position to match the point cloud data with the local positioning data, thereby outputting the current actual position of the inspection robot.

[0038] Model building unit 3 includes a signal extraction module 301, a signal processing module 302, a feature vector determination module 303, a performance coefficient calculation module 304, and a weight setting module 305. The signal extraction module 301 determines the j-th type of device S in a specified relay station. j The corresponding sound signal is generated by s j,1 ,s j,2 ,...,s j,mAfterwards, where j represents the device type number and m represents the total number of causes of sound signal generation for that type of device, the corresponding sound signal is retrieved from the device voiceprint database according to the device type and cause. The signal processing module 302 performs pre-emphasis, framing, and windowing operations on the sound signal, and then performs a Fourier transform to obtain the frequency domain energy spectrum. The feature vector determination module 303 analyzes the frequency domain energy spectrum using a Mel filter bank to determine the energy of each frequency band. Based on the energy of each frequency band, it extracts the multidimensional features of the current sound signal and then uses principal component analysis to reduce its dimensionality, obtaining the final feature vector of the sound signal. After determining the feature vectors and causes of all sound signals, the performance coefficient calculation module 304 stores them as samples in a set. The samples in the set are divided into a training set and a test set in a 7:3 ratio. The samples in the training set are transmitted to the RNN model, LSTM model, and CNN model. After determining the model parameters, the performance coefficients γ1, γ2, and γ3 of each model are calculated using the samples in the test set. The weight setting module 305 analyzes γ1, γ2, and γ3 to determine the basic weights ω1, ω2, and ω3 of the RNN model, LSTM model, and CNN model. Repeat the operation until all types of devices have corresponding RNN, LSTM, and CNN models;

[0039] The sound analysis unit 4 includes a mode division module 401, a signal upload module 402, and a prediction result calculation module 403. The mode division module 401 divides the inspection robot's working mode into two types: routine inspection and special inspection. When the inspection robot is in the routine inspection mode, the signal upload module 402 collects the corresponding visual reference label image information, obtains the current device type based on the label image information, acquires the real-time sound signal of the current sound source point using the microphone array on the inspection robot, and uploads both the real-time sound signal and the device type to the edge server. The prediction result calculation module 403 retrieves the corresponding RNN model, LSTM model, and CNN model from the edge server based on the device type, determines the feature vector of the real-time sound signal, and inputs it into the three models for analysis. Based on the model prediction results and the corresponding basic weights, it optimizes the corresponding weight coefficients and uses a sound recognition algorithm to calculate the optimized weight coefficients and prediction results, thereby outputting the cause of the current real-time sound signal. The sound recognition algorithm is as follows:

[0040]

[0041] Where F represents the final predicted output, P κ ω represents the prediction result of the κ-th model. κ C represents the basic weights of the κ-th model. κLet λ represent the prediction confidence of the κ-th model, where κ represents the model number and λ represents the adjustment coefficient.

[0042] The device joint analysis unit 5 includes a normal operation determination module 501, a distance calculation module 502, a candidate set generation module 503, a similarity determination module 504, and a priority coefficient comparison module 505. If the cause of the real-time signal generation is normal operation, the normal operation determination module 501 uploads the current device number, the cause of sound generation, and the acquisition time to the edge server, and then searches for the next sound source point according to the visual reference label number order. If the cause of the real-time signal generation is not normal operation, the distance calculation module 502 uses MVDR technology to calculate the actual sound source location. The distance d between the actual sound source location and the sound source point is then used to determine the location of the sound source. v If the distance exceeds the threshold, a timer is started, and a special inspection is conducted after the timer expires. The candidate set generation module 503 reads visual reference label image information from other devices within five meters of the current sound source point when the distance between the actual sound source location and the sound source point is less than or equal to the threshold, and collects the corresponding sound signal. The sound signals are analyzed to determine the cause of each sound signal. The feature vectors of sound signals whose cause is not normal operation are combined with the device number and stored in the candidate set. The similarity determination module 504 calculates the similarity of the sound signals u1, u2, ..., u in the candidate set. M The similarity sim1,...,sim between the feature vector and the feature vector of the sound signal v at the current sound source point t ,...,sim M Set the preset value μ, and determine u1, u2, ..., u M The distances d1, d2, ..., d between the corresponding sound source points and the current actual sound source locations are... t ,...,d M If sim exists t >μ and d t <d v If the timer is started, a special inspection will be conducted after the timer expires. Here, M represents the total number of sound signals in the candidate set, t represents the signal sequence number, and priority coefficient comparison module 505 selects u1,...,u signals with the same cause as sound signal v. N According to u1,...,u N After determining the historical failure rate and distance between the corresponding device number and the actual sound source location, a coefficient analysis algorithm is used to analyze it and obtain the priority coefficient of each sound signal. If the priority coefficient of v is higher than or equal to u1,...,u NWhere N represents the total number of filtered sound signals, the current device number, the cause of the sound, the actual location information of the visual reference tag, and the acquisition time are uploaded to the edge server as recorded information; otherwise, a timer is started, and a special inspection is performed after the timer expires. The coefficient analysis algorithm is as follows:

[0043]

[0044] Where W represents the priority coefficient, e represents the natural constant, h represents the attenuation coefficient, and d represents the attenuation coefficient. q ε represents the distance from the actual sound source location to the device, ε represents the adjustment factor, and R represents the number of failures in the past thirty days.

[0045] The recording information generation unit 6 includes a route generation module 601 and a special inspection module 602. The route generation module 601 searches for the next visual reference label according to the label order. When the device timeout ends, the inspection robot is in a special inspection and directly generates the shortest route based on the current position and the position of the visual reference label corresponding to the device whose timeout has ended. After the inspection robot arrives according to the route, the special inspection module 602 randomly selects five different positions inside the device, reads and analyzes the sound signals at the five positions and the sound source point, determines the cause of the sound generation at different positions, takes the prediction result with the highest confidence as the actual cause of the sound source point of the current device, and generates the corresponding recording information.

[0046] The fault verification unit 7 includes an abnormal receiving module 701 and a manual verification module 702. After the edge server receives the record information of all abnormal operation of the devices, the abnormal receiving module 701 reports the device number, the category of abnormality and the corresponding visual reference label position. After receiving the record information, the manual verification module 702 controls the inspection robot to verify the fault of the abnormal operation of the devices.

[0047] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0048] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An immersive remote maintenance system for unmanned railway relay stations, comprising a sound analysis unit (4), an equipment joint analysis unit (5), a record information generation unit (6), and a fault verification unit (7), characterized in that: Sound source identification unit (1) determines the sound source point of each device in the designated relay station, deploys corresponding visual reference tags at all sound source point locations, acquires global positioning data, initializes local positioning data and the actual positions of the four corners of the tags, determines the estimated position of the inspection robot at the current moment and the acquired point cloud data, analyzes the relative positions of the four corners of the visual reference tags, and calculates the adjustment matrix using the actual position and relative position; The position output unit (2) obtains the camera intrinsic parameter matrix and calculates the optimized estimated position of the current inspection robot based on the camera intrinsic parameter matrix and the adjustment matrix. It uses the estimated position to match all points in the current point cloud data with the points in the local positioning data, thereby calculating the actual position of the current inspection robot. It constructs new local positioning data based on the actual position and the point cloud data. The model building unit (3) determines the cause of sound signal generation for different types of equipment in a specified relay station. It then obtains the corresponding sound signal from the equipment voiceprint database according to the equipment type and cause of generation. After determining the feature vector and cause of generation of the sound signal, it stores it as a sample in a set. The sample is then transmitted to the RNN model, LSTM model and CNN model to determine the model parameters, performance coefficients and basic weights. The operation is repeated until all types of equipment have corresponding RNN models, LSTM models and CNN models.

2. The immersive remote maintenance system for unmanned railway relay stations according to claim 1, characterized in that: The sound source identification unit (1) includes a tag location determination module (101), a point cloud data acquisition module (102), an image acquisition module (103), and a coefficient matrix construction module (104). The tag location determination module (101) determines the sound source point of each device in the specified relay station and deploys corresponding visual reference tags c1, c2, ..., c at all sound source point locations. i ,...,c n ,in i represents the tag number, and O represents the tag sequence number. These represent the actual positions of the top-left, top-right, bottom-left, and bottom-right corners corresponding to the i-th visual reference label, respectively. This represents the actual x-coordinate of the top-left corner corresponding to the i-th visual reference label. This represents the actual ordinate of the upper left corner corresponding to the i-th visual reference label. After acquiring global positioning data and initializing local positioning data, the point cloud data acquisition module (102) determines the estimated position of the inspection robot at the current moment and the acquired point cloud data, and projects the point cloud data into the local positioning data. The image acquisition module (103) calculates the distance between the estimated position and the positions of all visual reference labels in the global positioning data. If the value is lower than the threshold, the corresponding visual reference label image information is acquired using the camera mounted on the inspection robot; otherwise, the image acquisition operation is not performed. If the coefficient matrix construction module (104) acquires the image information of the visual reference label, it analyzes the relative positions of the four corners of the current visual reference label based on the image information. in This represents the top-left corner relative to the x-coordinate of the i-th visual reference label. This represents the relative ordinate of the top-left corner of the i-th visual reference label, based on... and Set the corresponding adjustment matrix D, where use and Calculate the coefficient matrix in the upper left corner. in Repeat the operation until all coefficient matrices are analyzed, and then combine them to obtain the actual values ​​of each parameter in the adjustment matrix D.

3. The immersive remote maintenance system for unmanned railway relay stations according to claim 1, characterized in that: The position output unit (2) includes an estimated position optimization module (201) and a transformation matrix calculation module (202). After obtaining the camera intrinsic parameter matrix, the estimated position optimization module (201) calculates the first transformation matrix α and the second transformation matrix β based on the camera intrinsic parameter matrix and the adjustment matrix D, where α = (α0, α1, α2). The offset angle θ at the current moment is calculated using β, where The optimized estimated position L of the current inspection robot is output using α and θ, where L = (α0, α1, θ). The transformation matrix calculation module (202) uses the estimated position to match all points in the current point cloud data with the points in the local positioning data, calculates the first transformation matrix and the second transformation matrix corresponding to the point with the minimum distance between points, and deduces the actual position of the current inspection robot based on the first transformation matrix and the second transformation matrix.

4. The immersive remote maintenance system for unmanned railway relay stations according to claim 3, characterized in that: The position output unit (2) further includes a number reading module (203), a record storage module (204), and a data matching module (205). The number reading module (203) reads the historical path set stored by the robot and determines the number of the visual reference label. If the number is the same as the number in the historical path set, the position corresponding to the historical record number is extracted, and the difference between the position and the current actual position is calculated. The actual position corresponding to all historical record numbers is optimized based on the difference. The record storage module (204) constructs new local positioning data based on the actual position and point cloud data. The original local positioning data, actual position, and the number, four corner positions, and acquisition time of the visual reference label are stored as historical records in the historical path set. The number of the visual reference label is the number of the current historical record. If the data matching module (205) does not acquire the image information of the visual reference label, it directly uses the unoptimized estimated position to match the point cloud data with the local positioning data, thereby outputting the current actual position of the inspection robot.

5. The immersive remote maintenance system for unmanned railway relay stations according to claim 1, characterized in that: The model building unit (3) includes a signal extraction module (301), a signal processing module (302), a feature vector determination module (303), a performance coefficient calculation module (304), and a weight setting module (305). The signal extraction module (301) determines the j-th type of device S in a specified relay station. j The corresponding sound signal is generated by s j,1 ,s j,2 ,...,s j,m Then, where j represents the device type number and m represents the total number of causes of sound signal generation for that type of device, the corresponding sound signal is retrieved from the device voiceprint database according to the device type and cause. The signal processing module (302) performs pre-emphasis, framing, and windowing operations on the sound signal, and then performs Fourier transform on it to obtain the frequency domain energy spectrum. The feature vector determination module (303) analyzes the frequency domain energy spectrum through the Mel filter bank to determine the energy of each frequency band. After extracting the multidimensional features of the current sound signal based on the energy of each frequency band, it uses principal component analysis to reduce the dimensionality of the sound signal to obtain the final feature vector of the sound signal. The performance coefficient calculation module (304) determines the feature vectors and causes of all sound signals, stores them as samples in a set, divides the samples in the set into a training set and a test set in a 7:3 ratio, and transmits the samples in the training set to the RNN model, LSTM model, and CNN model. After determining the model parameters, the performance coefficients γ1, γ2, and γ3 of each model are calculated using the samples in the test set. The weight setting module (305) analyzes the basic weights ω1, ω2, and ω3 of the RNN model, LSTM model, and CNN model based on γ1, γ2, and γ3. Repeat the process until all types of devices have corresponding RNN, LSTM, and CNN models.

6. The immersive remote maintenance system for unmanned railway relay stations according to claim 1, characterized in that: The sound analysis unit (4) includes a mode division module (401), a signal upload module (402), and a prediction result calculation module (403). The mode division module (401) divides the working mode of the inspection robot into two types: routine inspection and special inspection. When the inspection robot is in the routine inspection working mode, the signal upload module (402) collects the corresponding visual reference label image information, obtains the current device type based on the label image information, obtains the real-time sound signal of the current sound source point using the microphone array mounted on the inspection robot, and uploads the real-time sound signal and the device type to the edge server. The prediction result calculation module (403) retrieves the corresponding RNN model, LSTM model, and CNN model from the edge server according to the device type, determines the feature vector of the real-time sound signal, inputs it into the three models for analysis, optimizes the corresponding weight coefficients based on the model prediction results and the corresponding basic weights, and calculates the optimized weight coefficients and prediction results using the sound recognition algorithm, thereby outputting the cause of the current real-time sound signal.

7. The immersive remote maintenance system for unmanned railway relay stations according to claim 1, characterized in that: The device joint analysis unit (5) includes a normal operation determination module (501), a distance calculation module (502), a candidate set generation module (503), a similarity determination module (504), and a priority coefficient comparison module (505). If the real-time signal is generated due to normal operation, the normal operation determination module (501) uploads the current device number, the cause of sound generation, and the acquisition time to the edge server, and then searches for the next sound source point according to the visual reference label number order. If the real-time signal is not generated due to normal operation, the distance calculation module (502) uses MVDR technology to calculate the actual sound source location. When the distance d between the actual sound source location and the sound source point is... v If the distance is greater than the threshold, a timer is started, and a special inspection is carried out after the timer expires. When the distance between the actual sound source location and the sound source point is less than or equal to the threshold, the candidate set generation module (503) reads the visual reference label image information of other devices within five meters of the current sound source point, collects the sound signal of the corresponding sound source point, analyzes the sound signal, determines the cause of each sound signal, combines the feature vector of the sound signal whose cause is not normal operation with the device number, and stores it in the candidate set. The similarity determination module (504) calculates the sound signals u1, u2, ..., u in the candidate set. M The similarity sim1,...,sim between the feature vector and the feature vector of the sound signal v at the current sound source point t ,...,sim M Set the preset value μ, and determine u1, u2, ..., u M The distances d1, d2, ..., d between the corresponding sound source points and the current actual sound source locations are... t ,...,d M If sim exists t >μ and d t <d v If the timer is started, a special inspection will be conducted after the timer expires. Here, M represents the total number of sound signals in the candidate set, and t represents the signal sequence number. If the priority coefficient comparison module (505) does not exist, then u1,...,u1, which have the same cause as the sound signal v, will be selected. N According to u1,...,u N After determining the historical failure rate and distance between the corresponding device number and the actual sound source location, a coefficient analysis algorithm is used to analyze it and obtain the priority coefficient of each sound signal. If the priority coefficient of v is higher than or equal to u1,...,u N Where N represents the total number of filtered sound signals, the current device number, the cause of the sound, the actual location information of the visual reference tag, and the acquisition time are uploaded to the edge server as recorded information; otherwise, a timer is started, and a special inspection is carried out after the timer expires.

8. The immersive remote maintenance system for unmanned railway relay stations according to claim 1, characterized in that: The recording information generation unit (6) includes a route generation module (601) and a special inspection module (602). The route generation module (601) searches for the next visual reference label according to the label order. When the device timeout ends, the inspection robot is in a special inspection and directly generates the shortest route based on the current position and the position of the visual reference label corresponding to the device whose timeout has ended. After the inspection robot arrives according to the route, the special inspection module (602) randomly selects five different positions inside the device, reads and analyzes the sound signals at the five positions and the sound source point, determines the cause of the sound generation at different positions, takes the prediction result with the highest confidence as the actual cause of the sound source point of the current device, and generates the corresponding recording information.

9. The immersive remote maintenance system for unmanned railway relay stations according to claim 1, characterized in that: The fault verification unit (7) includes an abnormal receiving module (701) and a manual verification module (702). After the edge server receives the record information of all abnormal equipment, the abnormal receiving module (701) reports the equipment number and the visual reference label position corresponding to the abnormal category. After receiving the record information, the manual verification module (702) manually controls the inspection robot to verify the abnormal equipment.

Images