Steel rail flaw detection audio and video automatic synchronization merging and filing method, equipment and medium

Abstract

The invention relates to the technical field of rail detection and maintenance, and discloses a rail flaw detection audio and video automatic synchronization merging and filing method, equipment and a medium. The method comprises the following steps: S1, acquiring a multi-channel audio and video file which is generated by flaw detection operation and is configured with standardized identification information through a processing end; s2, analyzing the standardized identification information and then establishing a position group, a time group and a type group; s3, establishing a file global time reference in the same time group, and respectively complementing time intervals for the audio and video files in the group; s4, merging the multiple paths of audio and video files after completion of the time interval to generate an output file, and generating a merged file name including the standardized identification information; and S5, uploading the output file to a server, and binding and filing the output file and the corresponding flaw detection operation record. According to the invention, a traditional low-efficiency mode of manually copying and splicing audios and videos is replaced by full-process automatic operation, the labor cost can be saved, and the processing efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of track inspection and maintenance technology, specifically to a method, equipment, and medium for automatic synchronization, merging, and archiving of audio and video recordings from rail flaw detection. Background Technology

[0002] During rail flaw detection, two types of audio and video files are typically generated simultaneously: one type is waveform data video output from dedicated flaw detection recording equipment, which shows the internal defects of the rail; the other type is multi-view operation video from the field, such as probe position video and endoscope video, used to locate the physical location of the defects. These multiple audio and video streams need to be played back synchronously and compared to accurately determine rail damage.

[0003] Currently, the processing of audio and video recordings for rail flaw detection is mostly done manually. This requires manually copying the audio and video files recorded by each device to a computer, then manually aligning the timelines and stitching the images using video editing tools, and finally manually associating the merged video with the flaw detection operation record for storage. A single rail flaw detection operation will record a large amount of audio and video for each inspection location. It is difficult to manually organize, archive, and merge these audio and video recordings according to management requirements such as inspection location and inspection time. Furthermore, the current mainstream playback method uses multiple players to simultaneously play the flaw detection waveform video and multiple video streams recorded in the same batch of operations. Due to factors such as network latency, network fluctuations, and computer performance limitations, there will be delays of several seconds or tens of seconds between these videos. Summary of the Invention

[0004] In view of this, the present invention provides a method, equipment and medium for automatic synchronization, merging and archiving of audio and video of rail flaw detection, in order to solve the above problems.

[0005] To address the above technical problems, this invention provides a method for automatic synchronization, merging, and archiving of audio and video recordings from rail flaw detection, comprising the following steps: S1. Obtain multi-channel audio and video files with standardized identification information generated during flaw detection operations through the processing terminal; S2. After parsing the standardized identification information, establish location grouping, time grouping, and type grouping; S3. Establish a global time reference for files within the same time group, calculate the time offset of each file in the group relative to the global time reference, and fill in the time interval for the audio and video files in the group respectively. S4. Merge the multiple audio and video files after completing the time interval to generate an output file, and generate a merged file name including standardized identification information; S5. Upload the output file to the server, and bind and archive the output file with the corresponding flaw detection operation record based on standardized identification information.

[0006] As an optional method, when the processing end is not a recording device, S1 also includes: Identify the accessed recording devices, perform a deep traversal of the storage paths of the recording devices, and filter out audio and video files containing standardized identification information; A unique file fingerprint is generated based on the file path, file size, and file modification time. The file fingerprint is compared with the fingerprint of the file already stored locally, and the newly added file is copied to a non-recording device for storage. The original audio and video files in the recording device are renamed.

[0007] As an optional method, when the processing end is a recording device, S1 also includes: Scan the local storage for audio and video files according to the preset local storage path; When an audio or video file without a configured processed tag is detected, it is determined to be a new file to be processed, and the standardized identification information of the new file is read directly.

[0008] As an optional approach, after parsing the standardized identification information, location grouping, time grouping, and type grouping can be established, including: Extract the job location parameters, time range, and media type of the file; Files are grouped by location based on job location parameters, and files within the same location group are further grouped by time interval overlap, while files within the same time group are grouped by media type.

[0009] As an optional approach, grouping files within the same location based on their overlapping time intervals includes using a disjoint-set data structure algorithm. The grouping rules include: For any two files within the same location group, if the start time of the first file is less than or equal to the end time of the second file and the start time of the second file is less than or equal to the end time of the first file, then they are considered to have overlapping times and are grouped into the same group.

[0010] As an optional method, when completing the time interval for audio and video files within a group, the processing of video files includes calculating the completion time interval using a chained time difference method. The chain time difference calculation includes: The earliest start time of all files in the same time group is defined as the global earliest start time, and the latest end time is defined as the global latest end time. For the first video of the same media type, its time offset is the difference between its own start time and the earliest global start time; For subsequent videos of the same media type, the time offset is the difference between its own start time and the end time of the previous video; For the last video of the same media type, its end offset is the difference between the global latest end time and its own end time.

[0011] As an optional method, when completing the time interval for audio and video files within a group, the processing of audio files includes: Calculate the time offset of the audio file relative to the global time base, generate a silent segment with the same duration as the time offset, and splice the silent segment to the beginning of the audio file to complete the time interval padding; The silent segment is a zero-value audio signal, and its sampling rate and number of channels are consistent with the corresponding audio file.

[0012] As an optional method, if no corresponding file is available for a certain media type, a compensation video will be generated: A background video is generated at a preset resolution, and the corresponding media type is marked with "No Video". The duration of the compensated video is consistent with the difference between the latest global end time and the earliest global start time.

[0013] As an optional method, the output file upload includes: The output file is split into multiple pieces according to a preset size, and a list of piece information is generated. Upload the fragments sequentially according to the fragment information list and record the upload completion status of each fragment; After all fragments have been uploaded, the server merges the fragments based on the fragment information list to obtain the complete output file.

[0014] As an optional method, binding and archiving the output file with the corresponding flaw detection operation record includes: Extract the device serial number, operation location parameters, and global time interval from the standardized identification information in the output file; Query the flaw detection operation record database to match flaw detection operation records that contain the same equipment serial number, operation location parameters and operation time covering the global time interval; Write the storage path and filename of the output file into the matched flaw detection operation record; Mark the output file and the corresponding flaw detection operation record as archived.

[0015] On the other hand, the present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of an automatic synchronization merging and archiving method for rail flaw detection audio and video.

[0016] On the other hand, the present invention also provides a storage medium storing a computer program, wherein when the computer program is executed by a processor, the steps of an automatic synchronization merging and archiving method for rail flaw detection audio and video are implemented.

[0017] The beneficial effects of this invention are as follows: This invention replaces the inefficient traditional method of manually copying and splicing audio and video with fully automated operation, significantly saving labor costs and improving processing efficiency. It achieves audio and video synchronization through global time reference and time offset calculation, and automatically binds and archives audio and video with flaw detection operation records based on standardized identification information. This solves the problems of scattered files and chaotic information, facilitating subsequent traceability and management, and overall improving the safety assurance capability of rail flaw detection operations. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the automatic synchronization, merging, and archiving method for audio and video of rail flaw detection according to the present invention; Figure 2 This is a schematic diagram of the video stream processing flow of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been presented in the various embodiments of the present invention to enable the reader to better understand the present invention. However, the technical solutions claimed in the present invention can be implemented even without these technical details and various changes and modifications based on the following embodiments.

[0020] Rail flaw detection relies on the combined analysis of multiple audio and video streams to determine damage. These streams include waveform data from recording equipment and multi-view footage from on-site operations. Traditional methods require manual intervention in copying, splicing, and recording, which not only consumes significant manpower but is also prone to timing discrepancies due to human error, leading to inaccurate audio and video correspondence. Furthermore, the lack of standardized file management results in low efficiency for subsequent retrieval and traceability. This solution is designed based on these practical needs, using automation technology to replace manual operations, making audio and video processing and archiving more efficient.

[0021] like Figure 1As shown, in one embodiment, the processing end collects all audio and video files generated during the flaw detection operation. In this embodiment, all collected audio and video files are configured with standardized identification information. This standardized identification information is a set of information configured on each audio and video file to support the automated processing of the entire audio and video process for rail flaw detection. As an optional approach, the identifier includes device information to distinguish files recorded by different recording devices, operation location information such as the mileage of the rail weld, start and end times of the audio and video files, and type information to distinguish the audio and video sources and video sources from different perspectives or functions. In this embodiment, the standardized identification information is named in the format "devicerecord-device serial number-weld number-line number-rail number-mileage identifier-track number-side identifier-start recording time-media type". This information is presented in a unified manner, enabling the processing end to quickly parse and extract it, and establish a complete file metadata structure. The standardized identification information integrates key content related to the flaw detection operation and file attributes, ensuring that files can be accurately identified and processed in all stages such as acquisition, grouping, synchronization, merging, and archiving.

[0022] After parsing the standardized identification information, the files are first grouped by work location to ensure that files from different work areas are not confused. Next, they are grouped by time intervals to process related files within the same flaw detection period. Finally, they are grouped by media type to prepare for targeted audio and video processing. For files in the same time group, a unified global time reference is established. By calculating the time offset of each file relative to this reference, missing time intervals are added to both video and audio, ensuring consistency across multiple audio and video streams on the same timeline. Then, the screen layout is automatically adjusted based on the number of media types within the group, merging all audio and video files after time interval completion into a single complete file. The file name retains the standardized identification information for easy identification later. Finally, the merged file is uploaded to the server, where the corresponding flaw detection work record is automatically located and associated based on the key information in the standardized identification information, completing the entire processing.

[0023] Based on the aforementioned technical issues, in actual rail flaw detection operations, the recording equipment equipped by different work units varies, and the data processing capabilities of these devices also differ. Some recording devices only have basic audio and video recording functions and cannot perform complex data processing operations such as file grouping, time synchronization, and audio / video merging within the device. Other recording devices, however, integrate dedicated data processing modules, possessing the ability to independently complete the entire automated processing process. Based on this current equipment application situation, this solution designs two different implementation methods for the processing end to adapt to more work scenarios. It can meet the automation upgrade needs of work units equipped with basic recording equipment, and also fully leverage the local processing advantages of intelligent recording equipment, enabling work units with different hardware configurations to achieve automated processing and archiving of rail flaw detection audio and video through this solution.

[0024] Therefore, when the processing end is a non-recording device such as a computer, the device identification and monitoring issues need to be considered first, and the information connection needs to be completed in order to correctly implement the automation scheme of this embodiment.

[0025] As an optional approach, the processing end first obtains information on all storage devices currently connected to it, including various storage media such as local disks, USB devices, and MTP devices, and then establishes a corresponding access handle for each device. This embodiment uses handles as identifiers to cover any device access method, avoiding the omission of target recording devices due to different device types.

[0026] After acquiring information from all storage devices, the processing unit performs a name feature matching operation on each device. By using preset keywords related to the flaw detection operation, a string matching algorithm is employed to detect whether the device name contains these specific keywords, thereby filtering out target recording devices. These are then added to the target device list for further processing. The reason for using keyword matching is that the names of recording devices specifically designed for flaw detection typically integrate operation-related identifiers.

[0027] After identifying the target recording device, the processing unit further obtains key parameters of the device, including the device serial number, total storage capacity, and remaining available space, and calculates the used space and utilization rate. Furthermore, the parameters selected in this embodiment are based on the following considerations: the device serial number serves as a unique identifier for the device, avoiding file confusion between different recording devices; storage capacity-related parameters provide prerequisites for subsequent file copying, determining in advance whether the processing unit has sufficient space to receive the file, or whether the recording device has a valid readable file, thus preventing copy failure due to insufficient storage.

[0028] Furthermore, to address situations where recording devices may temporarily or unexpectedly disconnect at the work site, the processing end of this embodiment is configured with a timed monitoring thread. By setting a fixed time interval, it periodically performs device scanning to track the connection status of the target recording device in real time, thereby preventing interruptions or data loss during file acquisition.

[0029] Specifically, when a device is detected to be connected, the subsequent file filtering and copying process is immediately triggered; when a device is detected to be disconnected, the relevant operations are paused and the current progress is recorded, so that the process can be quickly resumed when the device is reconnected.

[0030] In this embodiment, the processing end for non-recording devices first scans all currently connected storage devices using a string matching algorithm for identification. This algorithm quickly distinguishes recording devices from unrelated storage devices such as ordinary USB flash drives and external hard drives, preventing the processing end from mistakenly operating on non-target devices. After identifying the target recording device, the processing end establishes a timed monitoring thread to periodically scan the device's connection status. This addresses potential temporary disconnections and reconnections of equipment during flaw detection operations, ensuring the processing end can respond promptly to changes in device connection status and preventing interruptions or omissions in file acquisition.

[0031] After the device is stably connected, the processing unit performs a depth-first search on the storage path of the recording device. This search traverses multi-level nested directories to find all possible locations within the device where flaw detection audio and video files might be stored, avoiding file omissions due to complex directory hierarchies. During the traversal, the processing unit filters valid audio and video files based on file extensions and preset file size thresholds, with a focus on files containing standardized identification information (as mentioned earlier, only target files generated during flaw detection operations are configured with this identifier), excluding invalid data such as system files and irrelevant documents within the device.

[0032] To avoid duplicate copying of already transmitted files, the processing end generates a unique file fingerprint based on the file path, file size, and file modification time (optionally, an MD5 hash algorithm can be used to integrate these three dimensions to generate a unique fingerprint). The file fingerprint uniquely identifies the content and state of the file. When a file remains unchanged, its fingerprint information does not change. By comparing the file fingerprint on the device with the fingerprints of files already stored locally on the processing end, the processing end can accurately determine which are newly added, untransmitted files and only perform copy operations on these files, excluding already copied files with the "copied-" prefix and irrelevant files that do not meet the file extension requirements.

[0033] After identifying newly added, untransmitted files, the processing unit copies these files to the local file storage directory while waiting for all files to complete the copying process. During file copying, the processing unit performs a file integrity verification operation, comparing the file size and hash value before and after copying to confirm that the local files are completely consistent with the original audio and video files on the recording device. Once the copying is complete and verification is successful, the processing unit updates the local file index, recording information such as the storage path, file fingerprint, and copying time of the newly added files. Updating the local file index allows subsequent processing steps, such as group synchronization, to quickly obtain the status of available files, ensuring that processing is carried out based on accurate file information.

[0034] After each file is copied, the processing end will rename the original audio and video files in the recording device. The modified file name will have a fixed prefix "copied-" added to it. This allows the processing end to directly identify the copied files by their file names during subsequent device scanning, eliminating the need for file fingerprint generation and comparison. This skips the filtering process for such files, avoids repeated searching and copying of already transmitted files, improves the processing speed of the processing end, and enhances the overall efficiency of the file acquisition process.

[0035] When the processing end is a recording device with integrated data processing capabilities, its file processing flow is as follows: after obtaining the file list, it scans its own storage medium according to the preset local storage path, filters out audio and video files that are not marked as "processed", and reads the standardized identification information of these files to form a list of files to be processed.

[0036] After obtaining the audio and video files through any of the aforementioned processing terminals, each processing terminal in this embodiment is also configured with a preset grouping script to call the parsing function and perform a complete parsing operation on the standardized identification information of each file to be processed. As described above, this embodiment still parses media files according to the standardized naming format. The parsed file name format is "devicerecord-device serial number-weld number-line number-rail number-mileage marker-track number-side marker-start recording time-media type", extracting key information such as device serial number, weld number, line number, rail number, mileage marker, track number, side marker, start recording time, and media type to establish a complete file metadata structure including location, time, and type. This structured design allows the script to parse in batches according to fixed rules, avoiding recognition errors caused by chaotic identification formats. The extraction of the three types of information directly determines the accuracy of grouping, and the entire process is executed automatically by the script without manual intervention.

[0037] Furthermore, in one optional method, the filename uniformly follows a fixed format of "devicerecord-device serial number-weld number-track number-rail number-mileage marker-track number-side marker-start recording time-media type", with each field separated by a hyphen, consisting of 10 components, as detailed below: The prefix "devicerecord" is used to distinguish audio and video files specifically for flaw detection operations, preventing them from being confused with other unrelated files. Device serial number: uniquely identifies the recording device and prevents files from being mixed up with those from different recording devices; Weld number, track number, rail number, mileage marker, track number, side marker: rail information used in flaw detection operations to distinguish different rail markings; Start recording time: The time when video recording begins, used to identify the uniform start time of the video; Media type: Differentiate file attributes and use fixed numerical encoding. 00 is on-site audio, 01 is left camera video, 02 is right camera video, 03 is flaw detector screen video, 04 is endoscope video, and so on to identify different video types.

[0038] After the processing unit starts the video merging task, it first obtains the list of files to be processed through the preset file path (local save directory for non-recording devices, and device-preset storage path for recording devices), and then performs the parsing operation according to the following steps: The acquired file list undergoes initial filtering, automatically excluding files whose filenames contain the "Converted-" prefix. In this embodiment, the "Converted-" prefix is ​​added to the original audio and video files to differentiate processing progress. Specifically, once a group of audio and video files completes the synchronous merging operation, the processing end immediately renames all the original source files involved in that group. This prefix serves as a physical marker for the "processed" status. When the next round of automatic scanning task starts, the system uses a filename matching algorithm to automatically exclude files with the "Converted-" identifier from the processing list, thus avoiding repeated reading and processing of the same data during the cyclic incremental processing.

[0039] The filenames of each file to be parsed are split by string splitting, based on hyphens, generating a parameter array `parts_array` containing 10 elements. The splitting logic is based on a fixed structure of standardized naming rules, with each element corresponding to a preset information field.

[0040] For example, if a file is named "devicerecord-SN001-W001-L001-R001-M100-T1-L-20250201110313100-01.mp4", the automatic parsing process in this embodiment is as follows: After performing string splitting, the parameter array is: parts_array=['devicerecord','SN001','W001','L001','R001','M100','T1','L','20250201110313100','01']; The information corresponding to each element is as follows: prefix devicerecord, device serial number SN001, weld number W001, line number L001, rail number R001, mileage marker M100, track number T1, side marker L, start recording time 20250201110313100, media type 01 (left camera / video); Based on the file extension ".mp4", the file can be fully identified as: recorded at 11:03:13:100 milliseconds on February 1, 2025, by a device with serial number SN001, corresponding to the left camera video of line L001, weld seam W001, mileage M100, track T1, and side L.

[0041] Furthermore, the start recording time field is a fundamental benchmark for time synchronization. This embodiment first checks the length of the timestamp string, retaining only strings with a length of 17 characters. The design logic is that a 17-character string corresponds to the complete format of year (first 4 characters), month (5-6 characters), day (7-8 characters), hour (9-10 characters), minute (11-12 characters), second (13-14 characters), and millisecond (15-17 characters). If the length is less than 17 characters, it indicates that the time information of the file is missing or incomplete during recording, and such files will be automatically excluded to avoid affecting subsequent synchronization and merging.

[0042] The specific steps are as follows: For example, in the above 20250201110313100, the first four digits represent the year, the 5th and 6th digits represent the month, the 7th and 8th digits represent the day, the 9th and 10th digits represent the hour, the 11th and 12th digits represent the minute, the 13th and 14th digits represent the second, and the 15th and 17th digits represent the millisecond. Based on the split time fields, a standardized time object is constructed to form a unified time base data.

[0043] Considering that different files from the same work site may have overlapping or consecutive recording times, after obtaining the standardized time object, this embodiment also needs to further obtain the accurate duration, start time, and complete time interval of the file. Optionally, a video length extraction tool can be called to read the accurate duration of each video file, the extracted file duration can be added to the standardized start time, the end time of the file can be calculated, and finally a complete time interval object containing the start time, duration, and end time can be generated.

[0044] For example, let's take a video file with a start time of 2025-02-01 11:03:13.100 and an extraction duration of 30.5 seconds as an example: The end time is calculated as 11:03:13.100 + 30.5s = 11:03:43.600; The time interval object is calculated as: [2025-02-01 11:03:13.100,30.5,2025-02-01 11:03:43.600].

[0045] Based on the technical field of this embodiment, in actual scenarios, flaw detection operations are usually carried out in segments according to specific lines / welds (spatial dispersion), multiple devices record simultaneously within the same operation segment (multiple data sources), and the recording process may generate scattered files due to device switching or pausing (discontinuous time). Therefore, the script in this embodiment will also perform three-level grouping to organize the above-mentioned disordered audio and video files into an ordered set of the same operation area, the same flaw detection time period, and the same type of data.

[0046] Specifically, for location grouping, this embodiment aims to isolate audio and video files from different work areas to ensure that all subsequently merged files correspond to the same rail work point. This avoids mismatches between output files and flaw detection records caused by mixed processing, which directly affects the accuracy of subsequent damage verification. The processing unit in this embodiment constructs equipment information key-value pairs by extracting seven key parameters from the parameter array: First, the standardized identification information of each file to be processed is parsed. The seven types of parameters mentioned above are extracted through string segmentation to construct a composite identification key, `device_key` (device serial number, weld number, line number, rail number, mileage, track number, and side identifier). Then, files with completely identical composite identification keys are grouped into the same location group. For example, all files with the parsed parameter "SN001-W001-L001-R001-M100-T1-L" will be aggregated into one location group; while files with the parameter "SN001-W001-L002-R001-M100-T1-L" will be grouped into another location group due to their different line numbers. The reason for choosing these seven types of parameters is that a single parameter cannot uniquely pinpoint the work point (the same weld may correspond to different lines, and the same mileage may involve different tracks), while the combination of the seven types of parameters can achieve the location of the work point from three dimensions: equipment, line, and specific location.

[0047] The purpose of time grouping is to aggregate all files belonging to the same flaw detection period within the same location group. Since the start and end times of recording for each video file are not the same, there will inevitably be one file recorded at the beginning of flaw detection and another file recorded at the end of flaw detection. Furthermore, flaw detection at the same work point is a continuous process, but multi-device recording may have differences in start / stop times, or multiple scattered files may be generated due to equipment pauses. Although these files are not completely continuous in time, they belong to the same flaw detection task and need to be included in the same time period for processing.

[0048] Therefore, this embodiment uses a disjoint-set data structure (DFS) algorithm to implement time grouping. Firstly, the DFS algorithm solves the problem of transitive time overlap in this scenario (e.g., files A and B overlap, B and C overlap; even if A and C do not directly overlap, they still need to be grouped into the same time period), which aligns with the actual scenario of continuous flaw detection. Secondly, the DFS algorithm can achieve query and merging operations with approximately constant time, enabling rapid processing of massive numbers of files, which is superior to traditional traversal comparison methods. Specifically, the time overlap determination rule in this embodiment is: for any two files within the same position group, if the start time of file A is less than or equal to the end time of file B, and the start time of file B is less than or equal to the end time of file A, then the two files are determined to have time overlap. This rule covers partial overlap, complete inclusion, and endpoint connection cases, ensuring that no files within the same continuous flaw detection time period are missed.

[0049] As an optional approach, during implementation, based on the start timestamp parsed from the file identifier and combined with the file duration obtained from the video length extraction tool, the precise time interval [start time, end time] (accurate to milliseconds) for each file is calculated. For example, a file with a start time of 11:03:13.100 and a duration of 30.5s has a time interval of [11:03:13.100, 11:03:43.600].

[0050] Subsequently, a disjoint-set data structure is initialized within the same group, all file pairs are traversed, and the grouping is determined according to the overlap rule to decide whether to merge the groups. A specific example is as follows: File A (11:00:00.000-11:10:00.000) and File B (11:05:00.000-11:15:00.000): The start time of A is less than or equal to the end time of B, and the start time of B is less than or equal to the end time of A. They are considered to overlap and are merged into Group 1. File B and File C (11:12:00.000-11:20:00.000): The start time of B is less than or equal to the end time of C, and the start time of C is less than or equal to the end time of B. Therefore, they are considered to overlap. Since B is already in group 1, C is assigned to group 1. The final group 1 contains files A, B, and C, covering the entire continuous flaw detection period (11:00:00.000-11:20:00.000).

[0051] Media type grouping categorizes files within the same time group according to their media type. In this embodiment, the grouping is based on the "media type" field in the standardized file identification information. This field uses a fixed numerical code to ensure that the processing end can automatically identify the files: 00 represents on-site audio, 01 represents left camera video, 02 represents right camera video, 03 represents flaw detector screen video, 04 represents endoscope video, and so on.

[0052] In one implementation, all files are traversed within the same time group, the media type code of each file is extracted, and the files are divided into different subgroups according to the code: files with code 00 are assigned to the audio subgroup, and files with codes 01-04, etc., are assigned to the corresponding video subgroups (left camera subgroup, right camera subgroup, etc.).

[0053] Subsequently, the files within each subgroup are sorted by their start timestamp: the audio subgroup generates a time-sorted sequence of audio files, with each file assigned the identifier "A-{serial number}" (A-{1}, A-{2}); each video subgroup generates a time-sorted sequence of video files, with each file assigned the identifier V-{camera ID}-{serial number} (e.g., V-{01}-{1}, V-{02}-{2}). The reason for this sorting is that files of the same type may generate multiple fragments due to recording pauses; sorting by time allows for chained splicing, ensuring the continuity of audio and video streams of the same type on the timeline.

[0054] For example, a certain time group contains the following files: two audio files with code 00, three left camera files with code 01, and one flaw detector screen display file with code 03. After sorting the group, three sequences are formed: Audio sequence: A-{1} (starting at 11:00:00.000), A-{2} (starting at 11:08:00.000); Left camera video sequence: V-{01}-{1} (starting at 11:00:00.000), V-{01}-{2} (starting at 11:05:00.000), V-{01}-{3} (starting at 11:12:00.000); The flaw detector screen displays the video sequence: V-{03}-{1} (starting at 11:00:00.000).

[0055] Therefore, the grouping design in this embodiment fully considers the actual operation scenario of rail flaw detection, which not only avoids the problem of document confusion at different operation points and at different times, but also the grouping process is fully automated without human intervention, thus solving the problem of low efficiency and reliance on manual document organization in the prior art.

[0056] After grouping the files, a problem arises: although audio and video files within the same time group belong to the same flaw detection period, their recording start / stop times differ, and fragmented segments may occur due to equipment pauses. Therefore, it is necessary to align all audio and video streams on the same timeline through unified benchmark calibration and interval compensation. To solve this problem, this embodiment establishes a global time benchmark, providing a unique time reference for all audio and video files within the same time group, eliminating time start deviations between different files. Thus, this embodiment can overcome practical problems such as potential start delays during multi-device recording or discontinuous file times caused by segmented recording on a single device.

[0057] The implementation method of this embodiment is as follows: Using milliseconds as the precision unit, the standardized identification information of audio and video files within the same group is parsed, the start time of all files within the time group is traversed, and the minimum value is extracted as the earliest global start time (so that the timeline of all files can cover the entire flaw detection period, and avoid the loss of early content caused by individual files starting late). Iterate through the end times of all files in the time group (calculated by adding the file duration to the start time), and extract the maximum value as the global latest end time (to ensure that the timeline of all files can continue until the end of the flaw detection task, and to avoid the loss of later content due to the early termination of individual files). The total global duration is calculated as follows: Latest global end time - Earliest global start time. This total global duration is the standard duration of the final merged output file.

[0058] After determining the global time base, this embodiment needs to perform time compensation on video and audio files separately. For video files, considering that videos from the same perspective need to maintain temporal continuity in order to present a smooth picture after merging, this embodiment will use chained time difference calculation to identify and fill in the time intervals between segments to avoid scene jumps. Finally, the scattered video segments of the same media type will be completed into a continuous video stream with the same total global duration.

[0059] Specifically, for the first video in the sequence, its time offset is equal to the video's start time minus the earliest global start time. This calculation is used to fill in the blank period from the global time reference start point to before the video starts. If the video's start time is the earliest global start time, then its start time offset is 0.

[0060] For subsequent videos in the sequence: Time offset = Start time of the current video - End time of the previous video. If the offset is positive, fill in the blank segment; if it is negative, it is determined to be a time overlap and is directly spliced. This calculation is used to fill in the blank period from the end of the previous video to the start of the current video.

[0061] For the last video in the sequence: End offset = Global latest end time - End time of this video. This calculation is used to fill in the gap between the end of the current video and the end of the global time base.

[0062] The chained computation described above does not directly align with the global time base because videos of the same media type may be recorded in multiple segments (e.g., starting and stopping the camera as needed to avoid invalid recording). The method described in this embodiment can fill in each segment interval. If the global time base were directly aligned at this time, it might cause a break in the inherent temporal continuity between segments.

[0063] Based on the time deviation calculations described above, the processing unit also needs to perform automated compensation operations on video sequences of the same media type according to the aforementioned rules. The steps are as follows: Figure 2As shown, it includes: Initialize the video stream buffer to store the completed continuous video data; Iterate through the video sequence and determine if the currently processed video is the first video in the sequence: If this is the first video: Calculate the time offset. If the time offset > 0, generate a black screen compensation video (optional parameters: resolution 640×480 pixels, duration = time offset, pure black background, add "No Video" white text prompt in the center of the screen, font size 25 pixels), and concatenate the compensation video with the current video in chronological order and store it in the cache (as input for the next video); if the time offset ≤ 0, directly store the current video in the cache, without compensation.

[0064] If it is a subsequent video: Calculate the time offset. If the time offset > 0, generate a black screen compensation video with the same parameters as above. First, stitch the compensation video to the end of the existing video stream in the buffer, and then stitch it with the current video. If the time offset ≤ 0, do not perform compensation and directly use it as the stitching source. Stitch the current video with the previous video to ensure time continuity.

[0065] When processing up to the last video in the sequence, calculate the time offset of the last video: If the time offset of the last video is greater than 0, generate a black screen compensation video with the same parameters (this is the end compensation video), and splice it to the end of the cached video stream to fill the gap between the end of the last video and the global end point; If the time offset of the last video is less than or equal to 0, no additional compensation is needed. The video stream in the cache is already consistent with the total global duration, and the current type has been completed.

[0066] Assumption: The global time reference parameters for a certain time group are: earliest global start time = 0s, latest global end time = 90s, and total global duration = 90s; the left camera video sequence is one continuous segment (0s-90s), the right camera video sequence is one segment (5s-80s), and the endoscope video sequence is two segments (10s-30s and 70s-90s). The compensation processing results are as follows: Left camera: Time offset = 0s, time offset of the last video = 0s, no compensation, output 90s continuous video stream; Right camera: The first video time offset = 5s (5s-0s), generating a 5s black screen compensation video; the last video time offset = 10s (90s-80s), generating a 10s black screen compensation video; the final output is a 90s continuous stream of 5s black screen + 75s video + 10s black screen. Endoscopy: The first video time offset is 10s (10s-0s), generating a 10s black screen compensation video, which is then stitched together with the first 30s video; the second video time offset is 40s (70s-30s), generating a 40s black screen compensation video, which is then stitched together with the second 20s video; the last video time offset is 0s; the final output is a 90s continuous stream consisting of 10s black screen + 30s video + 40s black screen + 20s video.

[0067] The audio file time compensation uses a silence segment generation method to ensure that each audio stream is aligned with the global time base, and then a unified audio track is formed through mixing. Furthermore, since multiple audio streams need to be mixed for playback, it is not necessary to maintain the continuity of individual audio segments; simply aligning each stream with the global time base as its starting point is sufficient to avoid sound misalignment after mixing. Therefore: The processing unit independently performs time compensation operations for each audio file (media type encoded as "00") within the same time group. The specific steps include: The time offset for each audio stream is calculated as: start time of that audio stream - earliest global start time. Unlike video, audio uses a global time base directly. Chained calculation is not used because multiple audio streams will eventually be mixed.

[0068] If the time offset is greater than 0, a silence segment with the same duration as the time offset is generated. The silence segment uses a zero-value audio signal (to ensure no noise interference). To avoid audio quality distortion or format incompatibility during mixing, its sampling rate and number of channels must be consistent with the original audio file. At this point, the silence segment is spliced ​​to the beginning of the original audio, thus aligning the audio start point with the earliest global start time.

[0069] Calculate the end offset for each audio track = global latest end time - end time of that audio track. If the end offset > 0, generate a silent segment of the corresponding duration and append it to the end of the original audio track to ensure that the total duration of each audio track is consistent with the global total duration.

[0070] Specific steps: This embodiment uses the above-mentioned time grouping reference parameters as an example (earliest global start time = 0s, latest global end time = 90s): The audio sequence contains 3 files: Audio 1 (0s-90s), Audio 2 (30s-90s), Audio 3 (20s-50s), and the compensation processing results are as follows: Audio 1: Time offset = 0s, end offset = 0s, no compensation, output 90s of original audio; Audio 2: Time offset = 30s (30s-0s), generate a 30s silent segment and splice it to the front end; End offset = 0s, finally output a 90s audio stream consisting of 30s of silence + 60s of original audio; Audio 3: Time offset = 20s (20s-0s), generate a 20s silent segment and splice it to the beginning; End offset = 40s (90s-50s), generate a 40s silent segment and splice it to the end; the final output is a 90s audio stream consisting of 20s silence + 30s original audio + 40s silence.

[0071] After all audio compensation is completed, the processing unit merges the multi-channel aligned audio streams into a single mixed audio stream using a mixing filter, thus automatically synchronizing the sound and picture during playback.

[0072] After the merging step is completed, the processing end initiates the upload process. This process determines the output files to be uploaded by automatically identifying and filtering specific filename characteristics. As an optional implementation, the filtering rules are based on filename characteristics, adding only files with the "-Cvted" suffix and without the "uploaded-" prefix to the list of files to be uploaded.

[0073] In the above logic, the "-Cvted" suffix is ​​specifically used to identify that the file is a finished product file generated after simultaneous merging of multiple audio and video streams, thus distinguishing it from the original source file. Correspondingly, the aforementioned "Converted-" prefix is ​​only added to the original source file that has completed the merging process to prevent the source file from being read repeatedly. The "uploaded-" prefix is ​​used to indicate that the file has been archived on the server. For example, a file that meets the above characteristics, such as "devicerecord-TS2024110405-0FFB-0000-0000-0895-0000-00-0-20250514162126900-Cvted.mp4", will be added to the upload list, and once the "uploaded-" prefix is ​​detected, the system will automatically skip it to avoid duplicate uploads.

[0074] In this embodiment, a chunked upload strategy is adopted to reduce the pressure of single transmission of large files, including the following methods: The system reads the fragment size parameter from the configuration file, automatically calculates the required total number of fragments based on the total size of the output file and the preset fragment size, assigns a unique sequence number and corresponding byte range to each fragment, and generates a fragment information list. This list includes information such as the sequence number, start byte, end byte, and size of each fragment.

[0075] The processor sends a multi-part upload request to the server. Upon receiving the request, the server creates a new upload session and returns a unique upload identifier. The upload identifier uniquely identifies the current upload task, ensuring that subsequent operations such as multi-part uploads, breakpoint resumption, and multi-part merging all point to the same file, thus avoiding confusion between multi-part upload tasks.

[0076] Considering unforeseen circumstances such as network interruptions and device restarts during the upload process, this embodiment also includes breakpoint resume logic: When an upload task starts, the processing end first queries the server for the list of incomplete upload tasks corresponding to that file. The query is based on unique and standardized identifier information such as file name and file size; If the server has an incomplete task for this file, the processing end calls the server interface to obtain a list of uploaded fragments (including the sequence number, hash value, etc. of the uploaded fragments). By comparing the local fragment information list with the server's list of uploaded fragments, the breakpoint is located, which is the sequence number of the first fragment that has not been uploaded. For unuploaded fragments, the processing end resumes uploading from the breakpoint, without needing to re-upload the completed fragments. For example, if a file has a total of 20 fragments, and 10 fragments were uploaded before the network interrupted, upon recovery, only the remaining 10 fragments need to be uploaded, not the entire file.

[0077] In addition, to prevent data loss or corruption due to network fluctuations, the upload process for each fragment also includes integrity verification, including: Before uploading fragments, the processing end calculates the hash value of the current fragment and uploads it along with the fragment data to the server. After receiving the fragment, the server calculates its hash value and compares it with the hash value uploaded by the processing end. If they match, the fragment transmission is complete, and the processing end continues uploading the next fragment. If they do not match, it indicates a data error during transmission, the server returns a verification failure notification, and the processing end re-uploads the current fragment until verification succeeds.

[0078] Once all fragments have been uploaded, the processing end sends a fragment merging request to the server. The server queries all uploaded fragments based on the upload identifier and merges all fragments into a complete output file according to the fragment sequence number and byte range.

[0079] After merging the files, the server calculates the hash value of the complete file and returns it to the processing end. The processing end compares the hash value of the complete file returned by the server with the original hash value of the output file.

[0080] If the two are consistent, it means the file upload is complete and has not been tampered with, and the upload process ends successfully; if the two are inconsistent, it means there is a data anomaly during the merging process, the server deletes the merged file, and the processing end re-executes the chunked upload process to ensure that the final uploaded file is completely consistent with the local file.

[0081] Additionally, after a successful upload, the processing end automatically renames the local output file, adding the prefix "uploaded-" to the merged filename. For example, the original output file "devicerecord-TS2024110405-0FFB-0000-0000-0895-0000-00-0-20250514162126900-Cvted.mp4" is renamed to "uploaded-devicerecord-TS2024110405-0FFB-0000-0000-0895-0000-00-0-20250514162126900-Cvted.mp4".

[0082] Finally, once the file upload is complete, a file upload record is automatically added to the backend database and bound to the existing flaw detection operation record.

[0083] The binding process employs a multi-dimensional matching algorithm: First, query conditions are constructed based on information such as equipment serial number, weld number, line number, rail number, mileage marker, track number, and side marker in the file name; then, matching flaw detection operation records are searched in the flaw detection operation record database; through time matching, flaw detection operation records that match the time interval are found, and the uploaded video file path, file size, upload time, and other information are written into the associated fields of that flaw detection operation record.

[0084] After the files are linked with the flaw detection operation records, users can view the uploaded videos. Since the videos have been synchronized during merging, there will be no time delay between videos due to network or computer performance issues, ensuring that flaw detection experts can accurately determine the rail damage status through the videos.

[0085] Based on the above scheme, this embodiment achieves millisecond-level time synchronization accuracy using intelligent grouping technology based on the disjoint-set data structure algorithm, and can handle complex time overlap relationships and transitive grouping.

[0086] On the other hand, this embodiment also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of an automatic synchronization merging and archiving method for rail flaw detection audio and video.

[0087] On the other hand, this embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of an automatic synchronization, merging, and archiving method for rail flaw detection audio and video.

[0088] The embodiments of the present invention have been described in detail above. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A method for automatic synchronization, merging, and archiving of audio and video recordings from rail flaw detection, characterized in that, Includes the following steps: S1. Obtain multi-channel audio and video files with standardized identification information generated during flaw detection operations through the processing terminal; S2. After parsing the standardized identification information, establish location grouping, time grouping, and type grouping; S3. Establish a global time reference for files within the same time group, calculate the time offset of each file in the group relative to the global time reference, and fill in the time interval for the audio and video files in the group respectively. S4. Merge the multiple audio and video files after completing the time interval to generate an output file, and generate a merged file name including the standardized identification information; S5. Upload the output file to the server, and bind and archive the output file with the corresponding flaw detection operation record based on the standardized identification information.

2. The method for automatic synchronization, merging, and archiving of audio and video recordings from rail flaw detection according to claim 1, characterized in that, When the processing terminal is a non-recording device, step S1 further includes: Identify the accessed recording device, perform a depth-first traversal of the storage path of the recording device, and filter out audio and video files containing the standardized identification information; A unique file fingerprint is generated based on the file path, file size, and file modification time. The file fingerprint is compared with the fingerprint of a file already stored locally, and the newly added file is copied to a non-recording device for storage. The original audio and video files in the recording device are then renamed.

3. The method for automatic synchronization, merging, and archiving of audio and video recordings from rail flaw detection according to claim 1, characterized in that, When the processing terminal is a recording device, step S1 further includes: Scan the local storage for audio and video files according to the preset local storage path; When an audio or video file without a configured processed tag is detected, it is determined to be a new file to be processed, and the standardized identification information of the new file is directly read.

4. The method for automatic synchronization, merging, and archiving of audio and video recordings from rail flaw detection according to claim 1, characterized in that, The process of parsing the standardized identification information and establishing location grouping, time grouping, and type grouping includes: Extract the job location parameters, time range, and media type of the file; Files are grouped by location based on job location parameters, and files within the same location group are further grouped by time interval overlap, while files within the same time group are grouped by media type.

5. The method for automatic synchronization, merging, and archiving of audio and video recordings from rail flaw detection according to claim 4, characterized in that, The grouping of files within the same location group according to their overlapping time intervals includes using a disjoint-set data structure algorithm for grouping, wherein the grouping rules include: For any two files within the same location group, if the start time of the first file is less than or equal to the end time of the second file and the start time of the second file is less than or equal to the end time of the first file, then they are considered to have overlapping times and are grouped into the same group.

6. The method for automatic synchronization, merging, and archiving of audio and video recordings from rail flaw detection according to claim 4, characterized in that, When completing the time interval for audio and video files within a group, the processing of video files includes calculating the time interval for completing the video files using a chained time difference. The chain time difference calculation includes: The earliest start time of all files in the same time group is defined as the global earliest start time, and the latest end time is defined as the global latest end time. For the first video of the same media type, its time offset is the difference between its own start time and the earliest global start time; For subsequent videos of the same media type, the time offset is the difference between its own start time and the end time of the previous video; For the last video of the same media type, its end offset is the difference between the global latest end time and its own end time.

7. The method for automatic synchronization, merging, and archiving of audio and video recordings from rail flaw detection according to claim 4, characterized in that, When completing the time interval for audio and video files within a group, the processing of audio files includes: Calculate the time offset of the audio file relative to the global time base, generate a silent segment with a duration consistent with the time offset, and splice the silent segment to the beginning of the audio file to complete the time interval padding; The silent segment is a zero-value audio signal, and its sampling rate and number of channels are consistent with the corresponding audio file.

8. The method for automatic synchronization, merging, and archiving of audio and video recordings from rail flaw detection according to claim 6, characterized in that, If a certain media type does not have a corresponding file, a compensation video will be generated: A background video is generated at a preset resolution, and the corresponding media type is marked with "No Video". The duration of the compensated video is consistent with the difference between the latest global end time and the earliest global start time.

9. The method for automatic synchronization, merging, and archiving of audio and video recordings from rail flaw detection according to claim 1, characterized in that, Uploading the output file to the server includes: The output file is split into multiple pieces according to a preset size, and a list of piece information is generated. Upload the fragments sequentially according to the fragment information list and record the upload completion status of each fragment; After all fragments have been uploaded, the server merges the fragments based on the fragment information list to obtain the complete output file.

10. The method for automatic synchronization, merging, and archiving of audio and video recordings from rail flaw detection according to claim 1, characterized in that, The step of binding and archiving the output file with the corresponding flaw detection operation record includes: Extract the device serial number, operation location parameters, and global time interval from the standardized identification information in the output file; Query the flaw detection operation record database to match flaw detection operation records that contain the same equipment serial number, operation location parameters and operation time covering the global time interval; Write the storage path and filename of the output file into the matched flaw detection operation record; Mark the output file and the corresponding flaw detection operation record as archived.

11. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the automatic synchronization, merging, and archiving method for rail flaw detection audio and video as described in any one of claims 1-10.

12. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the automatic synchronization, merging, and archiving method for rail flaw detection audio and video as described in any one of claims 1-10.

Images