File synchronous playing method, system and device, storage medium and storage product

Abstract

The invention provides a file synchronous playing method, system and device, a storage medium and a program product, and the method comprises the steps: obtaining a target device file corresponding to each data device in a plurality of data devices of different types according to a first playing request instruction, and carrying out the timestamp alignment processing of data frames of a plurality of target device files, the plurality of files to be played comprise communication bus files, and data frames in different files to be played in the plurality of files to be played are synchronized according to generation timestamps; a plurality of to-be-played files are sent to a file receiving device, the file receiving device is used for synchronously sending data frames which are from a plurality of data devices and have different types but the same timestamp in the plurality of to-be-played files to a playing device, and the playing device is used for performing multi-channel synchronous playing on the plurality of data frames with the same timestamp. The synchronism of various types of data is ensured, the accuracy of vehicle scene reconstruction is improved, and the vehicle data analysis effect is ensured.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a file synchronous playback method, system, device, storage medium and storage product. Background Art

[0002] With the rapid development of multimedia technology, the diversity of multimedia content display is becoming more and more extensive, and the application scenarios of synchronous playback of multiple types of files are becoming more and more extensive.

[0003] The current method of synchronous playback of multiple types of files is usually to use a device to decode and render multimedia data, and then distribute the rendered multimedia data, such as audio and video data, to the playback device for playback to achieve synchronous playback of multiple types of files. However, the traditional method of synchronous playback of multiple types of files is not suitable for vehicles. Different types of vehicle files are difficult to play synchronously, resulting in the inability to reconstruct vehicle scenes based on multiple types of vehicle files, affecting the effect of vehicle data analysis. Summary of the invention

[0004] The present invention provides a file synchronous playback method, system, device, storage medium and storage product to solve the problem that different types of vehicle files are difficult to play synchronously, resulting in the inability to reconstruct vehicle scenes based on multiple types of vehicle files, affecting the effect of vehicle data analysis.

[0005] In a first aspect, an embodiment of the present application provides a method for synchronously playing a file, characterized by comprising:

[0006] receiving a first play request instruction, the first play request instruction requesting synchronous play of target device files of multiple data devices;

[0007] According to the first play request instruction, a target device file corresponding to each data device among the multiple data devices is obtained, and the types of the target device files corresponding to different data devices are different;

[0008] Performing timestamp alignment processing on data frames of multiple target device files to obtain multiple to-be-played files of different types, wherein the multiple to-be-played files include communication bus files, and data frames in different to-be-played files of the multiple to-be-played files are synchronized according to the generated timestamps;

[0009] A plurality of files to be played are sent to a file receiving device, which is used to synchronously send data frames of different types but with the same timestamp from a plurality of data devices in the plurality of files to be played to a playing device, and the playing device is used to perform multi-channel synchronous playing of the plurality of data frames with the same generated timestamp.

[0010] In a second aspect, an embodiment of the present application provides a file synchronous playback system, including:

[0011] The client is used to send a first play request instruction to the file processing device;

[0012] A file processing device, used to execute the above-mentioned file synchronous playback method;

[0013] A file receiving device, used for synchronously sending data frames of different types but with the same generation timestamp received from a plurality of data devices to a playback device;

[0014] The playback device is used for performing multi-channel synchronous playback of multiple data frames with the same generation timestamp sent by the file receiving device.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned file synchronization playback method when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned file synchronization playback method are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the above-mentioned file synchronization playback method.

[0018] In a sixth aspect, an embodiment of the present application provides a file synchronous playback device, characterized in that it includes:

[0019] A receiving unit, configured to receive a first play request instruction, wherein the first play request instruction requests synchronous play of target device files of multiple data devices;

[0020] an acquisition unit, configured to acquire a target device file corresponding to each data device among the plurality of data devices according to the first play request instruction, wherein the target device files corresponding to different data devices are of different types;

[0021] A processing unit is used to perform timestamp alignment processing on data frames of multiple target device files to obtain multiple to-be-played files of different types, wherein the multiple to-be-played files include communication bus files, and data frames of different to-be-played files among the multiple to-be-played files are synchronized according to the generated timestamps;

[0022] The sending unit is used to send multiple files to be played to the file receiving device, the file receiving device is used to synchronously send data frames of different types but with the same timestamps from multiple data devices in the multiple files to be played to the playing device, and the playing device is used to perform multi-channel synchronous playback of multiple data frames with the same generated timestamps.

[0023] In a scheme provided by the above-mentioned file synchronous playback method, system, device, storage medium and program product, a first playback request instruction is received, the first playback request instruction requests the synchronous playback of target device files of multiple data devices, and according to the first playback request instruction, the target device file corresponding to each data device in the multiple data devices is obtained, the types of target device files corresponding to different data devices are different, and the data frames of the multiple target device files are timestamp aligned to obtain multiple files to be played of different types, the multiple files to be played include communication bus files, and the data frames of different files to be played in the multiple files to be played are synchronized according to the generated timestamp; the multiple files to be played are sent to a file receiving device, the file receiving device is used to synchronously send data frames of different types but with the same timestamps from multiple data devices in the multiple files to be played to the playback device, and the playback device is used to perform multi-channel synchronous playback of multiple data frames with the same generated timestamp; multiple types of vehicle files including communication bus files can be classified and timestamp aligned, so that data frames from different types of devices but with the same timestamps can be synchronously sent and played, ensuring the synchronization of various types of data, improving the accuracy of vehicle scene reconstruction, and ensuring the vehicle data analysis effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0025] Figure 1 is a structural schematic diagram of a file synchronous playback system in one embodiment of the present invention;

[0026] Figure 2 is another structural schematic diagram of a file synchronous playback system in one embodiment of the present invention;

[0027] Figure 3 is a signaling interaction diagram of a file synchronous playback method in one embodiment of the present invention;

[0028] Figure 4 yes Figure 3 A schematic diagram of an implementation process of step S20;

[0029] Figure 5 yes Figure 3 A schematic diagram of an implementation process of step S30;

[0030] Figure 6 yes Figure 3A schematic diagram of an implementation process of step S40;

[0031] Figure 7 is another schematic diagram of the implementation process of the file synchronous playback method in one embodiment of the present invention;

[0032] Figure 8 is a structural schematic diagram of a file processing device in one embodiment of the present invention;

[0033] Fig. 9 It is a schematic diagram of the structure of an electronic device in one embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or their collections. It should also be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0036] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0037] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0038] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0039] The synchronous file playback method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In the file synchronous playback system shown, the file synchronous playback system includes a client, a file processing device, a file receiving device and a playback device. The file processing device communicates with the client through a network, the file processing device communicates with the file receiving device through a network or a bus, and the file receiving device communicates with the playback device through a network or a bus.

[0040] The client is used to send a first play request instruction (eg, a synchronous play request instruction, a play state control instruction (eg, a loop play instruction)) to the file processing device. For example, the synchronous play request instruction may include file play request information of multiple data devices of different types.

[0041] The file processing device is used to receive a first playback request instruction, and to obtain multiple target device files of different types according to the first playback request instruction (from a server or from multiple data devices), and to perform timestamp alignment processing on the data frames of the multiple target device files to obtain multiple files to be played of different types, so that the data frames in each file to be played are synchronized according to the generated timestamp. The file processing device is also used to send multiple files to be played to the file receiving device, and the files to be played include communication bus files. The file receiving device is used to synchronously send data frames of different types but with the same generated timestamps from multiple data devices in multiple playback files to the playback device. The playback device is used to perform multi-channel synchronous playback of multiple data frames with the same generated timestamps sent by the file receiving device.

[0042] As an example, in the embodiments of the present application, the file processing device, the file receiving device and the playback device may belong to different modules of the same device, or the file processing device and the file receiving device may belong to different modules of the same device, while the playback device belongs to a module of another device, or the file processing device belongs to one device, while the file receiving device and the playback device belong to a module of another device. The embodiments of the present application are not limited to this.

[0043] The file synchronization playback method in this embodiment can classify and align timestamps of multiple types of vehicle files including communication bus files, so that data frames from different types of devices but with the same timestamps can be sent and played synchronously, ensuring the synchronization of various types of data, improving the accuracy of vehicle scene reconstruction, and ensuring the effectiveness of vehicle data analysis.

[0044] In one embodiment, the file processing device may include multiple file sending systems of different types, each file sending system is used to send files of one type; through the multiple file sending systems of different types, the to-be-played files of the corresponding file types of the file sending systems are synchronously sent to the file receiving device in a frame-by-frame manner, and each time, data frames of different types but with the same timestamps from multiple data devices in the multiple files to be played are sent to the file receiving device.

[0045] In other embodiments, the file synchronization playback system also includes other devices, such as a server (such as a cloud), multiple data devices of different types. Such as video devices, audio devices, bus interaction devices (devices that interact through a communication bus) and other types of devices. Among them, the server is used to store the file processing device to perform timestamp alignment processing on the data frames of multiple target device files to obtain multiple files of different types to be played.

[0046] For example, Figure 2 As shown, the file processing device may include a video sending system, an audio sending system, a bus file sending system, and other types of file sending systems (eg, XX sending system).

[0047] Among them, the video sending system is used to send video files to the file receiving device; the audio sending system is used to send audio files to the file receiving device; and the bus file sending system is used to send communication bus files to the file receiving device.

[0048] Among them, the client, also known as the user end, is a program that provides local services to customers. The client can be installed on but not limited to various personal computers, laptops, smart phones, tablets and portable wearable devices. Multiple data devices may include various sensors and vehicle control systems of the vehicle. The file receiving device and the playback device may be related devices or functions of the audio and video playback system on the vehicle. The file processing device and the file processing device may be a data processing device in the vehicle, or a server that communicates with the vehicle. The server may be implemented as an independent server or a server cluster consisting of multiple servers.

[0049] In one embodiment, if Figure 3 As shown, a method for synchronously playing files is provided, and the method is applied in Figure 1 The file synchronization playback system in the example is described as follows, including the following steps:

[0050] S10: The file processing device receives a first play request instruction.

[0051] The first play request instruction requests synchronous play of target device files of multiple data devices. For example, the file processing device may receive a first play request instruction from a client for synchronous play of target device files of multiple data devices. For example, the first play request instruction may be a synchronous play request instruction.

[0052] When it is necessary to construct a historical scene for the vehicle, that is, when it is necessary to synchronously play device files of different types of data devices on the vehicle, the user can send a first play request instruction to the file processing device through the client, and the first play request instruction is used to request synchronous play of target device files of multiple data devices. The client can send a request message of the first play request instruction to the file processing device in the form of Ethernet.

[0053] S20: The file processing apparatus obtains a target device file corresponding to each data device among the multiple data devices according to the first play request instruction.

[0054] The file processing device receives a first play request instruction sent by the client, and acquires a plurality of target device files of different types from a plurality of data devices of different types according to the first play request instruction. The types of target device files corresponding to different data devices are different.

[0055] The first play request instruction includes file request play information, and the file request play information includes a target time period and device information of multiple data devices. The target time period includes the data start time (begin time) and the data end time (end time) that need to be played synchronously. The device information is used to specify the information of the data device. For example, the device information may include one or more of the following information: device serial number (serial number, sn), device type (device type), device channel number (cin), and device file storage medium (torage edium).

[0056] After receiving the first play request instruction sent by the client, the file processing device parses the first play request instruction to obtain file play request information, which includes the target time period and device information of multiple data devices. The file processing device obtains the target device file within the target time period from the device file storage medium of each data device according to the device information of the data device, and obtains multiple target device files of different types.

[0057] Optionally, in a possible implementation of the present application, after parsing and obtaining the file request playback information, the file processing device may persistently store the file request playback information in a database, so that when the service is subsequently restarted abnormally, the file request playback information can be read from the database and the first playback request instruction can be re-responded to perform synchronous playback of multiple types of files on multiple data devices; when the file playback ends and the first playback request instruction is completed, the file processing device can delete the persistent file request playback information from the database.

[0058] S30: The file processing device performs timestamp alignment processing on data frames of multiple target device files to obtain multiple to-be-played files of different types.

[0059] Timestamp alignment refers to synchronizing data frames with different timestamps in different data sources so that data frames with the same timestamps in different data sources are associated and matched. After acquiring multiple target device files of different types, the file processing device performs timestamp alignment processing on the data frames of the multiple target device files, that is, one-to-one correspondence is made between the data frames with the same generation timestamp, so that the data frames in each target device file are synchronized according to the generation timestamp, and multiple files to be played of different types are obtained, that is, the data frames of different files to be played among the multiple files to be played are synchronized according to the generation timestamp.

[0060] For example, the types of multiple target device files are all different. According to the generation timestamps of each data frame in a target device file, the data frames in the target device file are timestamp aligned, and the multiple data frames after the timestamp alignment are arranged in sequence according to the order of the generation timestamps to obtain a file to be played; the target device files are traversed to obtain multiple files to be played of different types.

[0061] It can be understood that in the embodiment of the present application, if the type of a file to be played is a video type, then the file to be played can also be called a video file. If the type of a file to be played is a communication bus type, then the file to be played can also be called a communication bus file. If the type of a file to be played is an audio type, then the file to be played can also be called an audio file.

[0062] The plurality of files to be played include at least a video file and a communication bus file. The communication bus file may be a CAN file. In other embodiments, the communication bus file may also be a LIN bus or other vehicle communication bus file. In other embodiments, the plurality of files to be played also include an audio file.

[0063] S40: The file processing device sends a plurality of files to be played to the file receiving device.

[0064] After the file processing device performs timestamp alignment processing on the data frames of the files of the multiple target devices, the file processing device sends the processed multiple files to be played to the file receiving device. The file processing device is provided with multiple file sending systems of different types, and each file sending system can send files of one type. Through the multiple file sending systems of different types, the files to be played of the corresponding file types of the file sending system are sent to the file receiving device synchronously in a frame-by-frame manner, and each time, the data frames of the multiple files to be played from multiple data devices with different types but the same timestamps are sent to the file receiving device. The frame-by-frame sending means that when sending the files to be played, only one data frame of the files to be played is sent each time, and after sending the current data frame, the sending of new data frames is stopped, and the waiting state is entered to wait for the next data frame to be sent.

[0065] For example, the multiple files to be played include video files, audio files and communication bus files; correspondingly, the multiple file sending systems include a video sending system, an audio sending system and a bus file sending system. The file processing device sends the data frames of the video file frame by frame through the video sending system, synchronously sends the data frames of the audio file frame by frame through the audio sending system, and synchronously sends the data frames of the communication bus file frame by frame through the bus file sending system.

[0066] Each file sending system is responsible for sending a specific file type, such as a video file, an audio file, or a communication bus file. The file processing system selects the appropriate transmission protocol and processing method by identifying the type of each file to be played. Among them, the video sending system can adopt protocols such as Real Time Streaming Protocol (RTSP), Real-Time Transport Protocol (RTP) RTP or HTTP-based streaming media transmission protocol HTTPStreaming, and send video data frame by frame to ensure real-time transmission of continuous images. The audio sending system can adopt the Voice over Internet Protocol (VoIP) protocol based on IP or Web Real-Time Communications WebRTC (Web Real-Time Communications) to send audio files frame by frame, provide low-latency transmission optimization, and maintain real-time voice or audio synchronization. The bus sending system can use a lightweight protocol to send communication bus files frame by frame to ensure that the communication bus files are synchronized with other data.

[0067] Among them, in order to ensure that the system timestamps of the file processing device and the file receiving device can be consistent and improve the data control accuracy when sending multiple files to be played, it is necessary to perform time synchronization calibration on the file processing device and the file receiving device. The file processing device can be a chrony server, and the file receiving device can be used as a chrony client. The chrony client is calibrated by the chrony server, and the synchronization frequency of the system timestamp takes the minimum interval, that is, once every 64 seconds. In this way, the timestamps of the file sending device and the file receiving device can be consistent, and preparations can be made for the subsequent synchronous sending of multiple video files in the video sending system, multiple audio files in the audio sending system, and multiple CAN files in the bus file system (such as the CAN file system).

[0068] S50: The file receiving device synchronously sends data frames of different types but with the same timestamp from multiple data devices in multiple files to be played to the playing device.

[0069] The file receiving device receives data frames of multiple files to be played sent by the file processing device, and synchronously sends data frames of different types but with the same timestamps from multiple data devices among the received multiple files to be played to the playing device.

[0070] For example, multiple files to be played include a video file, an audio file and a CAN file. The generation timestamps of video frame 1 in the video file, audio frame 1 in the audio file, and communication data frame 1 in the CAN file are the same. Then, the file receiving device synchronously sends video frame 1 in the video file, audio frame 1 in the audio file, and communication data frame 1 in the CAN file to the playback device.

[0071] S60: The playback device performs multi-channel synchronous playback on multiple data frames with the same generation timestamp sent by the file receiving device.

[0072] The playback device receives data sent to the file receiving device, and performs multi-channel synchronous playback of multiple data frames with the same generation timestamp sent by the file receiving device, so that data frames with different types but the same timestamp from multiple data devices can be played synchronously, ensuring the synchronization of various types of data playback. Among them, the playback device includes multiple playback devices of the same type as each data device, the number of playback devices is greater than the number of data devices, and at least one playback device can play data frames from corresponding types of data devices in multiple files to be played. The playback devices in the playback device include video playback devices, audio playback devices, and bus data playback devices for communication bus files. The playback device receives multiple data frames with the same generation timestamp but different types sent by the file receiving device (including video frames collected by multiple video acquisition devices, audio frames collected by multiple audio acquisition devices, and multiple control systems interacting through the communication bus to generate communication data frames), and synchronizes the multiple data frames with the same generation timestamp but different types through different types of playback devices, and the timestamps displayed on all playback screens are the same.

[0073] By performing multi-channel synchronous playback of multiple data frames of different types with the same timestamp, the synchronous presentation of multimedia content or multi-dimensional data can be achieved. For example, different types of data such as video, audio, and communication bus data can be played simultaneously and maintain a consistent timeline, thereby providing users with a unified, multi-dimensional information display experience. The multi-type file synchronization and timestamp alignment technology in this embodiment can support complex data fusion scenarios, especially in autonomous driving scenarios, where data such as bus messages of cameras, radars, lidars, and related control systems need to be strictly synchronized to ensure that all input data can be analyzed based on the same timeline in subsequent data processing.

[0074] In this embodiment, the file processing device receives a first play request instruction, and the first play request instruction requests the synchronous play of target device files of multiple data devices. After receiving the first play request instruction, the file processing device obtains the target device file corresponding to each data device in the multiple data devices according to the first play request instruction, and the types of target device files corresponding to different data devices are different, and the data frames of the multiple target device files are aligned by time stamps to obtain multiple files to be played of different types, and the data frames of different files to be played in the multiple files to be played are synchronized according to the generated timestamps, and the multiple files to be played are sent to the file receiving device. Then, the file receiving device synchronously sends the data frames of different types but the same generated timestamps from multiple data devices in the multiple files to be played to the playing device, and the playing device performs multi-channel synchronous play on the multiple data frames with the same generated timestamps sent by the file receiving device. The file synchronous play method can classify and timestamp-align multiple types of vehicle files including communication bus files, so that the data frames from different types of devices but with the same timestamps can be synchronously sent and played, so that the display of data matches the actual acquisition time, and the synchronization of various types of data is ensured, thereby enhancing the accuracy of data display, improving the accuracy of vehicle scene reconstruction, and ensuring the vehicle data analysis effect.

[0075] In addition, multiple files to be played are time stamped and then sent to the playback device for synchronous playback, which reduces the need for additional synchronization processing of data in the subsequent synchronous playback of file data frames, reduces the computational complexity of the system, and optimizes the performance of the system.

[0076] In one embodiment, the first play request instruction includes file play request information, and the file play request information includes a target time period and device information of multiple data devices of the same type. Figure 4 As shown, in step S20, according to the first play request instruction, a target device file corresponding to each data device in a plurality of data devices is obtained, which specifically includes the following steps:

[0077] S21: The file processing apparatus determines the storage address of the device file in the target time period in each data device according to the device information of each data device, and obtains the target file address of each data device.

[0078] In this embodiment, the first play request instruction includes file play request information, and the file play request information includes a target time period and device information of multiple data devices of the same type. The target time period includes the data start time (begin time) and data end time (end time) that need to be played synchronously; the device information includes the device serial number (sn), device type (device type), channel number (cin), and device file storage medium (storage medium).

[0079] The file processing device determines the data device corresponding to the device serial number, searches for the device file in the target time period in the channel number (cin) file in the device file storage medium of the data device, determines the storage address of the device file of the data device in the device file storage medium, and obtains the target file address of the device file of the data device; traverses all data devices in the file request playback information to obtain the target file address of each data device.

[0080] S22: The file processing device controls the file sending system of the corresponding type to create multiple download clients according to the target file addresses of the data devices of the same type, and controls each download client to establish a link with each data device of the same type.

[0081] The file processing device controls the file sending system of the corresponding type to create multiple download clients according to the target file addresses of each data device of the same type, and controls each download client to establish a link with each data device of the same type. Among them, a download client is linked to a data device through a preset streaming protocol. The download client is a download program created by the file processing device.

[0082] Each download client is linked to a data device separately, which can effectively distribute the network load and device load, avoid a single device from taking on too many data transmission tasks, and improve the stability and performance of the overall system. The file processing device dynamically creates download clients according to the number of data devices, supports the access of large-scale devices, improves the flexibility of data acquisition, and can adapt to the growth of the number of devices and data volume.

[0083] S23: The file processing device obtains the device files of each data device in the target time period through each download client in the corresponding file sending system, and aggregates them to obtain target device files of the same type.

[0084] The file processing device obtains the device files corresponding to each data device through each download client of the file sending system. The device file is the target device file corresponding to the target file address, and the target device file of the same type can be obtained through multiple download clients, wherein the target device file includes the device files of multiple data devices of the same type within the target time period. Files are obtained for different types of data devices through different file sending systems to obtain multiple target device files of different types. By obtaining data in parallel through multiple download clients, the file acquisition speed of the same type of device can be greatly improved, avoiding the bottleneck problem of single-point data collection.

[0085] For example, the data device is a camera device, and the file sending system of the same type is a video sending system; the file processing device first determines the storage address of the device file in the target time period in the camera device according to the device information of the camera device, obtains the target file address of the camera device, and then creates a download client according to the target file address of the camera device, and establishes a specific streaming protocol between the download client and the camera device, such as the Real Time Streaming Protocol RTSP (Real Time Streaming Protocol), the download client establishes a link with the camera device through the streaming protocol, and then obtains the device file corresponding to the target file address from the camera device.

[0086] In this embodiment, the file processing device determines the storage address of the device file in the target time period in each data device according to the device information of each data device of the same type, and obtains the target file address of each data device; according to the target file address of each data device of the same type, controls the file sending system of the corresponding type to create multiple download clients, and controls each download client to establish a link with each corresponding data device respectively, and one download client is linked to one data device through a preset streaming protocol; each download client of the file sending system obtains the device file corresponding to each data device respectively, and obtains the target device file of the same type, and the target device file includes the device files of multiple data devices of the same type in the target time period. The acquisition process of the target device file is clarified, and the data files in the specified time period are obtained through the streaming protocol, which improves the efficiency and reliability of data acquisition, especially when facing the data collection task of a large number of devices, it can effectively disperse the load and maintain data consistency.

[0087] In one embodiment, if Figure 5 As shown, in step S30, the data frames of multiple target device files are aligned with the timestamps to obtain multiple files to be played of different types, which specifically includes the following steps:

[0088] S31: The file processing device aligns the timestamps of the data frames in each target device file according to the generation timestamps of each data frame in each target device file, and caches the multiple data frames after the timestamp alignment in the order of the generation timestamps into the file sending system of the corresponding file type, thereby obtaining multiple files to be played of different types.

[0089] Multiple data devices (such as cameras, sensors, recording devices, etc.) generate a data frame within a predetermined time. Each data frame carries a generation timestamp to record the exact time when the data frame was generated. Before device file transmission, the data frames in each target device file are timestamped according to the generation timestamps of each data frame in the target device file using a timestamp alignment algorithm. The multiple data frames in each target device file with aligned timestamps are cached in the consumption queue of the file sending system of the corresponding file type in the order of the generation timestamps, and multiple files to be played of different types are obtained.

[0090] In this embodiment, through timestamp alignment, data frames collected by different data devices can be uniformly processed according to the actual time sequence generated, which can ensure that the data of multiple devices can be displayed synchronously during playback, avoid data asynchronization problems caused by time differences between different devices, and improve the accuracy of data synchronization. After the data frames are aligned with timestamps, they are cached to the file sending system in time sequence, and are distinguished based on file types, which can ensure that in the subsequent file synchronization playback process, the data frames of each type of file to be played can be processed in the correct order and timing, avoiding problems caused by file disorder or delay.

[0091] In one embodiment, the target device file includes device files of multiple data devices of the same type within the target time period. In step S31, the data frames in each target device file are timestamped according to the generation timestamps of each data frame in each target device file, and the multiple data frames after the timestamp alignment are cached in the order of the generation timestamps to the file sending system of the corresponding file type, so as to obtain multiple files to be played of different types, which specifically includes the following steps:

[0092] S311: The file processing device controls each file sending system to decrypt the device file of each data device in the target device file of the corresponding file type, and obtains the decrypted data frame of each data device.

[0093] The target device file includes the device files of multiple data devices of the same type within the target time period. To ensure data security, the data device encrypts and stores the data after collecting it. After the file sending system obtains the target device file, it needs to be decrypted. The file processing device controls each file sending system to decrypt the device files of each data device in the target device file of the corresponding file type, and obtains the decrypted data frames of each data device.

[0094] Different file types may use different encryption methods, and each file sending system selects a suitable decryption algorithm according to the file type (such as video, audio, sensor data, etc.). The file processing device can process the decryption operations of multiple device files in the target device file in parallel. For example, multiple device files can be decrypted separately using multi-threading or multi-processing to reduce decryption time and improve efficiency.

[0095] S312: The file processing device stores the decrypted data frames of the data device in the producer queue of the data device in the order of the generation timestamps of the data device, and splices the device channel number of the data device to each data frame in the producer queue of the data device to obtain the files to be processed of the data device.

[0096] The file processing device stores the decrypted data frames of the data device in the producer queue of the data device in the order of the generation timestamp of the data device, and splices the device type and device channel number of the data device to each data frame in the producer queue of the data device to obtain the to-be-processed file of the data device. The information of the data frame includes: generation timestamp, frame sequence number ID, frame length, frame binary content, etc.

[0097] By storing data frames in the order of generated timestamps, the consistency of time sequence is ensured, so that subsequent data splicing and playback are carried out according to a unified timeline. Each data frame is spliced ​​with the device type and device channel number of the data device. During subsequent processing, the data frames can be sorted according to the device type and device channel number, which facilitates the subsequent playback device to install the channel number sequence to allocate data frames, reduce the complexity of data processing, and improve playback synchronization.

[0098] S313: The file processing device splices the data frames in the to-be-processed files of each data device with the same generation timestamp in ascending order according to the device channel numbers of the data devices, and puts the spliced ​​data frames into the consumption queue of the file sending system of the corresponding file type in the order of the generation timestamps to obtain the to-be-played files of the corresponding file type.

[0099] The file processing device splices the data frames in the to-be-processed files of each data device with the same generation timestamp in ascending order according to the device channel numbers of the data devices, and puts the spliced ​​data frames into the consumption queue of the file sending system of the corresponding file type in the order of the generation timestamps to obtain the to-be-played files of the corresponding file type.

[0100] When stitching, the frames can be aligned according to the frame ID and frame length. For example, when processing a video file, stitching may include combining the images of multiple cameras into one video frame and arranging the data frames of different cameras in the order of the device channel number.

[0101] The consumption queue is used to store data frames that are finally arranged in the order of generation timestamp and device channel number. This queue can be based on the First In First Out (FIFO) structure to ensure that subsequent data frames can be played in the order of generation timestamp and device channel number. In order to ensure that there is no delay during playback, the consumption queue should have a certain buffer depth, especially when the transmission or decryption speed is unstable, the cache can ensure the continuity of playback.

[0102] By generating a dual sorting mechanism of timestamp and channel number, it is ensured that the data frames of each data device can be spliced ​​and processed in the correct order, avoiding data loss or sequence errors; the spliced ​​data frames enter the consumption queue in the order of generation timestamps, ensuring that the data streams of multiple devices can be played synchronously, providing a smooth multi-device data display effect.

[0103] In this embodiment, decryption, timestamp alignment, splicing and sorting are used to ensure the synchronization and sequential playback of multiple data devices and multiple types of data. Through queue design and device channel number splicing, efficient data processing and transmission can be guaranteed, while ensuring data integrity and synchronization, which is suitable for complex multimedia scenarios or multi-device collaboration environments.

[0104] In one embodiment, if Figure 6 As shown, in step S40, multiple files to be played are sent to the file receiving device, that is, multiple file sending systems of different types send files to be played of corresponding file types of the file sending system to the file receiving device in a frame-by-frame manner, specifically including the following steps:

[0105] S41: The file processing device synchronously sends the first data frame of the to-be-played file corresponding to the file sending system to the file receiving device through the file sending system corresponding to each to-be-played file.

[0106] In this embodiment, the file processing device is provided with multiple file sending systems, each of which can send a type of file. After obtaining multiple files to be played, the file processing device synchronously sends the first data frame of the file to be played corresponding to the file sending system to the file receiving device through the file sending system corresponding to each file to be played, thereby realizing synchronous sending of data frames.

[0107] For example, multiple files to be played include video files, audio files and communication bus files. Correspondingly, multiple file sending systems include a video sending system, an audio sending system and a bus file sending system. The file processing device sends the first data frame of the video file through the video sending system, synchronously sends the first data frame of the audio file through the audio sending system, and synchronously sends the first data frame of the communication bus file through the bus file sending system.

[0108] Among them, the file request playback information of the first playback request instruction also includes the file playback trigger time and / or the file playback rate. The file processing device can control the file sending system corresponding to each file to be played, and synchronously send the first data frame in the file to be played corresponding to the file sending system to the file receiving device at the file playback trigger time. The synchronous playback of different types of files at the user-specified time point improves the user experience.

[0109] S42: The file processing device receives the data frame receiving time of each to-be-played file from the file receiving device.

[0110] After the file processing device synchronously sends the data frames in the file to be played corresponding to the file sending system to the file receiving device through the file sending system corresponding to each file to be played, the file receiving device receives different types of data frames sent by each file sending system in the file processing device, and records the receiving time of each type of data frame, which is fed back to the file processing device as the data frame receiving time. The file processing device receives the data frame receiving time of each file to be played from the file receiving device.

[0111] Among them, after the file receiving device receives different types of data frames sent by each file sending system in the file processing device, it feeds back data frame reception information of different files to be played to the file processing device, and the data frame reception information includes the data frame type, the data frame reception time, the time when the data frame is sent to the playback device (such as time stamp = 1698647253000), the playback device and device type of the data frame and the unique identification ID (such as camera id = 100, camera is the device type, id = 100 is the unique identification ID).

[0112] Among them, in order to reduce the load of the file processing device and ensure the stability of data frame transmission, the client can monitor the information feedback of the file receiving device; when the file receiving device receives different types of data frames sent by each file sending system in the file processing device, the file receiving device sends the data frame receiving time of each file to be played to the client, and after the client monitors the data frame receiving time of each file to be played, it sends the data frame receiving time of each file to be played to the file processing device, and the file processing device obtains the data frame receiving time of each file to be played.

[0113] S43: The file processing device determines the data frame receiving time deviation of each to-be-played file according to the data frame receiving time of the to-be-played file.

[0114] After receiving the data frame receiving time fed back from the file receiving device, the file processing device determines the data frame receiving time deviation of each to-be-played file according to the received data frame receiving time of each to-be-played file.

[0115] For example, based on the data frame last received by the file receiving device in the current data frame receiving time of each file to be played, and taking the receiving time of the last received data frame as the reference time, the difference between the receiving time of the data frame of other types of files to be played and the reference time is determined as the data frame receiving time deviation of the file to be played.

[0116] S44: The file processing device adjusts the time for each file sending system to send the next data frame of the to-be-played file of the corresponding file type to the file receiving device according to the data frame receiving time deviation of each to-be-played file, so that the file receiving device receives data frames with the same timestamp in multiple to-be-played files.

[0117] Since the transmission links of data frames of different files are different, the time required for data transmission of the same data size is different, and the data frame sizes of different types of files are different, when sending the next data frame, the file processing device adjusts the time for each file sending system to send the next data frame of the corresponding file type to the file receiving device according to the data frame receiving time deviation of each file to be played as needed, so that the file receiving device receives data frames with the same timestamp in multiple files to be played, thereby realizing dynamic compensation of the receiving time of different data transmission links; then, steps S142 to S44 are looped until all data frames in multiple files to be played are sent.

[0118] For example, based on the data frame of the file to be played that was last received during the current data frame receiving time of each file to be played, the file sending system of the type corresponding to the data frame that was last received can be controlled to send the next data frame of the file to be played of that type; after waiting for the data frame receiving time deviation of other files to be played, the next data frame of the file to be played of the corresponding type can be sent through the file sending systems of the corresponding types of other files to be played.

[0119] In this embodiment, the file processing device synchronously sends the first data frame of the file to be played corresponding to the file sending system to the file receiving device through the file sending system corresponding to each file to be played, then receives the data frame receiving time of each file to be played from the file receiving device, and determines the data frame receiving time deviation of each file to be played according to the data frame receiving time of each file to be played, and then adjusts the time of each file sending system sending the next data frame of the file to be played of the corresponding file type to the file receiving device according to the data frame receiving time deviation, so that the file receiving device receives data frames with the same timestamp in multiple files to be played until all data frames in multiple files to be played are sent; when multiple (long) links synchronously play multiple types of files of different types of data devices, by real-time monitoring the difference in data frame receiving timestamps of different types of files, dynamically compensating the receiving time of subsequent file data frames, it can ensure that the file receiving device synchronously receives data frames with the same timestamp generated in various types of files to be played, ensure the synchronization and continuity of the playback of different types of data frames, and realize long-term synchronous playback of multiple types of files of multiple data devices.

[0120] In one embodiment, the plurality of files to be played include a first file to be played, and the type of the first file to be played is a video type, that is, the first file to be played is a video file. In step S42, the data frame receiving time deviation of each file to be played is determined according to the data frame receiving time of each file to be played, which specifically includes the following steps:

[0121] S421: Determine the data frame receiving time of the video file among the data frame receiving times of each file to be played;

[0122] S422: Taking the data frame receiving time of the video file among the data frame receiving times of the files to be played as the time reference point, determining the time difference between the data frame receiving time of other types of files to be played and the time reference point, and obtaining the data frame receiving time deviation of other types of files to be played.

[0123] For example, multiple files to be played include a video file, a communication bus file, and an audio file. When determining the data frame receiving time of the video file after sending the first data frame of the video file, the data frame receiving time of the video file can be used as the time reference point, and then the time difference between the data frame receiving time corresponding to the communication bus file and the audio file and the time reference point is determined. Then, based on the time difference between the data frame receiving time of the communication bus file and the time reference point, the data frame receiving time deviation of the communication bus file is determined. Based on the time difference between the data frame receiving time of the audio file and the time reference point, the data frame receiving time deviation of the audio file is determined.

[0124] After the file sending system corresponding to each file to be played synchronously sends the data frame of the file to be played corresponding to the file sending system to the file receiving device, the file processing device needs to receive the data frame receiving time of each file to be played fed back from the file receiving device, and determine the video frame receiving time of the video file in the data frame receiving time of each file to be played; before sending the next data frame of each file to be played, it is necessary to use the video frame receiving time of the video file in the data frame receiving time of each file to be played as the time reference point, determine the time difference between the data frame receiving time of other types of files to be played and the time reference point, and obtain the data frame receiving time deviation of other types of files to be played. Among them, each data frame receiving time is the time when the file processing device has received the corresponding type of data frame.

[0125] For example, the multiple files to be played include video files, audio files and communication bus files, and the data frame receiving time includes the video frame receiving time of the video file, the audio frame receiving time of the audio file and the communication data frame receiving time of the communication bus file. When the file processing device receives the data frame receiving time fed back from the file receiving device each time, the video frame receiving time in the current data frame receiving time is used as the time reference point to determine the time difference between the audio frame receiving time and the video frame receiving time to obtain the data frame receiving time deviation of the audio file; determine the time difference between the communication data frame receiving time and the video frame receiving time to obtain the data frame receiving time deviation of the communication bus file.

[0126] In this embodiment, since the data of the video frame is large and the data transmission time is long, the video frame reception time of the video file is used as the time reference point, and the time difference between the data frame reception time of other types of files to be played and the time reference point is determined to obtain the data frame reception time deviation of other types of files to be played, and then adjust the subsequent sending interval of the data frames of other types of files to be played, so that the file receiving device can receive different types of data frames with the same generated timestamp as synchronously as possible, and send them to the player for synchronous playback, so as to correct the time difference of the synchronous playback of different types of files such as video files, audio files, CAN files, etc. in real time, and improve the synchronization of the playback of different files.

[0127] In one embodiment, the multiple files to be played also include video files. In step S44, the time for each file sending system to send the next data frame of the corresponding file type to the file receiving device is adjusted so that the file receiving device receives the data frames with the same timestamp in the multiple files to be played, which specifically includes the following steps:

[0128] S441: The file processing device sends the next data frame of the first file to be played through the file sending system corresponding to the first file to be played.

[0129] It is understandable that the next data frame of the first file to be played may be the second data frame of the first file to be played, that is, a data frame that is temporally located after the first data frame and temporally adjacent to the first data frame. The first file to be played is a video file.

[0130] S442: the file processing device sends the next data frame of the second file to be played to the file receiving device through the file sending system corresponding to the second file to be played after delaying the data frame receiving time deviation of the second file to be played.

[0131] It can be understood that the next data frame of the second file to be played may be the second data frame of the second file to be played, that is, a data frame that is temporally located after the first data frame and temporally adjacent to the first data frame.

[0132] The second file to be played is any other type of file to be played except the first file to be played among the multiple files to be played. For example, the second file to be played may be an audio file or a CAN file.

[0133] Among them, the file processing device receives the data frame reception time fed back from the file receiving device, and uses the video frame reception time of the video file in the data frame reception time as the time reference point to determine the time difference between the data frame reception time of other types of files to be played and the time reference point, and obtains the data frame reception time deviation of other types of files to be played; then, the file processing device first controls the file sending system (i.e., the video sending system) corresponding to the video file to send the next video frame of the video file, and for any other type of files to be played, after delaying the data frame reception time deviation of the file to be played, the next data frame in the file to be played is sent to the file receiving device through the file sending system corresponding to the file to be played.

[0134] For example, after obtaining the data frame receiving time deviation of other types of files to be played, the file processing device notifies other types of file sending systems (such as audio sending systems and communication file sending systems) other than the file sending system (i.e., video sending system) corresponding to the video file, and sets a timer. The duration of the timer timing of each file sending system is the data frame receiving time deviation of the corresponding type of file to be played; the file processing device first controls the file sending system (i.e., video sending system) corresponding to the video file to send the next video frame of the video file, and synchronously notifies the file sending systems of other types of files to be played (such as audio sending systems and communication file sending systems) to start the timer for timing, so as to send the next data frame of the corresponding file to be played when the timer ends.

[0135] In this embodiment, the video frame receiving time of the video file in the data frame receiving time is used as the time reference point, and the time difference between the data frame receiving time of other types of files to be played and the time reference point is determined, and then the next video frame of the video file is sent through the file sending system corresponding to the video file. For any other type of file to be played, after delaying the data frame receiving time deviation of the file to be played, the next data frame in the file to be played is sent to the file receiving device through the file sending system corresponding to the file to be played, which can ensure that the file receiving device can synchronously receive data frames of different types of files with the same generation timestamp, and ensure that subsequent data frames of different types can be played synchronously.

[0136] In one embodiment, each to-be-played file is stored in a consumption queue of a corresponding file type, and the consumption queue of each file sending system consumes the stored to-be-played files frame by frame, so as to send data frames of the to-be-played files of the corresponding file type to the file receiving device frame by frame. In step S40, multiple to-be-played files are sent to the file receiving device, that is, multiple file sending systems of different types send the to-be-played files of the corresponding file type of the file sending system to the file receiving device in a synchronous manner frame by frame, specifically including the following steps:

[0137] Determine the generation timestamp interval of two adjacent data frames in each file to be played, and determine the actual sending timestamp interval of two adjacent data frames in each file to be played. When the generation timestamp interval of two adjacent data frames in the file to be played is greater than or equal to the actual sending timestamp interval, the difference between the generation timestamp interval and the actual sending timestamp interval is used as the sleep time of the consumption queue in the file sending system corresponding to the file type of the file to be played when consuming two adjacent data frames.

[0138] When the interval between the generated timestamps of two adjacent data frames in the file to be played is less than the interval between the actual sent timestamps, the time required for the consumption queue in the file sending system to consume multiple data frames within a preset time interval (such as 5ms) is determined, and the total consumption time of the data frames in the preset time interval is obtained; the time difference between the preset time interval and the total consumption time of the data frames in the preset time interval is used as the sleep time of the consumption queue in the file sending system corresponding to the file type of the file to be played when consuming two adjacent data frames. This can not only reduce system calls, but also improve the accuracy of the sleep timestamp of the consumer thread.

[0139] In one embodiment, the file request playback information of the first playback request instruction also includes the file playback trigger time and / or the file playback rate. In step S40, that is, sending multiple files to be played to the file receiving device, specifically includes the following steps:

[0140] S401: According to the file play trigger time and / or the file play rate in the first play request instruction, the to-be-played file of the corresponding file type is sent to the file receiving device in a frame-by-frame manner through different types of file sending systems.

[0141] Among them, the first play request instruction includes file request play information, and the file request play information will include a predetermined file play trigger time, that is, the specific time when each to-be-played file should start to be sent and played. The file play trigger time is usually based on the global clock of the entire file synchronous play system to ensure that different types of file sending systems can start sending data at the same time. The file request play information also includes a file play rate, which refers to the rate at which each to-be-played file is sent and played (such as 30 frames per second), which determines the time interval for each type of file sending system to send the data frames of the corresponding file type to be played frame by frame. For example, 30 frames / second means that the interval between each frame sent is 33.33 milliseconds.

[0142] Each file sending system is responsible for sending a specific file type, such as video files, audio files or communication bus files. The file processing system selects the appropriate transmission protocol and processing method by identifying the type of each file to be played.

[0143] When sending multiple files to be played to the file receiving device, the file processing device needs to send the files to be played of the corresponding file type of the file sending system to the file receiving device in a frame-by-frame manner through the file sending systems of different types according to the file playing trigger time and / or file playing rate in the file playing request information. That is, the file processing device needs to control each file sending system to start sending the first data frame of the file to be played of the corresponding file type to the file receiving device at the file playing trigger time, and / or, in the process of controlling each file sending system to send each file to be played, the file processing device needs to control each file sending system to adjust the sending interval of adjacent data frames in each file to be played according to the file playing rate.

[0144] When the file playback trigger time arrives, the file processing device controls each file sending system to simultaneously start sending the data frames of the corresponding file types to be played, ensuring that the file receiving device can synchronously receive different types of data frames and synchronously send them to the playback device for synchronous playback. When each file sending system sends the corresponding file type to be played frame by frame, it is necessary to send the file to be played frame by frame at a certain time interval according to the set file playback rate. By controlling the sending rate of each file data frame, the playback rate control of the data frames of different types of files to be played can be achieved.

[0145] In this embodiment, according to the file playback trigger time and / or file playback rate in the first playback request instruction, the to-be-played files of the file sending system corresponding to the file type are sent to the file receiving device in a frame-by-frame manner through the file sending systems of different types. The time range can be customized for synchronous playback, or a specific time point can be specified for synchronous playback to meet the needs of different users; during the synchronous transmission and playback of different types of files, the playback rate of each to-be-played file can be switched according to different scenarios, achieving multiple-speed synchronous playback or slow playback of multiple to-be-played files of different types, thereby improving the user experience.

[0146] In one embodiment, step S401, that is, according to the file play trigger time and / or the file play rate in the first play request instruction, the to-be-played file of the file sending system corresponding to the file type is sent to the file receiving device through different types of file sending systems in a frame-by-frame manner, specifically including the following steps:

[0147] S4011: The file processing device determines the sending timestamp interval between two adjacent data frames of the file to be played according to the file playing rate, and determines the generating timestamp interval between two adjacent data frames in the file to be played.

[0148] For each file to be played, the file processing device needs to determine the generation timestamp interval between two adjacent data frames in the file to be played based on the generation timestamps of all data frames in the file to be played; at the same time, it is necessary to determine the sending timestamp interval between two adjacent data frames of the file to be played based on the file playback rate. The sending timestamp interval is the sending time interval of the data frames, which is determined according to the file playback rate.

[0149] S4012: The file processing device controls each file sending system to send data frames of the to-be-played files of the corresponding file type frame by frame to the file receiving device according to the sending timestamp interval and the generating timestamp interval of each to-be-played file.

[0150] After determining the sending timestamp interval between two adjacent data frames of the file to be played according to the file playing rate, the file processing device controls each file sending system to send data frames of the file to be played of the corresponding file type frame by frame to the file receiving device according to the sending timestamp interval and the generation timestamp interval of each file to be played.

[0151] Among them, each file to be played is stored in a consumption queue of the corresponding file type. When sending data frames of the files to be played of the corresponding file type to the file receiving device, the consumption queue of each file sending system consumes the stored files to be played frame by frame to send data frames of the files to be played of the corresponding file type to the file receiving device frame by frame.

[0152] For each file sending system, if the generation timestamp interval of two adjacent data frames in the file to be played in the file sending system is greater than or equal to the sending timestamp interval determined according to the file playing rate, the consumption queue of the file sending system is controlled to consume the stored file to be played frame by frame according to the sending timestamp interval, so as to send the data frames of the file to be played of the corresponding file type to the file receiving device frame by frame; if the generation timestamp interval of two adjacent data frames in the file to be played in the file sending system is less than the sending timestamp interval determined according to the file playing rate, the consumption queue of the file sending system is controlled to consume the stored file to be played frame by frame according to the sending timestamp interval, so as to send the data frames of the file to be played of the corresponding file type to the file receiving device frame by frame.

[0153] In this embodiment, the file processing device determines the sending timestamp interval of two adjacent data frames of the file to be played according to the file playback rate, and determines the generation timestamp interval of two adjacent data frames in the file to be played, and controls each file sending system to send the data frames of the file to be played of the corresponding file type frame by frame to the file receiving device according to the sending timestamp interval and the generation timestamp interval of each file to be played. The frame sending time interval of each file sending system can be dynamically adjusted according to different file playback rates to ensure that the data frame received by the file receiving device is consistent with the set file playback rate, thereby ensuring the flexibility of the playback rate and the consistency of the playback, and adapting to the requirements for playback fluency in different scenarios.

[0154] In one embodiment, step S40, that is, sending a plurality of to-be-played files to the file receiving device, specifically includes the following steps:

[0155] S01: If the file processing device receives a play status control instruction, the file processing device controls the data sending state of each to-be-played file according to the play status control instruction when sending multiple to-be-played files to the file receiving device, so as to control the data play status of the playback device.

[0156] The user may send a play status control instruction to the file processing device through the client, wherein the play status control instruction includes at least one of a loop play instruction, a pause play instruction, a stop play instruction, and a resume play instruction.

[0157] Before sending multiple files to be played to the file receiving device or during the process of sending multiple files to be played to the file receiving device, if the file processing device receives a playback status control instruction from the client, then when sending multiple files to be played to the file receiving device, the file processing device controls the data sending status of each file to be played according to the playback status control instruction, so as to control the data playback status of the playback device. The synchronous playback status control of multiple types of files on multiple data devices and the arbitrary switching of playback status can be realized. The synchronous playback can be looped, the synchronous playback can be paused, the synchronous playback can be resumed, or the synchronous playback can be terminated. The pause playback state can be switched to the resume playback state, and the resume playback state can be switched to the pause resume playback state, etc., to meet the different requirements of users and improve the user experience.

[0158] In one embodiment, if Figure 7 As shown, after step S30, that is, after obtaining a plurality of files to be played after data frame alignment, the method further includes the following steps:

[0159] S70: The file processing device stores a plurality of to-be-played files of different types in the server, and sends the storage address of each to-be-played file in the server to the client.

[0160] After obtaining the multiple files to be played after the data frames are aligned, the file processing device stores the multiple files to be played of different types in the database of the server, and sends the storage address of each file to be played in the server to the client. The server can be a remote server, that is, a cloud. By storing multiple files to be played of different types in the server.

[0161] The file processing device assembles and stores each to-be-played file on the server in a specific file format. For example, the file format of the video file among the multiple to-be-played files is mp4; the file format of the audio file among the multiple to-be-played files is mp3; the file format of the communication bus file among the multiple to-be-played files is dat. Then, the storage address of each to-be-played file in the server is sent to the client, so that the client maintains the storage address of each to-be-played file in the server.

[0162] S80: The file processing device receives a loop play instruction or a replay instruction from the client.

[0163] The user can send a loop play instruction or a replay instruction to the file processing device through the client. The file processing device receives the loop play instruction or the replay instruction from the client, and the loop play instruction or the replay instruction can also be sent through the first play request instruction. The loop play instruction or the replay instruction includes the storage address of each file to be played in the server.

[0164] S90: the file processing device obtains multiple to-be-played files of different types according to the storage addresses of the to-be-played files in the server, and sends the multiple to-be-played files to the file receiving device again, so that the multiple to-be-played files are played frame by frame through the playing device.

[0165] After the file processing device receives a loop playback instruction or a replay instruction from the client, the file processing device obtains multiple files to be played of different types according to the storage address of each file to be played in the server, and sends the multiple files to be played to the file receiving device again, so that the multiple files to be played can be played frame by frame through the playback device.

[0166] In this embodiment, the file processing device stores multiple to-be-played files of different types in the server, and sends the storage address of each to-be-played file in the server to the client. When there is a need for loop playback and replay, the file processing device can directly read each to-be-played file according to the storage address of each to-be-played file in the server, without the need to obtain it from the data device in real time, thereby reducing the data processing process. In addition, the impact of data device anomalies on data acquisition can be reduced, without relying on data devices, and improving system reliability.

[0167] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0168] In one embodiment, a file sending device is provided, which corresponds one-to-one to the file synchronous playback method in the above embodiment. Figure 8 As shown, the file sending device includes a receiving module 801, an acquisition module 802, an alignment module 803 and a sending module 804. The functional modules are described in detail as follows:

[0169] The receiving module 801 is used to receive a first play request instruction, where the first play request instruction requests synchronous play of target device files of multiple data devices;

[0170] An acquisition module 802 is used to acquire a target device file corresponding to each data device among a plurality of data devices according to the first play request instruction, and the target device files corresponding to different data devices are of different types;

[0171] An alignment module 803 is used to perform timestamp alignment processing on data frames of multiple target device files to obtain multiple to-be-played files of different types, wherein the multiple to-be-played files include communication bus files, and data frames of different to-be-played files among the multiple to-be-played files are synchronized according to the generated timestamps;

[0172] The sending module 804 is used to send multiple files to be played to the file receiving device, the file receiving device is used to synchronously send data frames of different types but with the same timestamps from multiple data devices in the multiple files to be played to the playing device, and the playing device is used to perform multi-channel synchronous playback of multiple data frames with the same generation timestamp sent by the file receiving device.

[0173] Optionally, the sending module 804 is specifically configured to:

[0174] The first data frame of the to-be-played file corresponding to the file sending system is synchronously sent to the file receiving device through the file sending system corresponding to each to-be-played file;

[0175] Receiving the data frame receiving time of each to-be-played file from the file receiving device, and determining the data frame receiving time deviation of each to-be-played file according to the data frame receiving time of each to-be-played file;

[0176] According to the data frame receiving time deviation, the time for each file sending system to send the next data frame of the corresponding file type to be played to the file receiving device is adjusted so that the file receiving device receives data frames with the same timestamp in multiple files to be played.

[0177] Optionally, the first play request instruction includes file play request information, the file play request information includes a target time period and device information of multiple data devices of the same type, and the acquisition module 802 is specifically used to:

[0178] Determine the file address of the device file in the target time period in each data device according to the device information of each data device of the same type, and obtain the target file address of each data device of the same type;

[0179] According to the target file addresses of the same type of data devices, the file sending system of the corresponding type is controlled to create multiple download clients, and each download client is controlled to establish a link with each corresponding data device respectively, and one download client is connected to one data device through a preset streaming protocol;

[0180] Each download client of the file sending system obtains the stream pulling file corresponding to each data device respectively, and obtains the target device file of the same type. The target device file includes the device files of multiple data devices of the same type within the target time period.

[0181] Optionally, the target device file includes device files of multiple data devices of the same type within a target time period. According to the generation timestamp of each data frame in each target device file, the alignment module 803 is specifically used to:

[0182] Control each file sending system to decrypt the device file of each data device in the target device file of the corresponding file type, and obtain the decrypted data frame of each data device;

[0183] The decrypted data frames of the data device are stored in the producer queue of the data device in the order of the generation timestamps of the data device, and the device channel number of the data device is spliced ​​to each data frame in the producer queue of the data device to obtain the to-be-processed file of the data device;

[0184] The data frames in the to-be-processed files of each data device with the same generation timestamp are spliced ​​in ascending order according to the device channel numbers of the data devices, and the spliced ​​data frames are placed in the consumption queue of the file sending system of the corresponding file type in the order of the generation timestamps to obtain the to-be-played files of the corresponding file type.

[0185] Optionally, the sending module 802 is further specifically configured to:

[0186] According to the file play trigger time and / or file play rate in the first play request instruction, the to-be-played file of the file type corresponding to the file sending system is sent to the file receiving device in a frame-by-frame manner through different types of file sending systems.

[0187] Optionally, the sending module 802 is further specifically configured to:

[0188] If a playback status control instruction is received, when multiple files to be played are sent to the file receiving device, the data sending status of each file to be played is controlled according to the playback status control instruction to control the data playback status of the playback device; wherein the playback status control instruction includes at least one of a loop playback instruction, a pause playback instruction and a resume playback instruction.

[0189] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0190] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0191] The present application also provides an electronic device, such as Fig. 9 As shown, the electronic device includes: at least one processor, a memory, and a computer program stored in the memory and executable on at least one processor. When the processor executes the computer program, the steps in any of the above-mentioned method embodiments are implemented, or when the processor executes the computer program, the functions of each module / unit in the above-mentioned device embodiments are implemented.

[0192] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in a memory and executed by a processor to complete the present application. One or more modules / units may be a series of computer program instruction segments capable of completing a specific function, and the instruction segments are used to describe the execution process of the computer program in a computer device.

[0193] Those skilled in the art will understand that Fig. 9It is only an example of a computer device and does not constitute a limitation of the computer device. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0194] The processor may be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, field programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0195] The memory may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. The memory may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card, a secure digital card, a flash memory card, etc. equipped on the computer device. Furthermore, the memory may also include both an internal storage unit of a computer device and an external storage device.

[0196] An embodiment of the present application further provides a readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0197] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0198] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include at least: any entity or device capable of carrying the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium.

[0199] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0200] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0201] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0202] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for synchronously playing a file, characterized in that: include: receiving a first play request instruction, wherein the first play request instruction requests synchronous play of target device files of multiple data devices; According to the first play request instruction, a target device file corresponding to each of the plurality of data devices is acquired, wherein the target device files corresponding to different data devices are of different types; Performing timestamp alignment processing on data frames of the plurality of target device files to obtain a plurality of to-be-played files of different types, wherein the plurality of to-be-played files include a communication bus file, and data frames of different to-be-played files among the plurality of to-be-played files are synchronized according to a generated timestamp; Sending multiple files to be played to a file receiving device, the file receiving device is used to synchronously send data frames of different types but with the same timestamp from multiple data devices in the multiple files to be played to a playing device, and the playing device is used to perform multi-channel synchronous playback of the multiple data frames with the same generation timestamp.

2. The file synchronous playback method according to claim 1, characterized in that: The sending the plurality of files to be played to the file receiving device comprises: Synchronously sending, by the file sending system corresponding to each of the files to be played, the first data frame in the file to be played corresponding to the file sending system to the file receiving device; Receiving the data frame receiving time of each of the to-be-played files from the file receiving device; Determining a data frame reception time deviation of each of the files to be played according to the data frame reception time of each of the files to be played; According to the data frame reception time deviation of each file to be played, the time for each file sending system to send the next data frame of the file to be played of the corresponding file type to the file receiving device is adjusted, so that the file receiving device receives multiple data frames with the same timestamp in the files to be played.

3. The file synchronous playback method according to claim 2, characterized in that: The multiple files to be played also include a first file to be played, the type of the first file to be played is a video type, and the time for each file sending system to send the next data frame of the file to be played of the corresponding file type to the file receiving device is adjusted according to the data frame reception time deviation of each file to be played, so that the file receiving device receives the data frames with the same timestamp in the multiple files to be played, including: Sending the next data frame of the first file to be played through the file sending system corresponding to the first file to be played; For the second file to be played, after delaying the data frame receiving time deviation of the second file to be played, the next data frame in the second file to be played is sent to the file receiving device through the file sending system corresponding to the second file to be played, wherein the second file to be played is any other type of file to be played among the multiple files to be played except the first file to be played.

4. The method according to claim 2, characterized in that: The plurality of files to be played further include a first file to be played, the type of the first file to be played is a video type, and the determining of the data frame receiving time deviation of each file to be played according to the data frame receiving time of each file to be played includes: Taking the data frame receiving time of the first file to be played as the time reference point, determining the time difference between the data frame receiving time of other types of files to be played and the time reference point, and obtaining the data frame receiving time deviation of other types of files to be played.

5. The file synchronous playback method according to claim 1, characterized in that: The first play request instruction includes file request play information, the file request play information includes a target time period and device information of the plurality of data devices, and acquiring a target device file corresponding to each of the plurality of data devices according to the first play request instruction includes: Determine the storage address of the device file in the target time period of each data device according to the device information of each data device, and obtain the target file address of each data device; According to the target file addresses of the data devices of the same type, control the file sending system of the corresponding type to create multiple download clients, and control each of the download clients to establish links with the data devices of the same type, one download client is linked to one data device; By corresponding to each download client in the file sending system, the device files corresponding to each data device in the target time period are obtained respectively, and the target device files of the same type are summarized, and the target device files include the device files of multiple data devices of the same type in the target time period.

6. The file synchronous playback method according to claim 1, characterized in that: The step of performing timestamp alignment processing on the data frames of the plurality of target device files to obtain a plurality of to-be-played files of different types includes: According to the generation timestamp of each data frame in each target device file, the data frames in each target device file are timestamp aligned, and the multiple data frames after timestamp alignment are cached in the order of the generation timestamps into the file sending system of the corresponding file type, so as to obtain multiple files to be played of different types.

7. The file synchronous playback method according to claim 6, characterized in that: The target device file includes device files of multiple data devices of the same type within the target time period, and the data frames in each target device file are timestamp aligned according to the generation timestamp of each data frame in each target device file, and the multiple data frames after the timestamp alignment are cached in the order of the generation timestamps to the file sending system of the corresponding file type, to obtain multiple files to be played of different types, including: Control each of the file sending systems to decrypt the device files of each of the data devices in the target device files of the corresponding file types, respectively, to obtain the decrypted data frames of each of the data devices; The decrypted data frames of the data device are stored in the producer queue of the data device in the order of the generation timestamps of the data device, and the device channel number of the data device is spliced ​​to each data frame in the producer queue of the data device to obtain the to-be-processed file of the data device; The data frames in the to-be-processed files of each data device with the same generation timestamp are spliced ​​in ascending order according to the device channel numbers of the data devices, and the spliced ​​data frames are placed in the consumption queue of the file sending system of the corresponding file type in the order of the generation timestamps to obtain the to-be-played files of the corresponding file type.

8. The file synchronous playback method according to claim 1, characterized in that: The sending the plurality of files to be played to the file receiving device comprises: According to the file play trigger time and / or file play rate in the first play request instruction, the to-be-played file of the corresponding file type of the file sending system is sent to the file receiving device in a frame-by-frame manner through each of the file sending systems of different types.

9. The file synchronous playback method according to any one of claims 1 to 8, characterized in that: The sending the plurality of files to be played to the file receiving device comprises: If a playback state control instruction is received, the data transmission state of each of the to-be-played files is controlled according to the playback state control instruction, so as to control the data playback state of the playback device; The playback status control instruction includes at least one of a loop playback instruction, a pause playback instruction and a resume playback instruction.

10. A file synchronous playback system, characterized in that: include: A client, used for sending a first play request instruction to the file processing device, wherein the first play request instruction requests synchronous play of target device files of multiple data devices; The file processing device is used to execute the file synchronous playback method according to any one of claims 1 to 9; A file receiving device, used for synchronously sending data frames of different types but with the same generation timestamp received from a plurality of data devices to a playback device; The playback device is used to perform multi-channel synchronous playback of the multiple data frames with the same generation timestamp sent by the file receiving device.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the file synchronous playback method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the file synchronous playback method as claimed in any one of claims 1 to 9 are implemented.

13. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the steps of the file synchronous playback method according to any one of claims 1 to 9.

Images