Audio and video storage device, audio and video playing device, audio and video storage method and audio and video playing method

Abstract

The invention discloses an audio and video storage device, an audio and video playing device, an audio and video storage method and an audio and video playing method, and belongs to the technical field of audios. The device comprises an acquisition module which responds to an audio and video acquisition request, acquires audio data according to audio channel distribution and acquires video data according to video track distribution; the coding module is used for coding all the audio data and each piece of video data by using the same clock, and recording all the coded audio data in the same audio track data; the packaging module is used for respectively performing data packaging operation on the audio track data and each piece of coded video data to generate a plurality of packaging files; sending the packaged files into a memory queue according to a collection coding sequence; and the audio and video write-in module sequentially writes each packaged file into a disk based on the memory queue to generate an audio and video file. According to the invention, the audio data and the video data can be stored in one audio and video file, so that the complexity of audio and video file management of the vehicle-mounted panoramic image system can be reduced.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of audio and video technology, and in particular to an audio and video storage device, a playback device, a storage method, and a playback method. Background Art

[0002] An in-vehicle panoramic imaging system, also known as a 360-degree surround view system, typically uses four cameras (front, back, left, and right) to combine and create a panoramic bird's-eye view of the vehicle's surroundings using an intelligent stitching algorithm. Its application scenarios include driving records, reversing images, parking monitoring alarms, and remote monitoring. Some of these scenarios require real-time viewing, while others require recording for later review. Sometimes, a panoramic image is required, while others require viewing a specific direction. Currently, when storing video footage from multiple cameras, it is typically stored as multiple files, and when browsing, multiple video files are simultaneously manipulated.

[0003] However, this storage solution has many disadvantages. First, file management is difficult. Several cameras require several video files. If surround-view composite videos are stored simultaneously, an additional file will be needed. Multiple files require an association scheme to be designed for association processing. Second, audio management is difficult. Multiple videos can be stored, but only one audio file can be stored. Audio must be stored in a separate file or in one of the videos. If a video without audio is selected during subsequent playback, and the audio needs to be displayed synchronously, the video must be extracted from other files. If each video file stores an audio file, then when browsing multiple videos simultaneously, the redundant audio must be filtered out. Finally, audio and video synchronization is difficult. Since the audio and video required for playback may not be in the same file, cross-file processing is required to synchronize audio and video during playback, which is cumbersome and may even make synchronization impossible. Summary of the Invention

[0004] Embodiments of the present invention provide a video storage device, a playback device, a storage method, and a playback method to at least solve the problems of difficult file management, difficult audio management, and difficult audio and video synchronization during the audio and video storage process of existing vehicle-mounted panoramic imaging systems, thereby reducing the complexity of audio and video file management in vehicle-mounted panoramic imaging systems and improving the simplicity of audio and video synchronization.

[0005] In a first aspect, an embodiment of the present invention provides an audio and video storage device, the audio and video storage device comprising at least an acquisition module, an encoding module, a packaging module, and an audio and video writing module;

[0006] The acquisition module is at least used to respond to the audio and video acquisition request, collect audio data according to the audio channel distribution, and collect video data according to the video track distribution;

[0007] The encoding module is at least used to encode all the audio data and each video data using the same clock, and record all the encoded audio data into the same audio track data;

[0008] The packaging module is configured to perform data packaging operations on the audio track data and each of the encoded video data to generate a plurality of packaged files; and to send each of the packaged files into a memory queue according to an acquisition and encoding order;

[0009] The audio and video writing module is at least used to write each of the packaged files into the disk in sequence based on the memory queue in response to the termination of the collection of the audio data and the video data, so as to generate an audio and video file.

[0010] Optionally, the packaging module is at least specifically configured to:

[0011] Performing single data packaging on the audio track data and each of the encoded video data according to a specified data size to generate a packaged file; and, during any data packaging process, if the packaged file does not meet the specified data size, padding the data that does not meet the specified data size.

[0012] Optionally, the audio and video file includes at least a file header and at least one packaged file;

[0013] The file header at least includes a file identifier, the number of audio channels, the number of video tracks, duration information and a reserved field;

[0014] The package file at least includes a data flag, a valid data length, and the audio track data or the video data;

[0015] Among them, the file identifier is at least used to distinguish whether the file is an audio or video file; the data flag is at least used to distinguish whether the packaged file corresponds to the audio data or the video data, and after determining that the packaged file corresponds to the video data, determine the acquisition module source of the video data.

[0016] In a second aspect, an embodiment of the present invention further provides an audio and video playback device, the audio and video playback device comprising at least a parsing module, a decoding module, and an audio and video playback module;

[0017] The parsing module is at least used to respond to the user's play instruction and determine whether the file header of the audio and video file meets the preset requirements; if it does, parse the audio and video file and read the package file; if it does not, issue an error prompt and directly exit the audio and video playback program;

[0018] The decoding module is configured to determine that the audio and video files meet the preset requirements, calculate the position of the first packaged file in the packaged file according to the file header, read each packaged file in sequence according to a specified data size, and then send the audio track data or video data in the packaged file to a specified audio decoding queue or video decoding queue based on the data flag of each packaged file;

[0019] The audio and video playback module is at least used to render and play the video data at the same clock after the video data and the audio track data are decoded, and to render and play the audio data generated by decoding the audio track data.

[0020] Optionally, the audio and video playback device further includes a surround view synthesis module;

[0021] The surround synthesis module is at least used to generate a surround video picture according to the decoded video data, and upload the surround video picture to the audio and video playback module;

[0022] The audio and video playback module is at least further configured to render and play the surround video image at the same clock, and to play the audio data generated by decoding the audio track data.

[0023] Optionally, the audio and video playback module is at least further configured to:

[0024] Receiving and modifying layout parameters to control the playback mode of the video data;

[0025] The playback mode at least includes controlling whether a certain video is displayed; and controlling the display position and size of one or more videos.

[0026] In a third aspect, an embodiment of the present invention further provides an audio and video real-time playback device, the audio and video real-time playback device comprising at least an acquisition module, a surround synthesis module and an audio and video playback module;

[0027] The acquisition module is at least used to respond to the audio and video acquisition request, collect audio data according to the audio channel distribution, and collect video data according to the video track distribution;

[0028] The surround synthesis module is at least used to generate a surround video picture according to the collected video data, and upload the surround video picture to the audio and video playback module;

[0029] The audio and video playback module is at least used to render and play the video data and the surround video image at the same clock, and to render and play the audio data.

[0030] In a fourth aspect, this embodiment further provides an audio and video storage method, which is performed using the audio and video storage device described in the first aspect. The method at least includes:

[0031] S1. In the audio and video storage process, the acquisition module responds to an audio and video acquisition request, collects audio data according to the audio channel distribution, and collects video data according to the video track distribution;

[0032] S2. During the audio and video storage process, the encoding module encodes all of the audio data and each of the video data using the same clock, and records all of the encoded audio data into the same audio track data;

[0033] S3. During the audio and video storage process, the packaging module performs a data packaging operation on the audio track data and each of the encoded video data to generate a plurality of packaged files; and each of the packaged files is sent to a memory queue in the order of acquisition and encoding;

[0034] S4. During the audio and video storage process, in response to the termination of the collection of the audio data and the video data, the audio and video writing module writes each of the packaged files into the disk in sequence based on the memory queue to generate an audio and video file.

[0035] In a fifth aspect, this embodiment further provides an audio and video playback method, which is performed using the audio and video playback device described in the second aspect. The method at least includes:

[0036] S11. During audio and video playback, the parsing module responds to the user's playback instruction to determine whether the file header of the audio and video file meets the preset requirements; if so, the audio and video file is parsed and the package file is read; if not, an error message is displayed and the audio and video playback program is directly exited;

[0037] S12. During audio and video playback, after determining, through the decoding module, that the audio and video files meet the preset requirements, the position of the first packaged file in the packaged files is calculated according to the file header, and each of the packaged files is read in sequence according to a specified data size, and then, based on the data flags of each packaged file, the audio track data or video data in the packaged files is sent to a specified audio decoding queue or video decoding queue;

[0038] S13. During the audio and video playback process, after the video data and the audio track data are decoded, the audio and video playback module renders and plays the video data at the same clock, and renders and plays the audio data generated by decoding the audio track data.

[0039] In a sixth aspect, this embodiment further provides a method for real-time audio and video playback, which is performed using the real-time audio and video playback device described in the third aspect. The method at least includes:

[0040] S21. During the real-time audio and video playback process, the acquisition module responds to the audio and video acquisition request and collects audio data according to the audio channel distribution and video data according to the video track distribution;

[0041] S22. During the real-time audio and video playback process, a surround video image is generated according to the collected video data by a surround synthesis module, and the surround video image is uploaded to the audio and video playback module;

[0042] S23. During the real-time audio and video playback process, the video data and the surround video image are rendered and played at the same clock by the audio and video playback module, and the audio data is rendered and played.

[0043] In summary, the embodiments of the present invention provide an audio and video storage device, a playback device, a storage method, and a playback method. First, the acquisition module responds to the audio and video acquisition request, collects audio data according to the audio channel distribution, and collects video data according to the video track distribution; second, the encoding module uses the same clock to encode all audio data and each video data, and records all encoded audio data in the same audio track data; third, the packaging module performs data packaging operations on the audio track data and each encoded video data respectively, generating multiple packaged files; then, the packaging module sends each packaged file to the memory queue in the order of acquisition and encoding; finally, the audio and video writing module responds to the termination of audio data and video data acquisition, and writes each packaged file to the disk in sequence based on the memory queue to generate an audio and video file.

[0044] Based on the technical solution provided by the embodiments of the present invention, when the vehicle-mounted panoramic imaging system needs to store the recorded audio and video files, the audio data and video data can be encoded using the same clock and packaged in a unified format to generate an audio and video file, which at least solves the problems of difficult audio and video file management and audio and video synchronization during the audio and video storage process, and is conducive to reducing the complexity of audio and video file management of the vehicle-mounted panoramic imaging system and improving the simplicity of audio and video synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 1 is a structural diagram of an audio and video storage device provided by an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of the structure of an audio and video file provided by an embodiment of the present invention;

[0048] Figure 3 1 is a schematic diagram of the architecture of an audio and video storage device provided by an embodiment of the present invention;

[0049] Figure 4 This is a structural diagram of an audio and video playback device provided by an embodiment of the present invention;

[0050] Figure 5 This is a display mode diagram of an audio and video playback device provided by an embodiment of the present invention;

[0051] Figure 6 This is a display mode diagram of another audio and video playback device provided by an embodiment of the present invention;

[0052] Figure 7 This is a schematic diagram of a playback process of an audio and video playback device provided by an embodiment of the present invention;

[0053] Figure 8 This is a structural diagram of a real-time audio and video playback device provided by an embodiment of the present invention;

[0054] Figure 9 This is a flow chart of an audio and video storage method provided by an embodiment of the present invention;

[0055] Figure 10 This is a flow chart of an audio and video playback method provided by an embodiment of the present invention;

[0056] Figure 11 The present invention provides a flowchart of a method for real-time audio and video playback. DETAILED DESCRIPTION

[0057] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0058] Figure 1 This is a structural diagram of an audio and video storage device provided by an embodiment of the present invention. This embodiment is applicable to any type of vehicle audio and video storage scenario, such as new energy vehicles, fuel vehicles, etc. The device can be implemented in software and / or hardware. Figure 1As shown, the audio and video storage device 100 provided in this embodiment includes at least a collection module 110 , an encoding module 120 , a packaging module 130 and an audio and video writing module 140 .

[0059] The acquisition module 110 is at least configured to respond to an audio or video acquisition request, collect audio data according to the audio channel distribution, and collect video data according to the video track distribution.

[0060] The encoding module 120 is at least configured to encode all audio data and each video data using the same clock, and record all encoded audio data into the same audio track data.

[0061] The packaging module 130 is at least used to perform data packaging operations on the audio track data and each encoded video data to generate multiple packaged files; and send each packaged file to the memory queue according to the acquisition and encoding order.

[0062] The audio and video writing module 140 is at least used to write each packaged file to the disk in sequence based on the memory queue in response to the termination of the collection of audio data and video data, so as to generate an audio and video file.

[0063] Responding to an audio and video capture request may refer to a user manually clicking an audio and video recording button on the human-computer interaction interface of a device or system with an audio and video capture function. An audio channel may refer to the fact that under normal circumstances, two left and right channel microphones are set for the capture of audio data, and a microphone of one channel is an audio channel. A video track may refer to the fact that a vehicle-mounted panoramic imaging system usually uses four cameras in different orientations to capture video data, that is, each camera corresponds to a video track. Encoding may refer to compressing the captured video data or audio data and reconstructing it into a data type or data format that can be recognized by a packager. Recording all encoded audio data in the same audio track data may refer to integrating and storing multiple audio data in a single audio track. A memory queue may refer to a continuous memory used to temporarily store packaged files (which can be understood as temporarily storing packaged files). Responding to the termination of the capture of audio data and video data may refer to a user clicking an audio and video recording stop button on the human-computer interaction interface of a device or system with an audio and video capture function. Writing each packaged file to disk sequentially based on the memory queue can mean writing them to disk in the order in which the packaged files are written in the memory queue, i.e., according to the first-in, first-out principle (i.e., the packaged file that enters the memory queue first is written to disk first). As can be seen, a packaged file is a collection of data generated by packaging audio or video data according to a specified format. Therefore, a packaged file can also be interpreted as packaged data.

[0064] In a specific embodiment, the user manually clicks on the vehicle-mounted panoramic imaging system on the human-computer interaction interface of the car's central control and selects the audio and video recording function to turn on the audio and video storage device 100. Furthermore, the audio and video storage device 100 uses four vehicle-mounted cameras in different directions to collect video data, and uses microphones of two left and right channels to collect audio data. Furthermore, the audio and video storage device 100 encodes the video data and audio data, compresses the video data and audio data, and converts the data format of the video data and audio data into a data format that can be recognized by the packager. Furthermore, the audio and video storage device 100 integrates and stores the encoded audio data in a single audio track. Furthermore, the audio and video storage device 100 uses a packager to package each video data and audio data separately to generate multiple packaged files. Furthermore, the audio and video storage device 100 writes the packaged files into the memory queue in the order in which the packaged files are generated. Furthermore, after the user manually clicks the audio and video recording stop function button on the car's in-vehicle panoramic imaging system, the audio and video storage device 100 writes the packaged files to the disk in a first-in-first-out manner according to the packaging order of the packaged files in the memory queue to generate an audio and video file.

[0065] To sum up, based on the technical solution provided by this embodiment, this embodiment can use the same clock to encode and package the audio data and video data respectively when the vehicle-mounted panoramic imaging system stores the recorded audio and video files, thereby generating an audio and video file from the video data and audio data, at least solving the problem of difficult file management and audio management during the audio and video storage process, and is conducive to improving the simplicity of audio and video file management and the convenience of audio management in the vehicle-mounted panoramic imaging system.

[0066] In another specific embodiment, optionally, the packaging module 130 is further configured to:

[0067] Perform single data packaging on the audio track data and each encoded video data according to the specified data size to generate a packaged file; and in any data packaging process, if the packaged file does not meet the specified data size, fill the data that does not meet the specified data size.

[0068] Exemplarily, during any data packing process, if the packed file does not meet the specified data size, the processing flow for padding the data that does not meet the specified data size may refer to, assuming that the packer packs the audio data or video data according to the length of a packed file of 128 bytes, and each generated packed file is called a block (the length of each block is 128 bytes). Furthermore, when the length of the last block does not meet 128 bytes, "0" (audio data or video data is transmitted or stored in the computer as binary data, and the end of the data is padded with a "0" value, which does not affect or change the video data or audio data itself) or other specific characters are used to fill the end of the data to make its length meet 128 bytes, thereby ensuring that the length of each block can meet the 128-byte requirement.

[0069] In another specific embodiment, optionally, the audio and video file includes at least one file header and at least one package file;

[0070] The file header at least includes a file identifier, the number of audio channels, the number of video tracks, duration information, and a reserved field;

[0071] The package file at least includes a data flag, a valid data length, and audio track data or video data;

[0072] Among them, the file identifier is at least used to distinguish whether the file is an audio or video file; the data identifier is at least used to distinguish whether the packaged file corresponds to audio data or video data, and after determining that the packaged file corresponds to video data, determine the source of the video data acquisition module 110.

[0073] In a specific embodiment, Figure 2 It is a structural diagram of an audio and video file, see Figure 2 The audio and video file starts from the file identifier and ends at the last package file of the audio data or video data (the package file can be multiple Figure 2 Only two packaged files are given in the , and the file identifier, number of audio channels, number of video tracks, duration information, reserved field, audio and video number, valid data length and audio track data or video data are connected in sequence (wherein the video number, valid data duration and audio track data or video data constitute a packaged file, and multiple packaged files are arranged in sequence according to the writing order after the reserved field).

[0074] Exemplarily, the data flag is at least used to distinguish whether the audio and video data belongs to audio data or video data; and after determining that the audio and video data is video data, determining the source of the video data can mean, assuming that the audio number is "1", and the video data collected by the four cameras "F", "L", "B" and "R" are numbered "2", "3", "4" and "5" respectively, first determine whether the packaged file data flag is "1", if it is not "1", it is determined to be a non-audio file, and continue to determine whether the data flag is any value of "2", "3", "4" and "5", assuming that the audio flag value is "3", it can be confirmed that the video data is taken by the "L" camera.

[0075] For example, the number of audio channels and video tracks can refer to the specific number of onboard microphones or cameras used to capture the corresponding audio or video data. Duration information can refer to the specific duration required to play the audio or video file. A reserved field can refer to special scenarios where adjustments can be made to meet the needs of the specific scenario. For example, in scenarios where the integrity of audio and video data is highly required, a CRC (Cyclic Redundancy Check) field can be added to the reserved field to ensure the integrity of the audio and video data. In scenarios where the security of audio and video data is highly required, an MD5 authentication field can be added to the reserved field to ensure the security of the audio and video files. The effective data length can refer to the actual length of the packaged audio or video data. During the acquisition process, this value is generally assigned a default value of 126 (assuming a packaged file is 128 bytes, of which the audio and video number and effective data length account for 2 bytes). Only at the end of the acquisition, the actual data length of the last block may be less than 126 bytes.

[0076] In a specific embodiment, Figure 3 This is a schematic diagram of the architecture of an audio and video storage device. Figure 1 and Figure 3 , the user manually clicks on the car panoramic imaging system on the human-machine interface of the car's central control and selects the audio and video recording function to start the audio and video storage device 100. Further, the audio and video storage device 100 uses four car cameras "F", "L", "B" and "R" to collect video data and uses two car microphones ("LV" and "RV") to collect audio data (this step can be done by Figure 1The acquisition module 110 completes the process. Furthermore, the audio and video storage device 100 sends the video data to its own VDC (Vehicle Dynamics Control) via LVDS (Low-Voltage Differential Signaling). Furthermore, the audio and video storage device 100 uses VDC to compress the video data and convert the data format of the video data and audio data into a data format that can be recognized by the packager (this step can be performed by Figure 1 Furthermore, the audio and video storage device 100 sends the encoded video data to the Sentinel module inside its own CSC (Cockpit Space Center) via RSTP (Real Time Streaming Protocol); the audio and video storage device encodes the audio data using the Sentinel module inside the CSC (this step can be performed by Figure 1 Furthermore, the audio and video storage device 100 uses the Sentinel module to package each video data and audio data separately (this step can be completed by the encoding module 120). Figure 1 When the packing module 130 in the middle generates multiple blocks (i.e., packaged files, each packaged file is 128 bytes long), the first 2 bytes of each block are the block header (i.e., data flag and valid data duration). Further, when generating the last block of video data, the remaining video data length is 124 bytes, which does not meet the requirement of 128 bytes, and the Sentinel module fills the end of the video data with a "0" value of 4 bytes in length. Furthermore, the audio and video storage device 100 writes all blocks into the memory queue in the Sentinel module in the order of packing. Furthermore, after the user manually clicks the audio and video recording stop function button on the car's on-board panoramic imaging system, the audio and video storage device 100 writes the blocks in the memory queue into the disk in sequence according to the write order of the memory queue (this step can be performed by Figure 1 The audio and video writing module 140 completes the process. Furthermore, after all blocks are written to disk, the audio and video storage device 100 integrates all blocks and sequentially encapsulates the reserved field (connected to the block), duration information, the video track value of "4", the audio channel value of "2", and the file identifier of "fawmedia" (i.e., the file header). After all fields are encapsulated, the audio and video file is successfully generated.

[0077] To sum up, based on the technical solution provided by this embodiment, when a vehicle needs to collect audio and video data, an audio and video storage device can be used to package the audio data and video data in a specified format and store them in the same audio and video folder, which at least solves the problems of difficult file management and audio management during the audio and video storage process, and is conducive to improving the simplicity of audio and video file management and the convenience of audio management in the vehicle-mounted panoramic imaging system.

[0078] It should be noted that, in addition to automotive audio and video storage scenarios, the audio and video storage device can also be used in other scenarios requiring audio and video storage, such as mobile phone recording scenarios, computer screen recording scenarios, etc. In other words, the audio and video storage device provided in this embodiment can also be applied to other technical fields to meet the collection needs of corresponding devices, and will not be further described.

[0079] Based on the above embodiments or implementations, this embodiment further provides an audio and video playback device. Figure 4 FIG. 1 is a structural diagram of an audio and video playback device provided by an embodiment of the present invention. Figure 4 As shown, the audio and video playback device 200 provided in this embodiment includes at least a parsing module 210 , a decoding module 220 and an audio and video playback module 230 .

[0080] The parsing module 210 is at least used to respond to the user's playback instruction and determine whether the file header of the audio and video file meets the preset requirements; if it does, it parses the audio and video file and reads the package file; if it does not, it gives an error prompt and directly exits the audio and video playback program.

[0081] The decoding module 220 is at least used to determine that the audio and video files meet the preset requirements, calculate the position of the first packaged file in the packaged file based on the file header, read each packaged file in sequence according to the specified data size, and then send the audio track data or video data in the packaged file to the specified audio decoding queue or video decoding queue based on the data flags of each packaged file.

[0082] The audio and video playback module 230 is at least used to render and play the video data at the same clock after the video data and the audio track data are decoded, and to render and play the audio data generated by decoding the audio track data.

[0083] Responding to a user's play instruction may refer to the user manually double-clicking an audio or video file in an audio or video folder. Meeting a preset requirement may refer to determining whether a file identifier in a header of the acquired audio or video file meets a preset requirement.

[0084] For example, the specific workflow of the decoding module 220 may be that, after determining that the file identifier of the audio or video file meets the preset requirements of the audio or video file, the decoding module 220 calculates the position of the first packaged file according to the fixed length of the file header, and then reads each packaged file in sequence according to the fixed size of the packaged file (for example, if the fixed length of the file header is 64 bytes and the fixed length of the packaged file is 128 bytes, then the first packaged file is from byte 65 to byte 192). Furthermore, when reading the packaged file, the decoding module 220 first reads the data flag in the packaged file and sends the corresponding packaged file to the designated queue based on the data flag (for example, when the data flag value of the first packaged file is "1", the first packaged file is placed in the first decoding queue).

[0085] In a specific embodiment, the user manually double-clicks the audio and video file in the audio and video folder to prepare to play the audio and video file. Furthermore, the audio and video playback device 200 obtains the file identifier of the audio and video file. After determining that the file identifier is an audio and video file, the audio and video playback device 200 calculates the position of the first packaged file according to the fixed length of the file header, and then reads each packaged file in the fixed size order of the packaged file. Furthermore, when reading the packaged file, the audio and video playback device 200 first obtains the data flag in the packaged file and sends the packaged file to the designated decoding queue according to the data flag value. Furthermore, after all packaged files are decoded, all video data and audio data are obtained. Furthermore, the audio playback device 200 renders and plays the video data, and renders and plays the audio data through the same clock. Furthermore, the user can browse the audio and video file on the display screen of the audio and video playback device 200.

[0086] To sum up, based on the technical solution provided by this embodiment, this embodiment can read all video data and audio data from an audio and video file when the vehicle-mounted panoramic imaging system plays the recorded audio and video files, and use the same clock to play the audio data and video data, thereby realizing the playback of audio and video files, at least solving the problems of audio management difficulties and audio and video synchronization difficulties during audio and video playback, which is conducive to reducing the complexity of operations when playing audio and video files in the vehicle-mounted panoramic imaging system and improving user experience.

[0087] Continue to see Figure 3 and Figure 4 In another specific implementation, the audio and video playback device 200 further includes a surround view synthesis module 240 .

[0088] The surround synthesis module 240 is at least configured to generate surround video images according to the decoded video data, and upload the surround video images to the audio and video playback module 230 .

[0089] The audio and video playback module 230 is at least further configured to render and play the surround video images at the same clock, and to play the audio data generated by decoding the audio track data.

[0090] The surround video image may refer to a surround view of the video image in different directions around the vehicle body that the user can watch by moving the viewing angle (the surround view image can be viewed by Figure 2 (check with AVM in the .

[0091] It can be understood that the aforementioned AVM is the abbreviation of Around View Monitor, which means panoramic monitoring imaging system.

[0092] In another specific embodiment, optionally, the audio and video playback module 230 is further configured to:

[0093] Receive and modify layout parameters to control the playback mode of the video data.

[0094] The playback mode at least includes controlling whether a certain video is displayed; and controlling the display position and display size of one or more videos.

[0095] As shown in the figure, Figure 5 It is a display mode diagram of an audio and video playback device. Figure 6 This is another display mode diagram of the audio and video playback device. Figure 5 and Figure 6 , wherein receiving and modifying the layout parameters to achieve control over the playback mode of the video data can mean that two functions of the video screen router (i.e., two display modes) can be achieved by modifying the layout parameters. The first display mode is that 4 sub-video data screens (i.e., left view, front view, rear view, and right view) can be displayed at the same time. In this mode, the screen of a certain channel of video data can be controlled not to decode the video data corresponding to the camera, thereby achieving the goal of not displaying a certain channel of video screen, and its corresponding position is displayed in black (in other words, in some cases, the user only wants to observe the screen taken by the front camera and the rear camera, and only decodes the data flag representing the package file of the front camera and the rear camera); the second mode is that any 2 of the 4 channels (for example, front view and rear view) can be displayed at the same time, plus the spliced panoramic video (i.e. Figure 5 Central View).

[0096] For example, Figure 7 This is a schematic diagram of the playback process of an audio and video player. Figure 4 and Figure 7 The actual working principle of the audio and video playback device 200 provided by the embodiment of the present invention can be specifically described as follows:

[0097] First, the user manually sets the video image router to display the front camera and rear camera, and opens the surround synthesis module 240. Furthermore, the user manually double-clicks an audio or video file in the audio and video folder (i.e., disk data). Furthermore, the audio and video playback device 200 uses the audio and video data router to obtain the file identifier of the audio and video file. After determining that the file identifier is an audio or video file, it calculates the location of the first packaged file according to the fixed length of the file header and then reads each packaged file in order according to the fixed size of the packaged file. Furthermore, when reading the packaged file, the audio and video playback device 200 obtains the data flag in the packaged file and sends the packaged file to the corresponding decoding queue according to the data flag (only the packaged files representing the front camera and rear camera are decoded). Furthermore, after all packaged files are decoded, all video data and audio data are obtained. Furthermore, the audio and video playback device 200 uses the surround view synthesizer to generate surround video images based on the video data. Furthermore, the audio playback device 200 uses the video image router to render and play the video data and surround video images; and, using the same clock, uses the rendering device to play the audio data. Furthermore, the user can browse the images captured by the front camera on the upper left side of the central control screen; and preview the images captured by the rear camera on the lower left side of the central control screen; and observe the surround video images around the vehicle body on the right side of the central control screen.

[0098] To sum up, based on the technical solution provided by this embodiment, this embodiment can read video data and audio data from a folder when the vehicle-mounted panoramic imaging system plays the recorded audio and video files. It can not only play the audio data and video data using the same clock, but also control the display screen to meet the different needs of different users. At least it solves the problem of difficulty in audio and video synchronization during audio and video playback, which is conducive to reducing the complexity of the vehicle-mounted panoramic imaging system's audio and video file playback operations and improving user experience.

[0099] Figure 8 This is a structural diagram of a real-time audio and video playback device provided by an embodiment of the present invention. Figure 8 As shown, the real-time audio and video playback device 300 provided in this embodiment includes at least a collection module 110 , a surround synthesis module 240 and an audio and video playback module 230 .

[0100] The acquisition module 110 is at least configured to respond to an audio or video acquisition request, collect audio data according to the audio channel distribution, and collect video data according to the video track distribution.

[0101] The surround synthesis module 240 is at least configured to generate a surround video image based on the collected video data, and upload the surround video image to the audio and video playback module 230 .

[0102] The audio and video playback module 230 is at least used to render and play the video data and the surround video image at the same clock, and to render and play the audio data.

[0103] In summary, this embodiment provides a real-time audio and video playback device. First, the real-time audio and video playback device uses a capture module to respond to audio and video capture requests, collecting audio data according to audio channel distribution and video data according to video track distribution. Then, the real-time audio and video playback device uses a surround synthesis module to generate surround video images based on the collected video data and upload the surround video images to the audio and video playback module. Finally, the real-time audio and video playback device uses the audio and video playback module to render and play the video data and surround video images at the same clock, and also render and play the audio data.

[0104] In view of this, this embodiment can directly obtain the real-time video data collected by the camera and the audio data collected by the microphone and use the same clock for rendering and playback when playing audio and video in real time. At the same time, the multi-channel videos played synchronously can perform real-time surround video synthesis and synchronously restore the surround effect during recording, thereby realizing seamless switching between the surround video screen and the video display in each direction, which is beneficial to improving the user experience.

[0105] The embodiment of the present invention also provides an audio and video storage method, Figure 9 This is a flow chart of an audio and video storage method provided by an embodiment of the present invention. This embodiment is applicable to various vehicle audio and video storage scenarios, such as fuel vehicles, pure electric vehicles, hybrid vehicles, plug-in hybrid vehicles, extended-range electric vehicles, unmanned vehicles, etc. The audio and video storage method can be, but is not limited to, executed by the audio and video storage device in the embodiment of the present invention as the execution subject, and the execution subject can be implemented in software and / or hardware. Figure 9 As shown, the audio and video storage method includes at least the following steps:

[0106] S1. During the audio and video storage process, the acquisition module responds to the audio and video acquisition request, collects audio data according to the audio channel distribution, and collects video data according to the video track distribution.

[0107] S2. During the audio and video storage process, all audio data and each video data are encoded using the same clock through the encoding module, and all the encoded audio data are recorded in the same audio track data.

[0108] S3. During the audio and video storage process, the data packaging operation is performed on the audio track data and each encoded video data through the packaging module to generate multiple packaged files; and each packaged file is sent to the memory queue according to the acquisition and encoding order.

[0109] S4. During the audio and video storage process, in response to the termination of the collection of audio data and video data, the audio and video writing module writes each packaged file into the disk in sequence based on the memory queue to generate an audio and video file.

[0110] Optionally, step S3 at least further includes:

[0111] S31. Perform single data packaging on the audio track data and each encoded video data according to the specified data size through the packaging module to generate a package file.

[0112] S32. During any data packing process, if the packed file does not meet the specified data size, the data that does not meet the specified data size is padded by the packing module.

[0113] Optionally, the audio and video file includes at least one file header and at least one package file:

[0114] The file header includes at least a file identifier, the number of audio channels, the number of video tracks, duration information, and a reserved field.

[0115] The package file at least includes a data flag, a valid data length, and audio track data or video data.

[0116] Among them, the file identifier is at least used to distinguish whether the file is an audio or video file; the data identifier is at least used to distinguish whether the packaged file corresponds to audio data or video data, and after determining that the packaged file corresponds to video data, determine the source of the acquisition module of the video data.

[0117] The technical solution provided by this embodiment is as follows: first, during the audio and video storage process, the acquisition module responds to the audio and video acquisition request and acquires audio data according to the audio channel distribution, and acquires video data according to the video track distribution; second, during the audio and video storage process, the encoding module uses the same clock to encode all audio data and each video data, and records all encoded audio data in the same audio track data; then, during the audio and video storage process, the packaging module performs data packaging operations on the audio track data and each encoded video data respectively to generate multiple packaged files; and each packaged file is sent to the memory queue in the order of acquisition and encoding; finally, during the audio and video storage process, the audio and video writing module responds to the termination of audio data and video data acquisition and writes each packaged file to the disk in sequence based on the memory queue to generate an audio and video file.

[0118] To sum up, based on the technical solution provided by this embodiment, this embodiment can use the same clock to encode and package the audio data and video data respectively when the vehicle-mounted panoramic imaging system stores the recorded audio and video files, thereby generating an audio and video file from the video data and audio data, at least solving the problem of difficult file management and audio management during the audio and video storage process, and is conducive to improving the simplicity of audio and video file management and the convenience of audio management in the vehicle-mounted panoramic imaging system.

[0119] The embodiment of the present invention also provides an audio and video playback method. Figure 10 This is a flow chart of an audio and video playback method provided by an embodiment of the present invention. Figure 10 As shown, the audio and video playback method includes at least the following steps:

[0120] S11. During the audio and video playback process, the parsing module responds to the user's playback instruction to determine whether the file header of the audio and video file meets the preset requirements; if it does, the audio and video file is parsed and the package file is read; if it does not, an error prompt is given and the audio and video playback program is directly exited.

[0121] S12. During the audio and video playback process, after the decoding module determines that the audio and video files meet the preset requirements, the position of the first packaged file in the packaged file is calculated according to the file header, and each packaged file is read in sequence according to the specified data size. Then, based on the data flags of each packaged file, the audio track data or video data in the packaged file is sent to the specified audio decoding queue or video decoding queue.

[0122] S13. During the audio and video playback process, after the video data and the audio track data are decoded, the audio and video playback module renders and plays the video data at the same clock, and renders and plays the audio data generated by decoding the audio track data.

[0123] Optionally, the audio and video playback method further includes:

[0124] S14: Generate a surround video picture according to the decoded video data through the surround synthesis module, and upload the surround video picture to the audio and video playback module.

[0125] S15. Rendering and playing the surround video images at the same clock through the audio and video playing module, and playing the audio data generated by decoding the audio track data.

[0126] Optionally, step S13 further includes:

[0127] S131. Receive and modify layout parameters through the audio and video playback module to control the video data playback mode.

[0128] The playback mode at least includes controlling whether a certain video is displayed; and controlling the display position and display size of one or more videos.

[0129] The technical solution provided by this embodiment is as follows: first, during the audio and video playback process, the parsing module responds to the user's playback instruction to determine whether the file header of the audio and video file meets the preset requirements; if it does, the audio and video file is parsed and the packaged file is read; if it does not, an error prompt is given and the audio and video playback program is directly exited; secondly, during the audio and video playback process, after the decoding module determines that the audio and video file meets the preset requirements, the position of the first packaged file in the packaged file is calculated according to the file header, and each packaged file is read in sequence according to the specified data size, and then based on the data flags of each packaged file, the audio track data or video data in the packaged file is sent to the specified audio decoding queue or video decoding queue; finally, during the audio and video playback process, after the video data and audio track data are decoded, the audio and video playback module renders and plays the video data at the same clock, and renders and plays the audio data generated by decoding the audio track data.

[0130] To sum up, based on the technical solution provided by this embodiment, this embodiment can read all video data and audio data from an audio and video file when the vehicle-mounted panoramic imaging system plays the recorded audio and video files, and use the same clock to play the audio data and video data, thereby realizing the playback of audio and video files, at least solving the problems of audio management difficulties and audio and video synchronization difficulties during audio and video playback, which is conducive to reducing the complexity of operations when playing audio and video files in the vehicle-mounted panoramic imaging system and improving user experience.

[0131] The embodiment of the present invention also provides a method for real-time audio and video playback. Figure 11 This is a flow chart of a method for real-time audio and video playback provided by an embodiment of the present invention. Figure 11 As shown, the real-time audio and video playback method includes at least the following steps:

[0132] S21. During the real-time audio and video playback process, the acquisition module responds to the audio and video acquisition request, collects audio data according to the audio channel distribution, and collects video data according to the video track distribution.

[0133] S22. During the real-time audio and video playback process, a surround video picture is generated according to the collected video data through the surround synthesis module, and the surround video picture is uploaded to the audio and video playback module.

[0134] S23. During the real-time audio and video playback process, the video data and the surround video image are rendered and played at the same clock by the audio and video playback module, and the audio data is rendered and played.

[0135] The technical solution provided by this embodiment is as follows: first, during the real-time audio and video playback process, the acquisition module responds to the audio and video acquisition request and collects audio data according to the audio channel distribution, and collects video data according to the video track distribution; second, during the real-time audio and video playback process, the surround synthesis module generates a surround video screen based on the collected video data, and uploads the surround video screen to the audio and video playback module; and, during the real-time audio and video playback process, the audio and video playback module renders and plays the video data and the surround video screen at the same clock, and renders and plays the audio data.

[0136] In view of this, this embodiment can directly obtain the real-time video data collected by the camera and the audio data collected by the microphone and use the same clock for rendering and playback when playing audio and video in real time. At the same time, the multi-channel videos played synchronously can perform real-time surround video synthesis and synchronously restore the surround effect during recording, thereby realizing seamless switching between the surround video screen and the video display in each direction, which is beneficial to improving the user experience.

[0137] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. An audio and video storage device, characterized in that: The audio and video storage device at least includes an acquisition module, an encoding module, a packaging module and an audio and video writing module; The acquisition module is at least used to respond to the audio and video acquisition request, collect audio data according to the audio channel distribution, and collect video data according to the video track distribution; The encoding module is at least used to encode all the audio data and each video data using the same clock, and record all the encoded audio data into the same audio track data; The packaging module is at least used to perform data packaging operations on the audio track data and each of the encoded video data to generate multiple packaged files; And, sending each of the packaged files into a memory queue according to the acquisition and encoding order; The audio and video writing module is at least used to write each of the packaged files into the disk in sequence based on the memory queue in response to the termination of the collection of the audio data and the video data, so as to generate an audio and video file.

2. The audio and video storage device according to claim 1, wherein: The packaging module is specifically used at least for: Performing single data packaging on the audio track data and each of the encoded video data according to a specified data size to generate a packaged file; and, during any data packaging process, if the packaged file does not meet the specified data size, padding the data that does not meet the specified data size.

3. The audio and video storage device according to claim 2, characterized in that: The audio and video file includes at least a file header and at least one package file; The file header at least includes a file identifier, the number of audio channels, the number of video tracks, duration information and a reserved field; The package file at least includes a data flag, a valid data length, and the audio track data or the video data; Among them, the file identifier is at least used to distinguish whether the file is an audio or video file; the data flag is at least used to distinguish whether the packaged file corresponds to the audio data or the video data, and after determining that the packaged file corresponds to the video data, determine the acquisition module source of the video data.

4. An audio and video playback device, characterized in that: The audio and video playback device at least includes a parsing module, a decoding module and an audio and video playback module; The parsing module is at least used to respond to the user's play instruction and determine whether the file header of the audio and video file meets the preset requirements; if it does, parse the audio and video file and read the package file; if it does not, issue an error prompt and directly exit the audio and video playback program; The decoding module is configured to determine that the audio and video files meet the preset requirements, calculate the position of the first packaged file in the packaged file according to the file header, read each packaged file in sequence according to a specified data size, and then send the audio track data or video data in the packaged file to a specified audio decoding queue or video decoding queue based on the data flag of each packaged file; The audio and video playback module is at least used to render and play the video data at the same clock after the video data and the audio track data are decoded, and to render and play the audio data generated by decoding the audio track data.

5. The audio and video playback device according to claim 4, characterized in that: The audio and video playback device also includes a surround view synthesis module; The surround synthesis module is at least used to generate a surround video picture according to the decoded video data, and upload the surround video picture to the audio and video playback module; The audio and video playback module is at least further configured to render and play the surround video image at the same clock, and to play the audio data generated by decoding the audio track data.

6. The audio and video playback device according to claim 4, characterized in that: The audio and video playback module is at least further used for: Receiving and modifying layout parameters to control the playback mode of the video data; The playback mode at least includes controlling whether a certain video is displayed; and controlling the display position and display size of one or more videos.

7. A real-time audio and video playback device, characterized in that: The real-time audio and video playback device comprises at least an acquisition module, a surround synthesis module and an audio and video playback module; The acquisition module is at least used to respond to the audio and video acquisition request, collect audio data according to the audio channel distribution, and collect video data according to the video track distribution; The surround synthesis module is at least used to generate a surround video picture according to the collected video data, and upload the surround video picture to the audio and video playback module; The audio and video playback module is at least used to render and play the video data and the surround video image at the same clock, and to render and play the audio data.

8. An audio and video storage method, characterized in that: The method is performed using the audio and video storage device according to claim 1, and the method at least comprises: S1. In the audio and video storage process, the acquisition module responds to an audio and video acquisition request, collects audio data according to the audio channel distribution, and collects video data according to the video track distribution; S2. During the audio and video storage process, the encoding module encodes all of the audio data and each of the video data using the same clock, and records all of the encoded audio data into the same audio track data; S3. During the audio and video storage process, the packaging module performs a data packaging operation on the audio track data and each of the encoded video data to generate a plurality of packaged files; and each of the packaged files is sent to a memory queue in the order of acquisition and encoding; S4. During the audio and video storage process, in response to the termination of the collection of the audio data and the video data, the audio and video writing module writes each of the packaged files into the disk in sequence based on the memory queue to generate an audio and video file.

9. An audio and video playback method, characterized in that: The method is performed using the audio and video playback device according to claim 4, and the method at least comprises: S11. During audio and video playback, the parsing module responds to the user's playback instruction to determine whether the file header of the audio and video file meets the preset requirements; if so, the audio and video file is parsed and the package file is read; if not, an error message is displayed and the audio and video playback program is directly exited; S12. During audio and video playback, after determining, through the decoding module, that the audio and video files meet the preset requirements, the position of the first packaged file in the packaged files is calculated according to the file header, and each of the packaged files is read in sequence according to a specified data size, and then, based on the data flags of each packaged file, the audio track data or video data in the packaged files is sent to a specified audio decoding queue or video decoding queue; S13. During the audio and video playback process, after the video data and the audio track data are decoded, the audio and video playback module renders and plays the video data at the same clock, and renders and plays the audio data generated by decoding the audio track data.

10. A method for real-time audio and video playback, characterized in that: The method is performed using the real-time audio and video device according to claim 6, and the method at least comprises: S21. During the real-time audio and video playback process, the acquisition module responds to the audio and video acquisition request and collects audio data according to the audio channel distribution and video data according to the video track distribution; S22. During the real-time audio and video playback process, a surround video image is generated according to the collected video data by a surround synthesis module, and the surround video image is uploaded to the audio and video playback module; S23. During the real-time audio and video playback process, the video data and the surround video image are rendered and played at the same clock by the audio and video playback module, and the audio data is rendered and played.

Images