Audio and video playing method and device, terminal equipment, in-vehicle infotainment system and storage medium

Abstract

The invention is suitable for the technical field of audio and video processing, and provides an audio and video playing method and device, terminal equipment, a vehicle machine system and a storage medium. The audio and video playing method comprises the following steps: when initial audio and video data transmitted by acquisition equipment is received, analyzing the initial audio and video data to obtain one or more pieces of audio and video frame data; according to the playing sequence of one or more pieces of audio and video frame data and the size of each piece of audio and video frame data, sequentially storing each piece of audio and video frame data into a storage block with a corresponding size one by one in the applied memory pool; according to the playing sequence of the one or more pieces of audio and video frame data, sequentially storing the multiple pieces of audio and video frame data in the multiple storage blocks in the memory pool into at least one storage block in the cache queue one by one; and playing the at least one piece of audio and video frame data extracted from the at least one storage block in the cache queue according to a first-in first-out sequence.

Description

Technical Field

[0001] The present application belongs to the field of audio and video processing technology, and in particular, relates to an audio and video playback method, device, terminal equipment, vehicle system and storage medium. Background Art

[0002] With the rapid development of the automotive industry, the cockpit usage scenarios in cars are becoming more and more humane and entertaining. Many cars contain central control screens, co-pilot screens, rear screens and other screens, providing a lot of entertainment scenarios for people in the car. However, there are still many shortcomings in multi-screen sharing and application ecology in cars. For example, when the car system is connected to the acquisition device for audio and video playback, the audio and video playback delay will be very large, affecting the user experience. Summary of the invention

[0003] The embodiments of the present application provide an audio and video playback method, apparatus, terminal equipment, vehicle system and storage medium, which can solve the technical problem of excessive delay in audio and video playback.

[0004] In a first aspect, an embodiment of the present application provides an audio and video playback method, comprising:

[0005] When receiving the initial audio and video data transmitted by the acquisition device, parsing the initial audio and video data to obtain one or more audio and video frame data;

[0006] According to the playback order of the one or more audio and video frame data and the size of each audio and video frame data, each audio and video frame data is sequentially stored in a storage block of a corresponding size in the applied memory pool, wherein each storage block stores one audio and video frame data;

[0007] According to the playback order of the one or more audio and video frame data, the multiple audio and video frame data in the multiple storage blocks in the memory pool are stored one by one in at least one storage block in the cache queue;

[0008] At least one audio or video frame data retrieved from the at least one storage block in the cache queue is played in a first-in-first-out order.

[0009] In a possible implementation manner of the first aspect, the method further includes:

[0010] When the connected acquisition device is detected, the size of the memory pool to be applied is determined according to the audio and video information supported by the acquisition device;

[0011] According to the size of the memory pool to be applied for, a memory application operation is performed on the memory manager to obtain a memory pool of corresponding size.

[0012] In a possible implementation manner of the first aspect, storing the multiple audio and video frame data in the multiple storage blocks in the memory pool one by one in sequence into at least one storage block in the cache queue according to the playback order of the one or more audio and video frame data includes:

[0013] For each audio and video frame data, when storing the current audio and video frame data in the current storage block in the memory pool into the storage block in the cache queue, determining whether the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is less than a storage block number threshold;

[0014] If it is less than the storage block quantity threshold, the current audio and video frame data in the current storage block in the memory pool is stored in the storage block in the cache queue.

[0015] In a possible implementation manner of the first aspect, the method further includes:

[0016] If the number of the storage blocks is greater than or equal to the storage block quantity threshold, one or more audio and video frame data to be played in the cache queue are discarded;

[0017] The current audio and video frame data in the current storage block in the memory pool is stored in the storage block in the cache queue.

[0018] In a possible implementation manner of the first aspect, the method further includes:

[0019] Modify the memory usage mark of the storage block corresponding to the one or more discarded audio and video frame data to be played from the memory pool to free;

[0020] The memory manager is notified of a message that the storage blocks corresponding to one or more discarded audio and video frame data to be played are available.

[0021] In a possible implementation manner of the first aspect, after playing at least one audio or video frame data taken out from the at least one storage block in the cache queue in a first-in-first-out order, the method further includes:

[0022] Modify the memory usage mark of the storage block corresponding to the played audio and video frame data from the memory pool to free;

[0023] The memory manager is notified that the storage block corresponding to the played audio and video frame data is available.

[0024] In a possible implementation manner of the first aspect, playing at least one audio and video frame data taken out from the at least one storage block in the cache queue in a first-in-first-out order includes:

[0025] Decoding at least one audio and video frame data retrieved from at least one storage block in the cache queue in a first-in-first-out order;

[0026] At least one decoded audio or video frame data is broadcasted in the order of decoding.

[0027] In a possible implementation manner of the first aspect, the audio and video playback method is applied to a native layer of an Android system.

[0028] In a second aspect, an embodiment of the present application provides an audio and video playback device, characterized in that it includes:

[0029] A data parsing module, for parsing the initial audio and video data transmitted by the acquisition device to obtain one or more audio and video frame data;

[0030] A memory storage module, used for storing each audio and video frame data in a storage block of a corresponding size in the applied memory pool one by one in accordance with the playback order of the one or more audio and video frame data and the size of each audio and video frame data, wherein each storage block stores one audio and video frame data;

[0031] A cache storage module, used for storing the multiple audio and video frame data in the multiple storage blocks in the memory pool into at least one storage block in the cache queue one by one in sequence according to the playback order of the one or more audio and video frame data;

[0032] The data playing module is used to play at least one audio or video frame data taken out from the at least one storage block in the cache queue in a first-in first-out order.

[0033] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the terminal device is connected to an acquisition device, and when the processor executes the computer program, the audio and video playback method as described in the first aspect above is implemented.

[0034] In a fourth aspect, an embodiment of the present application provides a vehicle system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the vehicle system is connected to an acquisition device, and when the processor executes the computer program, the audio and video playback method as described in the first aspect above is implemented.

[0035] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the audio and video playback method as described in the first aspect above is implemented.

[0036] In a sixth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the audio and video playback method described in the first aspect above.

[0037] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0038] The audio and video playback method provided by the present application, when receiving the initial audio and video data transmitted by the acquisition device, parses the initial audio and video data to obtain one or more audio and video frame data; according to the playback order of the one or more audio and video frame data, according to the size of each audio and video frame data, stores each audio and video frame data one by one in the applied memory pool into a storage block of corresponding size, wherein each storage block stores one audio and video frame data; according to the playback order of the one or more audio and video frame data, stores multiple audio and video frame data in multiple storage blocks in the memory pool one by one in at least one storage block in the cache queue; according to the first-in-first-out order, plays at least one audio and video frame data taken out from at least one storage block in the cache queue. On the one hand, the present application scheme optimizes the memory application and usage without sacrificing image quality. The memory application is adjusted to apply for a memory pool once when the program is initialized, and the memory pool is used to uniformly manage the program's memory. Therefore, memory is only requested once during the entire audio and video playback process, and there is no need to request memory multiple times. The strategy of exchanging space for time solves the technical problem of multiple accesses to the memory manager leading to a large number of I / O operations, reduces the frequency of memory access, reduces time overhead, and reduces latency. On the other hand, by parsing the initial audio and video data, one or more audio and video frame data are obtained, and in the order of playing one or more audio and video data, the audio and video frame data are stored in the storage block matching the size of the audio and video frame data in the memory pool, the storage block in the cache queue, and finally the audio and video frame data taken out of the storage block in the cache queue for playback. The multi-level storage mechanism can better store and buffer the audio and video frame data before playback, reducing the latency of the overall audio and video playback process and improving the quality of audio and video playback and user experience.

[0039] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 It is a flowchart of an audio and video playback method provided by an embodiment of the present application;

[0042] Figure 2 is a schematic diagram of a memory pool in an embodiment of the present application;

[0043] Figure 3 It is a schematic diagram of storing two audio and video frame data into a memory pool in one embodiment of the present application;

[0044] Figure 4 This is a schematic diagram of the structure of a cache queue in an embodiment of the present application;

[0045] Figure 5 is a schematic diagram of storing audio and video data in a cache queue provided by an embodiment of the present application;

[0046] Figure 6 is a schematic diagram of storing audio and video data in a cache queue provided by another embodiment of the present application;

[0047] Figure 7 is a schematic diagram of a memory pool in an embodiment of the present application;

[0048] Figure 8 It is a flowchart of an audio and video playback method provided by an application embodiment of the present application;

[0049] Fig. 9 It is a structural diagram of an audio and video playback device provided in an embodiment of the present application;

[0050] Fig.10 It is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0052] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0053] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0054] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0055] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0056] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0057] The audio and video playback method provided in the embodiments of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, vehicle-mounted systems, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiments of the present application do not impose any restrictions on the specific types of terminal devices.

[0058] Currently, audio and video files can be projected onto the screen through a capture card. For example, one end of the capture card is connected to the car system and the other end is connected to a game console such as XBOX, PlayStation, Switch, etc., so that users can use the screen in the car to play games. Another example is that a computer can be connected to multiple screens through a capture card, for example, one end of the capture card is connected to the computer and the other ends are connected to multiple screens, so that multiple screens can play audio and video in the computer at the same time. The delay generated when playing audio and video through a capture card is mostly around 120 milliseconds to 1 second. During the game user's play, especially when playing some high FPS (Frames Per Second) games, high latency will greatly affect the game user's gaming experience. In the prior art, a conventional strategy to reduce latency is to optimize the transmission protocol, changing the long byte protocol to a more compact transmission protocol. Another strategy is to reduce the picture parameters: for example, lower the FPS, such as from FPS = 60 to FPS = 30; lower the resolution, such as from 1080p to 480p; and lower the high bit rate to a low bit rate.

[0059] This application mainly reduces the audio and video playback delay by optimizing memory application and usage, video transcoding and rendering methods, reducing video transcoding links, video and audio decoding and playback methods, and adjusting the cache queue by dropping and supplementing frames.

[0060] Figure 1 It is a flowchart of an audio and video playback method provided in an embodiment of the present application.

[0061] S11, when receiving the initial audio and video data transmitted by the acquisition device, parsing the initial audio and video data to obtain one or more audio and video frame data.

[0062] The acquisition device includes but is not limited to various devices such as acquisition cards that can perform audio and video acquisition and transmission. The initial audio and video data transmitted by the acquisition device includes video data of the UVC protocol and audio data of the UAC protocol. By parsing the initial audio and video data, one or more frame-based audio and video frame data can be obtained. The video data of the UVC protocol can be parsed into video frame data in YUV format, and the audio data of the UAC protocol can be parsed into audio frame data in PCM format. That is, one or more audio and video frame data include YUV format video frame data based on each frame and PCM format audio frame data of the corresponding frame, wherein each frame corresponds to a YUV format video frame data and a PCM format audio frame data.

[0063] Among them, UVC stands for USB Video (device) Class, which is a protocol standard defined for USB video capture devices jointly launched by several equipment manufacturers. It has now become one of the USB org standards.

[0064] The YUV video format is a color encoding method that is mainly used to optimize the transmission of color video signals to make them backward compatible with old black-and-white TVs. The YUV format divides color information into two components: brightness (Y) and chrominance (U and V), where "Y" represents brightness (Luminance or Luma), that is, grayscale value; and "U" and "V" represent chrominance (Chrominance or Chroma), that is, color information.

[0065] UAC stands for USB Audio Class. UAC is sometimes also called UAD. UAD stands for USB Audio Device, which is the protocol standard for USB audio transmission and one of the USB org standards.

[0066] The full name of PCM is Pulse-Code Modulation, which is a technology widely used in audio digital processing.

[0067] S12, according to the playback order of one or more audio and video frame data and the size of each audio and video frame data, each audio and video frame data is stored one by one in a storage block of a corresponding size in the applied memory pool, wherein each storage block stores one audio and video frame data.

[0068] The memory pool may be a piece of memory space applied for in advance to the memory manager, for example, the size may be 2k BYTE (bytes) to 80M BYTE (bytes).

[0069] Take the size of 4k BYTE (bytes) and the address of 0-4095 as an example, the memory pool that has been applied for is as follows Figure 2 shown.

[0070] Here, a storage block of a corresponding size can be taken from the applied memory pool according to the required size of each audio and video frame data to store the corresponding audio and video frame data. For example, if the current audio and video frame data size is 256 BYTE (bytes), then a storage block of 256 BYTE (bytes) can be taken from the memory pool, and the obtained storage block can be marked as already used. The structure of the mark can include, for example, the starting address, the ending address, the current block number, and whether it is currently in use. Exemplarily, the structure of a mark is:

[0071] {"Start Address:":"0",

[0072] "End Address:": "255",

[0073] "Current block number": "0",

[0074] "IsCurrentlyInUsed": "true" / / "IsCurrentlyInUsed": "yes"}.

[0075] Since the storage block has been marked as "currently in use", the memory space corresponding to the storage block will no longer be used until the mark of the memory space is modified to "currently not in use", that is, "currently in use": "false".

[0076] Following the previous example, according to two audio and video frame data of 256 BYTE (bytes) in size, the two audio and video frame data are stored one by one in the storage blocks of corresponding sizes in the applied memory pool. The schematic diagram of the memory pool after storage is as follows: Figure 3 As shown in the figure, the starting address of the first audio and video frame data is 0 and the ending address is 255. The starting address of the second audio and video frame data is 256 and the ending address is 511.

[0077] In the prior art, for example, for an audio or video stream with FPS = 60 frames per second, the memory needs to be applied for 60 times per second when running, and the memory needs to be accessed 60 times, resulting in 60 I / O operations. A large number of I / O operations will increase the frequency of memory access, thereby causing a large amount of time overhead. In addition, when the data rendering speed of the audio or video stream is faster than the data parsing speed, it will cause rendering waiting, thereby causing delays.

[0078] This application scheme optimizes the memory application method without sacrificing image quality. The memory application needs to be applied for multiple times each time audio and video frame data is stored during runtime. It is adjusted to apply for a memory pool once when the program is initialized, and the memory pool manages the program's memory. Therefore, only one memory application is required during the entire audio and video playback process, and there is no need to apply for memory multiple times. The strategy of exchanging space for time solves the technical problem of multiple accesses to the memory manager leading to a large number of I / O operations, reduces the frequency of memory access, reduces time overhead, and reduces latency. In addition, by using a tag structure for storage blocks, the principle of dynamic application and taking as much space as required can be implemented, thereby reducing the rate of memory fragmentation and improving memory utilization.

[0079] In one embodiment, the audio and video playback method further includes:

[0080] When a connected capture device is detected, the size of the memory pool to be applied is determined based on the audio and video information supported by the capture device. According to the size of the memory pool to be applied, a memory application operation is performed to the memory manager to obtain a memory pool of the corresponding size.

[0081] The audio and video information supported by the capture device includes but is not limited to frames per second, resolution, color bit, and other information.

[0082] For example, the size of the memory pool to be applied can be determined based on the following formula according to the frames per second, resolution, and color bit supported by the acquisition device.

[0083]

[0084] Among them, M represents the size of the memory pool to be applied, FPS represents the number of frames per second, Resolution represents the resolution, Colourways represents the color bit, and N is an empirical value.

[0085] Taking FPS=60, resolution of 1024*768, color bit of 24-bit true color, and N=6 as an example, it can be calculated that the size of the memory pool to be applied is 22.5M BYTE (bytes).

[0086]

[0087] Since the size of audio frame data is usually small, according to the frames per second, resolution, and color bit supported by the acquisition device, based on the above formula, the size of the memory pool to be applied is sufficient to meet the storage requirements of video frame data and audio frame data.

[0088] In one embodiment, if the data parsing speed is greater than the data rendering speed, N is decreased; if the data parsing speed is less than the data rendering speed, N is increased.

[0089] In order to better utilize the memory space, the size of N can be dynamically adjusted based on the relationship between the data parsing speed and the data rendering speed. If the data parsing speed is greater than the data rendering speed, reduce N to increase the number of frames per second, so as to increase the memory space for a few frames and make the memory pool to be applied larger. Conversely, if the data parsing speed is less than the data rendering speed, increase N to reduce the number of frames per second, so as to reduce the memory space for a few frames and make the memory pool to be applied smaller. By dynamically adjusting the size of N based on the relationship between the data parsing speed and the data rendering speed, the problem of the need to expand the memory pool due to insufficient memory at runtime can be avoided, and the memory reuse efficiency can be better improved.

[0090] Compared with the prior art solution that requires multiple memory requests during operation, the present application solution calculates the size of the memory pool to be requested based on the audio and video information supported by the acquisition device, and can accurately estimate the required memory pool size, thereby satisfying the memory space required for audio and video data playback on the one hand, and improving memory utilization and reducing memory waste on the other.

[0091] S13, according to the playback order of one or more audio and video frame data, multiple audio and video frame data in multiple storage blocks in the memory pool are stored one by one in at least one storage block in the cache queue.

[0092] The structure of the cache queue is a sequential structure, as shown in the schematic diagram. Figure 4 As shown. A cache queue may include one or more storage blocks, such as Figure 4 As shown, for example, it includes storage blocks from the 1st to the 8th. The 1st storage block is the head of the team and the 8th storage block is the tail of the team. Those skilled in the art should understand that the storage blocks in the cache queue are not of fixed size, and the size of each storage block in the cache queue is determined according to the size of the corresponding audio and video frame data stored in the cache queue. Multiple audio and video frame data in multiple storage blocks in the memory pool are stored in the cache queue one by one. When there is no audio and video frame data in the cache queue, the 1st audio and video frame data enters the 1st storage block, the 2nd audio and video frame data enters the 2nd storage block, and so on. When there are audio and video frame data in all 8 storage blocks in the cache queue, the newly entered audio and video frame data will enter from the tail of the team. The audio and video frame data that enters the cache queue first will be taken out of the cache queue first, and the audio and video frame data playing step, i.e., step S14, will be performed. The audio and video frame data that enters the queue later will be played after the audio and video frame data.

[0093] In one embodiment, according to the playback order of one or more audio and video frame data, storing multiple audio and video frame data in multiple storage blocks in the memory pool one by one in at least one storage block in the cache queue includes:

[0094] For each audio and video frame data, when storing the current audio and video frame data in the current storage block in the memory pool into the storage block in the cache queue, determining whether the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is less than the storage block number threshold;

[0095] If the number of storage blocks is less than the storage block number threshold, the current audio and video frame data in the current storage block in the memory pool is stored in the storage block in the cache queue.

[0096] For the cache queue, the more storage blocks and storage space there are in the cache queue, the longer the waiting time will be for the audio and video frame data that enters the cache queue last, from entering the cache queue to entering the audio and video frame data playback step. Figure 4As shown in the figure, for the audio and video frame data in the 8th storage block at the end of the queue, it is necessary to wait for the audio and video frame data in the previous 7 storage blocks to be played before the audio and video frame data in the 8th storage block can be played. Assuming that the playing time of an audio and video frame data is 10ms, the waiting time for the audio and video frame data in the 8th storage block is 70ms. Therefore, the more storage blocks and the larger the storage space of the cache queue, the longer the waiting time for the audio and video frame data in the cache queue to be played.

[0097] Therefore, the setting of the threshold of the number of storage blocks in the cache queue is a key factor. According to experience, the threshold of the number of storage blocks can be set to Q = 2 q -1 (pieces) Wherein, Q is the storage block quantity threshold, q≥1. The storage block quantity threshold can be set to a value that reduces the waiting delay for playing the audio and video frame data and satisfies the user viewing experience during the audio and video frame data playing process.

[0098] In one embodiment, the storage block number threshold is 1. After the audio and video frame data of this storage block in the cache queue is dequeued, it enters the audio and video frame data playing step, and this storage block is immediately used to store the next audio and video frame data. When the storage block number threshold is 1, the waiting time for rendering is the lowest, for example: 10ms*1=10ms, which greatly reduces the waiting time for audio and video frame data to be played.

[0099] When the threshold value of the number of storage blocks in the cache queue is 1, assuming that the memory pool includes 3 audio and video frame data A, B, and C, the order of the audio and video frame data waiting to enter the cache queue and waiting to enter the playback step is also A, B, and C. For each audio and video frame data, when the current audio and video frame data in the current storage block in the memory pool is stored in the storage block in the cache queue, determine whether the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is less than the threshold value of the number of storage blocks (that is, 1). When the cache queue is initially empty, the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is 0, which is less than 1. At this time, the current audio and video frame data A in the current storage block in the memory pool directly enters the storage block in the cache queue. At this time, the audio and video frame data corresponding to the storage block in the cache queue is A. Figure 5As shown in (a) in the figure. The audio and video frame data playback process determines whether there is audio and video frame data in the cache queue. If so, the audio and video frame data A in the storage block is taken out from the cache queue to perform the step of playing the audio and video frame data. At this time, A is taken out from the cache queue, and the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is 0. For the audio and video frame data B, when the current audio and video frame data B in the current storage block in the memory pool is stored into the storage block in the cache queue, it is determined whether the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is less than the storage block number threshold. Since the audio and video frame data A has been taken out from the cache queue, the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is 0, which is less than 1 at this time, and the current audio and video frame data B in the current storage block in the memory pool is stored into the storage block in the cache queue, as shown in FIG. Figure 5 As shown in (b) in the figure. The audio and video frame data playback process determines whether there is audio and video frame data in the cache queue. If so, the audio and video frame data B in the storage block is taken out from the cache queue to perform the step of playing the audio and video frame data. At this time, B is taken out from the cache queue, and the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is 0. For the audio and video frame data C, when the current audio and video frame data C in the current storage block in the memory pool is stored into the storage block in the cache queue, it is determined whether the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is less than the storage block number threshold. Since the audio and video frame data B has been taken out from the cache queue, the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is 0, which is less than 1 at this time, and the current audio and video frame data C in the current storage block in the memory pool is stored into the storage block in the cache queue, as shown in FIG. Figure 5 As shown in (c) in .

[0100] In one embodiment, according to the playback order of one or more audio and video frame data, storing multiple audio and video frame data in multiple storage blocks in the memory pool one by one in at least one storage block in the cache queue includes:

[0101] For each audio and video frame data, when storing the current audio and video frame data in the current storage block in the memory pool into the storage block in the cache queue, determining whether the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is less than the storage block number threshold;

[0102] If the number of storage blocks is less than the storage block number threshold, the current audio and video frame data in the current storage block in the memory pool is stored in the storage block in the cache queue;

[0103] If it is greater than or equal to the storage block quantity threshold, one or more audio and video frame data to be played in the cache queue are discarded;

[0104] The current audio and video frame data in the current storage block in the memory pool is stored into the storage block in the cache queue.

[0105] Here, if the number of storage blocks is greater than or equal to the storage block number threshold, the step of discarding one or more audio and video frame data to be played in the cache queue includes:

[0106] If the number of frames is greater than or equal to the storage block number threshold, one or more audio and video frame data to be played from the head position in the cache queue are discarded. There are at least four specific methods.

[0107] (1) For example, if it is greater than or equal to the threshold of the number of storage blocks, an audio and video frame data to be played in the storage block at the head position in the cache queue is discarded. Here, by discarding an audio and video frame data to be played in the storage block at the head position in the cache queue, the audio and video frame data to be played in the storage blocks after the head position in the cache queue can be moved forward in sequence, thereby freeing up a storage block at the tail position in the cache queue to store the current audio and video frame data in the current storage block in the memory pool into the storage block at the tail position in the cache queue. Discarding an audio and video frame data to be played in the storage block at the head position in the cache queue can meet the basic queuing requirement of storing a current audio and video frame data in the current storage block in the memory pool into the cache queue, quickly and accurately meet the audio and video playback requirements, and only discard one audio and video frame data to be played, minimize the discontinuity of audio and video playback, and better meet the user's experience of watching audio and video.

[0108] (2) For another example, if it is greater than or equal to the threshold value of the number of storage blocks, multiple audio and video frame data to be played in the multiple continuous storage blocks starting from the head position in the cache queue are discarded. Here, the multiple audio and video frame data to be played in the multiple continuous storage blocks starting from the head position in the cache queue are discarded, and the audio and video frame data to be played in the storage blocks after the multiple discarded storage blocks in the cache queue can be moved forward in sequence, thereby freeing up a corresponding number of storage blocks at the tail position in the cache queue, so as to store the current audio and video frame data in the current storage block in the memory pool into the front position of the free storage blocks in the cache queue. When the audio and video parsing speed is greater than the audio and video playback speed, the multiple audio and video frame data to be played in the multiple continuous storage blocks starting from the head position in the cache queue are discarded, which can more quickly relieve the storage pressure of the memory pool, release memory space, and more quickly relieve the delay of audio and video playback after the audio and video frame data are played.

[0109] (3) For another example, if the number of storage blocks is greater than or equal to the storage block number threshold, multiple audio and video frame data to be played in multiple interval storage blocks starting from the head position in the cache queue are discarded. Figure 4As shown, for example, 4 audio and video frame data located in 4 storage blocks, namely the 1st, 3rd, 5th and 7th, are discarded, and the audio and video frame data to be played in the storage blocks after the multiple discarded storage blocks in the cache queue are moved forward in sequence until there are no intermittent free storage blocks, so that there are multiple continuous free storage blocks at the back of the cache queue, so as to store the current audio and video frame data in the current storage block in the memory pool into the storage block at the front of the multiple continuous free storage blocks in the cache queue. The frame skipping discarding method, i.e., discarding multiple audio and video frame data to be played in multiple intermittent storage blocks in the cache queue starting from the head position, can, on the one hand, reduce the discontinuity of audio and video playback to a lesser extent, better meet the user's experience of watching audio and video, and on the other hand, reduce the delay of audio and video playback, thereby balancing the playback requirements of both continuity and delay of audio and video playback.

[0110] (4) For another example, if the number of storage blocks is greater than or equal to the threshold, one or more audio and video frame data to be played from the head position in the cache queue is discarded and the threshold of the number of storage blocks is reduced. Here, by discarding one or more audio and video frame data to be played from the head position in the cache queue and reducing the threshold of the number of storage blocks, the actual queue length of the cache queue can be shortened, and the delay of audio and video data playback can be reduced more quickly.

[0111] In one embodiment, according to the playback order of one or more audio and video frame data, storing multiple audio and video frame data in multiple storage blocks in the memory pool one by one in at least one storage block in the cache queue includes:

[0112] For each audio and video frame data, when storing the current audio and video frame data in the current storage block in the memory pool into the storage block in the cache queue, determining whether the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is less than the storage block number threshold;

[0113] If the number of storage blocks is less than the storage block number threshold, the current audio and video frame data in the current storage block in the memory pool is stored in the storage block in the cache queue;

[0114] If it is greater than or equal to the storage block quantity threshold, one or more audio and video frame data to be played in the cache queue are discarded and the storage block quantity threshold is reduced;

[0115] The current audio and video frame data in the current storage block in the memory pool is stored into the storage block in the cache queue.

[0116] Here, if it is greater than or equal to the storage block number threshold, one or more audio and video frame data to be played in the cache queue will be discarded and the storage block number threshold will be reduced, which can shorten the actual queue length of the cache queue and reduce the audio and video data playback delay more quickly.

[0117] In the prior art, in order to pursue the smoothness of the picture and sound, no special restrictions are made on the processing of audio and video frame data in the cache queue. In this case, if the playback rate of the audio and video frame data is lower than the transmission rate of the audio and video frame data, more and more data will be accumulated in the cache queue, resulting in serious lag in the picture and sound. The present application optimizes the processing scheme of audio and video frame data in the cache queue by determining whether the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is less than the storage block number threshold. By adopting the strategy of discarding audio and video frame data and supplementing audio and video frame data, and dynamically adjusting the cache queue to dynamically adjust the audio and video frame data in the cache queue, it can take into account both the playback delay of audio and video frame data and the smoothness of the picture.

[0118] In one embodiment, the storage block number threshold is 1. After the audio and video frame data of this storage block in the cache queue is dequeued, it enters the audio and video frame data playing step, and this storage block is immediately used to store the next audio and video frame data. When the storage block number threshold is 1, the waiting time for rendering is the lowest, for example: 10ms*1=10ms, which greatly reduces the waiting time for audio and video frame data to be played.

[0119] When the threshold value of the number of storage blocks in the cache queue is 1, assuming that the memory pool includes 3 audio and video frame data A, B, and C, the order of the audio and video frame data waiting to enter the cache queue and waiting to enter the playback step is also A, B, and C. For each audio and video frame data, when the current audio and video frame data in the current storage block in the memory pool is stored in the storage block in the cache queue, determine whether the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is less than the threshold value of the number of storage blocks (that is, 1). When the cache queue is initially empty, the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is 0, which is less than 1. At this time, the current audio and video frame data A in the current storage block in the memory pool directly enters the storage block in the cache queue. At this time, the audio and video frame data corresponding to the storage block in the cache queue is A. Figure 6As shown in (a) in the figure. The audio and video frame data playing process determines whether there is audio and video frame data in the cache queue. If there is, the audio and video frame data A in the storage block is taken out from the cache queue to play the audio and video frame data. At this time, A is taken out from the cache queue, and the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is 0. For audio and video frame data B, when the current audio and video frame data B in the current storage block in the memory pool is stored in the storage block in the cache queue, it is determined whether the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is less than the storage block number threshold. Since the audio and video frame data A has been taken out from the cache queue, the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is 0, which is less than 1, and the current audio and video frame data B in the current storage block in the memory pool is stored in the storage block in the cache queue. For audio and video frame data C, when the current audio and video frame data C in the current storage block in the memory pool is stored in the storage block in the cache queue, it is determined whether the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is less than the storage block number threshold. If the audio and video frame data A is still being rendered, and the audio and video frame data B has not been taken out of the cache queue, the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is 1, which is equal to the storage block number threshold. Figure 6 At this time, the audio and video frame data B to be played in the cache queue is taken out of the queue and discarded, and then the current audio and video frame data C in the current storage block in the memory pool is stored in the storage block in the cache queue, as shown in (b). Figure 6 As shown in (c) in .

[0120] In one embodiment, the audio and video playback method further includes:

[0121] Modify the memory usage mark of the storage block corresponding to the discarded one or more audio and video frame data to be played to free from the memory pool;

[0122] The memory manager is notified of a message that the storage blocks corresponding to one or more discarded audio and video frame data to be played are available.

[0123] Taking the size of 4kBYTE (bytes) and the address of 0-4095 as an example, the applied memory pool is as follows Figure 2 shown.

[0124] Following the previous example, the structure marked by the used storage blocks in the memory pool can be, for example:

[0125] {"Start Address:":"0",

[0126] "End Address:": "255",

[0127] "Current block number": "0",

[0128] "IsCurrentlyInUsed": "true" / / "IsCurrentlyInUsed": "yes"}.

[0129] If the audio and video frame data corresponding to the storage block is discarded, the content of the storage block corresponding to the audio and video frame data is modified to be free using a mark. The corresponding mark structure may be, for example:

[0130] {"Start Address:":"0",

[0131] "End Address:": "255",

[0132] "Current block number": "0",

[0133] "IsCurrentlyInUsed": "false" / / "IsCurrentlyInUsed": "No"}.

[0134] Since the storage block has been marked as "currently not in use" (i.e., "currently in use": "false"), and the memory manager has been notified that the storage block corresponding to one or more discarded audio and video frame data to be played can be used, the memory space corresponding to the storage block can be used again.

[0135] In the memory pool, the memory usage mark of the storage block corresponding to the discarded audio and video frame data to be played is modified to free, and the memory manager is notified of the message that the storage block corresponding to one or more discarded audio and video frame data to be played is available, so that the storage space corresponding to the storage block can be used again and connected with other free storage spaces for use by other audio and video frame data, thereby reducing the fragmentation problem caused by inconsistent frame data size in the memory pool.

[0136] S14, playing at least one audio or video frame data retrieved from at least one storage block in the cache queue in a first-in-first-out order.

[0137] Under the condition of a YUV video frame data and PCM audio frame data playing environment, at least one YUV video frame data and PCM audio frame data taken out from at least one storage block in the cache queue can be directly played in a first-in-first-out order.

[0138] The audio and video playback method provided by the present application, when receiving the initial audio and video data transmitted by the acquisition device, parses the initial audio and video data to obtain one or more audio and video frame data; according to the playback order of the one or more audio and video frame data, according to the size of each audio and video frame data, stores each audio and video frame data one by one in the applied memory pool into a storage block of corresponding size, wherein each storage block stores one audio and video frame data; according to the playback order of the one or more audio and video frame data, stores multiple audio and video frame data in multiple storage blocks in the memory pool one by one in at least one storage block in the cache queue; according to the first-in-first-out order, plays at least one audio and video frame data taken out from at least one storage block in the cache queue. On the one hand, the present application scheme optimizes the memory application and usage without sacrificing image quality. The memory application is adjusted to apply for a memory pool once when the program is initialized, and the memory pool is used to uniformly manage the program's memory. Therefore, memory is only requested once during the entire audio and video playback process, and there is no need to request memory multiple times. The strategy of exchanging space for time solves the technical problem of multiple accesses to the memory manager leading to a large number of I / O operations, reduces the frequency of memory access, reduces time overhead, and reduces latency. On the other hand, by parsing the initial audio and video data, one or more audio and video frame data are obtained, and according to the playback order of one or more audio and video data, the audio and video frame data are stored in the storage block matching the size of the audio and video frame data in the memory pool, the storage block matching the size of the audio and video frame data in the cache queue, and finally the audio and video frame data taken out from the storage block in the cache queue is played. The multi-level storage mechanism can better store and buffer the audio and video frame data before playback, reducing the overall latency of the audio and video playback process and improving the quality of audio and video playback and user experience.

[0139] In one embodiment, after playing at least one audio or video frame data retrieved from at least one storage block in the cache queue in a first-in-first-out order, the method further includes:

[0140] Modify the memory usage mark of the storage block corresponding to the played audio and video frame data from the memory pool to free;

[0141] The memory manager is notified that the storage block corresponding to the played audio and video frame data is available.

[0142] Take the size of 4k BYTE (bytes) and the address of 0-4095 as an example, the memory pool that has been applied for is as follows Figure 2 shown.

[0143] Following the previous example, the structure marked by the used storage blocks in the memory pool can be, for example:

[0144] {"Start Address:":"256",

[0145] "End Address:": "511",

[0146] "Current block number": "1",

[0147] "IsCurrentlyInUsed": "true" / / "IsCurrentlyInUsed": "yes"}.

[0148] If the audio and video frame data corresponding to the storage block has been played, the content of the storage block corresponding to the audio and video frame data is modified to be free using a mark, and the corresponding structure may be, for example:

[0149] {"Start Address:":"256",

[0150] "End Address:": "511",

[0151] "Current block number": "1",

[0152] "IsCurrentlyInUsed": "false" / / "IsCurrentlyInUsed": "No"}.

[0153] Since the storage block has been marked as "currently not in use" (i.e. "currently in use": "false"), and the memory manager has been informed that the storage block corresponding to the played audio and video frame data is available, the memory space corresponding to the storage block can be used again. The current structure diagram of the memory pool is as follows Figure 7 shown.

[0154] In the memory pool, the memory usage mark of the storage block corresponding to the played audio and video frame data is modified to free, and the memory manager is notified of the message that the storage block corresponding to the played audio and video frame data is available, so that the storage space corresponding to the storage block can be used again and connected with other free storage spaces for use by other audio and video frame data, thereby reducing the fragmentation problem caused by inconsistent frame data size in the memory pool.

[0155] In one embodiment, playing at least one audio and video frame data retrieved from at least one storage block in the cache queue in a first-in-first-out order includes:

[0156] Decoding at least one audio and video frame data retrieved from at least one storage block in the cache queue in a first-in-first-out order;

[0157] At least one decoded audio or video frame data is broadcasted in the order of decoding.

[0158] Here, the audio and video frame data retrieved from at least one storage block in the cache queue can be decoded in a first-in-first-out order, that is, in the order in which the audio and video frame data is played, including decoding the video frame data and transcoding the audio frame data. Subsequently, in a decoding order, that is, in the order in which the audio and video frame data is played, the video frame data is rendered and displayed on the screen, and the audio frame data is broadcast through the audio equipment.

[0159] In one embodiment, decoding at least one audio and video frame data retrieved from at least one storage block in a cache queue in a first-in-first-out order includes:

[0160] At least one video frame data retrieved from at least one storage block in the cache queue is decoded in a first-in-first-out order through video format conversion.

[0161] Here, the video format can be converted based on the following formula to transcode the video frame data in YUV format into video frame data in RGB format:

[0162] R = Y + 1.402 * (V - 128);

[0163] ·G=Y-0.344136*(U-128)-0.714136*(V-128);

[0164] B = Y + 1.772*(U-128).

[0165] The video frame data in YUV format is the original data of the video frame, and the video frame data in RGB format is the rendering data format of the screen.

[0166] In the prior art, the transcoding method of video frame data is mostly to transcode the original YUV format video frame data into MJPEG format video frame data, and then transcode the MJPEG format video frame data into RGB format video frame data, and output it to the screen for rendering. The processing process of the prior art goes through two transcoding links, which will cause serious delays for multi-screen linkage and strict real-time requirements such as capture card screen projection. The present application scheme optimizes the transcoding and rendering methods of video frame data and reduces the video transcoding links. During the video frame data decoding process, the video frame data in YUV format is directly transcoded into the video frame data in RGB format, eliminating the intermediate link of transcoding the video frame data in YUV format to the video frame data in MJPEG format, and greatly reducing the transcoding delay.

[0167] In one embodiment, the audio and video playback method is applied to the native layer of the Android system.

[0168] Among them, the native layer of the Android system refers to the Native layer.

[0169] In the prior art, for audio frame data, the traditional audio playback method of the Android platform is to decode the audio frame data in PCM format, put the audio frame data in PCM format into the Native cache queue, copy the audio frame data in the Native cache queue to the Java business layer through JNI, and play it by the AudioTrack of the Java business layer. Among them, JNI is the full name of Java Native Interface, which is a universal communication framework between Android platform Native and Java.

[0170] The present application solution places all the decoding and rendering processes of video frame data transmission and the transcoding and broadcasting processes of audio frame data in the Native layer of the Android platform. The Native layer decodes the video frame data and plays the audio frame data in PCM format, optimizes the video decoding and playback mode and the audio decoding and playback mode, reduces the transmission link of PCM audio data, and reduces the sound playback delay and jamming as well as the video delay. In the prior art, if the audio is played using AudioTrack in the traditional way after parsing the data, the data conversion through JNI is required in the middle, which deepens the data transmission link in the entire link, and involves the action of copying data from the Native heap space to the Java space. One copy action is required for one data playback, which undoubtedly increases the access frequency of the memory and brings additional time overhead. The present application solution completes the entire audio and video frame data processing link in the Native layer, reduces the time overhead generated by the data copying action in the prior art, reduces the delay, and reduces the probability of sound jamming.

[0171] Figure 8 It is a flowchart of an audio and video playback method provided by an application embodiment of the present application.

[0172] S801, connect one end of the acquisition card to the acquisition device and the other end to the vehicle system.

[0173] S802, initialization program, applying for storage space to form a memory pool.

[0174] S803, when the initial audio and video data transmitted by the acquisition device is received, the initial audio and video data is parsed to obtain one or more audio and video frame data.

[0175] S804, calculating the size of the current audio and video frame data.

[0176] S805, when the program is running, according to the playback order of one or more audio and video frame data and the size of the current audio and video frame data, the corresponding space is obtained from the memory pool, and each audio and video frame data is stored one by one in the storage block of the corresponding size in the applied memory pool to load the audio and video frame data, wherein each storage block stores one audio and video frame data.

[0177] S806, for each audio and video frame data, when the current audio and video frame data in the current storage block in the memory pool is stored in the storage block in the cache queue, determine whether the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is greater than the storage block number threshold (whether it is greater than 1).

[0178] S8071, if it is greater than 1, discard the audio and video frame data to be played in the cache queue and store the current audio and video frame data in the current storage block in the memory pool into the storage block in the cache queue.

[0179] S8072, adjust the memory pool so that the space in the memory pool can be used for the next audio and video frame data. Change the memory usage mark of the storage block corresponding to the discarded one or more audio and video frame data to be played from the memory pool to free. Notify the memory manager of the message that the storage block corresponding to the discarded one or more audio and video frame data to be played is available.

[0180] S8081, if it is less than 1, store the current audio and video frame data in the current storage block in the memory pool into the storage block in the cache queue.

[0181] S8082: The Native layer decodes at least one audio or video frame data retrieved from at least one storage block in the cache queue in a first-in, first-out order.

[0182] S8083: In the decoding order, the Native layer broadcasts at least one decoded audio frame data and renders at least one decoded video frame data.

[0183] Fig. 9 It is a structural diagram of an audio and video playback device provided in one embodiment of the present application.

[0184] like Fig. 9 As shown, the audio and video playback device 9 includes:

[0185] The data parsing module 91 is used to parse the initial audio and video data when receiving the initial audio and video data transmitted by the acquisition device to obtain one or more audio and video frame data;

[0186] A memory storage module 92, configured to store each of the audio and video frame data one by one in a storage block of a corresponding size in the applied memory pool according to the playback order of the one or more audio and video frame data and the size of each audio and video frame data, wherein each of the storage blocks stores one audio and video frame data;

[0187] A cache storage module 93, configured to store the plurality of audio and video frame data in the plurality of storage blocks in the memory pool into at least one storage block in the cache queue one by one in sequence according to the playback order of the one or more audio and video frame data;

[0188] The data playing module 94 is used to play at least one audio or video frame data taken out from the at least one storage block in the cache queue in a first-in-first-out order.

[0189] Another embodiment of the present invention discloses an audio and video playback device 9. Fig. 9 On the basis of the corresponding embodiment, the audio and video playback device 9 further includes:

[0190] A memory size determination module, for determining the size of the memory pool to be applied for according to the audio and video information supported by the acquisition device when the connected acquisition device is detected;

[0191] The memory pool application module is used to perform a memory application operation on the memory manager according to the size of the memory pool to be applied for, so as to obtain a memory pool of corresponding size.

[0192] Another embodiment of the present invention discloses an audio and video playback device 9. Fig. 9 Based on the corresponding embodiment, the cache storage module 93 is used for:

[0193] For each audio and video frame data, when storing the current audio and video frame data in the current storage block in the memory pool into the storage block in the cache queue, determining whether the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is less than a storage block number threshold;

[0194] If it is less than the storage block quantity threshold, the current audio and video frame data in the current storage block in the memory pool is stored in the storage block in the cache queue.

[0195] Another embodiment of the present invention discloses an audio and video playback device 9. Fig. 9 Based on the corresponding embodiment, the cache storage module 93 is further used for:

[0196] If the number of the storage blocks is greater than or equal to the storage block quantity threshold, one or more audio and video frame data to be played in the cache queue are discarded;

[0197] The current audio and video frame data in the current storage block in the memory pool is stored in the storage block in the cache queue.

[0198] Another embodiment of the present invention discloses an audio and video playback device 9. Fig. 9 On the basis of the corresponding embodiment, the audio and video playback device 9 further includes:

[0199] A first memory modification module is used to modify the memory usage mark of the storage block corresponding to the one or more discarded audio and video frame data to be played from the memory pool to be free;

[0200] The first message notification module is used to notify the memory manager of a message that the storage blocks corresponding to one or more discarded audio and video frame data to be played are available.

[0201] Another embodiment of the present invention discloses an audio and video playback device 9. Fig. 9 On the basis of the corresponding embodiment, the audio and video playback device 9 further includes:

[0202] A second memory modification module is used to modify the memory usage mark of the storage block corresponding to the played audio and video frame data from the memory pool to free;

[0203] The second message notification module is used to notify the memory manager of the message that the storage block corresponding to the played audio and video frame data is available.

[0204] Another embodiment of the present invention discloses an audio and video playback device 9. Fig. 9 Based on the corresponding embodiment, the data playback module 94 is used to:

[0205] Decoding at least one audio and video frame data retrieved from at least one storage block in the cache queue in a first-in-first-out order;

[0206] At least one decoded audio or video frame data is broadcasted in the order of decoding.

[0207] Another embodiment of the present invention discloses an audio and video playback device 9. Fig. 9 On the basis of the corresponding embodiment, the audio and video playback device 9 is applied to the native layer of the Android system.

[0208] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0209] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0210] The present application also provides a terminal device, such as Fig.10 As shown, the terminal device 10 includes: at least one processor 100, a memory 101, and a computer program 102 stored in the memory 101 and executable on the at least one processor 100. The terminal device 10 is connected to a collection device, and the processor 100 implements the steps in any of the above-mentioned method embodiments when executing the computer program 102.

[0211] An embodiment of the present application also provides a vehicle system, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. The vehicle system is connected to an acquisition device, and the processor implements the steps in any of the above method embodiments when executing the computer program.

[0212] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0213] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0214] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0215] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0216] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0217] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0218] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0219] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An audio and video playback method, characterized in that: include: When receiving the initial audio and video data transmitted by the acquisition device, parsing the initial audio and video data to obtain one or more audio and video frame data; According to the playback order of the one or more audio and video frame data and the size of each audio and video frame data, each audio and video frame data is sequentially stored in a storage block of a corresponding size in the applied memory pool, wherein each storage block stores one audio and video frame data; According to the playback order of the one or more audio and video frame data, the multiple audio and video frame data in the multiple storage blocks in the memory pool are stored one by one in at least one storage block in the cache queue; At least one audio or video frame data retrieved from the at least one storage block in the cache queue is played in a first-in-first-out order.

2. The audio and video playback method according to claim 1, characterized in that: The method further includes: When the connected acquisition device is detected, the size of the memory pool to be applied is determined according to the audio and video information supported by the acquisition device; According to the size of the memory pool to be applied for, a memory application operation is performed on the memory manager to obtain a memory pool of corresponding size.

3. The audio and video playback method according to claim 1, characterized in that: The step of storing the plurality of audio and video frame data in the plurality of storage blocks in the memory pool one by one in sequence into at least one storage block in the cache queue according to the playback order of the one or more audio and video frame data comprises: For each audio and video frame data, when storing the current audio and video frame data in the current storage block in the memory pool into the storage block in the cache queue, determining whether the number of storage blocks corresponding to the audio and video frame data to be played in the cache queue is less than a storage block number threshold; If it is less than the storage block quantity threshold, the current audio and video frame data in the current storage block in the memory pool is stored in the storage block in the cache queue.

4. The audio and video playback method according to claim 3, characterized in that: The method further includes: If the number of the storage blocks is greater than or equal to the storage block quantity threshold, one or more audio and video frame data to be played in the cache queue are discarded; The current audio and video frame data in the current storage block in the memory pool is stored in the storage block in the cache queue.

5. The audio and video playback method according to claim 4, characterized in that: The method further includes: Modify the memory usage mark of the storage block corresponding to the one or more discarded audio and video frame data to be played from the memory pool to free; The memory manager is notified of a message that the storage blocks corresponding to one or more discarded audio and video frame data to be played are available.

6. The audio and video playback method according to claim 1, characterized in that: After playing at least one audio and video frame data taken out from at least one storage block in the cache queue in the first-in-first-out order, the method further includes: Modify the memory usage mark of the storage block corresponding to the played audio and video frame data from the memory pool to free; The memory manager is notified that the storage block corresponding to the played audio and video frame data is available.

7. The audio and video playback method according to claim 1, characterized in that: The step of playing at least one audio or video frame data retrieved from at least one storage block in the cache queue in a first-in-first-out order includes: Decoding at least one audio and video frame data retrieved from at least one storage block in the cache queue in a first-in-first-out order; At least one decoded audio or video frame data is broadcasted in the order of decoding.

8. The audio and video playback method according to claim 1, characterized in that: The audio and video playback method is applied to the native layer of the Android system.

9. An audio and video playback device, characterized in that: include: A data parsing module, for parsing the initial audio and video data transmitted by the acquisition device to obtain one or more audio and video frame data; A memory storage module, used for storing each audio and video frame data in a storage block of a corresponding size in the applied memory pool one by one in accordance with the playback order of the one or more audio and video frame data and the size of each audio and video frame data, wherein each storage block stores one audio and video frame data; A cache storage module, used for storing the multiple audio and video frame data in the multiple storage blocks in the memory pool into at least one storage block in the cache queue one by one in sequence according to the playback order of the one or more audio and video frame data; The data playing module is used to play at least one audio or video frame data taken out from the at least one storage block in the cache queue in a first-in first-out order.

10. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The terminal device is connected to a collection device, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

11. A vehicle computer system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The vehicle system is connected to a collection device, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images