Audio and picture tracking method and device, television and storage medium

Abstract

The invention relates to a sound and picture tracking method and device, a television and a storage medium, and the method comprises the steps: detecting first data and second data in a cache region in a process of carrying out the sound and picture tracking of a played video, so as to obtain a first detection result, the first data being audio data of the cached video, and the second data being audio data of the cached video; the second data is data obtained after sound effect processing is carried out on the first data; when the first detection result includes that both the first data and the second data are empty, obtaining load information corresponding to the processor; according to the load information, a first adjustment parameter corresponding to a preset cache amount of the cache region is determined, and the preset cache amount represents the data volume of the first data cached in the cache region; and according to the first adjustment parameter, adjusting the preset cache amount so as to complete audio and picture tracking of the played video by using the adjusted preset cache amount. According to the method and the device, the problems of sound interruption, noise and continuous silence during audio and picture tracking of the video are avoided, and the use experience of a user is improved.

Description

Technical Field

[0001] The present application relates to the field of television technology, and in particular to a method and device for tracking sound and picture, a television and a storage medium. Background Art

[0002] When we watch videos on TV, the objects in the video screen are usually moving, but the positions of the speakers installed in the TV are fixed. Therefore, the position of the synthesized sound and image produced by the sounds of the speakers is fixed. This causes the sound and image of the played video to be inconsistent with the positions of the objects in the video screen, which results in a different feeling from real life and a poor viewing experience for users.

[0003] At present, in order to make the sound and image of the played video consistent with the position of objects in the video screen, an audio and video tracking algorithm is integrated in the main chip of the TV. When playing the video, the video screen on the TV screen is captured at a certain frequency, the coordinates of the sound source position are determined according to the captured video screen, and the sound effect data cached in the buffer area are processed according to the sound source position coordinates, so as to control the amplitude and phase information of each speaker to make the synthesized sound and image position coincide with the real-time position of the moving object, thereby realizing audio and video tracking of the played video.

[0004] However, the above-mentioned audio and video tracking method has high requirements on the performance of the TV main chip. Since the main chips used in current high-end TVs are not very powerful and have limited computing power, when the audio data of the video cached in the cache and the data after the audio data is processed are empty, there will be no remaining audio data in the cache, resulting in disconnection, noise and continuous silence, which in turn affects the user experience. Summary of the invention

[0005] The present application provides a method, device, television and storage medium for audio and video tracking to solve the problems of disconnection, noise and continuous silence when performing audio and video tracking on a video in the prior art, thereby affecting the user experience.

[0006] In a first aspect, the present application provides a method for tracking audio and video, comprising:

[0007] In the process of performing audio and video tracking on the played video, first data and second data in the buffer area are detected to obtain a first detection result, where the first data is the audio data of the buffered video, and the second data is the data after the first data is processed with sound effects;

[0008] When the first detection result includes that both the first data and the second data are empty, acquiring load information corresponding to a processor, the processor being used to process the sound effect of the video;

[0009] Determine, according to the load information, a first adjustment parameter of a preset cache amount corresponding to the cache area, wherein the preset cache amount represents a data amount of the first data cached in the cache area;

[0010] The preset buffer size is adjusted according to the first adjustment parameter, so as to complete the audio and video tracking of the played video using the adjusted preset buffer size.

[0011] In an optional implementation, after performing the step of determining the first adjustment parameter corresponding to the preset cache amount of the cache area according to the load information, the method further includes:

[0012] Obtaining a first dynamic priority and a first static priority corresponding to a first process, and a second dynamic priority and a second static priority corresponding to a second process, wherein the first process is a process used for processing the displayed graphical interface, and the second process is a process used for processing the sound effects of the video;

[0013] The second dynamic priority and the second static priority are adjusted according to the first dynamic priority and the first static priority, so that the second total priority is one level lower than the first total priority, the second total priority is the sum of the adjusted second dynamic priority and the adjusted second static priority, and the first total priority is the sum of the first dynamic priority and the first static priority;

[0014] The step of adjusting the preset buffer size according to the first adjustment parameter so as to complete the audio and video tracking of the played video by using the adjusted preset buffer size includes:

[0015] According to the first adjustment parameter, the preset cache amount is adjusted to obtain the adjusted preset cache amount;

[0016] The audio and video tracking of the played video is completed using the adjusted preset cache amount, the adjusted second dynamic priority and the adjusted second static priority.

[0017] In an optional implementation, the adjusting the second dynamic priority and the second static priority according to the first dynamic priority and the first static priority so that the second total priority is one level lower than the first total priority includes:

[0018] The second dynamic priority and the second static priority are adjusted according to the first dynamic priority and the first static priority, so that the adjusted second dynamic priority is different from the first dynamic priority and the second total priority is one level lower than the first total priority.

[0019] In an optional implementation, before executing the step of detecting the first data and the second data in the buffer area to obtain a first detection result during the audio and video tracking of the played video, the method further includes:

[0020] When it is determined that the target mode has been turned on, detecting the first data in the buffer area to obtain a second detection result, wherein the target mode is used to track the audio and video of the played video;

[0021] When the second detection result includes that the first data is not empty, clearing the first data in the buffer area;

[0022] After the first data in the buffer area is cleared, the step of detecting the first data and the second data in the buffer area during the audio and video tracking of the played video is performed to obtain a first detection result.

[0023] In an optional implementation, after executing the step of detecting the first data and the second data in the buffer area to obtain a first detection result during the audio and video tracking of the played video, the method further includes:

[0024] When the first detection result includes that the first data is empty and the second data is not empty, or when the first detection result includes that both the first data and the second data are not empty, determining a target data volume corresponding to the second data;

[0025] Determining, according to the target data volume, a second adjustment parameter corresponding to a preset delay time, wherein the preset delay time is used to achieve audio and video synchronization during audio and video tracking of the played video;

[0026] The preset delay time is adjusted according to the second adjustment parameter, so as to use the adjusted preset delay time to complete the audio and video synchronization in the process of audio and video tracking of the played video.

[0027] In an optional implementation, determining the second adjustment parameter corresponding to the preset delay time according to the target data volume includes:

[0028] Determine a target difference between the target data volume and a preset data volume corresponding to the preset delay time;

[0029] Based on the target difference, query the first association relationship to obtain a second adjustment parameter corresponding to the preset delay time, wherein the first association relationship stores a plurality of corresponding relationships between the difference values ​​and the adjustment parameters, and the adjustment parameter in the first association relationship increases as the difference value increases;

[0030] The step of adjusting the preset delay time according to the second adjustment parameter so as to use the adjusted preset delay time to complete the audio and video synchronization in the process of audio and video tracking of the played video includes:

[0031] Determine the sum of the second adjustment parameter and the preset delay time to obtain the adjusted preset delay time;

[0032] The adjusted preset delay time is used to complete the audio and video synchronization during the audio and video tracking process of the played video.

[0033] In an optional implementation, the load information includes a usage rate;

[0034] The step of determining, according to the load information, a first adjustment parameter of a preset cache amount corresponding to the cache area includes:

[0035] Determining a target comparison result between the usage rate and a preset usage rate threshold, wherein the preset usage rate threshold represents a lower limit value of the usage rate corresponding to adjusting the preset cache amount corresponding to the cache area;

[0036] When the target comparison result is that the usage rate is greater than the preset usage rate threshold, the second association relationship is queried based on the usage rate to obtain the first adjustment parameter of the preset cache amount, and the second association relationship stores multiple sets of corresponding relationships between usage rates and adjustment parameters, and the adjustment parameters in the second association relationship increase with the increase of usage rate.

[0037] In a second aspect, the present application provides a sound and picture tracking device, comprising:

[0038] a detection module, configured to detect first data and second data in a buffer area during audio and video tracking of a played video to obtain a first detection result, wherein the first data is the buffered audio data of the video, and the second data is data after sound effects processing is performed on the first data;

[0039] an acquisition module, configured to acquire load information corresponding to a processor when the first detection result includes that both the first data and the second data are empty, and the processor is configured to process the sound effects of the video;

[0040] a determination module, configured to determine, according to the load information, a first adjustment parameter of a preset cache amount corresponding to the cache area, wherein the preset cache amount represents a data amount of the first data cached in the cache area;

[0041] The tracking module is used to adjust the preset cache amount according to the first adjustment parameter, so as to complete the audio and video tracking of the played video by using the adjusted preset cache amount.

[0042] In a third aspect, the present application provides a television, comprising: a processor and a memory, wherein the processor is used to execute an audio and video tracking program stored in the memory to implement the audio and video tracking method described above.

[0043] In a fourth aspect, the present application provides a storage medium storing one or more programs, and the one or more programs can be executed by one or more processors to implement the audio and video tracking method as described above.

[0044] The above-mentioned technical scheme provided by the embodiment of the present application has the following advantages compared with the prior art. The audio and video tracking method provided by the embodiment of the present application includes, in the process of audio and video tracking of the played video, detecting the first data and the second data in the cache area to obtain a first detection result, the first data is the audio data of the cached video, and the second data is the data after the first data is processed with sound effects; when the first detection result includes that the first data and the second data are both empty, obtaining the load information corresponding to the processor, and the processor is used to process the sound effects of the video; according to the load information, determining the first adjustment parameter corresponding to the preset cache amount of the cache area, the preset cache amount represents the data amount of the first data cached in the cache area; according to the first adjustment parameter, adjusting the preset cache amount to use the adjusted preset cache amount to complete the audio and video tracking of the played video. Through the above method, during the process of audio and video tracking of the played video, if the audio data of the video cached in the cache area and the data after the audio data is processed with sound effects are both empty, the load information of the sound effect of the processed video is obtained, and the preset cache amount for caching the first data in the cache area is dynamically adjusted according to the obtained load information, so as to leave a surplus of audio data of the video in the cache area, thereby avoiding the problems of disconnection, noise and continuous silence when tracking the audio and video of the video, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0048] Figure 1 A flowchart of a method for tracking sound and picture provided in an embodiment of the present application;

[0049] Figure 2 A flowchart of another audio and video tracking method provided in an embodiment of the present application;

[0050] Figure 3 A flowchart of another audio and video tracking method provided in an embodiment of the present application;

[0051] Figure 4 A processing flow chart of a sound and picture tracking algorithm provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of the structure of a sound and picture tracking device provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of the structure of a television provided in an embodiment of the present application;

[0054] In the above attached figure:

[0055] 501, detection module; 502, acquisition module; 503: determination module; 504: tracking module;

[0056] 600, television; 601, processor; 602, memory; 6021, operating system; 6022, application; 603, user interface; 604, network interface; 605, bus system. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0058] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0059] refer to Figure 1 , Figure 1 A flow chart of a method for tracking audio and video provided in an embodiment of the present application. A method for tracking audio and video provided in this embodiment includes the following steps:

[0060] S101: In the process of performing audio and video tracking on the played video, first data and second data in the buffer area are detected to obtain a first detection result.

[0061] In this embodiment, the method is applied to a processor in a television, and the processor is used to process the sound effects of the video during the audio and video tracking of the video being played. The first data in the buffer area mentioned above is the audio data of the cached video, and the second data in the buffer area is the data after the first data is processed with sound effects. Processing the sound effects can be understood as performing sound effect processing on the audio data of the video cached in the buffer area according to the coordinates of the moving object in the determined video screen, so as to cache the processed audio data in the buffer area, thereby sending the processed audio data in the buffer area to the audio driver interface, and finally controlling each speaker to sound, so that the synthetic sound and image position of all speakers coincides with the coordinate position of the object in the video screen, thereby achieving audio and video tracking of the played video.

[0062] Among them, the processing process of the audio and video tracking algorithm is as follows Figure 4 As shown, when the TV plays a video, it takes screenshots of the video image on the TV screen at a certain frequency, obtains the sound effect data of the video, caches the obtained sound effect data in a cache area, determines the coordinates of the sound source position according to the screenshot of the video image, performs sound effect processing on the sound effect data cached in the cache area according to the sound source position coordinates, and caches the processed sound effect data in the cache area, sends the processed audio data in the cache area to the audio driver interface, and finally controls each speaker to make a sound, so that the synthetic sound and image positions of all speakers coincide with the coordinate positions of the objects in the video image, thereby realizing the sound and image tracking of the played video.

[0063] Specifically, in the process of using the above-mentioned audio and video tracking algorithm to track the audio and video of the played video, if the audio data of the video cached in the buffer area and the data after the audio data is processed for sound effects are both empty, after processing a packet of audio data, a packet of audio data must be sent down immediately, and there is no remaining buffered audio data. When the processor used to process the sound effects of the video in the TV is under high load, the data may not be sent in time, resulting in disconnection or noise, and if the duration is longer, there will be a continuous silence problem. Therefore, in order to solve the above problem, in this embodiment, if the audio data of the video cached in the buffer area and the data after the audio data is processed for sound effects are both empty, the load situation of the processor is obtained, and the cache area is adjusted according to the obtained situation, so that when the processor is under high load, there is a surplus of video audio data in the cache area.

[0064] S102: When the first detection result includes that both the first data and the second data are empty, obtain load information corresponding to the processor.

[0065] In this embodiment, the processor is used to process the sound effects of the video. When the first detection result includes that the first data and the second data are both empty, the representation may cause a disconnection or noise, and a continuous silence problem may occur if the duration is longer. In order to determine whether the above problem occurs, the load information corresponding to the processor is obtained to determine whether the above problem occurs by processing the corresponding load information. The load information includes the usage rate.

[0066] S103: Determine a first adjustment parameter of a preset cache amount corresponding to the cache area according to the load information.

[0067] In this embodiment, the preset cache volume represents the amount of data of the first data cached in the cache area. When it is determined according to the load information that the sound will be disconnected or there will be noise, and if the duration is longer, there will be a continuous silence problem, the preset cache volume corresponding to the cache area needs to be adjusted so that when the processor is under high load, there is a surplus of audio data of the video in the cache area. Among them, when it is determined according to the load information that the processor is under high load, the first adjustment parameter of the preset cache volume corresponding to the cache area is determined according to the corresponding relationship between the load information and the adjustment parameter.

[0068] It should be noted that the preset cache amount can be set according to actual needs, and the specific value of the preset cache amount is not limited in this embodiment.

[0069] S104: adjusting the preset buffer size according to the first adjustment parameter, so as to complete the audio and video tracking of the played video by using the adjusted preset buffer size.

[0070] In this embodiment, after obtaining the first adjustment parameter, if the first adjustment parameter is a proportional parameter, the product value between the preset cache amount and the first adjustment parameter can be determined, so as to use the product value as the adjusted preset cache amount; if the first adjustment parameter is an incremental parameter, the sum value between the preset cache amount and the first adjustment parameter can be determined, so as to use the sum value as the adjusted preset cache amount. Among them, the specific form of the first adjustment parameter can be set according to actual needs, and this is not limited in this embodiment. After obtaining the adjusted preset cache amount, the audio data of the video can be cached to the cache area according to the adjusted preset cache amount. In this way, in the process of audio and video tracking, if the first detection result includes that both the first data and the second data are empty and the processor is in a high-load state, it is also possible to leave a surplus of audio data of the video in the cache area to avoid interruptions or noises, and the problem of continuous silence if the duration is longer, thereby improving the user experience.

[0071] The present embodiment provides a method for audio and video tracking. During the audio and video tracking process for a played video, if the audio data of the video cached in the cache area and the data after the audio data is processed with sound effects are both empty, the load information of the sound effect of the processed video is obtained, and the preset cache amount for caching the first data in the cache area is dynamically adjusted according to the obtained load information, so as to leave a surplus of audio data of the video in the cache area, thereby avoiding the problems of disconnection, noise and continuous silence when the audio and video are tracked, and improving the user experience.

[0072] refer to Figure 2 , Figure 2 A flowchart of another audio and video tracking method provided in an embodiment of the present application. The audio and video tracking method provided in this embodiment includes the following steps:

[0073] S201: In the process of performing audio and video tracking on the played video, first data and second data in the buffer area are detected to obtain a first detection result.

[0074] S202: When the first detection result includes that both the first data and the second data are empty, obtain load information corresponding to the processor.

[0075] Regarding the above-mentioned step S201 and step S202, step S201 is consistent with the above-mentioned step S101, and step S202 is consistent with the above-mentioned step S102. For details, please refer to the above-mentioned step S101 and step S102, which will not be described in detail in this embodiment.

[0076] In this embodiment, before executing step S201, the audio and video tracking method provided by this embodiment further includes the following steps:

[0077] When it is determined that the target mode has been turned on, detecting the first data in the buffer area to obtain a second detection result;

[0078] When the second detection result includes that the first data is not empty, clearing the first data in the buffer area;

[0079] After the first data in the buffer area is cleared, the first data and the second data in the buffer area are detected during the audio and video tracking of the played video to obtain a first detection result step (ie, step S201).

[0080] Among them, the target mode is used to track the audio and video of the played video. In the process of turning on the target mode to starting to track the audio and video of the played video, a lot of audio data of the video (i.e., the first data) has been cached in the buffer. If these audio data are directly processed with sound effects, the audio and video synchronization delay time will be longer during the audio and video tracking process, affecting the user experience. Therefore, in the process of turning on the target mode to starting to track the audio and video of the played video, the audio data of the video cached in the buffer is cleared to reduce the delay time of audio and video synchronization during the audio and video tracking process. After the audio data of the video cached in the buffer is cleared, the first data and the second data in the buffer are detected during the audio and video tracking of the played video to obtain the first detection result step (i.e., step S201).

[0081] In this embodiment, after executing step S201, the audio and video tracking method provided by this embodiment further includes the following steps:

[0082] When the first detection result includes that the first data is empty and the second data is not empty, or when the first detection result includes that both the first data and the second data are not empty, determining a target data volume corresponding to the second data;

[0083] Determine a second adjustment parameter corresponding to the preset delay time according to the target data volume;

[0084] The preset delay time is adjusted according to the second adjustment parameter, so as to use the adjusted preset delay time to complete the audio and video synchronization in the process of audio and video tracking of the played video.

[0085] Among them, when the first detection result includes that the first data is empty and the second data is not empty, or when the first detection result includes that both the first data and the second data are not empty, it indicates that there may be a large delay in the audio and video synchronization during the audio and video tracking process due to the presence of more second data in the cache area. Therefore, in order to solve the above problem, when the first detection result includes that the second data is not empty, the target data volume corresponding to the second data in the cache area is determined, so as to determine whether it is necessary to adjust the preset delay time according to the target data volume corresponding to the determined second data. When it is determined that the preset delay time needs to be adjusted, the second adjustment parameter corresponding to the preset delay time is determined according to the target data volume, so as to use the second adjustment parameter to adjust the preset delay time, so as to use the adjusted preset delay time to complete the audio and video synchronization in the process of audio and video tracking of the played video. Specifically, the preset delay time is a time pre-set for achieving audio and video synchronization during the audio and video tracking process. The preset delay time is fixed and is generally a negative value. The preset delay time is usually set according to the amount of second data in the cache area, and can be set according to actual needs. The specific value of the preset delay time is not limited in this embodiment.

[0086] In the above, determining the second adjustment parameter corresponding to the preset delay time according to the target data volume includes:

[0087] Determine a target difference between a target data volume and a preset data volume corresponding to a preset delay time;

[0088] Based on the target difference, the first association relationship is queried to obtain a second adjustment parameter corresponding to the preset delay time.

[0089] In this embodiment, the first association relationship stores a plurality of corresponding relationships between difference values ​​and adjustment parameters, and the adjustment parameters in the first association relationship increase as the difference value increases. After determining the target data volume corresponding to the second data in the cache area, the target data volume can be subtracted from the preset data volume corresponding to the preset delay time to obtain the target difference value, and the first association relationship is queried using the target difference value to obtain the adjustment parameter corresponding to the target difference value, and the adjustment parameter corresponding to the target difference value is determined as the second adjustment parameter corresponding to the preset delay time, and then the preset delay time is adjusted using the second adjustment parameter, so that the audio and video synchronization in the audio and video tracking process is achieved according to the adjusted preset delay time.

[0090] Specifically, the preset data volume corresponding to the preset delay time can be set according to actual needs, and the specific value of the preset data volume is not limited in this embodiment.

[0091] In the above, adjusting the preset delay time according to the second adjustment parameter so as to use the adjusted preset delay time to complete the audio and video synchronization in the audio and video tracking of the played video includes:

[0092] Determine the sum of the second adjustment parameter and the preset delay time to obtain the adjusted preset delay time;

[0093] The adjusted preset delay time is used to complete the audio and video synchronization in the process of audio and video tracking of the played video.

[0094] In this embodiment, the adjustment parameter in the second association relationship is usually a positive value, and the second adjustment parameter is summed with the preset delay time to obtain the adjusted preset delay time. When the target data volume is larger, the second adjustment parameter is larger, and the adjusted preset delay time is larger, thereby reducing the delay time of audio and video synchronization during the audio and video tracking process; when the target data volume is smaller, the second adjustment parameter is smaller, and the adjusted preset delay time is smaller, thereby reducing the delay time of audio and video synchronization during the audio and video tracking process. It should be noted that the use of the preset delay time to complete the audio and video synchronization in the audio and video tracking process of the played video can refer to the existing technology, which will not be repeated in this embodiment.

[0095] S203: Determine a first adjustment parameter of a preset cache amount corresponding to the cache area according to the load information.

[0096] In this embodiment, the load information includes a usage rate. The first adjustment parameter of the preset cache amount corresponding to the cache area is determined according to the load information in step S203, including:

[0097] determining a target comparison result between the utilization rate and a preset utilization rate threshold;

[0098] When the target comparison result is that the usage rate is greater than the preset usage rate threshold, the second association relationship is queried based on the usage rate to obtain a first adjustment parameter of the preset cache amount.

[0099] The preset usage rate threshold represents the lower limit of the usage rate corresponding to the adjustment of the preset cache amount corresponding to the cache area. The preset usage rate threshold can be set according to actual needs. In this embodiment, the specific value of the preset usage rate threshold is not limited. The second association relationship stores multiple sets of corresponding relationships between usage rates and adjustment parameters. The adjustment parameters in the second association relationship increase with the increase of usage rate.

[0100] Specifically, when the target comparison result is that the usage rate is greater than the preset usage rate threshold, it indicates that the processor is in a high-load state. At this time, in order to avoid disconnection, noise and continuous silence during the audio and video tracking process, the preset cache amount corresponding to the cache area needs to be adjusted so that when the processor is in a high-load state, the cache area has a surplus of video sound effect data, thereby solving the problems of disconnection, noise and continuous silence during the audio and video tracking process; when the target comparison result is that the usage rate is less than the preset usage rate threshold, it indicates that the processor is not in a high-load state. At this time, disconnection, noise and continuous silence will not occur during the audio and video tracking process, and there is no need to adjust the preset cache amount corresponding to the cache area.

[0101] More specifically, when the target comparison result is that the usage rate is greater than the preset usage rate threshold, the usage rate can be used to query the preset second association relationship to obtain the adjustment parameter corresponding to the usage rate, and the adjustment parameter corresponding to the usage rate is determined as the first adjustment parameter of the preset cache amount.

[0102] S204: Obtain a first dynamic priority and a first static priority corresponding to the first process, and a second dynamic priority and a second static priority corresponding to the second process.

[0103] S205: Adjust the second dynamic priority and the second static priority according to the first dynamic priority and the first static priority, so that the second total priority is one level lower than the first total priority.

[0104] S206: adjusting the preset cache amount according to the first adjustment parameter to obtain an adjusted preset cache amount.

[0105] S207: Complete the audio and video tracking of the played video by using the adjusted preset cache amount, the adjusted second dynamic priority and the adjusted second static priority.

[0106] For the above-mentioned steps S204 to S207, the second total priority is the sum of the adjusted second dynamic priority and the adjusted second static priority, and the first total priority is the sum of the first dynamic priority and the first static priority. The first process is the process used to process the displayed graphical interface, and the displayed graphical interface is the graphical interface displayed on the TV screen. The second process is the process used to process the audio of the video. Among them, the second process is actually the process for processing the sound effect process of the entire audio and video tracking, that is, the audio and video tracking method provided in this embodiment is executed by the second process. The first process is the SurfaceFlinger process, which is responsible for synthesizing the layers to obtain a graphical interface. The first dynamic priority corresponding to the SurfaceFlinger process is -3, and the first static priority corresponding to the SurfaceFlinger process is 0. The total priority of the SurfaceFlinger process is the sum of the first dynamic priority and the first static priority, which is -3. The second process is the AudioService process (audio service process). The AudioService process is a system service process responsible for sound management. The second dynamic priority corresponding to the AudioService process is 20, and the second static priority corresponding to the AudioService process is -20. The total priority of the AudioService process is the sum of the second dynamic priority and the second static priority, which is 0. Since the smaller the total priority, the higher the processor scheduling priority, when the processor is under high load, the AudioService process with a total priority of 0 will not be able to get the processor time slice, causing the processing time of the AudioService process to exceed the preset processing time (for example, the preset processing time is 5ms, and the processing time of the AudioService process is greater than 10ms). When the processing time of the AudioService process frequently exceeds the preset processing time, it will lead to the inability to process the sound effect data of the video in the buffer area, resulting in interruptions and noise.

[0107] In order to avoid the above problems, this embodiment determines the first adjustment parameter corresponding to the cache area according to the load information, indicating that the processor is in a high-load state. At this time, it is necessary to adjust the second dynamic priority and the second static priority corresponding to the second process according to the first dynamic priority and the first static priority corresponding to the first process, so that the second total priority is one level lower than the first total priority, that is, when the first total priority is -3, the second total priority is -2. By setting the second total priority to be one level lower than the first total priority, the priority of the AudioService process is only lower than the priority of the SurfaceFlinger process, and higher than the priority of other processes. Therefore, in the case of resource shortage, other processes will not seize resources, thereby avoiding the problem that the AudioService process cannot get the processor time slice, and further avoiding the problem of disconnection and noise when the processor is in a high-load state.

[0108] In the above, in step S205, the second dynamic priority and the second static priority are adjusted according to the first dynamic priority and the first static priority, so that the second total priority is one level lower than the first total priority, including:

[0109] The second dynamic priority and the second static priority are adjusted according to the first dynamic priority and the first static priority, so that the adjusted second dynamic priority is different from the first dynamic priority and the second total priority is one level lower than the first total priority.

[0110] Among them, when the adjusted second dynamic priority is the same as the first dynamic priority, if the processor is in a high-load state, during the audio and video tracking process, when the user is operating the graphical interface, since the adjusted second dynamic priority is the same as the first dynamic priority, it will not give priority to responding to the user's operation of the graphical interface, resulting in a freeze problem when the user operates the graphical interface, affecting the user's experience. Therefore, in order to ensure that the adjusted second dynamic priority does not affect the refresh of the graphical interface by the SurfaceFlinger process to improve the user's experience, when adjusting the second dynamic priority, the adjusted second dynamic priority needs to be set differently from the first dynamic priority. For example, when the first dynamic priority is -3, the adjusted second dynamic priority is -2. Since the first static priority is 0, the adjusted second static priority is 0, so that the second total priority is one level lower than the first total priority.

[0111] Specifically, after completing the adjustment of the second dynamic priority and the second static priority corresponding to the second process, the preset cache amount is adjusted using the first adjustment parameter to obtain the adjusted preset cache amount, thereby utilizing the adjusted preset cache amount, the adjusted second dynamic priority and the adjusted second static priority to complete the audio and video tracking of the played video, thereby avoiding noise, interruptions and continuous silence problems that occur during the audio and video tracking process, and improving the user experience.

[0112] The present embodiment provides a method for audio and video tracking. During the audio and video tracking process for a played video, if the audio data of the video cached in the cache area and the data after the audio data is processed with sound effects are both empty, the load information of the sound effect of the processed video is obtained, and the preset cache amount for caching the first data in the cache area is dynamically adjusted according to the obtained load information, so as to leave a surplus of audio data of the video in the cache area, thereby avoiding the problems of disconnection, noise and continuous silence when the audio and video are tracked, and improving the user experience.

[0113] As an example, refer to Figure 3 , let me introduce the whole process of audio and video tracking in detail, as follows:

[0114] Before determining that the audio and video tracking function for the played video has been turned on and the audio and video tracking for the played video has not been performed, clearing the audio data cached in the buffer area, so that the audio and video tracking for the played video is performed after the clearing of the audio data cached in the buffer area is completed;

[0115] In the process of audio and video tracking of the played video, the audio data cached in the buffer area and the data after the audio data is processed by sound effects are detected;

[0116] When the audio data in the buffer area and the data after the audio data is processed by sound effects are both empty, obtaining the load information corresponding to the processor;

[0117] When it is determined according to the load information that the processor is in a high load state, determining a first adjustment parameter of a preset cache amount corresponding to the cache area according to the load information;

[0118] Obtaining a first dynamic priority and a first static priority corresponding to a first process used for processing the displayed graphical interface and a second dynamic priority and a second static priority corresponding to a second process used for processing the sound effects of the video;

[0119] Adjusting the second dynamic priority and the second static priority according to the first dynamic priority and the first static priority, so that the adjusted second dynamic priority is different from the first dynamic priority and the second total priority is one level lower than the first total priority;

[0120] Tracking the audio and video of the played video using the adjusted preset cache amount, the adjusted second dynamic priority, and the adjusted second static priority;

[0121] When both the audio data in the buffer and the data after the audio data is processed with sound effects are not empty, or when the audio data in the buffer is empty and the data after the audio data is processed with sound effects is not empty, determining a target data volume of the data after the audio data is processed with sound effects in the buffer;

[0122] Determine a second adjustment parameter corresponding to the preset delay time according to the target data volume;

[0123] The preset delay time is adjusted according to the second adjustment parameter, so as to use the adjusted preset delay time to complete the audio and video synchronization in the process of audio and video tracking of the played video.

[0124] refer to Figure 5 , Figure 5 A structural diagram of a sound and picture tracking device provided in an embodiment of the present application. A sound and picture tracking device provided in an embodiment of the present application includes: a detection module 501, an acquisition module 502, a determination module 503 and a tracking module 504. Among them, the detection module 501 is used to detect the first data and the second data in the buffer area during the sound and picture tracking of the played video to obtain a first detection result, the first data is the audio data of the cached video, and the second data is the data after the sound effect processing of the first data; the acquisition module 502 is used to obtain the load information corresponding to the processor when the first detection result includes that the first data and the second data are both empty, and the processor is used to process the sound effect of the video; the determination module 503 is used to determine the first adjustment parameter of the preset cache amount corresponding to the cache area according to the load information, and the preset cache amount represents the amount of data of the cache area to cache the first data; the tracking module 504 is used to adjust the preset cache amount according to the first adjustment parameter, so as to complete the sound and picture tracking of the played video using the adjusted preset cache amount.

[0125] In this embodiment, the acquisition module 502 is further used for:

[0126] Obtain a first dynamic priority and a first static priority corresponding to a first process, and a second dynamic priority and a second static priority corresponding to a second process, wherein the first process is a process used to process the displayed graphical interface, and the second process is a process used to process the sound effects of the video.

[0127] The audio and video tracking device provided in this embodiment further includes an adjustment module, which is used to:

[0128] The second dynamic priority and the second static priority are adjusted according to the first dynamic priority and the first static priority so that the second total priority is one level lower than the first total priority, the second total priority is the sum of the adjusted second dynamic priority and the adjusted second static priority, and the first total priority is the sum of the first dynamic priority and the first static priority.

[0129] In this embodiment, the tracking module 504 is further used for:

[0130] According to the first adjustment parameter, the preset cache amount is adjusted to obtain the adjusted preset cache amount;

[0131] The audio and video tracking of the played video is completed using the adjusted preset cache amount, the adjusted second dynamic priority and the adjusted second static priority.

[0132] In this embodiment, the adjustment module is also used for:

[0133] The second dynamic priority and the second static priority are adjusted according to the first dynamic priority and the first static priority, so that the adjusted second dynamic priority is different from the first dynamic priority and the second total priority is one level lower than the first total priority.

[0134] In this embodiment, the detection module 501 is further used for:

[0135] When it is determined that the target mode has been turned on, detecting the first data in the buffer area to obtain a second detection result, wherein the target mode is used to track the audio and video of the played video;

[0136] When the second detection result includes that the first data is not empty, clearing the first data in the buffer area;

[0137] After the first data in the buffer area is cleared, during the audio and video tracking of the played video, the first data and the second data in the buffer area are detected to obtain a first detection result.

[0138] In this embodiment, the determination module 503 is further configured to:

[0139] When the first detection result includes that the first data is empty and the second data is not empty, or when the first detection result includes that both the first data and the second data are not empty, determining a target data volume corresponding to the second data;

[0140] According to the target data volume, a second adjustment parameter corresponding to a preset delay time is determined, and the preset delay time is used to achieve audio and video synchronization during audio and video tracking of the played video.

[0141] In this embodiment, the adjustment module is also used for:

[0142] The preset delay time is adjusted according to the second adjustment parameter, so as to use the adjusted preset delay time to complete the audio and video synchronization in the process of audio and video tracking of the played video.

[0143] In this embodiment, the determination module 503 is further configured to:

[0144] Determine a target difference between the target data volume and a preset data volume corresponding to the preset delay time;

[0145] Based on the target difference, a first association relationship is queried to obtain a second adjustment parameter corresponding to the preset delay time, wherein the first association relationship stores a plurality of corresponding relationships between difference values ​​and adjustment parameters, and the adjustment parameter in the first association relationship increases as the difference value increases.

[0146] In this embodiment, the adjustment module is also used for:

[0147] Determine the sum of the second adjustment parameter and the preset delay time to obtain the adjusted preset delay time;

[0148] The adjusted preset delay time is used to complete the audio and video synchronization during the audio and video tracking process of the played video.

[0149] In this embodiment, the load information includes a usage rate; the determination module 503 is further used to:

[0150] Determining a target comparison result between the usage rate and a preset usage rate threshold, wherein the preset usage rate threshold represents a lower limit value of the usage rate corresponding to adjusting the preset cache amount corresponding to the cache area;

[0151] When the target comparison result is that the usage rate is greater than the preset usage rate threshold, the second association relationship is queried based on the usage rate to obtain the first adjustment parameter of the preset cache amount, and the second association relationship stores multiple sets of corresponding relationships between usage rates and adjustment parameters, and the adjustment parameters in the second association relationship increase with the increase of usage rate.

[0152] The present embodiment provides an audio and video tracking device. During the audio and video tracking process for the played video, if the audio data of the video cached in the cache area and the data after the audio data is processed with sound effects are both empty, the load information of the sound effect of the processed video is obtained, and the preset cache amount for caching the first data in the cache area is dynamically adjusted according to the obtained load information, so as to leave a surplus of the audio data of the video in the cache area, thereby avoiding the problems of disconnection, noise and continuous silence when the audio and video tracking is performed on the video, and improving the user experience.

[0153] Figure 6 A schematic diagram of the structure of a television provided in an embodiment of the present application, Figure 6 The television 600 shown includes: at least one processor 601, a memory 602, at least one network interface 604 and other user interfaces 603. The various components in the television 600 are coupled together via a bus system 605. It is understood that the bus system 605 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 605 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 605 is not shown in FIG. Figure 6 Various buses are labeled as bus system 605 .

[0154] The user interface 603 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touch pad, or a touch screen).

[0155] It can be understood that the memory 602 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 602 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0156] In some implementations, the memory 602 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system 6021 and an application 6022 .

[0157] The operating system 6021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 6022 includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 6022.

[0158] In the embodiment of the present invention, by calling the program or instruction stored in the memory 602, specifically, the program or instruction stored in the application 6022, the processor 601 is used to execute the method steps provided by each method embodiment.

[0159] The method disclosed in the above embodiment of the present invention can be applied to the processor 601, or implemented by the processor 601. The processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 601 or the instruction in the form of software. The above processor 601 can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or the hardware and software units in the decoding processor can be executed. The software unit can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 602, and the processor 601 reads the information in the memory 602 and completes the steps of the above method in combination with its hardware.

[0160] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPDevice, DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0161] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0162] The television provided in this embodiment is as follows Figure 6 The television shown in FIG. 10 may perform the following operations: Figure 1 to Figure 3 All steps of the sound-picture tracking method are implemented Figure 1 to Figure 3 For details, please refer to Figure 1 to Figure 3 For the sake of brevity, the relevant description is not repeated here.

[0163] The embodiment of the present invention also provides a storage medium (computer-readable storage medium). The storage medium here stores one or more programs. The storage medium may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk or a solid-state drive; the memory may also include a combination of the above-mentioned types of memory.

[0164] When one or more programs in the storage medium can be executed by one or more processors, the audio and video tracking method executed on the audio and video tracking device side can be implemented.

[0165] The processor is used to execute the audio and video tracking program stored in the memory to implement the following steps of the audio and video tracking method executed on the audio and video tracking device side.

[0166] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0167] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0168] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for tracking sound and picture, characterized in that: include: In the process of performing audio and video tracking on the played video, first data and second data in the buffer area are detected to obtain a first detection result, where the first data is the audio data of the buffered video, and the second data is the data after the first data is processed with sound effects; When the first detection result includes that both the first data and the second data are empty, acquiring load information corresponding to a processor, the processor being used to process the sound effect of the video; Determine, according to the load information, a first adjustment parameter of a preset cache amount corresponding to the cache area, wherein the preset cache amount represents a data amount of the first data cached in the cache area; The preset buffer size is adjusted according to the first adjustment parameter, so as to complete the audio and video tracking of the played video using the adjusted preset buffer size.

2. The method according to claim 1, characterized in that After executing the step of determining, according to the load information, a first adjustment parameter corresponding to a preset cache amount of the cache area, the method further includes: Obtaining a first dynamic priority and a first static priority corresponding to a first process, and a second dynamic priority and a second static priority corresponding to a second process, wherein the first process is a process used for processing the displayed graphical interface, and the second process is a process used for processing the sound effects of the video; The second dynamic priority and the second static priority are adjusted according to the first dynamic priority and the first static priority, so that the second total priority is one level lower than the first total priority, the second total priority is the sum of the adjusted second dynamic priority and the adjusted second static priority, and the first total priority is the sum of the first dynamic priority and the first static priority; The step of adjusting the preset buffer size according to the first adjustment parameter so as to complete the audio and video tracking of the played video by using the adjusted preset buffer size includes: According to the first adjustment parameter, the preset cache amount is adjusted to obtain the adjusted preset cache amount; The audio and video tracking of the played video is completed using the adjusted preset cache amount, the adjusted second dynamic priority and the adjusted second static priority.

3. The method according to claim 2, characterized in that The step of adjusting the second dynamic priority and the second static priority according to the first dynamic priority and the first static priority so that the second total priority is one level lower than the first total priority includes: The second dynamic priority and the second static priority are adjusted according to the first dynamic priority and the first static priority, so that the adjusted second dynamic priority is different from the first dynamic priority and the second total priority is one level lower than the first total priority.

4. The method according to claim 1, characterized in that: Before executing the step of detecting the first data and the second data in the buffer area to obtain a first detection result during the audio and video tracking of the played video, the method further includes: When it is determined that the target mode has been turned on, detecting the first data in the buffer area to obtain a second detection result, wherein the target mode is used to track the audio and video of the played video; When the second detection result includes that the first data is not empty, clearing the first data in the buffer area; After the first data in the buffer area is cleared, the step of detecting the first data and the second data in the buffer area during the audio and video tracking of the played video is performed to obtain a first detection result.

5. The method according to claim 1, characterized in that After executing the step of detecting the first data and the second data in the buffer area to obtain a first detection result during the audio and video tracking of the played video, the method further includes: When the first detection result includes that the first data is empty and the second data is not empty, or when the first detection result includes that both the first data and the second data are not empty, determining a target data volume corresponding to the second data; Determining, according to the target data volume, a second adjustment parameter corresponding to a preset delay time, wherein the preset delay time is used to achieve audio and video synchronization during audio and video tracking of the played video; The preset delay time is adjusted according to the second adjustment parameter, so as to use the adjusted preset delay time to complete the audio and video synchronization in the process of audio and video tracking of the played video.

6. The method according to claim 5, characterized in that The determining, according to the target data volume, a second adjustment parameter corresponding to the preset delay time includes: Determine a target difference between the target data volume and a preset data volume corresponding to the preset delay time; Based on the target difference, query the first association relationship to obtain a second adjustment parameter corresponding to the preset delay time, wherein the first association relationship stores a plurality of corresponding relationships between the difference values ​​and the adjustment parameters, and the adjustment parameter in the first association relationship increases as the difference value increases; The step of adjusting the preset delay time according to the second adjustment parameter so as to use the adjusted preset delay time to complete the audio and video synchronization in the process of audio and video tracking of the played video includes: Determine the sum of the second adjustment parameter and the preset delay time to obtain the adjusted preset delay time; The adjusted preset delay time is used to complete the audio and video synchronization during the audio and video tracking process of the played video.

7. The method according to claim 1, characterized in that The load information includes a usage rate; The step of determining, according to the load information, a first adjustment parameter of a preset cache amount corresponding to the cache area includes: Determining a target comparison result between the usage rate and a preset usage rate threshold, wherein the preset usage rate threshold represents a lower limit value of the usage rate corresponding to adjusting the preset cache amount corresponding to the cache area; When the target comparison result is that the usage rate is greater than the preset usage rate threshold, the second association relationship is queried based on the usage rate to obtain the first adjustment parameter of the preset cache amount, and the second association relationship stores multiple sets of corresponding relationships between usage rates and adjustment parameters, and the adjustment parameters in the second association relationship increase with the increase of usage rate.

8. A sound and picture tracking device, characterized in that: include: a detection module, configured to detect first data and second data in a buffer area during audio and video tracking of a played video to obtain a first detection result, wherein the first data is the buffered audio data of the video, and the second data is data after sound effects processing is performed on the first data; an acquisition module, configured to acquire load information corresponding to a processor when the first detection result includes that both the first data and the second data are empty, and the processor is configured to process the sound effects of the video; a determination module, configured to determine, according to the load information, a first adjustment parameter of a preset cache amount corresponding to the cache area, wherein the preset cache amount represents a data amount of the first data cached in the cache area; The tracking module is used to adjust the preset cache amount according to the first adjustment parameter, so as to complete the audio and video tracking of the played video by using the adjusted preset cache amount.

9. A television, characterized in that: include: A processor and a memory, wherein the processor is used to execute the audio and video tracking program stored in the memory to implement the audio and video tracking method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the audio and video tracking method according to any one of claims 1 to 7.

Images