Data processing method, data processing device and electronic equipment

Abstract

The invention provides a data processing method, a data processing device and electronic equipment, and relates to the technical field of data encoding and decoding. The method comprises the following steps: when a hardware decoding module decodes to-be-decoded data, if it is detected that the hardware decoding module is abnormal when decoding the to-be-decoded data, obtaining corresponding current state information when the hardware decoding module is abnormal; determining a target decoding module in the hardware decoding module and the software decoding module through the obtained current state information when the hardware decoding module is abnormal; and decoding the data to be decoded through the target decoding module to obtain decoded data. On the basis of the scheme, when the hardware decoding module is abnormal, normal decoding can be performed on the to-be-decoded data while operation resources are saved.

Description

Technical Field

[0001] The present application relates to the field of data coding and decoding technology, and in particular to a data processing method, a data processing device and an electronic device in the field of data coding and decoding technology. Background Art

[0002] In order to facilitate data transmission, the original data can be encoded to obtain encoded data that occupies less memory. When the original data is needed, the encoded data can be decoded by the hardware decoding module to obtain the original data.

[0003] If the hardware decoding module encounters an exception while decoding the encoded data, it may cause decoding failure, affecting normal decoding of the encoded data. In this case, if you want to decode the original data normally, you can use the software decoding module to decode the encoded data. However, using the software decoding module to decode the encoded data may occupy a large number of operating resources, causing device operation to lag.

[0004] Therefore, when an abnormality occurs in the hardware decoding module, how to decode the data while saving operating resources is an urgent problem that needs to be solved. Summary of the Invention

[0005] The present application provides a data processing method, a data processing device, and an electronic device. The method can save operating resources while normally decoding the data to be decoded when an abnormality occurs in a hardware decoding module.

[0006] In a first aspect, the present application provides a data processing method, which is applied to an electronic device, wherein the electronic device includes a hardware decoding module and a software decoding module, and the method includes:

[0007] If it is detected that an exception occurs when the hardware decoding module decodes the data to be decoded, the current status information of the hardware decoding module is obtained;

[0008] Determine a target decoding module among the hardware decoding module and the software decoding module based on the current state information;

[0009] The data to be decoded is decoded based on the target decoding module to obtain decoded data.

[0010] In an embodiment of the present application, upon detecting that an abnormality occurs when the hardware decoding module decodes the data to be decoded, the current state information of the hardware decoding module can be first obtained, and the target decoding module capable of decoding the data to be decoded can be determined from the hardware decoding module and the software decoding module based on the current state information of the hardware decoding module, so that the data to be decoded can be decoded by the target decoding module to obtain the decoded data. Since the software decoding module requires a large amount of operating resources compared to the hardware decoding module when decoding the data to be decoded, it affects the operating performance of the electronic device. Therefore, in order to save the operating resources of the electronic device, upon detecting that an abnormality occurs when the hardware decoding module decodes the data to be decoded, the decoding module is not directly switched from the hardware decoding module to the software decoding module. Instead, the target decoding module capable of decoding the data to be decoded can be determined from the hardware decoding module and the software decoding module based on the current state information of the hardware decoding module, and the hardware decoding module can be used to decode the data to be decoded as much as possible, thereby achieving normal decoding of the data to be decoded while saving the operating resources of the electronic device.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes:

[0012] Get the corresponding exception type when an exception occurs in the hardware decoding module;

[0013] The above-mentioned determination of the target decoding module in the hardware decoding module and the software decoding module based on the current state information includes:

[0014] Based on the exception type and the current state information, a target decoding module among the hardware decoding module and the software decoding module is determined.

[0015] In this embodiment of the present application, while determining the target decoding module based on the current state information of the hardware decoding module, an exception type corresponding to an exception in the hardware decoding module is introduced. The exception type corresponding to the hardware decoding module exception and the current state information of the hardware decoding module are used together to determine the target decoding module in both the hardware decoding module and the software decoding module. This allows for more accurate determination of the target decoding module based on multi-dimensional data. Furthermore, with this more accurate target decoding module, normal decoding of the data to be decoded is achieved while conserving the operating resources of the electronic device.

[0016] In combination with the first aspect and the above implementations, in certain implementations of the first aspect, determining the target decoding module in the hardware decoding module and the software decoding module based on the exception type and the current state information includes:

[0017] If the exception type is the first exception type and the current state information indicates that the decoding function of the hardware decoding module is normal, the hardware decoding module is determined as the target decoding module;

[0018] If the exception type is the second exception type, and / or the current state indicates that the decoding function of the hardware decoding module is abnormal, the software decoding module is determined as the target decoding module;

[0019] The first abnormality type indicates a type in which the abnormality frequency is less than a preset frequency; the second abnormality type indicates a type in which the abnormality frequency is greater than or equal to a preset frequency.

[0020] In an embodiment of the present application, the target decoding module is determined in the hardware decoding module and the software decoding module by combining the corresponding exception type when an exception occurs in the hardware decoding module and the current state information of the hardware decoding module, so that the target decoding module determined by the multi-dimensional data can be more accurate.

[0021] In combination with the first aspect and the above implementations, in some implementations of the first aspect, the method further includes:

[0022] Determine a target timestamp of the first to-be-decoded data corresponding to when an exception occurs in the hardware decoding module;

[0023] Determining second data to be decoded in the data to be decoded based on the target timestamp and the first data to be decoded, wherein the timestamp of the second data to be decoded is after the target timestamp;

[0024] Decoding the first data to be decoded by a software decoding module to obtain first decoded data;

[0025] The target decoding module decodes the data to be decoded to obtain the decoded data, including:

[0026] Decoding the second data to be decoded based on the target decoding module to obtain second decoded data;

[0027] Decoded data is obtained based on the first decoded data and the second decoded data.

[0028] In an embodiment of the present application, in order to avoid abnormalities in the decoded data decoded by the hardware decoding module when an abnormality occurs, a software decoding module can be used to decode the decoded data corresponding to the hardware decoding module when an abnormality occurs, and the decoded data decoded by the software decoding module and the decoded data decoded by the target decoding module after the hardware decoding module abnormality are used to obtain the decoded data, thereby ensuring the normality of the decoded data.

[0029] In combination with the first aspect and the above implementation manner, in some implementation manners of the first aspect, when the target decoding module is the hardware decoding module, the method further includes:

[0030] Obtaining third decoded data, wherein the third decoded data represents decoded data corresponding to data whose timestamp is before the target timestamp in the data to be decoded;

[0031] The decoded data obtained based on the first decoded data and the second decoded data includes:

[0032] Decoding the second data to be decoded by the hardware decoding module to obtain second decoded data;

[0033] Based on the timestamp of the first decoded data, the timestamp of the second decoded data, and the timestamp of the third decoded data, the first decoded data, the second decoded data, and the third decoded data are combined to obtain decoded data.

[0034] In an embodiment of the present application, based on the decoded data (i.e., the first decoded data) corresponding to the hardware decoding module when an exception occurs, and the decoded data (i.e., the second decoded data) decoded by the target decoding module after the hardware decoding module exception occurs, the decoded data decoded by the hardware decoding module before its own exception occurs (i.e., the third decoded data) is also combined. The data of the three time periods (i.e., the first decoded data, the second decoded data, and the third decoded data) together constitute the decoded data corresponding to the data to be decoded, thereby ensuring the integrity of the decoded data and improving the user's audio-visual experience.

[0035] Furthermore, by combining the timestamps corresponding to the data of the three time periods, decoded data with the correct temporal sequence can be obtained, thereby improving the accuracy of the decoded data and thus enhancing the user's audio-visual experience.

[0036] In combination with the first aspect and the above implementation manner, in some implementation manners of the first aspect, the decoded data is video data, and the video data includes image data, and the method further includes:

[0037] Determine whether audio data exists in the video data;

[0038] If audio data exists in the video data, adjusting the playback speed of the image data based on the audio data so as to synchronize the image data with the audio data;

[0039] If there is no audio data in the video data, obtain the current system time and the playback frame rate of the image data; based on the current system time and the playback frame rate of the image data, adjust the playback speed of the image data so that the playback frame rate of the image data is the same in the same duration.

[0040] In an embodiment of the present application, when there is audio data in the video data, the playback speed of the image data in the video data can be adjusted based on the audio data, so that the image data and the audio data are synchronized, thereby achieving synchronous playback of the image data and the audio data, and improving the user's audio-visual experience.

[0041] Alternatively, when there is no audio data in the video data, the playback speed of the image data in the video data can be adjusted jointly by the current system time and the playback frame rate of the image data, so that the playback frame rate of the image data in the same duration is the same, thereby reducing the frame rate mutation caused by the different playback frame rates of the image data in the same duration, and the problems of image data playback buffering and / or stuttering, thereby improving the stability and compatibility of image data playback, allowing users to enjoy a coherent and natural visual experience, and improving the user's visual experience.

[0042] In combination with the first aspect and the above implementations, in certain implementations of the first aspect, adjusting the playback speed of image data in video data based on audio data to synchronize the image data with the audio data includes:

[0043] Obtaining a first playback time and a second playback time, wherein the first playback time indicates the playback time of the image data, the second playback time indicates the playback time of the audio data, and the timestamp of the image data is the same as the timestamp of the audio data;

[0044] If the first playback time is earlier than the second playback time, based on a first difference between the first playback time and the second playback time, reducing the playback speed of the image data so as to synchronize the image data with the audio data;

[0045] If the first play time is later than the second play time, the play speed of the image data is increased based on a second difference between the second play time and the first play time to synchronize the image data with the audio data.

[0046] In an embodiment of the present application, when the playback time of image data in the video data is earlier than the playback time of audio data, the playback speed of the image data can be reduced so that the playback time of the image data is the same as the playback time of the audio data; or, when the playback time of image data in the video data is later than the playback time of audio data, the playback speed of the image data can be increased so that the playback time of the image data is the same as the playback time of the audio data; thereby adjusting the asynchronous picture and audio to synchronized picture and audio, thereby improving the user's audio-visual experience.

[0047] In combination with the first aspect and the above implementations, in certain implementations of the first aspect, adjusting the playback speed of the image data based on the current system time and the playback frame rate of the image data so that the playback frame rate of the image data is the same for the same duration includes:

[0048] Determine the target playback time corresponding to each frame of image data based on the current system time and the playback frame rate of the image data;

[0049] If the actual playback time of each frame of image data is earlier than the target playback time, based on a third difference between the actual playback time and the target playback time, reducing the playback speed of the image data so that the playback frame rates of the image data in the same time interval are the same;

[0050] If the actual playback time of each frame of image data is later than the target playback time, the playback speed of the image data is increased based on the fourth difference between the target playback time and the actual playback time so that the playback frame rate of the image data in the same duration is the same.

[0051] In an embodiment of the present application, when the playback time of image data in video data is earlier or later than the set playback time (i.e., the target playback time), it may cause the image data to have different playback frame rates in the same duration, resulting in a sudden change in frame rate, and problems of image data playback buffering and / or stuttering. Therefore, when the playback time of image data in video data is earlier than the target playback time, the playback speed of the image data is reduced, or when the playback time of image data in video data is later than the target playback time, the playback speed of the image data is increased, so that the playback frame rate of the image data in the same duration is the same, thereby improving the stability and compatibility of image data playback, allowing users to enjoy a coherent and natural visual experience, and improving the user's visual experience.

[0052] In a second aspect, the present application provides a data processing device configured in an electronic device, wherein the electronic device includes a hardware decoding module and a software decoding module, and the device includes:

[0053] an acquisition module, configured to acquire current status information of the hardware decoding module if an exception is detected when the hardware decoding module decodes the data to be decoded;

[0054] A processing module, configured to determine a target decoding module among the hardware decoding module and the software decoding module based on current state information;

[0055] The decoding module is used to decode the data to be decoded based on the target decoding module to obtain decoded data.

[0056] In a third aspect, the present application provides an electronic device comprising a memory and a processor. The memory is configured to store executable program code, and the processor is configured to retrieve and execute the executable program code from the memory, so that the electronic device executes the method of the first aspect or any possible implementation of the first aspect.

[0057] In a fourth aspect, the present application provides a computer program product, which includes: computer program code, which, when running on a computer, enables the computer to execute the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0058] In a fifth aspect, the present application provides a computer-readable storage medium, which stores a computer program code. When the computer program code runs on a computer, the computer executes the method in the above-mentioned first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a schematic diagram of a data decoding scenario provided in an embodiment of the present application.

[0060] Figure 2 It is a flowchart of a data processing method provided in an embodiment of the present application.

[0061] Figure 3 This is another flow chart of a data processing method provided in an embodiment of the present application.

[0062] Figure 4 This is another flow chart of a data processing method provided in an embodiment of the present application.

[0063] Figure 5 It is an algorithm diagram of a data processing method provided in an embodiment of the present application.

[0064] Figure 6 It is a structural diagram of the data processing device provided in an embodiment of the present application.

[0065] Figure 7 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The following will clearly and thoroughly describe the technical solutions in this application in conjunction with the accompanying drawings. In the description of the embodiments of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more than two.

[0067] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0068] Figure 1 This is a schematic diagram of a data decoding scenario provided in an embodiment of the present application.

[0069] For example, Figure 1 As shown, Figure 1 The electronic device 110 includes a hardware decoding module 120 and a software decoding module 130. The electronic device 110 includes a hardware decoding module 120 and a software decoding module 130.

[0070] Hardware decoding module 120: This is a physical module in an electronic device and may include a central processing unit (CPU), a video processing unit (VPU), a graphics processing unit (GPU), etc. The CPU, VPU, and GPU work together to decode data to be decoded (referred to as "to-be-decoded data").

[0071] The software decoding module 130 is a virtual module in the electronic device, which can run on the CPU through a software algorithm to complete the decoding of the data to be decoded.

[0072] Although the software decoding module 130 (also referred to as the "software codec module") is more flexible than the hardware decoding module 120 (also referred to as the "hardware codec module") in decoding the data to be decoded, and can support new codec standards or algorithms through software updates, the software decoding module 130 generally occupies more CPU resources when decoding the data to be decoded, which may cause the performance of the operating system in the electronic device 110 to degrade, resulting in problems such as slow operation and freezing of the operating system. Since the hardware decoding module 120 has advantages over the software decoding module 130, such as faster decoding speed, less CPU resource occupation, low decoding power consumption, and stable operation, the hardware decoding module 120 is generally preferred for decoding the data to be decoded.

[0073] Optionally, when decoding the data to be decoded, a single-core CPU may discard part of the decoded data to reduce CPU resource usage due to its own low processing power and excessive CPU resource usage corresponding to the data to be decoded, resulting in problems such as data output being stuck or stuck. In order to avoid discarding the decoded data while reducing CPU resource usage, a multi-core CPU can be used to decode the data to be decoded, and the decoding work of the data to be decoded can be distributed to multiple CPU cores and threads for parallel processing (also known as "multi-core multi-threading"), thereby reducing CPU resource usage, avoiding the problem of discarding part of the decoded data when the CPU resources are too occupied, and ensuring the integrity of the decoded data.

[0074] It should be noted that the electronic device may be an intelligent device configured with a hardware decoding module and a software decoding module, as well as a decoding plug-in that controls the hardware decoding module and the software decoding module, including but not limited to: a personal computer, a tablet computer, a handheld device, an in-vehicle device, a wearable device, a computing device, or other processing device connected to a wireless modem, etc. Electronic devices may be called different names in different networks, such as: user equipment, access electronic device, subscriber unit, subscriber station, mobile station, mobile station, remote station, remote electronic device, mobile device, user electronic device, electronic device, wireless communication device, user agent or user device, cellular phone, cordless phone, electronic device in 5G network or future evolution network, etc., and the embodiments of the present application are not limited thereto.

[0075] Exemplarily, if there is data to be decoded in the electronic device 110, the data to be decoded can be decoded by the hardware decoding module 120 and / or the software decoding module 130 to obtain decoded data corresponding to the data to be decoded. The data to be decoded may include one or more of video data to be decoded, audio data to be decoded, text data to be decoded, and image data to be decoded. Furthermore, the data to be decoded may represent one or more types of data such as a key frame (Intra-coded Frame, I), a predicted frame (Predictive Frame, P), and a bidirectionally predicted frame (Bidirectionally Predictive Frame, B).

[0076] Optionally, in a vehicle, in order to meet the diverse needs of users, a multimedia device (i.e., an electronic device) with various functions can be configured in the vehicle cabin to provide users with a better audio-visual experience through the multimedia device. When playing multimedia data (e.g., video data to be decoded), the video data to be decoded can be decoded by a hardware decoding module configured in the multimedia device to obtain decoded video data, and the decoded video data can be played through the multimedia device. Among them, the multimedia device can include one or more of an in-vehicle infotainment system (IVI), a central control screen, a speaker, etc.

[0077] However, when decoding the video data to be decoded using the hardware decoding module, the hardware decoding module may fail to decode the video data to be decoded due to factors such as unsupported format of the video data to be decoded, poor hardware decoding compatibility, or hardware decoding module failure. This may cause abnormal video data playback, resulting in green screen, freezes, and other problems, reducing the user's audio-visual experience. In this case, if the data to be decoded is to be decoded normally, the software decoding module can be used to decode the data to be decoded. However, when the software decoding module is used to decode the data to be decoded, it may occupy a large amount of operating resources, causing the device to operate lag.

[0078] Therefore, in order to solve the problem that the hardware decoding module fails to decode the data to be decoded, the present application proposes a data processing method, a data processing device and an electronic device.

[0079] The following combination Figures 2 to 5 The data processing method provided in the embodiments of the present application is described in detail.

[0080] Figure 2 This is a flow chart of a data processing method provided by an embodiment of the present application. Figure 1 The electronic device 110 in the embodiment of the present invention executes the above-mentioned operation, or the decoding plug-in in the electronic device 110 executes the above-mentioned operation.

[0081] For example, Figure 2 As shown, the method 200 includes the following implementation process:

[0082] S210 : If it is detected that an exception occurs when the hardware decoding module decodes the data to be decoded, obtain current state information of the hardware decoding module.

[0083] For example, when decoding the data to be decoded (for example, the video data to be decoded) through the hardware decoding module, the real-time status of the decoding of the hardware decoding module can be monitored. If it is detected that an abnormality occurs when the hardware decoding module decodes the data to be decoded, the real-time status information of the hardware decoding module (which may be called "current status information") can be obtained.

[0084] S220 , determining a target decoding module among the hardware decoding module and the software decoding module based on the current state information.

[0085] For example, in order to decode the data to be decoded normally, when the current status information of the hardware decoding module is obtained, the current status information of the hardware decoding module can be used to determine the decoding module (which can be called the "target decoding module") in the hardware decoding module and the software decoding module that can be used to decode the data to be decoded.

[0086] It should be understood that what is determined in the hardware decoding module and the software decoding module is a target decoding module that can be used to decode the data to be decoded, that is, the decoding function of the target decoding module is normal (that is, not abnormal), thereby ensuring that the data to be decoded can be decoded normally through the target decoding module.

[0087] For example, if the target decoding module is a hardware decoding module, the decoding function of the hardware decoding module is normal, and it may be that other non-decoding functions are abnormal, which does not affect the normal decoding of the data to be decoded by the hardware decoding module. Alternatively, if the target decoding module is a software decoding module, the decoding function of the software decoding module is normal. It can be understood that if the hardware decoding module abnormality does not affect the decoding function, the hardware decoding module will continue to be used for decoding. If the hardware decoding module abnormality does affect the decoding function, the software decoding module will be switched to perform decoding.

[0088] In a possible implementation, in order to determine the target decoding module among the hardware decoding module and the software decoding module, the exception type when the hardware decoding module is abnormal may also be obtained.

[0089] Furthermore, when the exception type of the hardware decoding module is obtained, the target decoding module can be determined in the hardware decoding module and the software decoding module based on the current state information of the hardware decoding module and the exception type of the hardware decoding module.

[0090] The exception type includes a first exception type and a second exception type. The first exception type may indicate that the hardware decoding module has an exception frequency less than a preset frequency, i.e., the first exception type has a low frequency of exceptions. The second exception type may indicate that the hardware decoding module has an exception frequency greater than or equal to a preset frequency, i.e., the second exception type has a high frequency of exceptions.

[0091] It should be understood that the preset frequency can be 10 times per second, 1 time per second, 3 times per 2 seconds, etc., and should be set according to the requirements of data decoding. The embodiments of the present application do not limit this.

[0092] In this embodiment of the present application, while determining the target decoding module based on the current state information of the hardware decoding module, an exception type corresponding to an exception in the hardware decoding module is introduced. The exception type corresponding to the hardware decoding module exception and the current state information of the hardware decoding module are used together to determine the target decoding module in both the hardware decoding module and the software decoding module. This allows for more accurate determination of the target decoding module based on multi-dimensional data. Furthermore, with this more accurate target decoding module, normal decoding of the data to be decoded is achieved while conserving the operating resources of the electronic device.

[0093] Optionally, when the hardware decoding module is abnormal, the exception type is the first exception type, and the current status information of the hardware decoding module indicates that the decoding function of the hardware decoding module is normal, it means that the hardware decoding module can normally decode the data to be decoded. Therefore, in order to normally decode the data to be decoded, the hardware decoding module can be determined as the above-mentioned target decoding module. This is because when the exception type is the first exception type when the hardware decoding module is abnormal, the frequency of the exception is low, so that due to the additional processing requirements of the CPU caused by the hardware decoding module abnormality, a small amount of computing resources is used within the range that the CPU resources can bear to handle the abnormality of the hardware decoding module, and it will not interfere with the normal decoding of the data to be decoded by the hardware decoding module. Therefore, on the basis that the decoding function of the hardware decoding module is normal, the hardware decoding module can be determined as the above-mentioned target decoding module.

[0094] When the hardware decoding module is abnormal and the exception type is the second exception type, and / or the current status information of the hardware decoding module indicates that the decoding function of the hardware decoding module is abnormal, it means that the hardware decoding module may not be able to decode the data to be decoded normally. Therefore, in order to decode the data to be decoded normally, the software decoding module can be determined as the above-mentioned target decoding module. This is because when the hardware decoding module is abnormal and the exception type is the second exception type, the frequency of the exception is high, so that the additional processing requirements of the CPU caused by the hardware decoding module are beyond the tolerance range of the CPU resources. A large amount of computing resources are needed to handle the abnormality of the hardware decoding module, which is easy to interfere with the normal decoding of the data to be decoded by the hardware decoding module. Therefore, when the hardware decoding module is abnormal and the exception type is the second exception type, and / or the decoding function of the hardware decoding module is abnormal, the software decoding module can be determined as the above-mentioned target decoding module.

[0095] Optionally, when the hardware decoding module has an exception of the first and second types, the software decoding module may be determined as the target decoding module. This is because if the hardware decoding module has an exception of the second type, the hardware decoding module may be unable to properly decode the data to be decoded. Therefore, to ensure proper decoding of the data to be decoded, the software decoding module may be determined as the target decoding module.

[0096] In an embodiment of the present application, the target decoding module is determined in the hardware decoding module and the software decoding module by combining the corresponding exception type when an exception occurs in the hardware decoding module and the current state information of the hardware decoding module, so that the target decoding module determined by the multi-dimensional data can be more accurate.

[0097] S230 , decoding the data to be decoded based on the target decoding module to obtain decoded data.

[0098] Exemplarily, when a target decoding module is determined in the hardware decoding module and the software decoding module, the target decoding module may be used to normally decode the data to be decoded to obtain decoded data.

[0099] For example, when the target decoding module is a hardware decoding module, the data to be decoded is decoded normally by the hardware decoding module. Alternatively, when the target decoding module is a software decoding module, the data to be decoded is decoded normally by the software decoding module.

[0100] In such Figure 2 In the illustrated method 200, upon detecting an exception in a hardware decoding module while decoding data to be decoded, current state information of the hardware decoding module may be first obtained. A target decoding module capable of decoding the data to be decoded may be determined from the hardware decoding module and the software decoding module based on the current state information of the hardware decoding module. The target decoding module may then be used to decode the data to be decoded, thereby obtaining decoded data. Because the software decoding module requires a greater amount of operating resources than the hardware decoding module when decoding the data to be decoded, impacting the operating performance of the electronic device, to conserve operating resources of the electronic device, upon detecting an exception in the hardware decoding module while decoding the data to be decoded, the decoding module is not directly switched from the hardware decoding module to the software decoding module. Instead, the target decoding module capable of decoding the data to be decoded may be determined from the hardware decoding module and the software decoding module based on the current state information of the hardware decoding module. The hardware decoding module may then be used to decode the data to be decoded as much as possible, thereby enabling normal decoding of the data to be decoded while conserving operating resources of the electronic device.

[0101] Furthermore, while saving the operating resources of the electronic device, the device can be prevented from running stuck and the smooth operation of the device can be ensured.

[0102] In addition, since the electronic device includes a hardware decoding module and a software decoding module, the compatibility of the software decoding module for decoding the data to be decoded can be utilized to improve the compatibility of the electronic device for decoding the data to be decoded, so that the electronic device can decode data to be decoded of different data formats and types.

[0103] Optionally, due to the decoding process and network factors, the decoded data may be out of order, that is, the time order of the decoded data is disordered and is not in a chronological order from early to late. Therefore, when the decoded data is obtained, the timestamp generated by each decoded data can be obtained, and it can be determined whether the timestamp generated by each decoded data is sorted in a chronological order from early to late. If the timestamp generated by each decoded data is not sorted in a chronological order from early to late, it means that there is a problem of disorder in the decoded data. Therefore, the disordered decoded data can be re-sorted in a chronological order from early to late to avoid disorder in the decoded data and ensure that normal decoded data is obtained.

[0104] Optionally, when an exception is detected when the hardware decoding module decodes the data to be decoded, the data to be decoded corresponding to the time when the hardware decoding module encountered the exception (referred to as "first data to be decoded") can be obtained, i.e., the hardware decoding module encountered an exception when decoding the first data to be decoded. When the first data to be decoded is obtained, a timestamp carried by the first data to be decoded (referred to as "target timestamp") can be obtained. The second data to be decoded can then be determined from the data to be decoded using the target timestamp and the first data to be decoded. The timestamp of the second data to be decoded is after the target timestamp.

[0105] For example, when the target timestamp is 10:00:00, the timestamp of the second data to be decoded is after 10:00:00, which may be 10:00:01, 10:00:03, and so on.

[0106] Since an exception occurs in the hardware decoding module when decoding the first data to be decoded, in order to avoid the exception in the decoded data of the first data to be decoded decoded by the hardware decoding module, a software decoding module can be used to decode the first data to be decoded to obtain normal decoded data of the first data to be decoded (which can be called "first decoded data").

[0107] Furthermore, when the second data to be decoded and the target decoding module are obtained, the second data to be decoded can be decoded by the target decoding module to obtain normal decoded data of the second data to be decoded (which can be referred to as "second decoded data").

[0108] For example, when the target decoding module is a hardware decoding module, the second data to be decoded is decoded by the hardware decoding module. Alternatively, when the target decoding module is a software decoding module, the second data to be decoded is decoded by the software decoding module.

[0109] When the first decoded data and the second decoded data are obtained, the decoded data can be obtained through the first decoded data and the second decoded data.

[0110] In an embodiment of the present application, in order to avoid abnormalities in the decoded data decoded by the hardware decoding module when an abnormality occurs, a software decoding module can be used to decode the decoded data corresponding to the hardware decoding module when an abnormality occurs, and the decoded data decoded by the software decoding module and the decoded data decoded by the target decoding module after the hardware decoding module abnormality are used to obtain the decoded data, thereby ensuring the normality of the decoded data.

[0111] Optionally, when the target timestamp of the first data to be decoded is obtained, decoded data (referred to as "third decoded data") corresponding to the data to be decoded that precedes the target timestamp (referred to as "third data to be decoded") in the data to be decoded can be obtained. That is, the third decoded data represents the decoded data corresponding to the data to be decoded whose timestamp precedes the target timestamp. Furthermore, since the timestamp of the third data to be decoded precedes the target timestamp, this indicates that no abnormality has occurred when the hardware decoding module decodes the third data to be decoded, and the third decoded data corresponding to the decoded third data to be decoded is also normal.

[0112] For example, when the target timestamp is 10:00:00, the timestamp of the third data to be decoded is before 10:00:00, which may be 09:59:59, 09:59:50, and so on.

[0113] When the first decoded data, the second decoded data, and the third decoded data are obtained, the timestamp corresponding to the first decoded data, the timestamp corresponding to the second decoded data, and the timestamp corresponding to the third decoded data can also be obtained. The timestamps of the first decoded data, the second decoded data, and the third decoded data are then combined (also called "concatenated") in chronological order from earliest to latest to obtain combined decoded data, thereby obtaining complete decoded data (i.e., combined decoded data).

[0114] In an embodiment of the present application, based on the decoded data (i.e., the first decoded data) corresponding to the hardware decoding module when an exception occurs, and the decoded data (i.e., the second decoded data) decoded by the target decoding module after the hardware decoding module exception occurs, the decoded data decoded by the hardware decoding module before its own exception occurs (i.e., the third decoded data) is also combined. The data of the three time periods (i.e., the first decoded data, the second decoded data, and the third decoded data) together constitute the decoded data corresponding to the data to be decoded, thereby ensuring the integrity of the decoded data and improving the user's audio-visual experience.

[0115] Furthermore, by combining the timestamps corresponding to the data of the three time periods, decoded data with the correct temporal sequence can be obtained, thereby improving the accuracy of the decoded data and thus enhancing the user's audio-visual experience.

[0116] For example, when the decoded data is obtained, it can be determined whether the decoded data is video data. If the decoded data is video data, it can be determined whether the video data contains audio data. The video data includes image data.

[0117] When there is audio data in the video data, the audio data can be obtained, and the playback speed of the image data in the video data can be adjusted based on the playback time of the audio data to synchronize the image data with the audio data (i.e., picture and sound synchronization), thereby achieving synchronous playback of the image data and audio data.

[0118] Alternatively, when there is no audio data in the video data, the playback speed of the image data in the video data cannot be adjusted based on the audio data, so the current system time of the electronic device and the playback frame rate of the image data can be obtained.

[0119] When the current system time of the electronic device and the playback frame rate of the image data are obtained, the playback speed of the image data can be adjusted according to the current system time of the electronic device and the playback frame rate of the image data so that the playback frame rate of the image data in the same duration is the same.

[0120] To obtain the playback frame rate of the image data, the image data's metadata (META) can be obtained. This metadata may include information such as the image data's resolution and frame rate. The image data's frames per second (FPS) can then be extracted from the metadata.

[0121] In an embodiment of the present application, when there is audio data in the video data, the playback speed of the image data in the video data can be adjusted based on the audio data, so that the image data and the audio data are synchronized, thereby achieving synchronous playback of the image data and the audio data, and improving the user's audio-visual experience.

[0122] Alternatively, when there is no audio data in the video data, the playback speed of the image data in the video data can be adjusted jointly by the current system time and the playback frame rate of the image data, so that the playback frame rate of the image data in the same duration is the same, thereby reducing the frame rate mutation caused by the different playback frame rates of the image data in the same duration, and the problems of image data playback buffering and / or stuttering, thereby improving the stability and compatibility of image data playback, allowing users to enjoy a coherent and natural visual experience, and improving the user's visual experience.

[0123] Optionally, when audio data exists in the video data, the timestamp of the audio data in the video data and the timestamp of the image data in the video data can be obtained. The timestamp of the image data can indicate the time when the image data is generated, and the timestamp of the audio data can indicate the time when the audio data is generated, and both of them will not be changed.

[0124] When the timestamp of the audio data and the timestamp of the image data in the video data are obtained, the audio data (which can be called "target audio data") and the image data (which can be called "target image data") with the same timestamp can be determined. And the moment when the target image data is played (which can be called "first playback moment") and the moment when the target audio data is played (which can be called "second playback moment") are obtained. Since the target image data and the target audio data are data with the same timestamp, the playback moments of the target image data and the target audio data should also be the same to achieve picture and sound synchronization. However, due to various reasons such as decoding speed and CPU operating status, the picture and sound may be out of sync, thereby reducing the user's audio-visual experience.

[0125] Therefore, in order to solve the problem of the asynchrony between the picture and the audio of the video data, it can be determined whether the first playback time is the same as the second playback time.

[0126] If the first playback time is earlier than the second playback time, the image data will be played ahead of time. The difference between the first and second playback times (referred to as the "first difference") can be calculated, i.e., first difference = first playback time - second playback time. The playback of the image data is then delayed by the first difference (e.g., 0.01 seconds) to reduce the playback speed of the image data and synchronize the audio and video from asynchronous to synchronized, thereby improving the user's audiovisual experience.

[0127] Alternatively, when the first playback time is earlier than the second playback time, the playback of the audio data can also be advanced by the first difference value (for example, 0.01 seconds) to increase the playback speed of the audio data and adjust the asynchronous audio and video to synchronized audio and video, thereby improving the user's audio-visual experience.

[0128] If the first playback time is later than the second playback time, the image data will be played later. The difference between the second playback time and the first playback time (referred to as the "second difference") can be calculated (i.e., second difference = second playback time - first playback time). The image data playback is then advanced by the second difference (e.g., 0.02 seconds) to increase the playback speed of the image data and synchronize the image and audio from being out of sync, thereby improving the user's audio-visual experience.

[0129] Alternatively, when the first playback time is later than the second playback time, the playback of the audio data can also be delayed by the second difference (for example, 0.02 seconds) to reduce the playback speed of the audio data and adjust the asynchronous audio and video to synchronized audio and video, thereby improving the user's audio-visual experience.

[0130] In an embodiment of the present application, when the playback time of image data in the video data is earlier than the playback time of audio data, the playback speed of the image data can be reduced so that the playback time of the image data is the same as the playback time of the audio data; or, when the playback time of image data in the video data is later than the playback time of audio data, the playback speed of the image data can be increased so that the playback time of the image data is the same as the playback time of the audio data; thereby adjusting the asynchronous picture and audio to synchronized picture and audio, thereby improving the user's audio-visual experience.

[0131] Optionally, when there is no audio data in the video data, the playback time of each frame of image data (which may be referred to as a "target playback time") can be determined based on the current system time of the electronic device and the playback frame rate of the image data. The determined target playback time is then added to each frame of image data to add a playback timestamp to each frame of image data.

[0132] When the target playback time of each frame of image data is obtained, the actual playback time of each frame of image data can also be obtained, and it is determined whether the actual playback time of each frame of image data is the same as the target playback time.

[0133] If the actual playback time is earlier than the target playback time, it indicates that the image data will be played ahead of time. The difference between the actual playback time and the target playback time (referred to as the "third difference") can be calculated, i.e., third difference = actual playback time - target playback time. The playback of the image data is then delayed by the third difference (e.g., 0.005 seconds) to reduce the playback speed of the image data, ensuring that the playback frame rate of the image data remains the same over the same duration. This reduces image data playback buffering and / or lag caused by sudden frame rate changes, improves the stability and compatibility of image data playback, and enables users to enjoy a consistent and natural visual experience.

[0134] If the actual playback time is later than the target playback time, the image data will be played later. The difference between the target playback time and the actual playback time (referred to as the "fourth difference") can be calculated: fourth difference = target playback time - actual playback time. The image data playback is then advanced by the fourth difference (e.g., 0.007 seconds) to increase the playback speed of the image data, ensuring that the playback frame rate of the image data remains the same over the same duration. This reduces image data playback buffering and / or freezes caused by sudden frame rate changes, improves the stability and compatibility of image data playback, and enables users to enjoy a consistent and natural visual experience.

[0135] For example, when there is no audio data in the video data, the playback frame rate of the image data in the video data is 25FPS, and the current system time is 00:00:00, then the playback timestamp corresponding to the first frame of the image data is 00:00:00.00, the playback timestamp corresponding to the second frame of the image data is 00:00:00.04, the playback timestamp corresponding to the third frame of the image data is 00:00:00.08... until the playback timestamp corresponding to the nth frame of the image data.

[0136] In an embodiment of the present application, when the playback time of image data in video data is earlier or later than the set playback time (i.e., the target playback time), it may cause the image data to have different playback frame rates in the same duration, resulting in a sudden change in frame rate, and problems of image data playback buffering and / or stuttering. Therefore, when the playback time of image data in video data is earlier than the target playback time, the playback speed of the image data is reduced, or when the playback time of image data in video data is later than the target playback time, the playback speed of the image data is increased, so that the playback frame rate of the image data in the same duration is the same, thereby improving the stability and compatibility of image data playback, allowing users to enjoy a coherent and natural visual experience, and improving the user's visual experience.

[0137] It should be noted that Figure 2 All steps in Figure 3 and Figure 5The corresponding embodiments are described in detail in the accompanying drawings and will not be repeated here.

[0138] Figure 3 This is another flow chart of a data processing method provided by an embodiment of the present application. Figure 1 The electronic device 110 in the embodiment of the present invention executes the above-mentioned operation, or the decoding plug-in in the electronic device 110 executes the above-mentioned operation.

[0139] For example, Figure 3 As shown, the method 300 includes the following implementation process:

[0140] S301 , obtaining video data to be decoded and a decoding instruction for the video data to be decoded.

[0141] Exemplarily, when decoding is performed by a hardware decoding module and / or a software decoding module, video data to be decoded (ie, the above-mentioned data to be decoded) and decoding instructions for the video data to be decoded may be obtained first.

[0142] S302: Determine whether the device supports hardware decoding. If not, execute S303; if yes, execute S306.

[0143] For example, when the video data to be decoded and the decoding instruction for the video data to be decoded are obtained, the decoding instruction can be responded to to determine whether the current device (ie the electronic device) where the video data to be decoded is located supports hardware decoding.

[0144] S303: If the device supports software decoding, enable the software decoding module and enable system resource monitoring.

[0145] For example, when it is determined in S302 that the device does not support hardware decoding, it is possible to continue to determine whether the device supports software decoding. If the device supports software decoding (i.e., it supports software decoding but not hardware decoding), the software decoding module in the device can be enabled. Furthermore, system resource monitoring can be enabled so that if the software decoding module occupies a large amount of CPU resources when decoding the video data to be decoded, corresponding measures can be taken to avoid playback freezes of the decoded video corresponding to the video data to be decoded.

[0146] Optionally, when the device does not support hardware decoding or software decoding, it indicates that the device does not have the decoding function, and a prompt message can be output to the user to prompt the user that the device cannot decode the video data to be decoded, so that the user can change the device so that the video data to be decoded can be decoded normally.

[0147] S304: Decode the video data to be decoded by a software decoding module.

[0148] Exemplarily, when the software decoding module is turned on, the video data to be decoded can be decoded by the software decoding module to obtain decoded video data (ie, the decoded data mentioned above).

[0149] It should be noted that when the device does not support the hardware decoding module but only supports the software decoding module, all data in the video data to be decoded can be decoded by the software decoding module to obtain complete decoded video data, ensuring the normal decoding of the video data to be decoded.

[0150] S305 , obtaining the decoded video data, and checking the decoded video data. When the check result is normal, playing the decoded video data.

[0151] For example, when the software decoding module completes decoding of the video data to be decoded, the decoded video data can be obtained and checked to determine whether the decoded video data has problems such as disorder, distorted screen, and mosaic.

[0152] If the decoded video data does not have problems such as disorder, screen distortion, and mosaic, it indicates that the decoded video data is normal and can be played.

[0153] S306, start the hardware decoding module and start decoding status monitoring.

[0154] For example, when it is determined in S302 that the device supports hardware decoding, a hardware decoding module in the device may be enabled.

[0155] S307 , starting decoding status monitoring to monitor the real-time status of the hardware decoding module when the video data to be decoded is decoded by the hardware decoding module.

[0156] Exemplarily, when the hardware decoding module in the device is enabled, decoding status monitoring may also be enabled simultaneously to monitor the real-time decoding status of the hardware decoding module when the hardware decoding module decodes the video data to be decoded.

[0157] Exemplarily, when the hardware decoding module in the device is enabled, S303 may be further executed to enable the software decoding module and enable system resource monitoring if the device supports software decoding.

[0158] It should be noted that after the hardware decoding module in the device is turned on, if the software decoding module is turned on, the software decoding module does not decode all data frames in the video data to be decoded, but only decodes the data corresponding to the key frames in the video data to be decoded (which can be called "key frame data"), and discards the non-key corresponding data (which can be called "non-key frame data"), and does not decode the non-key frame data, so as to reduce the CPU resource occupation of the software decoding module during decoding.

[0159] Furthermore, the non-keyframe data discarded by the software decoding module is data already cached in the software decoding module and does not affect the original video data to be decoded. That is, when the software decoding module obtains the video data to be decoded, it must first cache the video data to be decoded, and then decode or discard the cached video data. It should be understood that when partially discarding the cached video data, it is only discarded after determining that the decoded data with the same timestamp in the hardware decoding module is not abnormal decoded data.

[0160] The key frame data may be marked in advance in the video data to be decoded, for example, marking the first frame of every 10 frames as a key frame, marking data with a large number of pixels as a key frame, etc., which is not limited in the present embodiment.

[0161] S308: Determine whether the real-time status is abnormal. If so, execute S309; ​​if not, execute S305.

[0162] For example, when the hardware decoding module decodes the real-time status, it can be determined whether there is an abnormality (also referred to as a "fault") in the real-time status.

[0163] For example, if the real-time decoding status of the hardware decoding module obtained in S308 is not abnormal, it indicates that the hardware decoding module has not failed and the video data to be decoded can be decoded normally to obtain decoded video data. Therefore, S305 can be executed to obtain the decoded video data, check the decoded video data, and play the decoded video data if the check result is normal.

[0164] S309: Obtain the fault type of the hardware decoding module.

[0165] For example, if the real-time decoding status of the hardware decoding module obtained in S308 is abnormal, it indicates that the hardware decoding module has failed. In order to determine whether the abnormality affects the decoding function of the hardware decoding module, the fault type of the hardware decoding module when it is currently abnormal (i.e., the above-mentioned abnormality type) can be obtained.

[0166] S310: Determine whether the fault type is an occasional fault. If so, execute S311; if not, execute S313.

[0167] For example, when the fault type of the hardware decoding module exception is obtained, it can be determined whether the fault type is an occasional fault (ie, the first exception type mentioned above).

[0168] S311 , obtaining synchronous decoding data corresponding to the software decoding module when the hardware decoding module is in an abnormal state, and using the synchronous decoding data to replace the abnormal decoding data of the hardware decoding module when the state is abnormal.

[0169] For example, when the fault type of the hardware decoding module abnormality is determined to be an occasional fault in S310 , it is also necessary to determine whether the occasional fault will cause the playback of the decoded video data to be stuck or frozen.

[0170] When occasional failures do not cause the decoded video data to be played back stuck or frozen, it means that the failure will not affect the normal decoding function of the hardware decoding module. However, in order to prevent the decoded video data from having problems such as flowery screen, green screen and mosaic when the hardware decoding module is abnormal, which makes the user's audio-visual experience poor, the software decoding module can obtain the corresponding decoding data (i.e., synchronous decoding data) when the hardware decoding module is in an abnormal state. If the hardware decoding module still decodes data (which can be called "abnormal decoding data") when it is abnormal, the synchronous decoding data can be used to replace the abnormal decoding data corresponding to the hardware decoding module when it is abnormal, or if the hardware decoding module does not decode data when it is abnormal, that is, there is no abnormal decoding data at this time, the synchronous decoding data can be used as the normal decoding data corresponding to the hardware decoding module when it is abnormal, so that normal decoding data can be obtained when the hardware decoding module is abnormal. Among them, the synchronous decoding data has the same timestamp as the abnormal decoding data.

[0171] When obtaining synchronized decoded data decoded by the software decoding module when the hardware decoding module is in an abnormal state, if the software decoding module only decodes key frame data in the video data to be decoded, it is also necessary to determine whether the abnormal video data is key frame data. If the abnormal video data is key frame data, a key frame data having the same timestamp as the abnormal video data can be identified from multiple key frames and used to replace the abnormal video data.

[0172] Conversely, when the software decoding module only decodes key frame data in the video data to be decoded, if the abnormal video data does not belong to the key frame data, the software decoding module needs to first obtain a piece of original data (referred to as "data A") with the same timestamp as the abnormal video data from the video data to be decoded, and then decode data A to obtain the decoded data corresponding to data A. The decoded data corresponding to data A is then used to replace the abnormal video data.

[0173] Alternatively, when obtaining the synchronous decoding data decoded by the software decoding module when the hardware decoding module is in an abnormal state, if the software decoding module decodes all data in the video data to be decoded, a decoding data with the same timestamp as the abnormal video data can be determined in the video data decoded by the software decoding module, and the abnormal video data can be replaced with the decoding data.

[0174] S312 , obtaining decoded video data by combining the replaced synchronous decoded data with the decoded data of the hardware decoding module when there is no abnormality.

[0175] For example, when the synchronous decoding data decoded by the software decoding module replaces the abnormal decoding data corresponding to the hardware decoding module when an abnormality occurs, the synchronous decoding data can be combined with the decoding data of the hardware decoding module when there is no abnormality (i.e., normal decoding data) to obtain the complete decoded video data.

[0176] When the complete decoded video data is obtained, step S305 is executed to obtain the decoded video data and check the decoded video data. When the check result is normal, the decoded video data is played.

[0177] S313, reset the hardware decoding module.

[0178] For example, if the fault type determined by S310 to be an abnormal hardware decoding module is not an occasional fault, then the fault type may be a frequent fault (i.e., the second abnormal type described above). A frequent fault may affect the normal decoding function of the hardware decoding module, rendering it unable to decode the video data to be decoded. Therefore, the hardware decoding module may be reset. After resetting the hardware decoding module, S308 is executed again to determine whether the hardware decoding module has returned to normal operation.

[0179] Optionally, if the fault type of the hardware decoding module abnormality determined in S310 is an occasional fault, and if this occasional fault causes playback of the decoded video data to become stagnant or freeze, this indicates that the occasional fault also affects the normal decoding function of the hardware decoding module, rendering the hardware decoding module unable to decode the video data to be decoded. Therefore, the hardware decoding module can be reset. After resetting the hardware decoding module, S308 is executed again to determine whether the hardware decoding module has returned to normal operation.

[0180] Furthermore, if the fault type determined in S310 to be an abnormal hardware decoding module is not an occasional fault, S304 may be further executed to switch the decoding module corresponding to the video data to be decoded from the hardware decoding module to the software decoding module, thereby implementing dynamic switching between the hardware decoding module and the software decoding module, so that the video data to be decoded is decoded by the software decoding module. Specifically, the decoding function of the hardware decoding module is replaced by the software decoding module to ensure normal decoding of the video data to be decoded.

[0181] It should be understood that Figure 1 The electronic device 110 may have functions such as checking whether it supports hardware decoding and software decoding, whether there are abnormalities in the hardware decoding module, optimizing and disorderly checking the decoded video data, monitoring system resources, and recording and applying abnormal information.

[0182] Optionally, when a hardware decoding module fails (i.e., an occasional failure and / or a frequent failure), an error code corresponding to the failure can be obtained. When the error code corresponding to the failure is obtained, the error code and the corresponding processing method can be recorded and counted to construct a fault database corresponding to the hardware decoding module. Moreover, since the constructed fault database corresponding to the hardware decoding module is obtained from a large amount of data, it can include different faults of the hardware decoding module and the corresponding processing methods for different faults, as well as different processing methods corresponding to the same fault. Therefore, when the hardware decoding module has the same fault and there are different processing methods, the optimal processing method can be selected from multiple processing methods, thereby improving the processing efficiency and processing effect of eliminating the fault.

[0183] The error code can be used to analyze the corresponding fault type when the hardware decoding module fails.

[0184] Among them, the fault information represented by the error code may include but is not limited to: relevant parameters of the hardware encoding module, parameters of the video data to be decoded, system parameters of the device, network parameters of the device, the number and type of hardware decoding module failures, and one or more of the processing methods for the video data to be decoded when the hardware decoding module fails.

[0185] For example, the parameters related to the hardware encoding module may include one or more of the hardware encoding module's vendor, operating instructions, and error messages. The parameters of the video data to be decoded may include one or more of the video format type, file encapsulation type, and resolution. The system parameters of the device may include one or more of the number of CPUs, CPU processing power, CPU usage, and memory usage. The network parameters of the device may include one or more of the network type, network load, and network speed.

[0186] Figure 4 This is another flow chart of a data processing method provided by an embodiment of the present application. Figure 1 The electronic device 110 in the embodiment of the present invention executes the above-mentioned operation, or the decoding plug-in in the electronic device 110 executes the above-mentioned operation.

[0187] For example, Figure 4 As shown, the method 400 includes the following implementation process:

[0188] S401: Obtain decoded video data, and check the decoded video data to obtain a check result.

[0189] For example, through Figure 3 When the decoded video data is obtained in S305, the decoded video data can be checked and a corresponding check result can be obtained.

[0190] The inspection result can indicate whether there is abnormal video data corresponding to problems such as green screen, flowery screen and mosaic in the decoded video data.

[0191] Optionally, since a mosaic problem occurs in the decoded video data, it is usually manifested as square pixel points in the decoded video data. Therefore, the mosaic problem in the decoded video data can be identified and detected by edge detection or block detection.

[0192] Optionally, since the decoded video data often exhibits a distorted screen due to color data confusion or pixel misalignment, the color difference between the decoded video data of adjacent frames can be calculated. If a frame differs significantly from the preceding and following frames, this may indicate color data confusion or pixel misalignment, leading to a distorted screen. Alternatively, the color histogram corresponding to each frame of the decoded video data can be used. If the color histogram exhibits abnormal peaks or distribution, this may also indicate color data confusion or pixel misalignment, leading to a distorted screen.

[0193] Optionally, since a green screen problem occurs in the decoded video data, it is usually manifested as a large area of ​​green blocks or green interference in the decoded video data. Therefore, the number of green pixels in the decoded video data can be detected. When the number of green pixels is too many, it may indicate that a green screen problem occurs in the decoded video data.

[0194] S402: Determine whether the inspection result is normal. If not, execute S403; if so, execute S410.

[0195] Exemplarily, when an inspection result corresponding to any frame of decoded video data is obtained, it can be determined whether the inspection result is normal.

[0196] When abnormal video data corresponding to problems such as green screen, flowery screen and mosaic exist in the decoded video data, it indicates that the inspection result is abnormal; and when abnormal video data corresponding to problems such as green screen, flowery screen and mosaic do not exist in the decoded video data, it indicates that the inspection result is normal.

[0197] S403: Acquire abnormal video data after decoding.

[0198] For example, when the inspection result corresponding to any frame of the decoded video data obtained through S402 is abnormal, it indicates that the decoded video data contains abnormal video data corresponding to problems such as green screen, flowery screen and mosaic. In order to repair the abnormal video data, the abnormal video data can be obtained.

[0199] S404: Determine whether the abnormal video data has a green screen problem. If so, execute S405; if not, execute S406.

[0200] Exemplarily, when abnormal video data is obtained, it can be determined whether a green screen problem exists in the abnormal video data.

[0201] S405 , compensating the abnormal data using the cached previous frame video data to obtain repaired video data.

[0202] For example, when a green screen problem is detected in the abnormal video data through S404, the cached previous frame data (i.e., the previous frame video data) corresponding to the abnormal video data can be obtained, and the abnormal video data can be compared and analyzed with the previous frame video data to find the pixel points where the green screen part appears in the abnormal video data.

[0203] In the previous frame video data, the pixel points at the same position as the pixel points of the green screen part in the abnormal video data are determined, and the pixel points at the same position are used to replace the pixel points of the green screen part in the abnormal video data, thereby eliminating or alleviating the green screen problem and obtaining the repaired video data.

[0204] S406: When the abnormal video data has a screen distortion and / or mosaic problem, determine the number of pixels corresponding to the screen distortion and / or mosaic in the abnormal video data.

[0205] For example, when the abnormal video data obtained in S404 does not have a green screen problem, it may have a distorted screen and / or mosaic problem. When the abnormal video data has a distorted screen and / or mosaic problem, in order to repair the abnormal video data, the number of pixels corresponding to the distorted screen and / or mosaic in the abnormal video data can be determined.

[0206] S407: Determine whether the number of pixels is greater than or equal to a preset number. If so, execute S405; if not, execute S408.

[0207] For example, when the number of pixels corresponding to the distorted screen and / or mosaic in the abnormal video data is obtained, it can be determined whether the number of pixels is greater than or equal to a preset number.

[0208] The pixels of the decoded video data can be displayed on the screen of the electronic device, corresponding to the display area on the screen of the electronic device. Different pixel positions of the decoded video data will result in different display areas on the screen of the electronic device.

[0209] For example, when the number of pixels obtained by S407 is ≥ the preset number, it means that the number of pixels corresponding to the flowery screen and / or mosaic in the abnormal video data is too large, and the fuzzy algorithm may affect the display effect. Therefore, S405 can be executed to compensate for the abnormal data through the cached previous frame video data to obtain the repaired video data.

[0210] The preset number is related to the total number of decoded video data and can be obtained by multiplying the total number by the preset percentage. In order to avoid affecting the display effect of the decoded video data, the preset percentage is generally set to a low value, such as 1%, 0.5%, or 2%.

[0211] S408: Perform a blurring algorithm on the video data corresponding to the distorted screen and / or mosaic to obtain restored video data.

[0212] For example, when the number of pixels obtained through S407 is less than the preset number, it means that the number of pixels corresponding to the flowery screen and / or mosaic in the abnormal video data is small, and the flowery screen and / or mosaic problems can be reduced or eliminated by a fuzzy algorithm with better computing performance and operating efficiency (also known as a "fast mean fuzzy algorithm") to obtain the repaired video data.

[0213] like Figure 5As shown, when a fuzzy algorithm is applied to the video data corresponding to the mosaic, the original data corresponding to the pixel values ​​of the mosaic in the abnormal video data are 25, 20, 22, 10, 2, 8, 16, 13, and 24, respectively, and the center data in the original data is 2. The fuzzy algorithm is used to calculate the average of 25, 20, 22, 10, 2, 8, 16, 13, and 24, and the average value is 15. The average value 15 is then used to replace the center data 2 in the original data to obtain the processed data. By averaging the local pixel values ​​and smoothing the pixel differences, the visual effect of the mosaic-like area in the abnormal video data is improved, the mosaic problem is reduced or eliminated, the display effect is improved, and the user's visual experience is enhanced, thereby improving the quality and competitiveness of electronic devices.

[0214] It should be understood that the process of applying the fuzzy algorithm to the video data corresponding to the flower screen is the same as the process of applying the fuzzy algorithm to the video data corresponding to the mosaic, and will not be repeated here.

[0215] S409: Play the restored video data.

[0216] Exemplarily, after obtaining the repaired video data, the repaired video data can be played.

[0217] S410: Play the decoded video data.

[0218] For example, when the inspection result corresponding to any frame of the decoded video data obtained in S402 is normal, it means that the decoded video data does not have problems such as green screen, distorted screen and mosaic, and the decoded video data can be played directly.

[0219] In summary, when an exception is detected in the hardware decoding module while decoding the data to be decoded, in order to save the operating resources of the electronic device, the decoding module is not directly switched from the hardware decoding module to the software decoding module. Instead, the target decoding module capable of decoding the data to be decoded is determined between the hardware decoding module and the software decoding module based on the current status information of the hardware decoding module. The hardware decoding module is used to decode the data to be decoded as much as possible, thereby achieving normal decoding of the data to be decoded while saving the operating resources of the electronic device. Furthermore, when the image data and audio data in the video data are out of sync, the speed of the image data or audio data is adjusted to synchronize the image data and audio data, thereby achieving synchronized playback of the image data and audio data, thereby improving the user's audio-visual experience. Furthermore, when the playback frame rates of the image data in the same duration are different in the video data, the playback speed of the image data is adjusted to ensure that the playback frame rates of the image data in the same duration are the same, thereby improving the stability and compatibility of the image data playback, allowing the user to enjoy a coherent and natural visual experience, thereby improving the user's visual experience.

[0220] In addition, since the electronic device includes a hardware decoding module and a software decoding module, the compatibility of the software decoding module for decoding the data to be decoded can be utilized to improve the compatibility of the electronic device for decoding the data to be decoded, so that the electronic device can decode data to be decoded of different data formats and types.

[0221] It should be understood that the above examples are intended to help those skilled in the art understand the embodiments of the present application, and are not intended to limit the embodiments of the present application to the specific numerical values ​​or specific scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or variations based on the above examples, and such modifications or variations also fall within the scope of the embodiments of the present application.

[0222] Combined with the above Figures 1 to 5 The data processing method provided by the embodiment of the present application is described in detail; Figure 6 and Figure 7 The device embodiments of the present application are described in detail. It should be understood that the devices in the embodiments of the present application can execute the various methods of the aforementioned embodiments of the present application, that is, the specific working processes of the following various products can refer to the corresponding processes in the aforementioned method embodiments.

[0223] Figure 6 It is a structural diagram of the data processing device provided in an embodiment of the present application.

[0224] For example, Figure 6 As shown, the apparatus 600 is configured in an electronic device, and the electronic device includes a hardware decoding module and a software decoding module, including:

[0225] An acquisition module 610 is configured to acquire current status information of the hardware decoding module if it is detected that an exception occurs when the hardware decoding module decodes the data to be decoded;

[0226] The processing module 620 is used to determine a target decoding module among the hardware decoding module and the software decoding module based on the current state information;

[0227] The decoding module 630 is configured to decode the data to be decoded based on the target decoding module to obtain decoded data.

[0228] In a possible implementation, the acquisition module 610 is further configured to:

[0229] Get the corresponding exception type when an exception occurs in the hardware decoding module;

[0230] The processing module 620 is specifically configured to:

[0231] Based on the exception type and the current state information, a target decoding module among the hardware decoding module and the software decoding module is determined.

[0232] In one possible implementation, the processing module 620 is specifically configured to:

[0233] If the exception type is the first exception type and the current state information indicates that the decoding function of the hardware decoding module is normal, the hardware decoding module is determined as the target decoding module;

[0234] If the exception type is the second exception type, and / or the current state indicates that the decoding function of the hardware decoding module is abnormal, the software decoding module is determined as the target decoding module;

[0235] The first abnormality type indicates a type in which the abnormality frequency is less than a preset frequency; the second abnormality type indicates a type in which the abnormality frequency is greater than or equal to a preset frequency.

[0236] In one possible implementation, the processing module 620 is further configured to:

[0237] Determine a target timestamp of the first to-be-decoded data corresponding to when an exception occurs in the hardware decoding module;

[0238] Determining second data to be decoded in the data to be decoded based on the target timestamp and the first data to be decoded, wherein the timestamp of the second data to be decoded is after the target timestamp;

[0239] Decoding the first data to be decoded by a software decoding module to obtain first decoded data;

[0240] The processing module 620 is specifically configured to:

[0241] Decoding the second data to be decoded based on the target decoding module to obtain second decoded data;

[0242] Decoded data is obtained based on the first decoded data and the second decoded data.

[0243] In a possible implementation, when the target decoding module is the hardware decoding module, the acquisition module 610 is further configured to:

[0244] Obtaining third decoded data, wherein the third decoded data represents decoded data corresponding to data whose timestamp is before the target timestamp in the data to be decoded;

[0245] The decoding module 630 is specifically used for:

[0246] Decoding the second data to be decoded by the hardware decoding module to obtain second decoded data;

[0247] Based on the timestamp of the first decoded data, the timestamp of the second decoded data, and the timestamp of the third decoded data, the first decoded data, the second decoded data, and the third decoded data are combined to obtain decoded data.

[0248] In one possible implementation, the processing module 620 is further configured to:

[0249] Determine whether audio data exists in the video data;

[0250] If audio data exists in the video data, adjusting the playback speed of the image data based on the audio data so as to synchronize the image data with the audio data;

[0251] If there is no audio data in the video data, obtain the current system time and the playback frame rate of the image data; based on the current system time and the playback frame rate of the image data, adjust the playback speed of the image data so that the playback frame rate of the image data is the same in the same duration.

[0252] In one possible implementation, the processing module 620 is further configured to:

[0253] Obtaining a first playback time and a second playback time, wherein the first playback time indicates the playback time of the image data, the second playback time indicates the playback time of the audio data, and the timestamp of the image data is the same as the timestamp of the audio data;

[0254] If the first playback time is earlier than the second playback time, based on a first difference between the first playback time and the second playback time, reducing the playback speed of the image data so as to synchronize the image data with the audio data;

[0255] If the first play time is later than the second play time, the play speed of the image data is increased based on a second difference between the second play time and the first play time to synchronize the image data with the audio data.

[0256] In one possible implementation, the processing module 620 is further configured to:

[0257] Determine the target playback time corresponding to each frame of image data based on the current system time and the playback frame rate of the image data;

[0258] If the actual playback time of each frame of image data is earlier than the target playback time, the playback speed of the image data is reduced based on a third difference between the actual playback time and the target playback time so that the playback frame rates of the image data are the same within the same duration;

[0259] If the actual playback time of each frame of image data is later than the target playback time, the playback speed of the image data is increased based on the fourth difference between the target playback time and the actual playback time so that the playback frame rate of the image data in the same duration is the same.

[0260] It should be noted that the above-mentioned device 600 is embodied in the form of a functional module. The term "module" here can be implemented in the form of software and / or hardware, and is not specifically limited to this.

[0261] For example, a "module" may be a software program, a hardware circuit, or a combination of the two that implements the aforementioned functionality. The hardware circuit may include an application-specific integrated circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a combination of processors) and memory for executing one or more software or firmware programs, combined logic circuits, and / or other suitable components that support the described functionality.

[0262] Therefore, the modules of each example described in the embodiments of this application can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0263] Figure 7 It is a structural diagram of an electronic device provided in an embodiment of the present application.

[0264] For example, Figure 7 As shown, the electronic device 700 includes: a memory 710 and a processor 720, wherein the memory 710 stores an executable program code 7101, and the processor 720 is used to call and execute the executable program code 7101 to perform a data processing method.

[0265] This application can divide the electronic device into functional modules based on the above-mentioned method examples. For example, each functional module can be mapped to a specific functional module, or two or more functions can be integrated into a processing module. The above-mentioned integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is illustrative and is only a logical functional division. In actual implementation, other division methods may be used.

[0266] When the functional modules are divided according to their functions, the electronic device may include an acquisition module, a processing module, a decoding module, etc. It should be noted that all relevant contents of the steps involved in the above method embodiment can be referred to the functional description of the corresponding functional modules and will not be repeated here.

[0267] The electronic device provided in this application is used to execute the above-mentioned data processing method, and thus can achieve the same effect as the above-mentioned implementation method.

[0268] In the case of an integrated unit, the electronic device may include a processing module and a storage module. The processing module may be used to control and manage the operation of the electronic device, and the storage module may be used to support the electronic device in executing relevant program codes and data.

[0269] The processing module may be a processor or controller that implements or executes the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processing (DSP) and a microprocessor, and the storage module may be a memory.

[0270] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the aforementioned methods. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD (Digital Video Disc), a CD-ROM (Compact Disc Read-Only Memory), a microdrive, a magneto-optical disk, a ROM (Read-Only Memory), a RAM (Random Access Memory), an EPROM (Erasable Programmable Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a DRAM (Dynamic Random Access Memory), a VRAM (Video Random Access Memory), a flash memory device, a magnetic or optical card, a nanosystem (including a molecular memory IC), or any other type of medium or device suitable for storing instructions and / or data.

[0271] The present application also provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the above-mentioned related steps to implement a data processing method in the above-mentioned embodiment.

[0272] In addition, the electronic device provided in the embodiments of the present application can specifically be a chip, component or module, and the electronic device may include a connected processor and memory; wherein the memory is used to store instructions, and when the electronic device is running, the processor can call and execute instructions so that the chip executes a data processing method in the above embodiment.

[0273] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0274] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0275] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0276] The above content is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A data processing method, characterized in that: Applied to an electronic device, the electronic device includes a hardware decoding module and a software decoding module, and the method includes: If it is detected that an exception occurs when the hardware decoding module decodes the data to be decoded, obtaining current state information of the hardware decoding module; Determining a target decoding module among the hardware decoding module and the software decoding module based on the current state information; The data to be decoded is decoded based on the target decoding module to obtain decoded data.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining the corresponding exception type when an exception occurs in the hardware decoding module; The determining, based on the current state information, a target decoding module among the hardware decoding module and the software decoding module includes: The target decoding module among the hardware decoding module and the software decoding module is determined based on the exception type and the current state information.

3. The method according to claim 2, characterized in that The determining the target decoding module in the hardware decoding module and the software decoding module based on the exception type and the current state information includes: If the exception type is the first exception type and the current state information indicates that the decoding function of the hardware decoding module is normal, determine the hardware decoding module as the target decoding module; If the exception type is the second exception type, and / or the current state indicates that the decoding function of the hardware decoding module is abnormal, the software decoding module is determined as the target decoding module; The first abnormality type indicates a type in which the abnormality frequency is less than a preset frequency; the second abnormality type indicates a type in which the abnormality frequency is greater than or equal to the preset frequency.

4. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Determine a target timestamp of the first to-be-decoded data corresponding to when an exception occurs in the hardware decoding module; Determining second data to be decoded in the data to be decoded based on the target timestamp and the first data to be decoded, wherein the timestamp of the second data to be decoded is after the target timestamp; Decoding the first data to be decoded by the software decoding module to obtain first decoded data; The decoding the to-be-decoded data based on the target decoding module to obtain decoded data includes: Decoding the second to-be-decoded data based on the target decoding module to obtain second decoded data; The decoded data is obtained based on the first decoded data and the second decoded data.

5. The method according to claim 4, characterized in that In the case where the target decoding module is the hardware decoding module, the method further includes: Acquire third decoded data, wherein the third decoded data represents decoded data corresponding to data whose timestamp in the data to be decoded is before the target timestamp; The obtaining the decoded data based on the first decoded data and the second decoded data includes: Decoding the second data to be decoded by the hardware decoding module to obtain second decoded data; The first decoded data, the second decoded data and the third decoded data are combined based on the timestamp of the first decoded data, the timestamp of the second decoded data and the timestamp of the third decoded data to obtain the decoded data.

6. The method according to any one of claims 1 to 3, characterized in that The decoded data is video data, and the video data includes image data. The method further includes: Determining whether audio data exists in the video data; If the audio data exists in the video data, adjusting the playback speed of the image data based on the audio data so as to synchronize the image data with the audio data; If the audio data does not exist in the video data, obtain the current system time and the playback frame rate of the image data; based on the current system time and the playback frame rate of the image data, adjust the playback speed of the image data so that the playback frame rate of the image data in the same duration is the same.

7. The method according to claim 6, characterized in that The adjusting the playback speed of the image data in the video data based on the audio data so as to synchronize the image data with the audio data includes: Obtaining a first playback time and a second playback time, wherein the first playback time indicates the playback time of the image data, the second playback time indicates the playback time of the audio data, and the timestamp of the image data is the same as the timestamp of the audio data; if the first playback time is earlier than the second playback time, reducing the playback speed of the image data based on a first difference between the first playback time and the second playback time so as to synchronize the image data with the audio data; If the first play time is later than the second play time, the play speed of the image data is increased based on a second difference between the second play time and the first play time, so as to synchronize the image data with the audio data.

8. The method according to claim 6, characterized in that The adjusting the playback speed of the image data based on the current system time and the playback frame rate of the image data so that the playback frame rates of the image data in the same duration are the same includes: Determining a target playback time corresponding to each frame of image data based on the current system time and the playback frame rate of the image data; If the actual playback time of each frame of image data is earlier than the target playback time, based on a third difference between the actual playback time and the target playback time, reducing the playback speed of the image data so that the playback frame rates of the image data in the same time interval are the same; If the actual playback time of each frame of image data is later than the target playback time, based on the fourth difference between the target playback time and the actual playback time, the playback speed of the image data is increased so that the playback frame rate of the image data in the same duration is the same.

9. A data processing device, characterized in that: The device is configured in an electronic device, wherein the electronic device includes a hardware decoding module and a software decoding module. The device includes: an acquisition module, configured to acquire current status information of the hardware decoding module if it is detected that an exception occurs when the hardware decoding module decodes the data to be decoded; a processing module, configured to determine a target decoding module among the hardware decoding module and the software decoding module based on the current state information; A decoding module is used to decode the data to be decoded based on the target decoding module to obtain decoded data.

10. An electronic device, characterized in that: The electronic device comprises: a memory for storing executable program code; A processor, configured to call and run the executable program code from the memory, so that the electronic device executes the method according to any one of claims 1 to 8.

Images