Audio and video data conversion method and device, electronic equipment and storage medium

Abstract

The invention discloses an audio and video data conversion method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining to-be-converted audio and video data and a man-machine interaction instruction, and determining a target format and an optimization coding algorithm according to the man-machine interaction instruction, and performing format conversion processing on the to-be-converted audio and video data according to the target format and the optimized coding algorithm to obtain audio data and video data, and performing synchronization processing on the audio data and the video data through a synchronization controller to obtain target audio and video data. The embodiment of the invention can improve the conversion efficiency, and can be widely applied to the technical field of data processing.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an audio and video data conversion method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of information technology, the processing and conversion of audio and video data plays a vital role in many fields. In the related art, digital audio and video conversion technology uses digital signal processing technology to efficiently and accurately convert and process audio and video data. However, due to the different requirements of different devices, platforms and formats for audio and video data, the conversion methods in the related art have the problems of slow speed, large quality loss, poor synchronization and inability to meet the conversion requirements. In summary, the technical problems existing in the related art need to be improved. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to provide an audio and video data conversion method, device, electronic device and storage medium, which can improve the conversion efficiency of audio and video data.

[0004] To achieve the above purpose, an embodiment of the present application provides a method for converting audio and video data, the method comprising:

[0005] Obtain audio and video data to be converted and human-computer interaction instructions;

[0006] Determine the target format and optimize the encoding algorithm according to the human-computer interaction instruction;

[0007] Performing format conversion processing on the audio and video data to be converted according to the target format and the optimized encoding algorithm to obtain audio data and video data;

[0008] The audio data and the video data are synchronously processed by a synchronization controller to obtain target audio and video data.

[0009] In some embodiments, performing format conversion processing on the audio and video data to be converted according to the target format and the optimized encoding algorithm to obtain audio data and video data includes the following steps:

[0010] Decoding the audio and video data to be converted to obtain decoded data;

[0011] The decoded data is encoded according to the target format, and the encoding process is optimized by the optimized encoding algorithm to obtain the audio data and the video data, wherein the optimized encoding algorithm includes at least one of a parallel processing algorithm, a coding prediction algorithm and an adaptive quantization algorithm.

[0012] In some embodiments, the optimized encoding algorithm adopts the parallel processing algorithm, and the encoding process is optimized by the optimized encoding algorithm to obtain the audio data and the video data, including the following steps:

[0013] Performing task segmentation processing on the decoded data to obtain subtasks;

[0014] Performing thread allocation processing on the subtasks, and performing access control and load balancing processing on the allocated threads through a thread synchronization algorithm to obtain parallel processing threads;

[0015] The decoded data is encoded according to the parallel processing thread to obtain the audio data and the video data.

[0016] In some embodiments, the optimized coding algorithm adopts the coding prediction algorithm, and the optimizing coding process is optimized by the optimized coding algorithm to obtain the audio data and the video data, including the following steps:

[0017] Performing statistical characteristic analysis on the decoded data to obtain statistical information;

[0018] Perform coding demand prediction processing on the statistical information according to a machine learning algorithm or a statistical analysis algorithm to obtain coding parameters;

[0019] The decoded data is encoded according to the encoding parameters to obtain the audio data and the video data.

[0020] In some embodiments, the optimized coding algorithm adopts the adaptive quantization algorithm, and the optimizing coding process by the optimized coding algorithm to obtain the audio data and the video data includes the following steps:

[0021] Performing local feature extraction processing on the decoded data to obtain feature data;

[0022] The decoded data is processed for data volume adjustment according to the characteristic data to obtain the audio data and the video data.

[0023] In some embodiments, the step of synchronously processing the audio data and the video data by a synchronization controller to obtain target audio and video data comprises the following steps:

[0024] Performing timestamp extraction processing on the audio data and the video data to obtain an audio frame timestamp and a video frame timestamp;

[0025] Comparing the audio frame timestamp with the video frame timestamp to obtain a timestamp difference;

[0026] The audio data and the video data are buffered and read according to the timestamp difference to obtain the target audio and video data.

[0027] In some embodiments, the buffering and reading processing of the audio data and the video data according to the timestamp difference to obtain the target audio and video data includes the following steps:

[0028] Storing the audio data in an audio buffer and storing the video data in a video buffer;

[0029] When the timestamp difference is that the audio frame timestamp is faster than the video frame timestamp, adjusting the reading speed of the audio buffer;

[0030] When the timestamp difference is that the audio frame timestamp is slower than the video frame timestamp, adjusting the reading speed of the video buffer or inserting an additional video frame into the video buffer;

[0031] The adjusted audio buffer and video buffer are read and processed to obtain the target audio and video data.

[0032] To achieve the above object, another aspect of the embodiment of the present application provides an audio and video data conversion device, the device comprising:

[0033] The first module is used to obtain audio and video data to be converted and human-computer interaction instructions;

[0034] The second module is used to determine the target format and optimize the encoding algorithm according to the human-computer interaction instruction;

[0035] The third module is used to perform format conversion processing on the audio and video data to be converted according to the target format and the optimized encoding algorithm to obtain audio data and video data;

[0036] The fourth module is used to synchronously process the audio data and the video data through a synchronization controller to obtain target audio and video data.

[0037] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0038] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0039] The embodiments of the present application include at least the following beneficial effects: the present application provides an audio and video data conversion method, device, electronic device and storage medium, the scheme performs format conversion processing on the audio and video data to be converted according to the target format and the optimized coding algorithm to obtain audio data and video data, and can improve the conversion efficiency and quality of audio and video data by optimizing the coding algorithm, and realize the rapid conversion of audio and video data. In addition, the scheme performs synchronous processing on audio data and video data through a synchronous controller to obtain target audio and video data, and can improve the synchronization of audio data and video data during the conversion process through the synchronous controller, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of an audio and video data conversion method provided by an embodiment of the present application;

[0041] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.

[0042] Figure 3 yes Figure 2 An implementation flow chart of step S202 in FIG.

[0043] Figure 4 yes Figure 2 Another implementation flow chart of step S202 in FIG.

[0044] Figure 5 yes Figure 2 Another implementation flow chart of step S202 in FIG.

[0045] Figure 6 yes Figure 1 Flow chart of step S104 in FIG.

[0046] Figure 7 yes Figure 6 Flow chart of step S603 in FIG.

[0047] Figure 8 It is an implementation flow chart provided by an embodiment of the present application;

[0048] Fig. 9 It is a structural schematic diagram of an audio and video data conversion device provided in an embodiment of the present application;

[0049] Fig.10 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0051] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0052] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0054] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0055] Human-Computer Interaction Techniques refers to the technology that realizes the dialogue between people and computers in an effective way through computer input and output devices. Human-Computer Interaction Techniques include machines providing people with a large amount of relevant information and prompts through output or display devices, and people input relevant information to machines through input devices, answer questions and prompts, etc.

[0056] Machine learning algorithms refer to a type of algorithms that automatically learn rules and patterns from data by analyzing and learning the data, and use these rules and patterns to perform tasks such as prediction, classification, clustering, and dimensionality reduction.

[0057] With the rapid development of information technology, the processing and conversion of audio and video data plays a vital role in many fields. Whether it is entertainment, education, conference, or advertising, news, security and other fields, audio and video data need to be efficiently and accurately converted and processed. Therefore, the research and development and optimization of audio and video conversion systems have always been the focus of the industry. In the early days, audio and video conversion was mainly achieved through analog signal processing technology. This method is limited by the limitations of hardware equipment and signal processing technology, with low conversion efficiency and poor quality, and cannot meet the needs of complex scenarios. With the rise of digital technology, digital audio and video conversion technology has gradually replaced analog technology and become the mainstream. Digital audio and video conversion technology uses digital signal processing technology to efficiently and accurately convert and process audio and video data. At the same time, with the continuous development of computer technology, audio and video conversion systems have gradually achieved automation and intelligence.

[0058] In the related technologies, although digital audio and video conversion technology has made significant progress, it still faces many challenges in practical applications. For example, how to ensure the synchronization of audio and video data, how to optimize the conversion efficiency, how to reduce system resource usage, etc., these problems limit the further development and application of audio and video conversion systems. Due to the different requirements of different devices, platforms and formats for audio and video data, the conversion of audio and video data has become an important technical issue. Traditional audio and video conversion methods have problems such as slow conversion speed, large quality loss, and poor synchronization, which cannot meet the needs of modern multimedia applications.

[0059] In view of this, an embodiment of the present application provides a method, device, equipment and medium for processing business data of an application. The solution performs format conversion processing on the audio and video data to be converted according to the target format and the optimized coding algorithm to obtain audio data and video data, and can improve the conversion efficiency and quality of the audio and video data by optimizing the coding algorithm, and realize the rapid conversion of audio and video data. In addition, the solution performs synchronous processing on the audio data and video data through a synchronous controller to obtain the target audio and video data, and can improve the synchronization of the audio data and video data during the conversion process through the synchronous controller, thereby improving the user experience.

[0060] An audio and video data conversion method provided in an embodiment of the present application relates to the field of data processing technology. An audio and video data conversion method provided in an embodiment of the present application can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements an audio and video data conversion method, etc., but is not limited to the above forms.

[0061] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0062] Figure 1 is an optional flowchart of an audio and video data conversion method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S104.

[0063] Step S101, obtaining audio and video data to be converted and human-computer interaction instructions;

[0064] Step S102, determining a target format and optimizing a coding algorithm according to the human-computer interaction instruction;

[0065] Step S103, performing format conversion processing on the audio and video data to be converted according to the target format and the optimized encoding algorithm to obtain audio data and video data;

[0066] Step S104: Synchronize the audio data and the video data through a synchronization controller to obtain target audio and video data.

[0067] Steps S101 to S104 shown in the embodiment of the present application are to perform format conversion processing on the audio and video data to be converted by obtaining human-computer interaction instructions. The target format and optimized coding algorithm to be converted can be obtained through the human-computer interaction instructions. The target format is used to perform format conversion on the audio and video data, and the optimized coding algorithm is used to optimize the coding process. Then, the audio and video data to be converted are format converted according to the target format and optimized coding algorithm determined by the user to obtain audio data and video data. And the audio data and video data are synchronously processed by the synchronization controller to obtain the target audio and video data, and data synchronization can be performed by buffer management and frame loss processing of the audio data and video data. The synchronization controller is used to monitor the synchronization status of the audio data and video data in real time, and adjust them as needed to obtain the adjusted audio and video data, and the processed audio and video data can be output to the target device or storage medium through the output module.

[0068] One of the above technical solutions has the following advantages or beneficial effects: the embodiment of the present application can monitor the synchronization status of audio and video data in real time by introducing a synchronization controller, and ensure audio and video synchronization by adjusting the reading speed of the buffer and frame loss processing, thereby improving the synchronization of audio data and video data during the conversion process through the synchronization controller. In addition, the embodiment of the present application can also improve the conversion efficiency and quality of audio and video data by optimizing the encoding algorithm, so that audio and video data can be quickly converted, and the quality loss of audio and video data after conversion is reduced.

[0069] In step S101 of some embodiments, the audio and video files to be converted can be selected by obtaining a human-computer interaction interface and issuing corresponding human-computer interaction instructions. The user can select the input and output audio and video formats through the system interface, for example, the H.264 / AVC encoding format can be selected as the input format, and the H.265 / HEVC encoding format can be selected as the output format.

[0070] In step S102 of some embodiments, the target format to be converted can be determined through the human-computer interaction instruction, and the corresponding optimized encoding algorithm can be selected. It is also conceivable that the embodiment of the present application can set the conversion speed and quality parameters through the human-computer interaction instruction. If the user pays more attention to the conversion speed, the relevant parameters can be adjusted to make the system adopt a faster encoding algorithm and processing method; if the user pays more attention to the quality, the parameters can be adjusted to make the system adopt a more accurate encoding algorithm and processing method. The embodiment of the present application can also view the conversion progress and error prompts through the human-computer interaction instruction. The system will display the progress percentage of the current conversion on the interface, and prompt the user with relevant error information when an error occurs. In addition, when the user performs operations such as fast forward, fast rewind, and pause, the synchronization controller will adopt a special algorithm to ensure that the audio and video can be restored to synchronization as soon as possible after the user's operation. For example, when the user resumes playback after pausing, the synchronization controller will recalculate the audio and video timestamps and adjust the playback rate to restore synchronization. The user can also set whether to automatically skip error frames and other operations during playback through the human-computer interaction instruction.

[0071] See also Figure 2 In step S103 of some embodiments, the format conversion processing of the audio and video data to be converted according to the target format and the optimized encoding algorithm to obtain audio data and video data includes the following steps:

[0072] Step S201, decoding the audio and video data to be converted to obtain decoded data;

[0073] Step S202, encoding the decoded data according to the target format, and optimizing the encoding process through the optimized encoding algorithm to obtain the audio data and the video data, wherein the optimized encoding algorithm includes at least one of a parallel processing algorithm, a coding prediction algorithm and an adaptive quantization algorithm.

[0074] In step S201 of some embodiments, the audio and video data to be converted is decoded to obtain decoded data, wherein the audio and video data to be converted is input audio and video data, and the input audio and video data stream can be received by an audio decoder and a video decoder, and the audio and video data is decoded using a corresponding decoding algorithm according to the encoding format of the file to obtain decoded data.

[0075] In step S202 of some embodiments, the decoded audio and video data is encoded, the decoded audio and video data is received by an audio encoder and a video encoder, the audio and video data is re-encoded using a corresponding encoding algorithm according to a target format specified by a user, and the encoding process is optimized by an optimized encoding algorithm, the optimized encoding algorithm includes at least one of a parallel processing algorithm, a coding prediction algorithm and an adaptive quantization algorithm. Among them, the parallel processing algorithm is implemented by dividing the processing task of the audio and video data into multiple subtasks, and using a multi-core processor or distributed computing resources to execute these subtasks simultaneously. The coding prediction algorithm mainly analyzes the statistical characteristics of the audio and video data, and the adaptive quantization method is a technology that dynamically adjusts the quantization parameters according to the local characteristics of the audio and video data.

[0076] One of the above technical solutions has the following advantages or beneficial effects: the embodiment of the present application optimizes the encoding process by optimizing the encoding algorithm, which can improve the conversion efficiency and quality of audio and video data and realize the rapid conversion of audio and video data.

[0077] See also Figure 3 In step S202 of some embodiments, the optimized encoding algorithm adopts the parallel processing algorithm, and the encoding process is optimized by the optimized encoding algorithm to obtain the audio data and the video data, including the following steps:

[0078] Step S301, performing task segmentation processing on the decoded data to obtain subtasks;

[0079] Step S302, performing thread allocation processing on the subtasks, and performing access control and load balancing processing on the allocated threads through a thread synchronization algorithm to obtain parallel processing threads;

[0080] Step S303: encoding the decoded data according to the parallel processing thread to obtain the audio data and the video data.

[0081] In step S301 of some embodiments, by performing task segmentation processing on the decoded data, a plurality of subtasks for processing the decoded data can be obtained, wherein the task segmentation can be segmented according to time slices, or can be segmented according to frames, or can be segmented according to data blocks. There are two types of segmentation according to time slices. For audio data, segmentation can be performed according to fixed time intervals, such as every 10 milliseconds. The specific algorithm can be: set a time window size of T, for example, T = 10 milliseconds, and start from the starting position of the audio stream, and extract audio data of a time length of T each time as a subtask. For video data, segmentation according to time slices can be based on the frame rate of the video. For example, if the video frame rate is 30 frames per second, then every thirtieth second of video data can be segmented as a time slice. When the algorithm is implemented, the corresponding video frame and related data can be extracted every thirtieth second as a subtask based on the timestamp information of the video. Assume that the audio sampling rate is f s (unit: Hz), the time slice length is T (unit: s), then the number of sampling points contained in each time slice is N = f s *T. For example, the audio sampling rate f s =44100H z , the time slice length T = 0.01s, then N = 44100 × 0.01 = 441 sampling points, and the corresponding audio frames and related data are extracted through the sampling points as a subtask. Frame segmentation can directly segment the video data by each frame. In the algorithm implementation, each frame can be extracted as an independent subtask by identifying the frame header and frame tail markers in the video data. This method is simple and direct and is applicable to most video encoding formats. For example, if the video frame rate is F (unit: frame / second), then the number of frames corresponding to the time slice n = F × T, where T is the time slice length (unit: s). Data segmentation is applicable to both audio and video data. For example, for audio data, a certain number of continuous audio samples (such as 1024 samples) can be segmented as a data block. In terms of algorithm, a counter can be set to extract each accumulated 1024 samples as a data block as a subtask. For video data, a part of the image (such as a 16X16 pixel block) can be segmented as a data block. The video image can be scanned two-dimensionally, and each time a 16X16 area is encountered, it is extracted as a data block as a subtask.

[0082] In step S302 of some embodiments, multithreading technology is used to manage parallel tasks. For example, in a multi-core processor system, an independent thread is assigned to each divided task to ensure that each task is executed in an independent thread. The data exchange and status update between different threads are coordinated by a thread synchronization algorithm. For example, when a thread completes the encoding of a frame of video data, it notifies other related threads through a semaphore that subsequent operations can be performed. Thread management ensures coordination and data consistency between various subtasks, avoids data conflicts and errors, and thus improves the accuracy of encoding and decoding. The thread synchronization algorithm includes a mutex algorithm and a semaphore algorithm, wherein the mutex algorithm: when a thread needs to access a shared resource (such as data in a buffer), it first tries to acquire a mutex lock. If the lock is not occupied, the thread acquires the lock and can access the resource; if the lock is occupied, the thread enters a waiting state until the lock is released. For example, when writing to an audio buffer, the thread needs to acquire a mutex lock to ensure that only one thread can write to the buffer at the same time. Let the mutex variable be muter. When the thread needs to acquire the lock, the operation P (muter) is executed. When the thread releases the lock, the operation V (muter) is executed. Here, the P and V operations are classic semaphore operations, which correspond to the lock acquisition and release functions of different programming languages ​​in the actual code. Semaphore algorithm: Semaphores can be used to control the number of accesses to resources by multiple threads. For example, setting the value of a semaphore to N means that at most N threads can access a certain resource at the same time. When a thread needs to access the resource, it first checks the value of the semaphore. If the value of the semaphore is greater than 0, the thread decrements the value of the semaphore by 1 and can access the resource; if the value of the semaphore is equal to 0, the thread enters a waiting state until the value of the semaphore is greater than 0. In the video encoding process, semaphores can be used to control the number of threads performing encoding operations at the same time to ensure the stability and efficiency of the system. Let the semaphore variable be semaphore, with an initial value of S. When a thread needs to access a resource, it executes the following operations: if semaphore>0, then semaphore=semaphore-1, and the thread can access the resource; if semaphore=0, the thread enters a waiting state until semaphore>0. Load balancing can use a static load balancing algorithm. A simple static load balancing algorithm is average distribution. For example, if there are N threads and M subtasks, then each thread is assigned M / N subtasks. This method is suitable for situations where the processing time of subtasks is roughly the same. Another static load balancing algorithm is based on task type allocation. For example, subtasks for audio data and subtasks for video data can be assigned to different thread groups. Because the processing characteristics of audio and video data are different, this can improve processing efficiency.Load balancing can also adopt a dynamic load balancing algorithm, wherein the dynamic load balancing algorithm can include a dynamic load balancing algorithm based on load monitoring: by monitoring the load of each thread (such as the number of subtasks processed or the processing time), the distribution of subtasks is dynamically adjusted. For example, if a thread is overloaded (the number of subtasks processed exceeds a certain threshold), some subtasks are transferred from the thread to threads with lighter loads. It can also include a dynamic load balancing algorithm based on performance prediction: the distribution of subtasks is adjusted according to the prediction of the future performance of each thread. For example, by analyzing the past processing speed of the thread and the current data characteristics, it can be predicted which threads will be processed faster in the future, and then more subtasks can be allocated to these threads.

[0083] In step S303 of some embodiments, the decoded data can be encoded through parallel processing threads to obtain the audio data and the video data. The encoding algorithm can be optimized to improve the conversion efficiency and quality of the audio and video data, and the rapid conversion of the audio and video data can be achieved. The advanced audio and video processing technology is used in the conversion process to minimize the quality loss of the converted audio and video data, and support multiple input and output formats to meet the needs of different devices and platforms.

[0084] See also Figure 4 In step S202 of some embodiments, the optimized coding algorithm adopts the coding prediction algorithm, and the optimized coding algorithm is used to optimize the coding process to obtain the audio data and the video data, including the following steps:

[0085] Step S401, performing statistical characteristic analysis on the decoded data to obtain statistical information;

[0086] Step S402, performing coding demand prediction processing on the statistical information according to a machine learning algorithm or a statistical analysis algorithm to obtain coding parameters;

[0087] Step S403: Encode the decoded data according to the encoding parameters to obtain the audio data and the video data.

[0088] In an embodiment of the present application, the decoded data is subjected to statistical characteristic analysis. For video data, the pixel changes between adjacent frames, motion vectors and other statistical information are analyzed; for audio data, the frequency, amplitude and other statistical information of the audio are analyzed. Then, a machine learning algorithm or a statistical analysis method is used to predict the encoding requirements of the audio and video data. The machine learning algorithm can use a neural network, a decision tree, etc., and the statistical analysis method can use a mean, a variance and other calculations. For example, by analyzing the pixel changes of adjacent frames of the video, the encoding parameters required for the next few frames, such as the quantization step size, the encoding mode, etc., are predicted. By predicting the encoding requirements in advance, the system can pre-process the audio and video data before encoding and adjust the encoding parameters to better adapt to the data characteristics. For example, if it is predicted that the pixel changes of the next frame of video data are small, the quantization step size can be appropriately reduced to improve the encoding quality. This advance pre-processing makes the encoding process more in line with the actual situation of the data, reduces errors and redundancy in the encoding process, and thus improves the accuracy and efficiency of encoding and decoding.

[0089] See also Figure 5 In step S202 of some embodiments, the optimized coding algorithm adopts the adaptive quantization algorithm, and the encoding process is optimized by the optimized coding algorithm to obtain the audio data and the video data, including the following steps:

[0090] Step S501, performing local feature extraction processing on the decoded data to obtain feature data;

[0091] Step S502: performing data volume adjustment processing on the decoded data according to the characteristic data to obtain the audio data and the video data.

[0092] In an embodiment of the present application, the adaptive quantization method can automatically adjust the quantization step size or quantization level according to the characteristics of different video frames or audio frames to achieve the best coding effect. The embodiment of the present application uses an adaptive quantization method to extract local features of decoded data to obtain feature data, thereby adjusting the data volume of the decoded data according to the feature data. The system can reduce the amount of encoded data as much as possible while maintaining the quality of audio and video, thereby improving the coding efficiency. Specific implementations may include quantization adjustment based on image content, quantization adjustment based on audio characteristics, etc. These methods can be dynamically adjusted according to different data characteristics.

[0093] See also Figure 6 In step S104 of some embodiments, the synchronous processing of the audio data and the video data by the synchronization controller to obtain the target audio and video data includes the following steps:

[0094] Step S601, performing timestamp extraction processing on the audio data and the video data to obtain an audio frame timestamp and a video frame timestamp;

[0095] Step S602, comparing the audio frame timestamp and the video frame timestamp to obtain a timestamp difference;

[0096] Step S603: Buffer and read the audio data and the video data according to the timestamp difference to obtain the target audio and video data.

[0097] In an embodiment of the present application, the synchronization controller first obtains the timestamps of the audio frame and the video frame. These timestamps can be extracted during the decoding process, and determines whether the audio and video are synchronized by comparing the difference between the two. If the timestamp of the audio frame is equal to the timestamp of the video frame, it means that the audio and video are synchronized; if there is a difference between the two, further processing is required. Then the system sets up audio and video buffers for temporarily storing the decoded audio and video data. The synchronization controller controls the reading speed of the audio buffer and the video buffer according to the timestamp difference, thereby outputting the audio data and video data and performing synthesis processing to obtain the aligned target audio and video data.

[0098] See also Figure 7 In step S603 of some embodiments, the buffering and reading processing of the audio data and the video data according to the timestamp difference to obtain the target audio and video data includes the following steps:

[0099] Step S701, storing the audio data in an audio buffer, and storing the video data in a video buffer;

[0100] Step S702: When the timestamp difference is that the audio frame timestamp is faster than the video frame timestamp, adjusting the reading speed of the audio buffer;

[0101] Step S703, when the timestamp difference is that the audio frame timestamp is slower than the video frame timestamp, adjusting the reading speed of the video buffer or inserting an additional video frame into the video buffer;

[0102] Step S704, reading and processing the adjusted audio buffer and video buffer to obtain the target audio and video data.

[0103] In an embodiment of the present application, when audio data arrives, it is stored in an audio buffer waiting to be played; when video data arrives, it is stored in a video buffer waiting to be decoded and played. The synchronization controller dynamically adjusts the reading speed of the buffer according to the timestamp difference between audio and video. If the video data lags behind the audio data, the synchronization controller will adjust the reading speed of the audio buffer to slow down the audio playback and wait for the video data to catch up; if the video data is faster than the audio data, the synchronization controller will speed up the reading speed of the video buffer or insert additional video frames to fill the time difference between the audio and video. It is conceivable that if the audio and video synchronization cannot be maintained by comparing the timestamps and buffer management, the synchronization controller will adjust the audio or video. When adjusting the playback rate of the audio or video, the adjustment range will be determined according to the size of the time difference. For example, if the time difference is large, the adjustment range will also be correspondingly large. If the audio and video lag behind by more than a certain threshold and the previous adjustments have failed to restore synchronization, the synchronization controller will start the feedback mechanism. For audio sources, the synchronization controller will send instructions to slow down or pause audio playback until the video data catches up; for video sources, the synchronization controller will send instructions to drop frames to reduce the amount of video data so that the video can catch up with the audio as soon as possible.

[0104] The following is a detailed description of the embodiments of the present invention with reference to specific application examples:

[0105] See also Figure 8 In the embodiment of the present application, the user starts the audio and video conversion system, and the system initializes and loads the necessary components and algorithms. The components are the system interface, buttons, etc., and the algorithms are encoding algorithms, decoding algorithms, optimized encoding algorithms, etc. The user selects the audio and video files to be converted through the input module, and the system receives the files selected by the user, and imports the file data into the system, ready for subsequent processing. The input audio data is decoded by the audio decoder to extract the audio frames and their timestamps, and the input video data is decoded by the video decoder to extract the video frames and their timestamps. The user selects or specifies the target format of the output audio and video through the system interface, and the decoded audio data is re-encoded according to the target format by the audio encoder, and the decoded video data is re-encoded according to the target format by the video encoder. At this time, the timestamps of the video frames and audio frames are compared by the synchronization controller to determine the time difference between the two. According to the time difference, the synchronization controller can adjust the reading speed of the audio buffer or video buffer to ensure the synchronization of audio and video. It can also perform frame dropping or frame duplication processing through the synchronization controller to fill the time difference according to the actual situation. The processed audio and video data is output to the target device or storage medium through the output module. The user can choose to shut down the system or continue other operations. The system cleans up resources and prepares to exit.

[0106] See also Fig. 9 The embodiment of the present application also provides an audio and video data conversion device, which can implement the above-mentioned audio and video data conversion method, and the device includes:

[0107] The first module 901 is used to obtain audio and video data to be converted and human-computer interaction instructions;

[0108] The second module 902 is used to determine the target format and optimize the encoding algorithm according to the human-computer interaction instruction;

[0109] The third module 903 is used to perform format conversion processing on the audio and video data to be converted according to the target format and the optimized encoding algorithm to obtain audio data and video data;

[0110] The fourth module 904 is used to synchronously process the audio data and the video data through a synchronization controller to obtain target audio and video data.

[0111] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0112] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned audio and video data conversion method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0113] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0114] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0115] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0116] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the audio and video data conversion method of the embodiment of this application;

[0117] Input / output interface 1003, used to implement information input and output;

[0118] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0119] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0120] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0121] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned audio and video data conversion method is implemented.

[0122] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0123] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0124] The embodiment of the present application provides an audio and video data conversion method, device, electronic device and storage medium. The solution performs format conversion processing on the audio and video data to be converted according to the target format and the optimized coding algorithm to obtain audio data and video data, and can improve the conversion efficiency and quality of audio and video data by optimizing the coding algorithm, and realize the rapid conversion of audio and video data. In addition, the solution performs synchronous processing on audio data and video data through a synchronous controller to obtain target audio and video data, and can improve the synchronization of audio data and video data during the conversion process through the synchronous controller, thereby improving the user experience.

[0125] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0126] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0127] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0129] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0130] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0131] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0132] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0133] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0134] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0135] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for converting audio and video data, characterized in that: The method comprises the following steps: Obtain audio and video data to be converted and human-computer interaction instructions; Determine the target format and optimize the encoding algorithm according to the human-computer interaction instruction; Performing format conversion processing on the audio and video data to be converted according to the target format and the optimized encoding algorithm to obtain audio data and video data; The audio data and the video data are synchronously processed by a synchronization controller to obtain target audio and video data.

2. The method according to claim 1, characterized in that The step of performing format conversion processing on the audio and video data to be converted according to the target format and the optimized encoding algorithm to obtain audio data and video data comprises the following steps: Decoding the audio and video data to be converted to obtain decoded data; The decoded data is encoded according to the target format, and the encoding process is optimized by the optimized encoding algorithm to obtain the audio data and the video data, wherein the optimized encoding algorithm includes at least one of a parallel processing algorithm, a coding prediction algorithm and an adaptive quantization algorithm.

3. The method according to claim 2, characterized in that The optimized coding algorithm adopts the parallel processing algorithm, and the optimized coding algorithm is used to optimize the coding process to obtain the audio data and the video data, including the following steps: Performing task segmentation processing on the decoded data to obtain subtasks; Performing thread allocation processing on the subtasks, and performing access control and load balancing processing on the allocated threads through a thread synchronization algorithm to obtain parallel processing threads; The decoded data is encoded according to the parallel processing thread to obtain the audio data and the video data.

4. The method according to claim 2, characterized in that: The optimized coding algorithm adopts the coding prediction algorithm, and the coding process is optimized by the optimized coding algorithm to obtain the audio data and the video data, including the following steps: Performing statistical characteristic analysis on the decoded data to obtain statistical information; Perform coding demand prediction processing on the statistical information according to a machine learning algorithm or a statistical analysis algorithm to obtain coding parameters; The decoded data is encoded according to the encoding parameters to obtain the audio data and the video data.

5. The method according to claim 2, characterized in that: The optimized coding algorithm adopts the adaptive quantization algorithm, and the encoding process is optimized by the optimized coding algorithm to obtain the audio data and the video data, including the following steps: Performing local feature extraction processing on the decoded data to obtain feature data; The decoded data is processed for data volume adjustment according to the characteristic data to obtain the audio data and the video data.

6. The method according to claim 1, characterized in that The step of synchronously processing the audio data and the video data through a synchronization controller to obtain target audio and video data comprises the following steps: Performing timestamp extraction processing on the audio data and the video data to obtain an audio frame timestamp and a video frame timestamp; Comparing the audio frame timestamp with the video frame timestamp to obtain a timestamp difference; The audio data and the video data are buffered and read according to the timestamp difference to obtain the target audio and video data.

7. The method according to claim 6, characterized in that The step of performing buffer reading processing on the audio data and the video data according to the timestamp difference to obtain the target audio and video data comprises the following steps: Storing the audio data in an audio buffer and storing the video data in a video buffer; When the timestamp difference is that the audio frame timestamp is faster than the video frame timestamp, adjusting the reading speed of the audio buffer; When the timestamp difference is that the audio frame timestamp is slower than the video frame timestamp, adjusting the reading speed of the video buffer or inserting an additional video frame into the video buffer; The adjusted audio buffer and video buffer are read and processed to obtain the target audio and video data.

8. An audio and video data conversion device, characterized in that: The device comprises: The first module is used to obtain audio and video data to be converted and human-computer interaction instructions; The second module is used to determine the target format and optimize the encoding algorithm according to the human-computer interaction instruction; The third module is used to perform format conversion processing on the audio and video data to be converted according to the target format and the optimized encoding algorithm to obtain audio data and video data; The fourth module is used to synchronously process the audio data and the video data through a synchronization controller to obtain target audio and video data.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images