Data processing method and device and electronic equipment

Abstract

The invention discloses a data processing method and device and electronic equipment. The method comprises the steps that a data collection task executed on a target object is determined according to a service type, and initial data of the target object is collected according to the data collection task; determining a target enhancement model according to the business type, and preprocessing the initial data according to a data processing requirement of the target enhancement model to obtain preprocessed data; inputting the preprocessed data into a target enhancement model, processing to obtain target data, and determining attribute information of a receiving end of the target data; and transmitting the target data to a receiving end according to the attribute information. According to the method and the device, the problem that the quality and the efficiency of business handling are low due to the fact that the quality of transmitted audio and video data is low when online business is handled in the prior art is solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a data processing method, apparatus, and electronic device. Background Technology

[0002] With the development of audio and video network technology, financial institutions are gradually shifting some of their offline business to online platforms, conducting business through remote conferencing, such as promoting wealth management products and providing remote customer service consultations, thereby reducing business operating costs and improving business processing efficiency.

[0003] However, in practical applications, due to the instability of the network environment, differences in equipment hardware, and quality loss during data transmission, the clarity, smoothness, and synchronization of audio and video data transmitted when conducting online business often fail to meet ideal standards. This leads to inconvenience for users when conducting financial transactions or seeking advice, such as audio noise interference, video stuttering, and audio-video asynchrony. These issues not only reduce the quality and efficiency of financial business processing but may also negatively impact users' sense of security and trust.

[0004] There is currently no effective solution to the problem of low quality and efficiency in online business processing due to the low quality of audio and video data transmitted during online transactions. Summary of the Invention

[0005] The main objective of this application is to provide a data processing method, apparatus, and electronic device to solve the problem in the related art where the quality of audio and video data transmitted during online business processing is low, resulting in low quality and efficiency of business processing.

[0006] To achieve the above objectives, according to one aspect of this application, a data processing method is provided. The method includes: determining a data acquisition task to be performed on a target object based on a business type, and acquiring initial data of the target object according to the data acquisition task; determining a target enhancement model based on the business type, and preprocessing the initial data according to the data processing requirements of the target enhancement model to obtain preprocessed data; inputting the preprocessed data into the target enhancement model, processing it to obtain target data, and determining the attribute information of the receiving end of the target data; and transmitting the target data to the receiving end according to the attribute information.

[0007] Optionally, the initial data includes audio data or video data. Preprocessing is performed on the initial data according to the data processing requirements of the target enhancement model to obtain preprocessed data, including: if the initial data includes audio data, denoising the audio data to obtain first data, and converting the first data into a Mel spectrogram to obtain preprocessed data; if the initial data includes video data, denoising the audio portion of the video data to obtain second data, and converting the second data into a Mel spectrogram to obtain third data; determining the video size according to the data processing requirements, and adjusting the size of the video portion of the video data according to the video size to obtain fourth data; aligning the third and fourth data according to timestamps to obtain preprocessed data.

[0008] Optionally, determining the target enhancement model based on the service type includes: when the service type is voice service, a first enhancement model is determined as the target enhancement model, wherein the first enhancement model is obtained by training an initial enhancement model using a first training set, the first training set including multiple first sample data, each first sample data consisting of a first original audio and a first transmitted audio, the first transmitted audio being the audio data obtained after the first original audio has undergone a transmission operation; when the service type is video service, a second enhancement model is determined as the target enhancement model, wherein the second enhancement model is obtained by training an initial enhancement model using a second training set, the second training set including multiple second sample data, each second sample data consisting of a first original video and a first transmitted video, the first transmitted video being the video data obtained after the first original video has undergone a transmission operation.

[0009] Optionally, the initial augmentation model includes, in sequence: an input layer, a multi-head self-attention layer, a feedforward neural network layer, a residual connection and normalization layer, a decoder layer, and an output layer.

[0010] Optionally, when the preprocessed data is video data, inputting the preprocessed data into the target augmentation model includes: obtaining the video portion of the video data, and using a graph convolutional network to perform optical flow estimation on the video portion to obtain optical flow feature information of the video portion; inputting the optical flow feature information and the preprocessed data into the target augmentation model.

[0011] Optionally, determining the attribute information of the receiving end of the target data includes: obtaining the device information of the receiving end, and obtaining feature information related to the service type from the device information, wherein the feature information includes at least one of the following: video reception type, video bitrate requirement, audio format requirement and network bandwidth information; and generating attribute information based on the feature information.

[0012] Optionally, transmitting the target data to the receiving end based on the attribute information includes: determining the transmission requirements of the target data based on the attribute information; processing the target data according to the transmission requirements to obtain the data to be transmitted, and transmitting the data to be transmitted to the receiving end.

[0013] To achieve the above objectives, according to another aspect of this application, a data processing apparatus is provided. The apparatus includes: a determining unit, configured to determine a data acquisition task to be performed on a target object based on a business type, and to acquire initial data of the target object according to the data acquisition task; a first processing unit, configured to determine a target enhancement model based on the business type, and to preprocess the initial data according to the data processing requirements of the target enhancement model to obtain preprocessed data; a second processing unit, configured to input the preprocessed data into the target enhancement model, process it to obtain target data, and determine the attribute information of the receiving end of the target data; and a transmission unit, configured to transmit the target data to the receiving end according to the attribute information.

[0014] Optionally, the initial data includes audio data or video data. The first processing unit includes: a first processing module, configured to perform noise reduction processing on the audio data when the initial data includes audio data to obtain first data, and convert the first data into a Mel spectrogram to obtain preprocessed data; a second processing module, configured to perform noise reduction processing on the audio portion of the video data when the initial data includes video data to obtain second data, and convert the second data into a Mel spectrogram to obtain third data; a third processing module, configured to determine the video size according to data processing requirements, and perform size adjustment operation on the video portion of the video data according to the video size to obtain fourth data; and a fourth processing module, configured to align the third data and the fourth data according to timestamps to obtain preprocessed data.

[0015] Optionally, the first processing unit includes: a first training module, used to determine a first enhancement model as a target enhancement model when the service type is voice service, wherein the first enhancement model is obtained by training an initial enhancement model through a first training set, the first training set including multiple first sample data, each first sample data consisting of a first original audio and a first transmitted audio, the first transmitted audio being audio data obtained after the first original audio has undergone a transmission operation; and a second training module, used to determine a second enhancement model as a target enhancement model when the service type is video service, wherein the second enhancement model is obtained by training an initial enhancement model through a second training set, the second training set including multiple second sample data, each second sample data consisting of a first original video and a first transmitted video, the first transmitted video being video data obtained after the first original video has undergone a transmission operation.

[0016] Optionally, the initial augmentation model includes, in sequence: an input layer, a multi-head self-attention layer, a feedforward neural network layer, a residual connection and normalization layer, a decoder layer, and an output layer.

[0017] Optionally, when the preprocessed data is video data, the second processing unit includes: a first acquisition module, used to acquire the video portion of the video data and perform optical flow estimation on the video portion using a graph convolutional network to obtain optical flow feature information of the video portion; and an input module, used to input the optical flow feature information and the preprocessed data into the target augmentation model.

[0018] Optionally, the second processing unit includes: a second acquisition module, used to acquire device information of the receiving end and acquire feature information related to the service type from the device information, wherein the feature information includes at least one of the following: video reception type, video bitrate requirement, audio format requirement and network bandwidth information; and a generation module, used to generate attribute information based on the feature information.

[0019] Optionally, the transmission unit includes: a determining module, used to determine the transmission requirements of the target data based on attribute information; and a transmission module, used to process the target data according to the transmission requirements to obtain the data to be transmitted, and to transmit the data to be transmitted to the receiving end.

[0020] To achieve the above objectives, according to another aspect of this application, an electronic device is provided, the electronic device including a memory storing an executable program; and a processor for running the program, wherein the program executes the above-described data processing method during runtime.

[0021] To achieve the above objectives, according to another aspect of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0022] In this embodiment, the method involves determining the data acquisition task to be performed on the target object based on the business type, and acquiring the initial data of the target object according to the data acquisition task; determining the target enhancement model based on the business type, and preprocessing the initial data according to the data processing requirements of the target enhancement model to obtain preprocessed data; inputting the preprocessed data into the target enhancement model to obtain target data, and determining the attribute information of the receiving end of the target data; and transmitting the target data to the receiving end according to the attribute information. By preprocessing the initial data transmitted during online business processing, using the target enhancement model to perform data enhancement operations, and transmitting the enhanced data with high quality according to the attribute information of the receiving end, the method achieves the goal of improving the data quality of audio and video data transmitted during online business processing. This achieves the technical effect of improving the quality and efficiency of online business processing, and solves the technical problem in related technologies where the low quality of audio and video data transmitted during online business processing leads to low quality and efficiency of business processing. Attached Figure Description

[0023] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0024] Figure 1 A hardware structure block diagram of a computer terminal for implementing a data processing method is shown.

[0025] Figure 2 This is a flowchart of the data processing method provided according to Embodiment 1 of this application;

[0026] Figure 3 This is a schematic diagram of the initial enhancement model provided according to Embodiment 1 of this application;

[0027] Figure 4 This is a schematic diagram of a data processing apparatus provided according to Embodiment 2 of this application;

[0028] Figure 5 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0029] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] It should be noted that the data processing methods, apparatuses, and electronic devices defined in this disclosure can be used in the field of artificial intelligence, or in any field other than artificial intelligence. The application fields of the data processing methods, apparatuses, and electronic devices defined in this disclosure are not limited.

[0033] It should be noted that all information, user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) used in this application are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with the relevant laws, regulations, and standards of the relevant regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse use. If the user chooses to refuse, the process will proceed to the expert decision-making process. For example, this system has an interface with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface. After receiving consent from the aforementioned user or organization, the relevant information is obtained. Users can view the purpose of data use in real time through the authorization interface and have the right to withdraw authorization or delete data at any time. After the authorization is withdrawn, the system will terminate the relevant data processing within 24 hours.

[0034] The embodiments or examples disclosed herein are not exhaustive, but merely illustrative of some embodiments or examples, and are not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment or example can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment or example can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment or example can be arbitrarily interchanged. Furthermore, optional methods or examples in a particular embodiment or example can be arbitrarily combined; moreover, embodiments or examples can be arbitrarily combined. For example, some or all steps of different embodiments or examples can be arbitrarily combined, and a particular embodiment or example can be arbitrarily combined with optional methods or examples of other embodiments or examples.

[0035] Example 1

[0036] According to an embodiment of this application, an embodiment of a data processing method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0037] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware structure block diagram of a computer terminal for implementing a data processing method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, processing devices such as microprocessors or programmable logic devices), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface, a universal serial bus port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0038] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0039] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned data processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0040] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0041] The display may be, for example, a touchscreen LCD display that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0042] Under the aforementioned operating environment, this application provides the following: Figure 2 The data processing method shown. Figure 2 This is a flowchart of the data processing method provided in Embodiment 1 of this application, as follows: Figure 2 As shown, the method includes:

[0043] Step S201: Determine the data collection task to be performed on the target object according to the business type, and collect the initial data of the target object according to the data collection task.

[0044] It should be noted that the execution entity in this embodiment can be a server. The server can receive video or voice data transmitted by two devices when conducting online business, and enhance the video or voice data before transmission, thereby improving data quality.

[0045] It should be noted that the target object can be a user engaging in audio or video conversations, the service type can be an audio conversation service or a video conversation service, the data collection task can be to collect the voice information of the target object in the case of an audio conversation service, and to collect the video information of the target object in the case of a video conversation service, and the initial data can be the collected audio data or video data.

[0046] For example, when the server detects that an online live streaming session has been started, it can automatically determine the type and specifications of the data to be collected based on specific business scenario requirements (such as voice session service or video session service). The data type can be audio data or video data.

[0047] Subsequently, the server initiates the data acquisition process, capturing the raw audio waveform stream and video frame sequence from the input pipeline of the live streaming system. The initial data acquired must meet certain requirements, such as a sampling rate of 16kHz for audio and a frame rate of 15fps for video.

[0048] Step S202: Determine the target enhancement model based on the business type, and preprocess the initial data according to the data processing requirements of the target enhancement model to obtain preprocessed data.

[0049] It should be noted that the target enhancement model is a deep learning model selected based on the business type to improve audio and video quality, including speech enhancement models and video enhancement models.

[0050] For example, after determining the service type, a target enhancement model for processing the initial data can be determined based on whether the service type is an audio session service or a video session service, wherein the enhancement model for processing the audio session service is different from the enhancement model for processing the video session service.

[0051] Furthermore, the server needs to perform in-depth preprocessing on the initial data, including time-frequency domain conversion of audio, conversion of waveform signals into Mel spectrograms, and scaling and normalization of video frames, so that all data is converted into a fixed format suitable for model input. The preprocessing steps clean and format the data as necessary, enabling the model to more effectively understand and process the input information.

[0052] Step S203: Input the preprocessed data into the target augmentation model, process it to obtain the target data, and determine the attribute information of the receiving end of the target data.

[0053] For example, the preprocessed data is fed into a selected target enhancement model, which enhances the preprocessed data to obtain the target data and acquires the attribute information of the receiving end. Based on the attribute information, it is determined how to transmit the target data with high quality.

[0054] For example, when the target enhancement model is a speech enhancement model, the preprocessed audio feature data is first input into the speech enhancement model. The model captures the global dependencies of the audio feature data through a multi-head self-attention mechanism, thereby performing feature association modeling. Subsequently, the model further refines the features through a feedforward neural network layer. The introduced cross-layer skip connections enhance the robustness of denoising and signal gain, achieving residual optimization. Finally, the audio features are reconstructed by the decoder layer, outputting a clear, high-quality audio waveform, i.e., the target data.

[0055] Furthermore, the server needs to collect detailed information about the receiving device, including device type, operating system version, and audio hardware specifications, to assess its audio processing capabilities. In addition, the server will monitor the bandwidth of the network where the receiving end is located in real time, ensuring that the transmission strategy meets both the device's needs and adapts to current network conditions.

[0056] For example, when the target enhancement model is a video enhancement model, preprocessed video feature data needs to be input into the video enhancement model. The graph convolutional network in the model first analyzes the motion vectors between video frames, constructs spatiotemporal consistency constraints, and achieves video frame optimization guided by optical flow. Subsequently, a multi-head self-attention mechanism is used to integrate spectrograms and optical flow features to generate cross-modal continuous representations and deeply mine the intrinsic correlations between audio and video data. After feature optimization is completed, the model simultaneously generates high-definition video streams and noise-reduced frequency streams, outputting high-quality audio and video target data.

[0057] Furthermore, the server also needs to collect the video processing capabilities and network conditions of the receiving device, and generate attribute information including video reception type (e.g., 1080p HD), video bitrate requirement (e.g., 4Mbps), and network bandwidth threshold (e.g., 8Mbps) to guide subsequent video data transmission.

[0058] Step S204: Transmit the target data to the receiving end according to the attribute information.

[0059] For example, after determining the attribute information, the server can convert the target audio data or target video data into an adapted format according to the attribute information of the receiving end (such as device type and network environment), and dynamically adjust the data packet size according to the network bandwidth to reduce real-time transmission latency, thereby ensuring efficient transmission of audio and video data, adapting to the needs of different receiving devices, and improving the real-time performance of online business processing operations.

[0060] The data processing method provided in this application embodiment improves the data quality of audio and video data transmitted during online business processing by preprocessing the initial data of the target object according to the business type, performing data enhancement operations using the target enhancement model, and transmitting the enhanced data with high quality according to the attribute information of the receiving end. This achieves the technical effect of improving the quality and efficiency of online business processing, and solves the technical problem in related technologies where the quality of audio and video data transmitted during online business processing is low due to low quality of audio and video data transmitted during online business processing.

[0061] To improve the quality of data processed by the model, optionally, in the data processing method provided in this application embodiment, the initial data includes audio data or video data. Preprocessing the initial data according to the data processing requirements of the target augmentation model to obtain preprocessed data includes: if the initial data includes audio data, performing noise reduction processing on the audio data to obtain first data, and converting the first data into a Mel spectrogram to obtain preprocessed data; if the initial data includes video data, performing noise reduction processing on the audio portion of the video data to obtain second data, and converting the second data into a Mel spectrogram to obtain third data; determining the video size according to the data processing requirements, and performing a size adjustment operation on the video portion of the video data according to the video size to obtain fourth data; aligning the third data and the fourth data according to the timestamp to obtain preprocessed data.

[0062] It should be noted that Mel spectrograms are a form of audio feature representation, extracted by converting the audio signal to the frequency domain and applying Mel frequency cepstral coefficients. Noise reduction is the process of removing noise and improving audio or video quality using signal processing techniques. Video resizing is used to scale or crop the video to ensure all input data has a uniform size. Timestamp alignment is used to synchronize audio and video data on the timeline, facilitating joint modeling of cross-modal features.

[0063] For example, when preprocessing the initial data, the server first collects the raw audio or video data from the live streaming system, i.e., the initial data. For the audio data, background noise and abnormal frequency components are effectively removed by employing time-domain impulse detection and frequency-domain filtering techniques to achieve noise reduction. After obtaining the first data, it is converted into a Mel spectrogram, which serves as the audio feature representation for subsequent model input.

[0064] When the initial data includes video data, it needs to be split into audio and video parts. The audio part undergoes the noise reduction process mentioned above to obtain the second data, which is then further converted into a Mel-spectrum, i.e., the third data. The video part needs to be resized to obtain the fourth data to fit the model's input requirements. This operation, through uniform scaling of video frames, ensures that all input videos have the same size, facilitating model processing. For example, when processing video data, video frames may come from different devices and have varying sizes. By adjusting them to a fixed size (e.g., 224×224), the input can be standardized, improving the model's processing efficiency and accuracy.

[0065] For example, after obtaining the third and fourth data points of the video data, it is necessary to align the third and fourth data points according to their timestamps to generate the final preprocessed data. This alignment process enables the audio and video in the video data to be synchronized in the time dimension. During the alignment process, the data can be adjusted by interpolation or cropping to match the frame rates of the audio and video, achieving consistent processing of cross-modal data.

[0066] Through the above implementation methods, audio and video data in the live streaming system can be preprocessed. This not only removes noise that interferes with audio and video quality, but also makes the audio and video features conform to the input requirements of the model through conversion and adjustment. Furthermore, through alignment processing, the synchronization of audio and video data is improved, providing high-quality, consistent, and synchronized input data for subsequent audio and video quality enhancement.

[0067] For example, when preprocessing the initial data, it is necessary to collect initial data metadata (device type, transmission protocol), the original audio waveform stream (sampling rate 16kHz), and the video frame sequence (frame rate 15fps), simultaneously acquiring the audio spectrogram and video frequency domain features. For audio, temporal impulse detection, frequency domain filtering (band-stopping to eliminate abnormal frequencies), and spectral subtraction to suppress background noise are performed; for video, spatial median filtering to eliminate impulse spikes and temporal optical flow smoothing to remove motion jitter are performed to ensure data clarity. If it is a pure audio session, an audio feature space is constructed and the feature content is projected into this space; if it is a video session, the audio Mel-spectrum and video optical flow features are concatenated to generate a cross-modal spatiotemporal tensor, which is finally integrated into data features adapted to the Transformer algorithm.

[0068] This embodiment improves the matching degree between data and model by preprocessing the initial data, thereby improving the accuracy of subsequent data processing by the model.

[0069] To improve the accuracy of the model output, optionally, in the data processing method provided in this application embodiment, determining the target enhancement model according to the service type includes: when the service type is voice service, determining a first enhancement model as the target enhancement model, wherein the first enhancement model is obtained by training an initial enhancement model using a first training set, the first training set including multiple first sample data, each first sample data consisting of a first original audio and a first transmitted audio, the first transmitted audio being audio data obtained after the first original audio has undergone a transmission operation; when the service type is video service, determining a second enhancement model as the target enhancement model, wherein the second enhancement model is obtained by training an initial enhancement model using a second training set, the second training set including multiple second sample data, each second sample data consisting of a first original video and a first transmitted video, the first transmitted video being video data obtained after the first original video has undergone a transmission operation.

[0070] For example, since different models are needed to process audio data and video data, when training the models, a first augmentation model for processing audio data and a second augmentation model for processing video data need to be trained separately.

[0071] For example, when training the first enhancement model, a first training set is required, where each set of first sample data consists of a first original audio and a corresponding first transmitted audio. The model learns how to recover key features of audio quality after transmission by comparing and analyzing the differences between these two types of audio data. Taking audio noise reduction as an example, during the training phase, the model focuses on identifying and simulating the clean characteristics of high-fidelity audio, while understanding the noise patterns generated during transmission. This enables the model to accurately remove background noise, echoes, and other common audio contaminants in communication during real-time processing, improving the user's call experience. When the model receives degraded audio input, it uses its learned knowledge to reconstruct the spectral characteristics of the audio signal, outputting clear and clean sound.

[0072] For example, when training the second enhancement model, a second training set is required, where each set of second sample data consists of the first original video and its transmitted first video. By comparing the differences between the original video and the transmitted video, the model learns how to compensate for and optimize the image quality degradation caused by factors such as network latency and compression distortion. In practical applications, the model can effectively identify and reduce image noise, restore detail clarity, and, through audio-video synchronization processing, ensure that the sound and image are consistent, avoiding lip-sync issues and improving the video quality of the output video.

[0073] This embodiment trains the initial augmentation model using different training sets, enabling accurate augmentation operations to be performed on different initial data, thus improving the accuracy of augmentation operations on the initial data.

[0074] Optionally, in the data processing method provided in the embodiments of this application, the initial enhancement model sequentially includes: an input layer, a multi-head self-attention layer, a feedforward neural network layer, a residual link and normalization layer, a decoder layer, and an output layer.

[0075] For example, Figure 3 This is a schematic diagram of the initial enhancement model provided in Embodiment 1 of this application, as shown below. Figure 3 As shown, the initial enhancement model used in this embodiment includes: an input layer, a multi-head self-attention layer, a feedforward neural network layer, a residual connection and normalization layer, a decoder layer, and an output layer, wherein:

[0076] The input layer receives preprocessed audio and video data, including Mel spectrograms (audio features) and adjusted video frames (video features). The main function of the input layer is to convert raw data from different sources and of different types into tensor forms that the model can understand, preparing it for subsequent processing. For example, when processing audio from a live financial broadcast, the Mel spectrogram is converted into a fixed-size matrix as input to the model, while the video frames need to be adjusted to a uniform size, also forming tensors for model analysis.

[0077] After receiving preprocessed data, the model captures and establishes long-range dependencies within the data through a multi-head self-attention layer. The multi-head self-attention layer takes a Mel-spectrum matrix or video frame tensor as input and analyzes the correlations between features from different perspectives through multiple parallel attention heads. It ultimately outputs a weighted and fused feature representation, thereby improving the model's flexibility and robustness in audio and video reconstruction tasks, enabling it to simultaneously focus on local details and global structure. Taking audio as an example, the multi-head self-attention layer effectively identifies and enhances the distinction between human voices and other background noise, laying a solid foundation for high-quality audio reconstruction.

[0078] The feedforward neural network layer, located after the multi-head self-attention layer, performs non-linear transformations and further extractions on the features output by the attention layer to enhance their expressive power. The feedforward neural network layer receives the feature representation output by the attention layer, processes it through a series of fully connected layers, and outputs a deeper feature representation, laying the groundwork for subsequent reconstruction tasks. In video quality improvement, the feedforward layer can enhance the model's understanding of details, such as facial expressions and background textures, resulting in more detailed and realistic video output.

[0079] The residual linking and normalization layer improves the stability of model training and accelerates convergence by introducing residual links and normalization operations, while maintaining a suitable feature scale for subsequent decoding. The residual linking and normalization layer receives features from the feedforward neural network layer, combines the original input with the processed features through residual links, and then normalizes them to output the final feature representation to be decoded. This mechanism is beneficial for handling scale inconsistencies that may exist in audio and video data, enabling deep learning models to effectively perform end-to-end learning.

[0080] The decoder layer transforms the feature representations output from the residual link and normalization layer into audible or visual audio and video signals. The input to the decoder layer is the normalized features. In audio quality enhancement tasks, it outputs a reconstructed Mel-spectrum, which is then converted into an audio waveform through operations such as inverse Fourier transform. In video quality enhancement, it generates a sequence of high-definition video frames. The decoding process emphasizes feature restoration and reconstruction capabilities, resulting in output audio and video quality close to the original high-standard data.

[0081] The output layer, as the end of the model, is used to convert the output of the decoder layer into the final audio and video stream. For audio, the output layer produces a clean and clear audio waveform, and for video, it generates high-definition and smooth video frames.

[0082] This embodiment sets up a six-layer structure for the initial enhancement model to achieve end-to-end audio and video quality improvement from input preprocessing, feature extraction, optimization to decoding output, thereby improving the accuracy of the model's output results.

[0083] To improve the accuracy of video data processing results, optionally, in the data processing method provided in this application embodiment, when the preprocessed data is video data, inputting the preprocessed data into the target augmentation model includes: obtaining the video portion of the video data, and using a graph convolutional network to perform optical flow estimation on the video portion to obtain optical flow feature information of the video portion; inputting the optical flow feature information and the preprocessed data into the target augmentation model.

[0084] It should be noted that optical flow estimation is a computation performed on a video frame sequence using a graph convolutional network to infer the motion direction and velocity of pixels between frames. Optical flow feature information is a set of features extracted from the video frame sequence that reflects the motion trend of pixels.

[0085] For example, when processing video data, it is also necessary to separate the video portion from the preprocessed data. This video portion has undergone uniform scaling and standardization, and an optical flow estimation operation is performed on the video frame sequence using a graph convolutional network. This process involves decomposing the video into time-series image frames, and then using the neighborhood aggregation capability of the graph convolutional network to analyze the displacement of pixels between adjacent frames. Specifically, the graph convolutional network establishes "edges" between video frames, processes each pixel as a node, and calculates the motion vector between pixels. By using the graph convolutional network to perform in-depth analysis of video frames, optical flow feature information reflecting the motion and changes of objects is extracted, thereby correcting motion blur, enhancing detail clarity, and improving video quality.

[0086] Furthermore, after completing the optical flow estimation operation, the obtained optical flow feature information can be compared with the Mel spectrum of the audio component. Figure 1 As input to the target enhancement model, cross-modal data fusion enables the model to comprehensively understand the conversation content from both audiovisual and visual dimensions, thereby improving its overall ability to process audio and video data.

[0087] This embodiment utilizes graph convolutional networks to perform optical flow estimation, capturing subtle motion changes between video frames. The obtained optical flow feature information is then integrated with the Mel spectrogram of the audio portion and used as input to a deep learning model, thereby improving the accuracy of the model's output.

[0088] To accurately determine attribute information, optionally, in the data processing method provided in this application embodiment, determining the attribute information of the receiving end of the target data includes: obtaining the device information of the receiving end, and obtaining feature information related to the service type from the device information, wherein the feature information includes at least one of the following: video reception type, video bitrate requirement, audio format requirement, and network bandwidth information; and generating attribute information based on the feature information.

[0089] It should be noted that video reception type refers to the video formats that the receiving end can process, such as standard definition, high definition, and ultra-high definition. Video bitrate requirement refers to the video data transmission rate requirements of the receiving device or network environment, usually measured in Mbps or Kbps. Audio format requirement refers to the audio encoding format preferred by the receiving device. Network bandwidth information refers to the available bandwidth of the network where the receiving device is located, monitored in real time, used to dynamically adjust data transmission strategies.

[0090] For example, when obtaining attribute information, it is first necessary to actively request and receive device information from the receiving end of the current session. This information covers hardware details such as device brand, model, operating system version, camera and microphone specifications, as well as software-level information such as browser type and software compatibility.

[0091] Furthermore, it is necessary to filter out feature information related to the current service type from the collected device information. Based on preset feature matching rules, the specific requirements of the receiving device in terms of audio and video reception can be identified. For example, for video services, the server will focus on checking information such as video reception type and bitrate requirements to determine the transmission format and speed.

[0092] Finally, the server comprehensively analyzes the selected feature information and generates a complete set of attribute information to guide subsequent target data transmission.

[0093] This embodiment identifies and matches the device characteristics and service requirements of the receiving end, dynamically generates and applies attribute information, and achieves the effect of accurately determining attribute information, thereby improving data transmission efficiency.

[0094] To improve data transmission efficiency, optionally, in the data processing method provided in the embodiments of this application, transmitting the target data to the receiving end according to the attribute information includes: determining the transmission requirements of the target data according to the attribute information; processing the target data according to the transmission requirements to obtain the data to be transmitted, and transmitting the data to be transmitted to the receiving end.

[0095] For example, when transmitting target data, the capabilities and preferences of the receiving device, as well as the real-time network status, are first analyzed based on key parameters in the attribute information to determine the transmission requirements of the target data. This process involves a comprehensive evaluation of factors such as video reception type, video and audio encoding formats, and network bandwidth thresholds to determine the most suitable transmission parameters.

[0096] Furthermore, the target data needs to be format-converted and encoded to optimize its performance based on transmission requirements. For example, if the transmission requirements specify audio format A and video needs to be compressed to H.265 encoding, the server will perform the corresponding encoding conversions on the audio and video data. In addition, to adapt to network bandwidth information, the server may adjust the size of data packets and the transmission rate to maintain stable data transmission even during network fluctuations. This processing flow not only needs to consider the encoding efficiency and fidelity of the audio and video data but also the decoding capabilities of the receiving device to achieve the best transmission effect.

[0097] After processing, the data to be transmitted is sent to the receiving end. This transmission process follows the encoding format and transmission rate set in the transmission requirements, ensuring that the audio and video data are delivered to the receiving end in the most suitable way. Simultaneously, the server can monitor network status in real time and dynamically adjust transmission strategies to cope with potential network fluctuations. For example, if network bandwidth information shows a slight decrease at a certain moment, the video bitrate can be immediately adjusted to 3Mbps to maintain smooth video playback and avoid stuttering or delays.

[0098] This embodiment improves data transmission efficiency and thus data quality by processing the target data according to transmission requirements before transmission.

[0099] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0100] Example 2

[0101] This application also provides a data processing apparatus. It should be noted that the data processing apparatus of this application can be used to execute the data processing methods provided in the above embodiments. The data processing apparatus provided in this application will be described below.

[0102] According to an embodiment of this application, an apparatus for implementing the above-described data processing method is also provided. Figure 4 This is a schematic diagram of a data processing apparatus according to Embodiment 2 of this application, as shown below. Figure 4 As shown, the device includes:

[0103] The determining unit 41 is used to determine the data collection task to be performed on the target object according to the business type, and to collect the initial data of the target object according to the data collection task.

[0104] The first processing unit 42 is used to determine the target enhancement model according to the business type, and to preprocess the initial data according to the data processing requirements of the target enhancement model to obtain preprocessed data.

[0105] The second processing unit 43 is used to input the preprocessed data into the target augmentation model, process it to obtain the target data, and determine the attribute information of the receiving end of the target data.

[0106] The transmission unit 44 is used to transmit the target data to the receiving end according to the attribute information.

[0107] The data processing apparatus provided in this application embodiment determines the data acquisition task to be performed on the target object according to the business type by the determining unit 41, and acquires the initial data of the target object according to the data acquisition task; the first processing unit 42 determines the target enhancement model according to the business type, and preprocesses the initial data according to the data processing requirements of the target enhancement model to obtain preprocessed data; the second processing unit 43 inputs the preprocessed data into the target enhancement model, processes it to obtain target data, and determines the attribute information of the receiving end of the target data; the transmission unit 44 transmits the target data to the receiving end according to the attribute information. By preprocessing the initial data transmitted during online business processing, using the target enhancement model to perform data enhancement operations, and transmitting the enhanced data with high quality according to the attribute information of the receiving end, the data quality of audio and video data transmitted during online business processing is improved, thereby achieving the technical effect of improving the quality and efficiency of online business processing, and solving the technical problem in related technologies where the quality of audio and video data transmitted during online business processing is low, resulting in low quality and efficiency of business processing.

[0108] Optionally, in the data processing apparatus provided in this application embodiment, the initial data includes audio data or video data. The first processing unit 42 includes: a first processing module, configured to perform noise reduction processing on the audio data when the initial data includes audio data to obtain first data, and convert the first data into a Mel spectrogram to obtain preprocessed data; a second processing module, configured to perform noise reduction processing on the audio portion of the video data when the initial data includes video data to obtain second data, and convert the second data into a Mel spectrogram to obtain third data; a third processing module, configured to determine the video size according to data processing requirements, and perform size adjustment operation on the video portion of the video data according to the video size to obtain fourth data; and a fourth processing module, configured to align the third data and the fourth data according to timestamps to obtain preprocessed data.

[0109] Optionally, in the data processing apparatus provided in this application embodiment, the first processing unit 42 includes: a first training module, used to determine a first enhancement model as a target enhancement model when the service type is voice service, wherein the first enhancement model is obtained by training an initial enhancement model through a first training set, the first training set including multiple first sample data, each first sample data consisting of a first original audio and a first transmitted audio, the first transmitted audio being audio data obtained after the first original audio has undergone a transmission operation; and a second training module, used to determine a second enhancement model as a target enhancement model when the service type is video service, wherein the second enhancement model is obtained by training an initial enhancement model through a second training set, the second training set including multiple second sample data, each second sample data consisting of a first original video and a first transmitted video, the first transmitted video being video data obtained after the first original video has undergone a transmission operation.

[0110] Optionally, in the data processing apparatus provided in the embodiments of this application, the initial enhancement model sequentially includes: an input layer, a multi-head self-attention layer, a feedforward neural network layer, a residual link and normalization layer, a decoder layer, and an output layer.

[0111] Optionally, in the data processing apparatus provided in this application embodiment, when the preprocessed data is video data, the second processing unit 43 includes: a first acquisition module, used to acquire the video portion of the video data and use a graph convolutional network to perform optical flow estimation on the video portion to obtain optical flow feature information of the video portion; and an input module, used to input the optical flow feature information and the preprocessed data into the target enhancement model.

[0112] Optionally, in the data processing apparatus provided in this application embodiment, the second processing unit 43 includes: a second acquisition module, used to acquire device information of the receiving end, and acquire feature information related to the service type from the device information, wherein the feature information includes at least one of the following: video reception type, video bitrate requirement, audio format requirement and network bandwidth information; and a generation module, used to generate attribute information based on the feature information.

[0113] Optionally, in the data processing apparatus provided in this application embodiment, the transmission unit 44 includes: a determining module, used to determine the transmission requirements of the target data according to attribute information; and a transmission module, used to process the target data according to the transmission requirements to obtain the data to be transmitted, and transmit the data to be transmitted to the receiving end.

[0114] It should be noted that the aforementioned determining unit 41, first processing unit 42, second processing unit 43, and transmission unit 44 correspond to steps S201 to S204 in Embodiment 1. The instances and application scenarios implemented by each of these units and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should also be noted that the aforementioned modules or units can be hardware or software components stored in a memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). These modules can also run as part of a device in the computer terminal 10 provided in Embodiment 1.

[0115] Example 3

[0116] Embodiments of this application may provide an electronic device. Figure 5 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 5 As shown, the electronic device may include: one or more ( Figure 5 (Only one is shown) processor 1002, memory 1004, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0117] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the above-described methods. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0118] Those skilled in the art will understand that Figure 5 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 5 This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0119] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0120] Example 4

[0121] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data processing method provided in Embodiment 1.

[0122] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0123] Embodiments of this application also provide a computer program product, which, when executed on a data processing device, is a program adapted to perform the steps of a data processing method.

[0124] Embodiments of this application also provide a computer-readable storage medium, which includes a stored executable program, wherein the executable program controls the device where the computer-readable storage medium is located to perform the above-described data processing method when it runs.

[0125] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0126] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0128] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0130] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0131] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: The data collection task to be performed on the target object is determined according to the business type, and the initial data of the target object is collected according to the data collection task; The target enhancement model is determined based on the business type, and the initial data is preprocessed according to the data processing requirements of the target enhancement model to obtain preprocessed data. The preprocessed data is input into the target enhancement model, processed to obtain target data, and the attribute information of the receiving end of the target data is determined. The target data is transmitted to the receiving end based on the attribute information.

2. The method according to claim 1, characterized in that, The initial data includes audio data or video data. The initial data is preprocessed according to the data processing requirements of the target enhancement model to obtain preprocessed data, including: If the initial data includes the audio data, the audio data is subjected to noise reduction processing to obtain first data, and the first data is converted into a Mel spectrogram to obtain the preprocessed data; If the initial data includes the video data, the audio portion of the video data is denoised to obtain the second data, and the second data is converted into a Mel spectrogram to obtain the third data. The video size is determined according to the data processing requirements, and the video portion in the video data is resized according to the video size to obtain the fourth data. The third and fourth data are aligned according to their timestamps to obtain the preprocessed data.

3. The method according to claim 1, characterized in that, Determining the target enhancement model based on the aforementioned business type includes: When the service type is voice service, the first enhancement model is determined as the target enhancement model. The first enhancement model is obtained by training the initial enhancement model with a first training set. The first training set includes multiple first sample data. Each first sample data consists of a first original audio and a first transmitted audio. The first transmitted audio is the audio data obtained after the first original audio has undergone a transmission operation. When the service type is video service, the second enhancement model is determined as the target enhancement model. The second enhancement model is obtained by training the initial enhancement model with a second training set. The second training set includes multiple second sample data. Each second sample data consists of a first original video and a first transmitted video. The first transmitted video is the video data obtained after the first original video has undergone a transmission operation.

4. The method according to claim 3, characterized in that, The initial enhancement model comprises, in sequence: an input layer, a multi-head self-attention layer, a feedforward neural network layer, a residual connection and normalization layer, a decoder layer, and an output layer.

5. The method according to claim 1, characterized in that, When the preprocessed data is video data, inputting the preprocessed data into the target augmentation model includes: The video portion of the video data is obtained, and an optical flow estimation operation is performed on the video portion using a graph convolutional network to obtain the optical flow feature information of the video portion. The optical flow feature information and the preprocessed data are input into the target enhancement model.

6. The method according to claim 1, characterized in that, Determining the attribute information of the receiving end of the target data includes: Obtain the device information of the receiving end, and obtain feature information related to the service type from the device information, wherein the feature information includes at least one of the following: video reception type, video bitrate requirement, audio format requirement, and network bandwidth information; The attribute information is generated based on the feature information.

7. The method according to claim 1, characterized in that, Transmitting the target data to the receiving end based on the attribute information includes: The transmission requirements of the target data are determined based on the attribute information; The target data is processed according to the transmission requirements to obtain the data to be transmitted, and the data to be transmitted is transmitted to the receiving end.

8. A data processing apparatus, characterized in that, include: The determining unit is used to determine the data collection task to be performed on the target object according to the business type, and to collect the initial data of the target object according to the data collection task; The first processing unit is used to determine the target enhancement model according to the business type, and to preprocess the initial data according to the data processing requirements of the target enhancement model to obtain preprocessed data; The second processing unit is used to input the preprocessed data into the target enhancement model, process it to obtain target data, and determine the attribute information of the receiving end of the target data; A transmission unit is used to transmit the target data to the receiving end according to the attribute information.

9. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the data processing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the data processing method according to any one of claims 1 to 7.

Images