A data processing method, device, apparatus, and readable storage medium

Abstract

The application discloses a data processing method, device and equipment and a readable storage medium. The method comprises the following steps: receiving a playing request for multimedia data sent by a terminal device, obtaining real-time network layer data and real-time application layer data of the multimedia data based on the playing request; performing online training on a code rate prediction enhancement model according to the real-time network layer data and the real-time application layer data, and obtaining a target code rate prediction enhancement model; determining a media prediction code rate of the multimedia data through the target code rate prediction enhancement model, the real-time network layer data and the real-time application layer data, and performing transcoding on the multimedia data based on the media prediction code rate. The application can improve the code rate prediction accuracy and optimize the presentation effect of the multimedia data. The application can be applied to cloud technology, artificial intelligence, intelligent transportation, assisted driving and the like.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, and readable storage medium. Background Technology

[0002] With the development of broadcasting technology and the application of online media, multimedia data (such as video) has become an important part of people's daily lives, and people use multimedia data for entertainment or learning.

[0003] In related technologies, to better present multimedia data, artificial intelligence models are mainly used to predict the bitrate, and then the multimedia data is transcoded based on the bitrate. Thus, when playing multimedia data, the transcoded data is presented. For the artificial intelligence models, current methods primarily use sample training data to train the models offline. However, due to the rapid development of the network and the fast pace of data updates, offline-trained models are likely to lag, failing to predict the accurate bitrate for multimedia data in a timely manner based on the latest relevant data, resulting in poor multimedia data presentation. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, and readable storage medium, which can improve the accuracy of bitrate prediction and optimize the presentation effect of multimedia data.

[0005] One embodiment of this application provides a data processing method, including: Receive playback requests for multimedia data sent by terminal devices, and obtain real-time network layer data and real-time application layer data of multimedia data based on the playback requests; Based on real-time network layer data and real-time application layer data, the bitrate prediction enhancement model is trained online to obtain the target bitrate prediction enhancement model. The bitrate prediction enhancement model refers to the model obtained by expanding the depth and width of the network structure of the bitrate prediction model that has been trained offline. The media prediction bitrate of multimedia data is determined by using a target bitrate prediction enhancement model, real-time network layer data, and real-time application layer data, and then the multimedia data is transcoded based on the media prediction bitrate.

[0006] This application provides another data processing method, including: Receive playback requests for multimedia data sent by terminal devices, and obtain real-time network layer data and real-time application layer data of multimedia data based on the playback requests; Real-time network layer data and real-time application layer data are input into the offline-trained bitrate prediction model. The initial media prediction bitrate of the multimedia data is determined by the offline-trained bitrate prediction model, real-time network layer data, and real-time application layer data. The real-time quality assessment feedback value of the multimedia data at the first moment is determined based on the initial media prediction bitrate; the first moment refers to the moment when the initial media prediction bitrate is determined. The prediction quality assessment feedback value of the multimedia data at the second time step is obtained. Based on the real-time quality assessment feedback value and the prediction quality assessment feedback value, the initial media prediction bitrate is optimized to obtain the media optimized bitrate of the multimedia data. The second time step is later than the first time step. Multimedia data transcoding based on media-optimized bitrate.

[0007] One embodiment of this application provides a data processing apparatus, including: The acquisition module is used to receive playback requests for multimedia data sent by terminal devices, and to acquire real-time network layer data and real-time application layer data of multimedia data based on the playback requests. The training module is used to train the bitrate prediction enhancement model online based on real-time network layer data and real-time application layer data to obtain the target bitrate prediction enhancement model. The bitrate prediction enhancement model refers to the model obtained by expanding the depth and width of the network structure of the bitrate prediction model that has been trained offline. The prediction module is used to determine the media prediction bitrate of multimedia data by using the target bitrate prediction enhancement model, real-time network layer data, and real-time application layer data. The transcoding module is used to transcode multimedia data based on media prediction bitrate.

[0008] One embodiment of this application provides another data processing apparatus, including: The acquisition module is used to receive playback requests for multimedia data sent by terminal devices, and to acquire real-time network layer data and real-time application layer data of multimedia data based on the playback requests. The prediction module is used to input real-time network layer data and real-time application layer data into the offline-trained bitrate prediction model, and determine the initial media prediction bitrate of the multimedia data through the offline-trained bitrate prediction model, real-time network layer data and real-time application layer data. The determining module is further configured to determine the real-time quality assessment feedback value of the multimedia data at the first moment based on the initial media prediction bitrate; the first moment refers to the moment when the initial media prediction bitrate is determined. The acquisition module is also used to acquire the prediction quality assessment feedback value of the multimedia data at the second moment; The optimization module is used to optimize the initial media prediction bitrate based on the real-time quality assessment feedback value and the predicted quality assessment feedback value to obtain the media optimized bitrate of the multimedia data; the second moment is later than the first moment. The transcoding module is used to transcode multimedia data based on media-optimized bitrates.

[0009] One embodiment of this application provides a computer device, including: a processor and a memory; The memory stores a computer program, which, when executed by a processor, causes the processor to perform the methods described in the embodiments of this application.

[0010] One aspect of this application provides a computer-readable storage medium storing a computer program, which includes program instructions. When executed by a processor, the program instructions perform the methods described in this application.

[0011] One aspect of this application provides a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the method provided in one aspect of the embodiments of this application.

[0012] In this embodiment, when playing multimedia data, a pre-trained offline bitrate prediction model can be used to adaptively adjust the bitrate of the multimedia data online. Specifically, this application can pre-train a bitrate prediction model offline. In real-time applications, if a playback request for multimedia data is received from a terminal device, the network layer data and application layer data of the multimedia data can be obtained. Based on this network layer data and application layer data, the model with an expanded network structure is trained online, and then bitrate prediction is performed based on the online-trained model. By adopting a combination of offline and online methods, real-time model training can be performed based on real-time data. The trained model can respond well to changes and updates in real-time data, and the output bitrate is adapted to real-time network layer data and real-time application layer data. At the same time, since online training is based on a model that has already been trained offline, training time can be reduced. In summary, this application can improve the accuracy of bitrate prediction and optimize the presentation effect of multimedia data. Attached Figure Description

[0013] Figure 1 This is a network architecture diagram provided in an embodiment of this application; Figure 2 This is a schematic diagram of a scenario provided in an embodiment of this application; Figure 3This is a flowchart illustrating a data processing method provided in an embodiment of this application; Figure 4 This is a communication architecture diagram for communication between devices provided in an embodiment of this application; Figure 5 This is a schematic diagram of a model extension provided in an embodiment of this application; Figure 6 This is a schematic diagram of the model training process provided in the embodiments of this application; Figure 7 This is a schematic diagram of a bitrate optimization process provided in an embodiment of this application; Figure 8 This is a flowchart illustrating another data processing method provided in an embodiment of this application; Figure 9 This is a system flowchart provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0014] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0015] The methods involved in the embodiments of this application can be applied to the field of intelligent transportation. For ease of understanding, intelligent transportation and its related concepts will be explained first below.

[0016] Intelligent Vehicle Infrastructure Cooperative Systems (IVICS) are a development direction of Intelligent Transportation Systems (ITS). IVICS utilizes advanced wireless communication and next-generation Internet technologies to implement comprehensive, real-time dynamic information exchange between vehicles and infrastructure. Based on the collection and fusion of dynamic traffic information across all times and spaces, it conducts active vehicle safety control and cooperative road management, fully realizing effective collaboration between people, vehicles, and roads. This ensures traffic safety, improves traffic efficiency, and ultimately forms a safe, efficient, and environmentally friendly road traffic system.

[0017] Intelligent Traffic Systems (ITS), also known as Intelligent Transportation Systems, effectively integrate advanced science and technology (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operations research, artificial intelligence, etc.) into transportation, service control, and vehicle manufacturing. This strengthens the connection between vehicles, roads, and users, thereby forming a comprehensive transportation system that ensures safety, improves efficiency, enhances the environment, and saves energy.

[0018] This application also relates to artificial intelligence and related technologies. For ease of understanding, the concepts of artificial intelligence and the like will be explained below.

[0019] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0020] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0021] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0022] The solutions provided in this application involve technologies such as machine learning (ML) in artificial intelligence.

[0023] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0024] For easier understanding, please refer to Figure 1 , Figure 1 This is a network architecture diagram provided in an embodiment of this application. For example... Figure 1 As shown, this network architecture may include a service server 1000 and a terminal device cluster. The terminal device cluster may include one or more terminal devices; the number of terminal devices is not limited here. Figure 1 As shown, multiple terminal devices may include terminal device 100a, terminal device 100b, terminal device 100c, ..., terminal device 100n; as Figure 1 As shown, terminal devices 100a, 100b, 100c, ..., 100n can each connect to the business server 1000 via a network, so that each terminal device can interact with the business server 1000 through the network connection.

[0025] It is understandable that, such as Figure 1 Each terminal device shown can have the target application installed. When the target application runs on each terminal device, it can interact with... Figure 1 The business servers 1000 shown interact with each other, enabling each business server 1000 to receive business data from each terminal device. The target application can include applications capable of displaying text, images, audio, and video data. These applications can include multimedia applications (e.g., video applications), entertainment applications (e.g., game applications), educational applications, etc., all of which can load and play multimedia data (e.g., video and music). For example, the application can be a social application, an educational application, a short video application, a game application, etc. The application can be a standalone application, installed on the terminal device and running on the terminal device's current operating system; it can also be an embedded sub-application within an application (such as a social application, an educational application, etc.). The form of the application is not limited here.

[0026] This application embodiment can select one terminal device from multiple terminal devices as the target terminal device. This terminal device may include: smartphones, tablets, laptops, desktop computers, smart TVs, smart speakers, desktop computers, smartwatches, smart in-vehicle devices, smart voice interaction devices, smart home appliances, and other smart terminals with multimedia data processing functions (e.g., video playback, music playback, text playback). However, it is not limited to these. For example, this application embodiment can... Figure 1 The terminal device 100a shown serves as the target terminal device. This target terminal device can integrate the aforementioned target application. In this case, the target terminal device can interact with the business server 1000 through the target application. The business server 1000 in this application can obtain business data based on these applications. For example, the business server 1000 can obtain business data through a user's bound account. Here, the bound account can refer to the account bound by the user in the application; the user can log in to the application, upload data, retrieve data, etc., through their corresponding bound account, and the business server can also obtain the user's login status, uploaded data, send data to the user, etc., through its bound account.

[0027] It should be understood that the business server 1000 may deploy a model for bitrate prediction. This model can refer to a model obtained by expanding both the width and depth of the network structure of a pre-trained bitrate prediction model (which can be called a bitrate prediction enhancement model). When a user plays multimedia data (such as video) through a target application (such as a video application) on a terminal device, the terminal device can generate a playback request for the multimedia data and collect network layer data (e.g., latency, data rate, modulation and coding strategies, which can be called real-time application layer data) and application layer data (e.g., download speed, download duration, buffer size, etc., which can be called real-time network layer data) in real time. Subsequently, the terminal device can send the playback request, the real-time network layer data, and the real-time application layer data to the business server 1000. The business server 1000 can perform real-time online training of the bitrate prediction enhancement model based on the real-time network layer data and the real-time application layer data. After training, the trained bitrate prediction enhancement model can be called the target bitrate prediction enhancement model. Furthermore, the service server 1000 can input the real-time network layer data and the real-time application layer data into the target bitrate prediction enhancement model. Through the target bitrate prediction enhancement model, a bitrate adapted to the real-time network layer data and the real-time application layer data (which can be called the media prediction bitrate) can be output. Based on the media prediction bitrate, the multimedia data can be transcoded. After obtaining the transcoded media data, the service server 1000 can return the transcoded media data to the terminal device. The terminal device can play the transcoded media data, and what the user sees is the transcoded media data.

[0028] For details on offline training of the bitrate prediction model and online training of the bitrate prediction enhancement model, please refer to the following sections. Figure 3 The description in the corresponding embodiments.

[0029] It should be understood that this application adopts a combination of offline and online methods, which allows the offline-trained model to be trained in real time based on real-time updated data. This enables the model to respond promptly to data changes, predict the bitrate that is suitable for both network layer data and application layer data, improve the bitrate prediction accuracy, and thus enhance the presentation effect of media data. At the same time, during online training, the network structure of the offline-trained model is expanded, making the model's network structure wider and deeper, resulting in higher accuracy. Furthermore, since this application trains on the basis of the offline-trained model, it can reduce training time and improve training efficiency.

[0030] It is understood that the methods provided in this application embodiment can be executed by computer devices, including but not limited to user terminals or business servers. The business server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0031] The user terminal and the business server can be connected directly or indirectly through wired or wireless communication, and this application does not impose any restrictions on this.

[0032] It should be noted that, in the specific embodiments of this application, data related to user information and user data (such as multimedia data played, user-uploaded data, etc.) requires user authorization before acquisition. In other words, when the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0033] The embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving. For ease of understanding, please refer to 2. Figure 2 This is a schematic diagram of a scenario provided in an embodiment of this application. Wherein, as... Figure 2 The terminal device shown can be used in the above-mentioned Figure 1 Any terminal device selected in the terminal device cluster of the corresponding embodiment, for example, the terminal device can be the aforementioned terminal device 100b; such as Figure 2 The service server shown can be one of the above-mentioned... Figure 1 The corresponding embodiment of the business server 1000 includes, for example, Figure 2 The terminal device shown is equipped with a data acquisition component.

[0034] like Figure 2 As shown, a target user (e.g., user m, hereinafter referred to as object m) logs into a target application (e.g., a video application) through terminal device 100b. Object m expects to watch the first episode of video A (the first episode of video A can be referred to as multimedia data). Figure 2After the playback control 2a generates a trigger operation, the terminal device 100b can respond to this trigger operation, generate a playback request for the first episode of video A, and collect network layer data and application layer data at the current moment through the data acquisition component. The network layer and application layer refer to two layers in the terminal device's data transmission system. The data transmission system of the terminal device can include the physical layer, data link layer, network layer, transport layer, session layer, presentation layer, application layer, etc., each with different functions. For example, the physical layer is mainly used for the transmission of raw bit streams on mechanical, electronic, and timing interface communication channels; the data link layer... The data link layer is mainly used for physical addressing and transforms the raw bit stream into logical transmission lines; the network layer is mainly used to control the operation of the subnet, such as logical addressing, packet transmission, and routing; the transport layer is mainly used to receive data from the upper layer, segment the data when necessary, and hand it over to the network layer, ensuring that these data segments effectively reach the other end; the session layer is mainly used to establish and manage sessions between users on different machines; the presentation layer is mainly used for the syntax and semantics of information, such as encryption and decryption, conversion and translation, compression and decompression; the application layer mainly includes various application protocols (such as HTTP protocol, FTP protocol, etc.).

[0035] In this embodiment, the network layer data may include, but is not limited to, Reference Signal Receiving Power (RSRP), Received Signal Strength Indicator (RSSI), Reference Signal Receiving Quality (RSRQ), Signal-to-Interference Plus Noise Ratio (SINR), latency, data rate, physical layer downlink shared channel transport block size, modulation and coding strategies, etc.; the application layer data may include, but is not limited to, the download segment rate, download segment duration, buffer size, number of remaining undownloaded segments, azimuth information, latitude and longitude information, etc. Both layers of data can be collected by the data acquisition component in the terminal device. The collected network layer data can be referred to as real-time network layer data, and the collected application layer data can be referred to as real-time application layer data.

[0036] Furthermore, terminal device 100b can send the playback request, the real-time network layer data, and the real-time application layer data together to the service server. The service server can deploy a bitrate prediction enhancement model (obtained by expanding the network structure of an offline-trained bitrate prediction model). The service server can perform real-time online training on this bitrate prediction enhancement model based on the real-time network layer data and the real-time application layer data. After training, a target bitrate prediction enhancement model can be obtained. Subsequently, the service server can input the real-time network layer data and the real-time application layer data into the target bitrate prediction enhancement model. This target bitrate prediction enhancement model can output a bitrate (which can be called the media prediction bitrate, assuming it's 480P) that matches the real-time network layer data and the real-time application layer data. The service server can then transcode the first episode of video A based on this media prediction bitrate and return the transcoded video data to terminal device 100b; terminal device 100b can then play the transcoded video data. Figure 2 As shown, what the user sees is video content with a bitrate of 480P.

[0037] It should be understood that the embodiments of this application use a combination of offline and online methods for bitrate adaptation, which enables the bitrate prediction model to respond promptly to changes in network layer data and application layer data (such as network changes), improves the accuracy of bitrate prediction, thereby enhancing the presentation effect of multimedia data and improving the quality of user experience.

[0038] Further, please see Figure 3 , Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of this application. Wherein, as... Figure 3 As shown, this method can be performed by a terminal device (e.g., the one described above). Figure 1 This can be executed by any terminal device in the terminal device cluster shown, or by a service server (e.g., the one mentioned above). Figure 1 The method can be executed by the service server 1000 shown, or jointly by the terminal device and the service server. For ease of understanding, this embodiment uses the execution of the method by the aforementioned service server as an example to illustrate the specific process of bitrate prediction in the service server. The method may include at least the following steps S101-S103: S101, receive a playback request for multimedia data sent by the terminal device, and obtain real-time network layer data and real-time application layer data of the multimedia data based on the playback request.

[0039] In this application, a target application may be installed on the terminal device. When a user launches and runs the target application through the terminal device, they can play or view any multimedia data. When the user performs a playback operation on a certain multimedia data, the terminal device can respond to this playback operation by generating a playback request for that multimedia data, and the terminal device can send the playback request to the business server. The target application can refer to an application with the function of displaying data information such as text, images, audio, and video. The application can include multimedia applications (e.g., video applications), entertainment applications (e.g., game applications), educational applications, etc., all of which can have the function of loading and playing multimedia data (e.g., video and music). For example, the application can be a social application, an educational application, a short video application, a game application, etc. The application can be a standalone application, installed on the terminal device and running on the terminal device's current operating system; or it can be an embedded sub-application within an application (such as a social application, an educational application, etc.). The form of the application is not limited here.

[0040] Furthermore, after receiving a playback request for multimedia data from a terminal device, the service server can obtain the network layer data and application layer data of the multimedia data from the transmission network environment. It is understood that different layers exist in the multimedia data transmission system for transmission, including but not limited to the physical layer, data link layer, network layer, transport layer, session layer, presentation layer, and application layer. Each layer has a different function. For example, the physical layer is mainly used for the transmission of raw bit streams over mechanical, electronic, and timing interface communication channels; the data link layer is mainly used for physical addressing and converting the raw bit stream into logical transmission lines; the network layer is mainly used for controlling the operation of the subnet, such as logical addressing, packet transmission, and routing; the transport layer is mainly used to receive data from the upper layer, segment the data when necessary, and deliver these data to the network layer, ensuring that these data segments effectively reach the other end; the session layer is mainly used for establishing and managing sessions between users on different machines; the presentation layer is mainly used for the syntax and semantics of information, such as encryption and decryption, conversion and translation, compression and decompression; and the application layer mainly includes various application protocols (such as HTTP protocol, FTP protocol, etc.).

[0041] In this embodiment, the network layer data can refer to network layer data in the transmission system, specifically including but not limited to Reference Signal Receiving Power (RSRP), Received Signal Strength Indicator (RSSI), Reference Signal Receiving Quality (RSRQ), Signal-to-Interference Plus Noise Ratio (SINR), latency, data rate, physical layer downlink shared channel transport block size, modulation and coding strategies, etc.; the application layer data can refer to application layer data, specifically including but not limited to download segment rate, download segment duration, buffer size, number of remaining undownloaded segments, azimuth information, latitude and longitude information, etc. Both layers of data can be collected by the data acquisition component in the terminal device. The collected network layer data can be called real-time network layer data, and the collected application layer data can be called real-time application layer data.

[0042] Optionally, and understandably, multimedia data can refer to a complete piece of media data (such as a complete video, for example, a specific episode of a video), while real-time network layer data and application layer data can refer to data within a specific duration (such as data for the entire duration, data within 10-20 seconds, data within 20-40 seconds, etc.). In other words, multimedia data (such as a video) can be divided into multiple media data blocks (such as video blocks, each with a corresponding duration). Specifically, real-time network layer data and real-time application layer data can refer to the network layer data (such as signal-to-noise ratio, latency, etc.) and application layer data (such as the size of the media data block, the download time of the media data block, the bitrate of the media data block, etc.) corresponding to a media data block (a specific duration).

[0043] Optionally, the terminal device may be equipped with a component for collecting data (such as network data) (referred to as a data acquisition component). This component allows the terminal device to collect real-time network layer data and real-time application layer data of multimedia data. The terminal device can then report this real-time network layer data and real-time application layer data to the service server via an upper-layer link. This upper-layer link may specifically include communication equipment and the core network. Data transmission between the terminal device and the service server can be achieved through data forwarding via the communication equipment and the core network. For clarity, please refer to [further details omitted]. Figure 4 , Figure 4 This is a communication architecture diagram for communication between devices provided in an embodiment of this application. Figure 4 As shown, the communication architecture specifically refers to the communication between terminal devices, communication devices, core network, and application servers. For ease of understanding, each device will be described below.

[0044] Terminal devices refer to devices operated by users, including but not limited to smartphones, in-vehicle cameras, and intelligent voice interaction devices. When a terminal device receives a playback signal from a user for certain multimedia data, it can generate a playback request for that multimedia data and collect real-time network layer data and real-time application layer data through data acquisition components.

[0045] Communication equipment refers to any active communication device that can act as a transmission source. Specifically, it can include, but is not limited to, one or more of the following: 4G / 5G base stations, roadside units (RSUs), Wi-Fi devices, etc. Terminal devices and communication equipment can be connected via a communication interface such as the Uu interface, allowing the terminal device to send real-time network layer data and real-time application layer data to the communication equipment.

[0046] The core network can include 4G / 5G core networks, cloud core networks, and so on. Communication equipment can connect to the core network through communication interfaces such as NGAP / N3 interfaces, allowing the communication equipment to send real-time network layer data and real-time application layer data to the core network.

[0047] A business server can be any electronic device with data processing capabilities, and this electronic device can be implemented as a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, and big data and artificial intelligence platforms. Servers can be connected directly or indirectly via wired or wireless communication; this application does not impose specific limitations. The core network and the business server can be connected via a communication interface such as the N6 interface, through which the core network can send real-time network layer data and real-time application layer data to the business server. After obtaining the real-time network layer data and real-time application layer data, the business server can perform online model training based on the real-time network layer data and real-time application layer data (see subsequent descriptions for details), and perform online bitrate prediction based on the trained model. After transcoding the media data based on the predicted bitrate, the transcoded multimedia data can be returned to the terminal device.

[0048] S102, Based on real-time network layer data and real-time application layer data, the bitrate prediction enhancement model is trained online to obtain the target bitrate prediction enhancement model; the bitrate prediction enhancement model refers to the model obtained by expanding the depth and width of the network structure of the bitrate prediction model that has been trained offline.

[0049] In this application, historical network layer data and historical application layer data can be obtained in advance, and the bitrate prediction model can be trained offline based on the historical network layer data and historical application layer data to obtain a pre-trained offline bitrate prediction model. Subsequently, this application can extend the network structure of the pre-trained offline bitrate prediction model. Specifically, the width of the pre-trained bitrate prediction model's network structure can be expanded. It should be understood that the network structure of the bitrate prediction model can be composed of multiple neural network layers. Therefore, when expanding the width, the dimension of each layer in the pre-trained bitrate prediction model's network structure can be expanded (for example, if the original bitrate prediction model's network structure has M dimensions per layer, it can now be expanded to (n+1)M dimensions). Simultaneously, the depth of the pre-trained bitrate prediction model's network structure can also be expanded. When expanding the depth, the network parameters of each layer in the pre-trained bitrate prediction model's network structure can be expanded (for example, if the original bitrate prediction model's network structure has W network parameters per layer, it can now be expanded to (n+1)W network parameters). The number of dimensions and network parameters expanded for each layer can be determined based on the actual scenario requirements (such as those determined based on human experience or experimental results, which will not be restricted here).

[0050] For ease of understanding, please refer to the following: Figure 5 , Figure 5 This is a schematic diagram illustrating a model extension provided in an embodiment of this application. For example... Figure 5 As shown, the bitrate prediction model can be a bitrate prediction neural network, which can specifically include recurrent neural networks (RNNs) and convolutional neural networks (CNNs). The bitrate prediction neural network can include a policy network (ActorNetwork) and a criterion network (Critic Network). The bitrate prediction neural network includes multiple neural network layers (e.g., ...). Figure 5 The neural network layers shown are 1, ..., 8.

[0051] The Actor Network is responsible for generating actions and interacting with the environment, while the Critic Network is responsible for evaluating the performance of the Actor Network and guiding the Actor's actions in the next stage.

[0052] The input to the bitrate prediction neural network is information such as real-time network layer data and real-time application layer data (or historical network layer data and historical application layer data), and the output of the bitrate prediction neural network is a media prediction bitrate value obtained based on these input information.

[0053] Understandably, the Critic Network can evaluate each bitrate selection value (e.g., 270P, 480P, 720P, 1080P) based on the input information, obtaining an evaluation value for each bitrate selection value. Among these evaluation values, an optimal evaluation value can be determined, and the bitrate selection value corresponding to this optimal evaluation value can be used as the final bitrate value to be output by the Actor Network. For example, if the optimal evaluation value corresponds to a bitrate selection value of 720P, then the Actor Network can output a media prediction bitrate of 720P. Therefore, the bitrate prediction neural network can output a media prediction bitrate. Furthermore, through the evaluation feedback function in the bitrate prediction neural network, the bitrate in the input information and the media prediction bitrate can be used to calculate the evaluation feedback value obtained under that media prediction bitrate. This evaluation feedback value can be fed back to the Critic Network, which will use it to instruct the adjustment of the network parameters in the bitrate prediction neural network.

[0054] It is understandable that the methods for training predictive neural networks include, but are not limited to, reinforcement learning-based methods, as well as other methods that have training optimization effects.

[0055] It should be understood that, such as Figure 5 The network structures corresponding to bounding boxes 50a and 50b shown can be the network structures of a bitrate prediction model trained offline. Specifically, neural network layers 1-4 can be the network structure of the Actor Network of the bitrate prediction model trained offline; neural network layers 8-11 are the network structures of the Critic Network of the bitrate prediction model trained offline. This application can extend the network structures of the two networks of the bitrate prediction model to obtain a bitrate prediction enhancement model. When extending the network structure, the dimensions and network parameters of each layer in the network structure can be extended, and the neural network layers of the network structure can also be extended. For example, as... Figure 5As shown, this application can extend the neural network layers of two networks. When training the bitrate prediction enhancement model online, the original network parameters of the bitrate prediction model that have already been trained offline can remain unchanged, and only the newly extended network parameters need to be updated online. See the following description for details.

[0056] Furthermore, the service server can extend the network structure model, termed the bitrate prediction enhancement model, and train it online based on received real-time application layer data and real-time network layer data. It should be understood that, in this application, the offline training method based on historical application layer data and historical network layer data, and the online training method based on real-time application layer data and real-time network layer data, can be the same. For example, both can employ reinforcement learning (such as A3C) for training. For a detailed implementation of online training of the bitrate prediction enhancement model based on real-time network layer data and real-time application layer data to obtain the target bitrate prediction enhancement model, please refer to the following sections. Figure 6 The description in the corresponding embodiments.

[0057] S103 determines the media prediction bitrate of multimedia data through the target bitrate prediction enhancement model, real-time network layer data, and real-time application layer data, and transcodes the multimedia data based on the media prediction bitrate.

[0058] In this application, after obtaining the target bitrate prediction enhancement model through online training, real-time network layer data and real-time application layer data can be input into the target bitrate prediction enhancement model. The target bitrate prediction enhancement model can output a bitrate value (which can be called the media prediction bitrate) that is adapted to the real-time network layer data and real-time application layer data. The service server can transcode the multimedia data based on the media prediction bitrate and return the transcoded multimedia data to the terminal device, thereby enabling the terminal device to play the transcoded multimedia data.

[0059] Optionally, and understandably, as described above, after obtaining real-time application layer data and real-time network layer data of multimedia data, this application can train the bitrate prediction enhancement model online based on the real-time application layer data and real-time network layer data, and then perform bitrate prediction based on the online-trained model. To better adapt to the media duration of multimedia data, this application can also propose another online bitrate optimization strategy. After obtaining a playback request for multimedia data, the business server can select a bitrate optimization strategy based on the media duration of the multimedia data. For example, if the media duration exceeds a certain threshold, a bitrate prediction enhancement model can be trained online, and a bitrate prediction optimization strategy based on the online-trained model can be used for accurate bitrate prediction; if the media duration is less than a certain threshold, a multi-step lookahead algorithm optimization strategy can be selected for accurate bitrate optimization. In other words, after receiving a multimedia data playback request, the business server can obtain the media duration of the multimedia data and compare it with a duration threshold. If the media duration is greater than the duration threshold, the server will execute the step of online training of the bitrate prediction enhancement model based on real-time network layer data and real-time application layer data to obtain the target bitrate prediction enhancement model, as well as subsequent steps. If the media duration is less than the duration threshold, online bitrate optimization can be performed according to another bitrate optimization strategy.

[0060] In this embodiment, when playing multimedia data, a pre-trained offline bitrate prediction model can be used to adaptively adjust the bitrate of the multimedia data online. Specifically, this application can pre-train a bitrate prediction model offline, and then expand the width and depth of its network structure. In real-time applications, if a playback request for multimedia data is received from a terminal device, the network layer data and application layer data of the multimedia data can be obtained. Based on this network layer data and application layer data, the model with the expanded network structure is trained online, and then bitrate prediction is performed based on the online-trained model. By adopting a combination of offline and online methods, real-time model training can be performed based on real-time data. The trained model can respond well to changes and updates in real-time data, and the output bitrate is adapted to real-time network layer data and application layer data. At the same time, since online training is based on a model that has already been trained offline, training time can be reduced. In summary, this application can improve the accuracy of bitrate prediction and optimize the presentation effect of multimedia data.

[0061] Further, please see Figure 6 , Figure 6 This is a schematic diagram of the model training process provided in an embodiment of this application. This process can correspond to the above-described... Figure 3The corresponding embodiment describes the process of training the bitrate prediction enhancement model based on real-time network layer data and real-time application layer data to obtain the target bitrate prediction enhancement model. For example... Figure 6 As shown, the process may include at least the following steps S601-S603: S601 inputs real-time network layer data and real-time application layer data into the bitrate prediction enhancement model, and outputs a sample media predicted bitrate that matches the real-time network layer data and real-time application layer data.

[0062] Specifically, after obtaining real-time network layer data and real-time application layer data, they can be input into the bitrate prediction enhancement model. This bitrate prediction enhancement model can output a bitrate value that matches the real-time network layer data and real-time application layer data (which can be called the sample media predicted bitrate).

[0063] S602, obtain the first quality assessment feedback function, and generate a sample quality assessment feedback value for the sample media prediction bitrate based on the sample media prediction bitrate and the first quality assessment feedback function.

[0064] In this application, the first quality assessment feedback function can refer to a function used to determine the Quality of Experience (QoE). User experience quality mainly refers to feedback on data related to products, devices, etc., so this QoE can also be called a quality assessment feedback value. This application embodiment can obtain the stuttering time information and bitrate fluctuation information corresponding to the predicted bitrate of the sample media. Based on the stuttering time information, the bitrate fluctuation information, the predicted bitrate of the sample media, and the first quality assessment feedback function, an assessment feedback value (which can be called a sample quality assessment feedback value) can be generated for the predicted bitrate of the sample media. It should be understood that this sample quality assessment feedback value can be used to reflect the bitrate prediction effect of the bitrate prediction neural network (bitrate prediction enhancement model) on the real-time network layer data and real-time application layer data. The larger the feedback value, the better the bitrate prediction effect of the bitrate prediction enhancement model.

[0065] The specific implementation method for determining the sample quality assessment feedback value corresponding to the predicted bit rate of the sample media can be as shown in formula (1): Formula (1) in, It can be used to characterize the sample quality assessment feedback value; n can be used to characterize the sequence number of the time segment of the collected data (for example, a multimedia data can be composed of multiple consecutive media data blocks, and each media data block can be assigned a sequence number in order). It can be used to characterize the first Bitrate value within a time segment; It can be used to characterize the use of a pre-defined nonlinear function. Regarding the bit rate value Perform nonlinear operations; It can be used to characterize the n time segments obtained Perform summation; It can be used to characterize the first Information on buffering time within a specific time segment; It can be used to characterize the summation of stuttering time information across n duration segments; It can be used to characterize the first The predicted bitrate of the sample media for the network layer data and application layer data within each time period; It can be used to characterize the use of a pre-defined nonlinear function. Predicted bitrate for sample media Perform nonlinear operations; It can be used to characterize the first Bitrate fluctuation information within a time segment; It can be used to characterize the sum of bitrate fluctuation information over n duration segments; It can be used to characterize the weight of the impact of stuttering time information on the feedback value. It can be used to characterize the weight of fluctuation bit rate information in the evaluation feedback value.

[0066] It should be understood that the network layer data and application layer data (including the bitrate values ​​within time segment 1) obtained in time segment 1 are... This can be used as real-time cross-layer data 1. This real-time cross-layer data 1 is input into the bitrate prediction enhancement model, which can then output a sample media predicted bitrate 1 (i.e.,...). The predicted bitrate of this sample media (1) can be adapted to the network layer data and application layer data (real-time cross-layer data 2) of the next duration segment (e.g., duration segment 2). Based on the predicted bitrate of this sample media (1) and the bitrate value corresponding to duration segment 1 (...), The bitrate fluctuation information corresponding to duration segment 1 can be determined. Then, the stuttering time information within duration segment 1 can be obtained by statistics. Then, the quality assessment feedback value for the sample media predicted bitrate 1 can be determined by using the stuttering time information within duration segment 1, the sample media predicted bitrate 1, the bitrate value obtained in duration segment 1 and formula (1). The larger the quality assessment feedback value, the better the sample media predicted bitrate 1 matches the real-time library layer data 2 corresponding to duration segment 2, and the better the bitrate prediction effect of the bitrate prediction enhancement model.

[0067] S603, adjust the bitrate prediction enhancement model based on the sample quality assessment feedback value to obtain the target bitrate prediction enhancement model.

[0068] Specifically, the bitrate prediction enhancement model can be adjusted using the aforementioned sample quality assessment feedback value. The specific method is as follows: Obtain the quality assessment feedback threshold and determine the assessment difference between the sample quality assessment feedback value and the threshold; define real-time network layer data and real-time application layer data as real-time cross-layer data; then compare the assessment difference with the difference range. If the assessment difference exceeds the range, it can be determined that the matching degree between the sample media predicted bitrate and the real-time cross-layer data does not meet the matching condition. The network parameters in the bitrate prediction enhancement model can then be adjusted based on the assessment difference to obtain the target bitrate prediction enhancement model. Conversely, if the assessment difference is within the range, it can be determined that the matching degree between the sample media predicted bitrate and the real-time cross-layer data meets the matching condition, and the bitrate prediction enhancement model can be determined as the target bitrate prediction enhancement model.

[0069] As can be seen from the above, this application can determine whether the model meets the iterative update conditions based on the evaluation difference between the sample quality evaluation feedback value and the quality evaluation feedback threshold. If the evaluation difference is within the range, the model can be determined to meet the iterative update conditions and can be put into use. If the evaluation difference exceeds the range, the model can be determined not to meet the iterative update conditions, and the network parameters of the model can be adjusted based on the evaluation difference until the model meets the iterative update conditions.

[0070] It is understandable that since the bitrate prediction enhancement model is obtained by extending the network structure of an offline-trained bitrate prediction model, which already has a high prediction accuracy, to improve training efficiency, during online training, the network parameters of the offline-trained bitrate prediction model can be kept unchanged, and only the extended network parameters can be iteratively updated. In other words, the specific implementation of adjusting the network parameters in the bitrate prediction enhancement model based on the evaluation difference to obtain the target bitrate prediction enhancement model can be as follows: The remaining network parameters can be obtained from the network parameters included in the bitrate prediction enhancement model; these remaining network parameters refer to the network parameters in the bitrate prediction enhancement model other than the fixed network parameters, which are the network parameters belonging to the offline-trained bitrate prediction model; subsequently, the remaining network parameters can be adjusted based on the evaluation difference to obtain the adjusted network parameters; the bitrate prediction enhancement model containing both the fixed and adjusted network parameters can be identified as the target bitrate prediction enhancement model.

[0071] In this embodiment, by expanding the network structure of the offline-trained model, the prediction accuracy of the model can be improved. At the same time, by training the expanded model online based on real-time network layer data and real-time application layer data, the prediction accuracy of the model can be greatly improved. This allows the model to respond promptly to changes in cross-layer data between the network layer and the application layer, and the model can accurately predict the bitrate that matches the real-time cross-layer data. In other words, the accuracy of the model's bitrate prediction can be improved, thereby enhancing the presentation effect of multimedia data.

[0072] As can be understood from the above, in addition to the bitrate optimization method based on offline + online training models, this application also proposes another bitrate optimization strategy. When performing bitrate prediction based on an offline-trained bitrate prediction model, an appropriate bitrate optimization strategy can be selected based on the media duration of the multimedia data, and combined with the offline-trained bitrate prediction model for online bitrate optimization. Specifically, this application proposes a bitrate adaptive algorithm based on Monte Carlo sampling for bitrate optimization. When the media duration of the multimedia data exceeds a duration threshold, the above-mentioned method of online training of the extended model and bitrate prediction based on the trained model can be used for bitrate optimization; while when the media duration of the multimedia data is less than the duration threshold, a bitrate adaptive algorithm based on Monte Carlo sampling can be used for bitrate optimization. For ease of understanding, please refer to [further details omitted]. Figure 7 , Figure 7 This is a schematic diagram of a bitrate optimization process provided in an embodiment of this application. This process can correspond to the above-mentioned... Figure 3 In the corresponding embodiment, when the media duration of the multimedia data is less than a duration threshold, the process of optimizing the bitrate of the multimedia data is performed by combining an offline-trained bitrate prediction model. For example... Figure 7 As shown, the process may include at least the following steps S701-S704: S701, if the media duration is less than the duration threshold, then the real-time network layer data and real-time application layer data are input into the offline-trained bitrate prediction model. The initial media prediction bitrate of the multimedia data is determined by the offline-trained bitrate prediction model, the real-time network layer data, and the real-time application layer data.

[0073] Specifically, through the above Figure 3As can be seen from the corresponding embodiment, after receiving a playback request for multimedia data from a terminal device, the service server can obtain real-time network layer data and real-time application layer data. The service server can also obtain the media duration of the multimedia data. If the media duration is greater than a duration threshold, the service server can train the bitrate prediction enhancement model online based on the real-time network layer data and real-time application layer data, and perform bitrate prediction processing based on the trained model. If the media duration is less than the duration threshold, the service server can use a Monte Carlo sampling-based bitrate adaptive algorithm to optimize the bitrate. Specifically, the service server can input the real-time network layer data and real-time application layer data into the offline-trained bitrate prediction model. Through the offline-trained bitrate prediction model, the real-time network layer data, and the real-time application layer data, the predicted bitrate of the multimedia data (which can be called the initial media prediction bitrate) can be determined. Subsequently, the service server can use a Monte Carlo sampling-based bitrate adaptive algorithm to optimize the initial media prediction bitrate.

[0074] S702 determines the real-time quality assessment feedback value of multimedia data at the first moment based on the initial media prediction bitrate; the first moment refers to the moment when the initial media prediction bitrate is determined.

[0075] Specifically, the real-time quality assessment feedback value of multimedia data at the first moment can refer to the quality assessment feedback value corresponding to the initial media prediction bitrate, or it can refer to the user experience quality. This real-time quality assessment feedback value can be determined based on the above formula (1).

[0076] The real-time quality assessment feedback value can be determined using the method shown in formula (2): Formula (2) Among them, r, as shown in formula (2) i It can be used to characterize the bitrate value at time i (such as the initial media prediction bitrate at time i); q(r) i ) can refer to r i A function that can represent the quality factor (usually r) i (log function); V ij C can be used to represent the code size (in bits) of multimedia data with a bit rate of j at time i. i It can be used to characterize the download speed (unit: bit / second) of multimedia data (such as video) at time i, b i The buffer size (in seconds) of multimedia data (such as video) at time i can be used to characterize the buffer size. α0 can be used to characterize the weight value for measuring stuttering duration. β0 can be used to characterize the weight value for measuring bitrate changes. iThis can be used to represent the bitrate value of the multimedia data at the previous time step (time step i-1). Formula (2) uses the form (x,y) + The expression can be used to represent the maximum value of x and y. The expression of the form |x| in formula (2) represents the absolute value of x.

[0077] It should be noted that formula (1) above can correspond to the first quality assessment feedback function, and formula (2) can correspond to the second quality assessment feedback function. In feasible embodiments, formula (1) and formula (2) can represent the same algorithm, that is, the first quality assessment feedback function and the second quality assessment feedback function can be the same function. Both are used to solve QoE.

[0078] S703: Obtain the prediction quality assessment feedback value of multimedia data at the second moment; based on the real-time quality assessment feedback value and the prediction quality assessment feedback value, perform online optimization processing on the initial media prediction bitrate to obtain the media optimized bitrate of the multimedia data; the second moment is later than the first moment.

[0079] Specifically, the implementation method for obtaining the prediction quality assessment feedback value of multimedia data at the second time point can be as follows: the prediction system state corresponding to the multimedia data at the second time point can be obtained; then, the initial variable coefficients corresponding to the prediction system state can be obtained, and the initial variable coefficients and the prediction system state can be processed to obtain the operation state value; then, the second quality assessment feedback function can be obtained, and the prediction quality assessment feedback value of the multimedia data at the second time point can be determined based on the second quality assessment feedback function, the prediction system state, and the operation state value.

[0080] The predicted system state characterizes the observed system state of multimedia data at a certain time after the first time step. Specifically, it can be a vector composed of one or more network parameters, including network information obtained by the service server, media download duration, symbol size, buffer size, etc. Based on this predicted system state, the quality assessment feedback value of the multimedia data at the second time step (which can be called the predicted quality assessment feedback value) can be determined. Taking the predicted system state at the second time step, which includes the media symbol information, media download rate, and media buffer capacity, as an example, the specific implementation of processing the initial variable coefficients with the predicted system state to obtain the calculated state value can be as follows: The initial variable coefficients can be processed with the media symbol information to obtain the processed symbol; subsequently, the initial variable coefficients can be processed with the media download rate to obtain the processed download rate; subsequently, the initial variable coefficients can be processed with the media buffer capacity to obtain the processed buffer capacity; finally, the processed symbol, processed download rate, and processed buffer capacity can be added together to obtain the calculated state value.

[0081] It is understandable that the real-time quality assessment feedback value of the multimedia data at the first moment can be expressed as q(o0,r0), where o0 can be used to characterize the system state of the multimedia data at the first moment, and r0 can be used to characterize the initial media prediction bitrate of the multimedia data at the first moment; q(o0,r0) can correspond to the second quality assessment feedback function mentioned above, and r0 can correspond to r in formula (2). i o0 can correspond to V in the above formula (2). ij C i b i Parameters, based on the above formula (2), r i And by finding o0, the real-time quality assessment feedback value of the multimedia data at the first moment can be obtained. Similarly, the real-time quality assessment feedback value of the multimedia data at the second moment can be expressed as... Among them, o n It can be used to characterize the predictive system state of multimedia data at time n (e.g., time 2). These can be used to characterize the variable coefficients used for bitrate optimization (such as the initial variable coefficients mentioned above). It can be used to characterize the operational state value obtained after processing the variable coefficients with the predicted system state. This corresponds to the second quality assessment feedback function mentioned above. This can correspond to r in formula (2) i , This can correspond to V in formula (2) above.ij C i b i Parameters, based on the above formula (2), as well as The prediction quality evaluation feedback value of the multimedia data at time n (e.g., time 2) can be obtained. In other words, for time 2, when the prediction system state includes media symbol information, media download rate, and media buffer capacity, the initial variable coefficients can be processed by operating on the media symbol information, media download rate, and media buffer capacity respectively, and then the results of each operation can be summed to obtain the operation state value; subsequently, based on the above formula (2), the operation state value, and the prediction system state, the prediction quality evaluation feedback value of the multimedia data at time 2 can be obtained.

[0082] Furthermore, based on the aforementioned real-time quality assessment feedback value and the aforementioned predicted quality assessment feedback value, the initial media prediction bitrate can be optimized online to obtain the optimized bitrate of the multimedia data. Specifically, the implementation method for optimizing the initial media prediction bitrate based on the aforementioned real-time quality assessment feedback value and the aforementioned predicted quality assessment feedback value to obtain the optimized bitrate of the multimedia data can be as follows: A bitrate optimization function corresponding to the multimedia data can be obtained; subsequently, the real-time quality assessment feedback value and the predicted quality assessment feedback value can be processed according to the bitrate optimization function to obtain a calculated feedback value; if the calculated feedback value is less than a feedback value threshold, the absolute value of the difference between the calculated feedback value and the feedback value threshold can be determined, and the initial media prediction bitrate and the initial variable coefficients can be adjusted according to the absolute value of the difference to obtain the transitional media prediction corresponding to the initial media prediction bitrate. The bitrate and the transition variable coefficients corresponding to the initial variable coefficients are determined. Based on the transition media prediction bitrate, the transition real-time quality assessment feedback value of the multimedia data at the first moment can be determined. The transition variable coefficients are then processed with the prediction system state to obtain the transition operation state value. Based on the second quality assessment feedback function, the prediction system state, and the transition operation state value, the transition prediction quality assessment feedback value of the multimedia data at the second moment can be determined. Based on the transition prediction quality assessment feedback value and the transition real-time quality assessment feedback value, the transition media prediction bitrate is optimized online until the transition prediction quality assessment feedback value is greater than the feedback value threshold, thus obtaining the optimized media bitrate of the multimedia data. If the operation feedback value is greater than the feedback value threshold, the initial media prediction bitrate can be directly determined as the media prediction bitrate of the multimedia data.

[0083] It should be understood that the bitrate optimization function here can refer to the bitrate adaptive optimization function corresponding to the bitrate adaptive algorithm based on Monte Carlo sampling. For ease of understanding, please refer to formula (3), which is the specific implementation method of the bitrate adaptive algorithm based on Monte Carlo sampling for optimizing the initial media prediction bitrate, as shown in formula (3): Formula (3) Among them, as in formula (3) Represents the observed system state at time n (e.g.) It can represent the system state at the current moment, such as the system state at the first moment. It can be used to characterize the real-time quality assessment feedback value of the aforementioned multimedia data at the first moment; It can be used to characterize the prediction quality assessment feedback value of the above multimedia data at the nth time after the first time (the subsequent T-1 time, such as the second time); It can be used to characterize the coefficients of variables at various time points; T can be used to characterize the look-ahead step size (which can be manually configured based on scenario requirements). It can be used to characterize the optimal value function, which can be defined manually according to the specific scenario. In some specific scenarios, the optimal value function may be an invalid value (such as the value 0). It can be used to characterize the discount factor, typically between 0 and 1. Specifically, for as well as The specific solution method can be found in the above description, and will not be repeated here. It should be understood that the function shown in formula (3) can be: through continuous adjustment and updating and the coefficients of the variables at each time point (including To ensure that the result obtained by solving formula (3) is the maximum result, the updated bitrate value can be determined. This represents the optimal bitrate at the first moment. In this embodiment, whether the calculation result is the maximum result can be determined based on a feedback value threshold; if it is adjusted and updated... And the variable coefficients at each time point, such that the calculation result determined by formula (3) is greater than the feedback value threshold, then the bit rate value at this time can be determined. This represents the optimal bitrate at the first moment; however, if the calculation result is not greater than the feedback threshold, adjustments and updates can continue. And the variable coefficients at each time point, until the calculation result determined based on formula (3) is greater than the feedback value threshold.

[0084] In other words, after determining as well as Afterwards, the optimal value at time T can be obtained by solving the optimal value function. Then, according to the operation method corresponding to the above formula (3), the real-time quality assessment feedback value, the predicted quality assessment feedback value, and the optimal value of the multimedia data at the first time can be processed. After obtaining the operation feedback value, it can be compared with the feedback value threshold. When the operation feedback value is greater than the feedback value threshold, the bitrate value at this moment can be determined as the media optimized bitrate at the first time. When the operation feedback value is less than the feedback value threshold, the initial media predicted bitrate and the initial variable coefficient can be adjusted. Then, based on the adjusted initial media predicted bitrate and the initial variable coefficient, the operation feedback value can be calculated again based on formula (3) until the operation feedback value is greater than the feedback value threshold. It should be noted that when the above optimal value function is invalid, when processing the real-time quality assessment feedback value, the predicted quality assessment feedback value, and the optimal value of the multimedia data at the first time according to the operation method corresponding to the above formula (3) to obtain the operation feedback value, it is only necessary to process the real-time quality assessment feedback value and the predicted quality assessment feedback value of the multimedia data at the first time to obtain the operation feedback value.

[0085] S704 is a media-optimized bitrate transcoding technology for multimedia data.

[0086] Specifically, after optimizing the initial media prediction bitrate to obtain the optimized media bitrate, the multimedia data can be transcoded based on the optimized media bitrate. After obtaining the transcoded multimedia data, the transcoded multimedia data can be returned to the terminal device, and the terminal device can play the transcoded multimedia data.

[0087] In this embodiment, a bitrate prediction model can be pre-trained offline. During real-time application, if a playback request for multimedia data is received from a terminal device, the network layer and application layer data of the multimedia data can be obtained. The pre-trained bitrate prediction model can then output an initial predicted bitrate adapted to the network layer and application layer data. Subsequently, a Monte Carlo sampling-based bitrate adaptation algorithm can be used for online bitrate optimization. By combining offline and online methods, real-time bitrate adaptation can be performed based on real-time data, and the final determined bitrate is also adapted to the real-time network layer and application layer data. In summary, this application can improve bitrate prediction accuracy and optimize the presentation effect of multimedia data.

[0088] Further, please see Figure 8 , Figure 8 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Wherein, as... Figure 3 As shown, this method can be performed by a terminal device (e.g., the one described above). Figure 1This can be executed by any terminal device in the terminal device cluster shown, or by a service server (e.g., the one mentioned above). Figure 1 The method can be executed by the service server 1000 shown, or jointly by the terminal device and the service server. For ease of understanding, this embodiment uses the execution of the method by the aforementioned service server as an example to illustrate the specific process of bitrate optimization in the service server. The method may include at least the following steps S801-S805: S801 receives a playback request for multimedia data sent by a terminal device, and obtains real-time network layer data and real-time application layer data of the multimedia data based on the playback request.

[0089] S802 inputs real-time network layer data and real-time application layer data into the offline-trained bitrate prediction model, and determines the initial media prediction bitrate of the multimedia data using the offline-trained bitrate prediction model, real-time network layer data, and real-time application layer data.

[0090] S803 determines the real-time quality assessment feedback value of multimedia data at the first moment based on the initial media prediction bitrate; the first moment refers to the moment when the initial media prediction bitrate is determined.

[0091] S804: Obtain the prediction quality assessment feedback value of the multimedia data at the second time step; optimize the initial media prediction bitrate based on the real-time quality assessment feedback value and the prediction quality assessment feedback value to obtain the media optimized bitrate of the multimedia data; the second time step is later than the first time step.

[0092] S805 is a media-optimized bitrate transcoding technology for multimedia data.

[0093] For the specific implementation methods of steps S801-S805, please refer to the above. Figure 7 The descriptions of steps S701-S705 in the corresponding embodiments will not be repeated here.

[0094] In this embodiment, a bitrate prediction model can be pre-trained offline. During real-time application, if a playback request for multimedia data is received from a terminal device, the network layer and application layer data of the multimedia data can be obtained. The pre-trained bitrate prediction model can then output an initial predicted bitrate adapted to the network layer and application layer data. Subsequently, a Monte Carlo sampling-based bitrate adaptation algorithm can be used for online bitrate optimization. By combining offline and online methods, real-time bitrate adaptation can be performed based on real-time data, and the final determined bitrate is also adapted to the real-time network layer and application layer data. In summary, this application can improve bitrate prediction accuracy and optimize the presentation effect of multimedia data.

[0095] Further, please see Figure 9 , Figure 9 This is a system flowchart provided in an embodiment of this application. For example... Figure 9 As shown, the system flow may include at least the following steps S91-S98: S91, terminal devices collect cross-layer historical data from the network layer and application layer.

[0096] Specifically, the cross-layer historical data here can include historical data of the network layer (which can be called historical network layer data) and historical data of the application layer (which can be called historical application layer data).

[0097] S92, the terminal device reports cross-layer historical data to the business server.

[0098] S93, the business server performs offline model training.

[0099] Specifically, the business server can train the bitrate prediction model based on cross-layer historical data. The bitrate prediction model can be trained using reinforcement learning; for details, please refer to the above. Figure 3 In the corresponding embodiments, the description of training the bitrate prediction enhancement model based on real-time application layer data and real-time network layer data is based on the same principle, and will not be repeated here.

[0100] S94, the terminal device sends a playback request for multimedia data to the service server.

[0101] S95, the business server determines whether the media duration of the multimedia data exceeds the duration threshold.

[0102] Specifically, after receiving a playback request, the service server can obtain the media duration of the multimedia data and determine whether it exceeds a duration threshold. If it does, the subsequent step S96 can be executed; if it does not, the subsequent step S97 can be executed.

[0103] S96, the business server executes the transfer learning algorithm.

[0104] Specifically, the transfer learning algorithm here can refer to acquiring real-time network layer data and real-time application layer data of multimedia data, training the bitrate prediction enhancement model online based on the real-time network layer data and real-time application layer data, and then using the online trained model to perform real-time bitrate prediction.

[0105] S97, the business server executes a multi-step look-ahead algorithm.

[0106] Specifically, the multi-step lookahead algorithm here can refer to an algorithm that optimizes bitrate based on the aforementioned Monte Carlo sampling algorithm. For example, based on the bitrate prediction model that has been trained offline, an initial media prediction bitrate can be predicted for real-time network layer data and real-time application layer data. Then, the Monte Carlo sampling algorithm can be used to optimize the bitrate of this initial media prediction bitrate.

[0107] S98, the business server performs transcoding based on an optimized bitrate.

[0108] Specifically, by using the algorithm in step S96 or step S97 above, an online optimized bitrate can be obtained, and the business server can transcode the multimedia data based on this online optimized bitrate.

[0109] For the specific implementation methods of steps S91-S98, please refer to the above. Figure 3 as well as Figure 7 The descriptions in the corresponding embodiments will not be repeated here.

[0110] In this embodiment, a combination of offline and online methods can be used, enabling real-time bitrate adaptation based on real-time data. The final determined bitrate is also adapted to real-time network layer and application layer data. In summary, this application can improve bitrate prediction accuracy and optimize the presentation effect of multimedia data.

[0111] Further, please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application. The data processing apparatus can be a computer program (including program code) running on a computer device; for example, the data processing apparatus is an application software. The data processing apparatus can be used to execute... Figure 3 The method shown. (As illustrated) Figure 10 As shown, the data processing device 1 may include: an acquisition module 11, a training module 12, a prediction module 13, and a transcoding module 14.

[0112] The acquisition module 11 is used to receive a playback request for multimedia data sent by the terminal device, and to acquire real-time network layer data and real-time application layer data of the multimedia data based on the playback request. Training module 12 is used to train the bitrate prediction enhancement model online based on real-time network layer data and real-time application layer data to obtain the target bitrate prediction enhancement model. The bitrate prediction enhancement model refers to the model obtained by expanding the depth and width of the network structure of the bitrate prediction model that has been trained offline. Prediction module 13 is used to determine the media prediction bitrate of multimedia data by using the target bitrate prediction enhancement model, real-time network layer data and real-time application layer data. Transcoding module 14 is used to transcode multimedia data based on media prediction bitrate.

[0113] The specific implementation methods of the acquisition module 11, training module 12, prediction module 13, and transcoding module 14 can be found in the above description. Figure 3 The descriptions of steps S101-S103 in the corresponding embodiments will not be repeated here.

[0114] In one embodiment, the training module 12 may include an input unit 121, an output unit 122, a generation unit 123, and an adjustment unit 124.

[0115] The input unit 121 is used to input real-time network layer data and real-time application layer data into the bit rate prediction enhancement model; Output unit 122 is used to output the sample media predicted bitrate that matches the real-time network layer data and the real-time application layer data through the bitrate prediction enhancement model. Generation unit 123 is used to obtain the first quality assessment feedback function; The generation unit 123 is also used to generate a sample quality assessment feedback value for the sample media prediction bitrate based on the sample media prediction bitrate and the first quality assessment feedback function. The adjustment unit 124 is used to adjust the bitrate prediction enhancement model based on the sample quality assessment feedback value to obtain the target bitrate prediction enhancement model.

[0116] The specific implementation methods of the input unit 121, output unit 122, generation unit 123, and adjustment unit 124 can be found in the above description. Figure 6 The descriptions of steps S601-S603 in the corresponding embodiments will not be repeated here.

[0117] In one embodiment, the generation unit 123 may include: an acquisition subunit 1231 and a generation subunit 1232.

[0118] Acquisition subunit 1231 is used to acquire the stuttering time information and bitrate fluctuation information corresponding to the predicted bitrate of the sample media. The generation subunit 1232 is used to generate a sample quality evaluation feedback value for the predicted bitrate of the sample media based on the stuttering time information, bitrate fluctuation information, sample media predicted bitrate, and evaluation feedback function.

[0119] For details on how to obtain subunit 1231 and generate subunit 1232, please refer to the above. Figure 6The description of step S602 in the corresponding embodiment will not be repeated here.

[0120] In one embodiment, the adjustment unit 124 may include: a determining subunit 1241 and an adjustment subunit 1242.

[0121] Subunit 1241 is defined to obtain the quality assessment feedback threshold and determine the assessment difference between the sample quality assessment feedback value and the quality assessment feedback threshold. The determining subunit 1241 is also used to determine real-time network layer data and real-time application layer data as real-time cross-layer data; The adjustment subunit 1242 is used to determine that the matching degree between the sample media prediction bitrate and the real-time cross-layer data does not meet the matching condition if the evaluation difference exceeds the difference range. The network parameters in the bitrate prediction enhancement model are adjusted according to the evaluation difference to obtain the target bitrate prediction enhancement model. The determining subunit 1241 is used to determine that the matching degree between the sample media predicted bitrate and the real-time cross-layer data meets the matching condition if the evaluation difference is within the difference range, and to determine the bitrate prediction enhancement model as the target bitrate prediction enhancement model.

[0122] The specific implementation methods for determining subunit 1241 and adjusting subunit 1242 can be found above. Figure 6 The description of step S604 in the corresponding embodiment will not be repeated here.

[0123] In one embodiment, the adjustment subunit 1242 is further specifically used to obtain the remaining network parameters from the network parameters included in the bitrate prediction enhancement model; the remaining network parameters refer to the network parameters other than the fixed network parameters included in the bitrate prediction enhancement model, and the fixed network parameters refer to the network parameters belonging to the bitrate prediction model that has been trained offline. The adjustment subunit 1242 is also specifically used to adjust the remaining network parameters based on the evaluation difference to obtain the adjusted network parameters. The model adjustment subunit 1242 is also specifically used to determine the bitrate prediction enhancement model, which includes fixed network parameters and adjusted network parameters, as the target bitrate prediction enhancement model.

[0124] In one embodiment, the data processing device 1 may further include a comparison module 15 and an execution module 16.

[0125] Comparison module 15 is used to obtain the media duration of multimedia data and compare the media duration with a duration threshold. Execution module 16 is used to perform the following steps if the media duration exceeds the duration threshold: online training of the bitrate prediction enhancement model based on real-time network layer data and real-time application layer data to obtain the target bitrate prediction enhancement model.

[0126] The specific implementation methods of the comparison module 15 and the execution module 16 can be found in the above description. Figure 3 The description of step S103 in the corresponding embodiment will not be repeated here.

[0127] In one embodiment, the data processing device 1 may further include a determination module 17 and an optimization module 18.

[0128] The determination module 17 is used to input real-time network layer data and real-time application layer data into the offline-trained bitrate prediction model if the media duration is less than the duration threshold, and determine the initial media prediction bitrate of the multimedia data through the offline-trained bitrate prediction model, real-time network layer data and real-time application layer data. The determining module 17 is further configured to determine the real-time quality assessment feedback value of the multimedia data at the first moment based on the initial media prediction bitrate; the first moment refers to the moment when the initial media prediction bitrate is determined. The determining module 17 is also used to obtain the prediction quality assessment feedback value of the multimedia data at a second time point; the second time point is later than the first time point. Optimization module 18 is used to perform online optimization processing on the initial media prediction bitrate based on the real-time quality assessment feedback value and the predicted quality assessment feedback value, so as to obtain the media optimization bitrate of the multimedia data. The optimization module 18 is also used to transcode multimedia data based on the media optimized bitrate.

[0129] For details on the specific implementation of module 17 and optimization module 18, please refer to the above. Figure 7 The descriptions of steps S701-S704 in the corresponding embodiments will not be repeated here.

[0130] In one embodiment, the determining module 17 may include: a data acquisition unit 171, a calculation unit 172, and a feedback determining unit 173.

[0131] The data acquisition unit 171 is used to acquire the prediction system state corresponding to the multimedia data at the second time point; The data acquisition unit 171 is used to acquire the initial variable coefficients corresponding to the predicted system state. The operation unit 172 is used to perform operations on the initial variable coefficients and the predicted system state to obtain the operation state value; Feedback determination unit 173 is used to obtain the second quality assessment feedback function; The feedback determination unit 173 is also used to determine the predicted quality assessment feedback value of the multimedia data at the second time step based on the second quality assessment feedback function, the predicted system state, and the operation state value.

[0132] The specific implementation methods of the data acquisition unit 171, the calculation unit 172, and the feedback determination unit 173 can be found in the above description. Figure 7 The description of step S703 in the corresponding embodiment will not be repeated here.

[0133] In one embodiment, the predicted system state includes media symbol information of multimedia data at a second time point, media download rate, and media buffer capacity. The arithmetic unit 172 is also specifically used to perform arithmetic processing on the initial variable coefficients and media code information to obtain arithmetic code; The arithmetic unit 172 is also specifically used to perform calculations on the initial variable coefficients and the media download rate to obtain the calculated download rate; The arithmetic unit 172 is also specifically used to perform calculations on the initial variable coefficients and the media buffer capacity to obtain the arithmetic buffer capacity. The arithmetic unit 172 is also specifically used to perform addition operations on the arithmetic code, the arithmetic download speed, and the arithmetic buffer capacity to obtain the arithmetic status value.

[0134] In one embodiment, the optimization module 18 may include: a function acquisition unit 181 and a bitrate adjustment unit 182.

[0135] Function acquisition unit 181 is used to acquire the bitrate optimization function corresponding to the multimedia data; The function acquisition unit 181 is also used to perform calculations on the real-time quality assessment feedback value and the predicted quality assessment feedback value according to the bitrate optimization function to obtain the calculation feedback value. The bitrate adjustment unit 182 is used to determine the absolute value of the difference between the operation feedback value and the feedback value threshold if the operation feedback value is less than the feedback value threshold, and adjust the initial media prediction bitrate and the initial variable coefficient according to the absolute value of the difference to obtain the transition media prediction bitrate corresponding to the initial media prediction bitrate and the transition variable coefficient corresponding to the initial variable coefficient. The bitrate adjustment unit 182 is also used to determine the transition real-time quality assessment feedback value of multimedia data at the first moment based on the transition media prediction bitrate, and to perform calculation processing on the transition variable coefficients and the prediction system state to obtain the transition calculation state value. The bitrate adjustment unit 182 is also used to determine the transition prediction quality evaluation feedback value of the multimedia data at the second time point based on the second quality evaluation feedback function, the prediction system state, and the transition operation state value. The bitrate adjustment unit 182 is also used to perform online optimization processing on the transition media prediction bitrate based on the transition prediction quality assessment feedback value and the transition real-time quality assessment feedback value, so as to obtain the media-optimized bitrate of the multimedia data.

[0136] The specific implementation methods of the function acquisition unit 181 and the bit rate adjustment unit 182 can be found in the above description. Figure 7 The description of step S703 in the corresponding embodiment will not be repeated here.

[0137] In one embodiment, the optimization module 18 may further include a bit rate determination unit 183.

[0138] The bitrate determination unit 183 is used to determine the initial media prediction bitrate as the media prediction bitrate of the multimedia data if the operation feedback value is greater than the feedback value threshold.

[0139] For the specific implementation of the bit rate determination unit 183, please refer to the above. Figure 7 The description of step S703 in the corresponding embodiment will not be repeated here.

[0140] In this embodiment, a bitrate prediction model can be pre-trained offline. Subsequently, the width and depth of its network structure can be expanded. In real-time applications, if a playback request for multimedia data is received from a terminal device, the network layer and application layer data of the multimedia data can be obtained. Based on this data, the expanded network structure model is trained online, and bitrate prediction is then performed using the online-trained model. This combined offline and online approach allows for real-time model training based on real-time data. The trained model responds well to data changes and updates, and the output bitrate is adapted to both network layer and application layer data. Furthermore, since online training is based on a model that has already been trained offline, training time can be reduced. In summary, this application can improve bitrate prediction accuracy and optimize the presentation effect of multimedia data.

[0141] Further, please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application. The data processing apparatus can be a computer program (including program code) running on a computer device; for example, the data processing apparatus is an application software. The data processing apparatus can be used to execute... Figure 8 The method shown. (As illustrated) Figure 11 As shown, the data processing device 2 may include: an acquisition module 31, a prediction module 32, a determination module 33, an optimization module 34, and a transcoding module 35.

[0142] The acquisition module 31 is used to receive a playback request for multimedia data sent by the terminal device, and to acquire real-time network layer data and real-time application layer data of the multimedia data based on the playback request. The prediction module 32 is used to input real-time network layer data and real-time application layer data into the offline-trained bitrate prediction model, and determine the initial media prediction bitrate of the multimedia data through the offline-trained bitrate prediction model, real-time network layer data and real-time application layer data. The determining module 32 is further configured to determine the real-time quality assessment feedback value of the multimedia data at the first moment based on the initial media prediction bitrate; the first moment refers to the moment when the initial media prediction bitrate is determined. The acquisition module 31 is also used to acquire the prediction quality assessment feedback value of multimedia data at the second moment; Optimization module 34 is used to optimize the initial media prediction bitrate based on the real-time quality assessment feedback value and the predicted quality assessment feedback value to obtain the media optimized bitrate of the multimedia data; the second moment is later than the first moment. Transcoding module 35 is used for transcoding multimedia data based on media-optimized bitrate.

[0143] The specific implementation methods of the acquisition module 31, prediction module 32, determination module 33, optimization module 34, and transcoding module 35 can be found above. Figure 8 The descriptions of steps S801-S805 in the corresponding embodiments will not be repeated here.

[0144] In this embodiment, a bitrate prediction model can be pre-trained offline. During real-time application, if a playback request for multimedia data is received from a terminal device, the network layer and application layer data of the multimedia data can be obtained. The pre-trained bitrate prediction model can then output an initial predicted bitrate adapted to the network layer and application layer data. Subsequently, a Monte Carlo sampling-based bitrate adaptation algorithm can be used for online bitrate optimization. By combining offline and online methods, real-time bitrate adaptation can be performed based on real-time data, and the final determined bitrate is also adapted to the real-time network layer and application layer data. In summary, this application can improve bitrate prediction accuracy and optimize the presentation effect of multimedia data.

[0145] Further, please see Figure 12 , Figure 12 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 12 As shown above, Figure 10 The data processing device 1 in the corresponding embodiment, or the one described above Figure 11The data processing device 2 in the corresponding embodiment can be applied to the aforementioned computer device 8000. The computer device 8000 may include a processor 8001, a network interface 8004, and a memory 8005. Furthermore, the computer device 8000 also includes a user interface 8003 and at least one communication bus 8002. The communication bus 8002 is used to enable communication between these components. The user interface 8003 may include a display screen and a keyboard; optionally, the user interface 8003 may also include a standard wired interface or a wireless interface. The network interface 8004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 8005 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 8005 may also be at least one storage device located remotely from the aforementioned processor 8001. Figure 12 As shown, the memory 8005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application program.

[0146] exist Figure 12 In the computer device 8000 shown, the network interface 8004 provides network communication functionality; the user interface 8003 is mainly used to provide an input interface for the user; and the processor 8001 can be used to call the device control application program stored in the memory 8005 to achieve: Receive playback requests for multimedia data sent by terminal devices, and obtain real-time network layer data and real-time application layer data of multimedia data based on the playback requests; Based on real-time network layer data and real-time application layer data, the bitrate prediction enhancement model is trained online to obtain the target bitrate prediction enhancement model. The bitrate prediction enhancement model refers to the model obtained by expanding the depth and width of the network structure of the bitrate prediction model that has been trained offline. The media prediction bitrate of multimedia data is determined by using a target bitrate prediction enhancement model, real-time network layer data, and real-time application layer data, and then the multimedia data is transcoded based on the media prediction bitrate.

[0147] Or achieve: Receive playback requests for multimedia data sent by terminal devices, and obtain real-time network layer data and real-time application layer data of multimedia data based on the playback requests; Real-time network layer data and real-time application layer data are input into the offline-trained bitrate prediction model. The initial media prediction bitrate of the multimedia data is determined by the offline-trained bitrate prediction model, real-time network layer data, and real-time application layer data. The real-time quality assessment feedback value of the multimedia data at the first moment is determined based on the initial media prediction bitrate; the first moment refers to the moment when the initial media prediction bitrate is determined. The prediction quality assessment feedback value of the multimedia data at the second time step is obtained. Based on the real-time quality assessment feedback value and the prediction quality assessment feedback value, the initial media prediction bitrate is optimized to obtain the media optimized bitrate of the multimedia data. The second time step is later than the first time step. Multimedia data transcoding based on media-optimized bitrate.

[0148] It should be understood that the computer device 8000 described in the embodiments of this application can execute the foregoing text. Figure 3 or Figure 8 The description of the data processing method in the corresponding embodiment can also be performed as described above. Figure 10 The corresponding embodiment refers to the data processing device 1, or the foregoing Figure 11 The description of the data processing device 2 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated here.

[0149] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium, which stores a computer program executed by the aforementioned data processing computer device 8000. The computer program includes program instructions, and when the processor executes the program instructions, it can execute the aforementioned... Figures 3 to 8 The description of the data processing method in the corresponding embodiments is already provided and will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application.

[0150] The aforementioned computer-readable storage medium can be an internal storage unit of the data processing apparatus or computer device provided in any of the foregoing embodiments, such as a hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0151] One aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in one aspect of the embodiments of this application.

[0152] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0153] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0154] The methods and related apparatuses provided in this application are described with reference to the method flowcharts and / or structural diagrams provided in this application. Specifically, each block of the method flowchart and / or structural diagram, as well as combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to create a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the process. Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 A process or multiple processes and / or structures illustrate the steps of the functions specified in one or more boxes.

[0155] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data processing method, characterized in that, include: Receive a playback request for multimedia data sent by a terminal device, and obtain real-time network layer data and real-time application layer data of the multimedia data based on the playback request; Obtain the media duration of the multimedia data and compare the media duration with a duration threshold; If the media duration exceeds the duration threshold, the bitrate prediction enhancement model is trained online based on the real-time network layer data and the real-time application layer data to obtain the target bitrate prediction enhancement model; the bitrate prediction enhancement model refers to the model obtained by expanding both the depth and width of the network structure of the bitrate prediction model that has been trained offline. The media prediction bitrate of the multimedia data is determined using the target bitrate prediction enhancement model, the real-time network layer data, and the real-time application layer data, and the multimedia data is transcoded based on the media prediction bitrate.

2. The method according to claim 1, characterized in that, The step of training the bitrate prediction enhancement model online based on the real-time network layer data and the real-time application layer data to obtain the target bitrate prediction enhancement model includes: The real-time network layer data and the real-time application layer data are input into the bitrate prediction enhancement model, and the bitrate prediction enhancement model outputs a sample media predicted bitrate that matches the real-time network layer data and the real-time application layer data. Obtain a first quality assessment feedback function, and generate a sample quality assessment feedback value for the predicted bitrate of the sample media based on the predicted bitrate of the sample media and the first quality assessment feedback function. The bitrate prediction enhancement model is adjusted based on the sample quality assessment feedback value to obtain the target bitrate prediction enhancement model.

3. The method according to claim 2, characterized in that, The step of generating a sample quality assessment feedback value for the predicted bitrate of the sample media based on the predicted bitrate of the sample media and the first quality assessment feedback function includes: Obtain the stuttering time information and bitrate fluctuation information corresponding to the predicted bitrate of the sample media; Based on the stuttering time information, the bitrate fluctuation information, the predicted bitrate of the sample media, and the evaluation feedback function, a sample quality evaluation feedback value is generated for the predicted bitrate of the sample media.

4. The method according to claim 2, characterized in that, The step of adjusting the bitrate prediction enhancement model based on the sample quality assessment feedback value to obtain the target bitrate prediction enhancement model includes: Obtain a quality assessment feedback threshold and determine the assessment difference between the sample quality assessment feedback value and the quality assessment feedback threshold; The real-time network layer data and the real-time application layer data are identified as real-time cross-layer data. If the evaluation difference exceeds the difference range, it is determined that the matching degree between the sample media predicted bitrate and the real-time cross-layer data does not meet the matching condition. The network parameters in the bitrate prediction enhancement model are adjusted according to the evaluation difference to obtain the target bitrate prediction enhancement model. If the evaluation difference is within the range of the difference, then the matching degree between the sample media predicted bitrate and the real-time cross-layer data is determined to meet the matching condition, and the bitrate prediction enhancement model is determined as the target bitrate prediction enhancement model.

5. The method according to claim 4, characterized in that, The step of adjusting the network parameters in the bitrate prediction enhancement model based on the evaluation difference to obtain the target bitrate prediction enhancement model includes: The remaining network parameters are obtained from the network parameters included in the bitrate prediction enhancement model. The remaining network parameters refer to the network parameters included in the bitrate prediction enhancement model other than the fixed network parameters. The fixed network parameters refer to the network parameters belonging to the bitrate prediction model that has been trained offline. The remaining network parameters are adjusted based on the evaluation difference to obtain the adjusted network parameters; The bitrate prediction enhancement model that includes the fixed network parameters and the adjusted network parameters is determined as the target bitrate prediction enhancement model.

6. The method according to claim 1, characterized in that, After comparing the media duration with a duration threshold, the method further includes: If the media duration is less than the duration threshold, the real-time network layer data and the real-time application layer data are input into the offline-trained bitrate prediction model. The initial media prediction bitrate of the multimedia data is determined by the offline-trained bitrate prediction model, the real-time network layer data, and the real-time application layer data. The real-time quality assessment feedback value of the multimedia data at the first moment is determined based on the initial media prediction bitrate; the first moment refers to the moment when the initial media prediction bitrate is determined. The predicted quality assessment feedback value of the multimedia data at the second time point is obtained. Based on the real-time quality assessment feedback value and the predicted quality assessment feedback value, the initial media predicted bitrate is optimized online to obtain the media optimized bitrate of the multimedia data. The second time point is later than the first time point. The multimedia data is transcoded based on the media-optimized bitrate.

7. The method according to claim 6, characterized in that, The step of obtaining the prediction quality assessment feedback value of the multimedia data at the second time point includes: Obtain the prediction system state corresponding to the multimedia data at the second time point; Obtain the initial variable coefficients corresponding to the prediction system state, and perform calculations between the initial variable coefficients and the prediction system state to obtain the calculated state value; Obtain a second quality assessment feedback function, and determine the prediction quality assessment feedback value of the multimedia data at the second time step based on the second quality assessment feedback function, the prediction system state, and the operation state value.

8. The method according to claim 7, characterized in that, The prediction system state includes the media code information, media download rate, and media buffer capacity of the multimedia data at the second time point; The step of performing calculations on the initial variable coefficients and the prediction system state to obtain the calculated state value includes: The initial variable coefficients are processed with the media code element information to obtain the computation code elements; The initial variable coefficients are processed in conjunction with the media download rate to obtain the calculated download rate; The initial variable coefficients are processed in conjunction with the media buffer capacity to obtain the calculation buffer capacity; The operation code, the operation download rate, and the operation buffer capacity are added together to obtain the operation status value.

9. The method according to claim 7, characterized in that, The step of optimizing the initial media prediction bitrate based on the real-time quality assessment feedback value and the predicted quality assessment feedback value to obtain the media-optimized bitrate of the multimedia data includes: Obtain the bitrate optimization function corresponding to the multimedia data; According to the bitrate optimization function, the real-time quality assessment feedback value and the predicted quality assessment feedback value are processed to obtain the calculation feedback value; If the computational feedback value is less than the feedback value threshold, then the absolute value of the difference between the computational feedback value and the feedback value threshold is determined, and the initial media prediction bitrate and the initial variable coefficient are adjusted according to the absolute value of the difference to obtain the transition media prediction bitrate corresponding to the initial media prediction bitrate and the transition variable coefficient corresponding to the initial variable coefficient. Based on the predicted bitrate of the transition media, the real-time quality assessment feedback value of the multimedia data at the first moment is determined, and the transition variable coefficients are processed with the state of the prediction system to obtain the transition operation state value. Based on the second quality assessment feedback function, the prediction system state, and the transition operation state value, the transition prediction quality assessment feedback value of the multimedia data at the second time point is determined; Based on the transition prediction quality assessment feedback value and the transition real-time quality assessment feedback value, the transition media prediction bitrate is optimized online to obtain the media-optimized bitrate of the multimedia data.

10. The method according to claim 9, characterized in that, The method further includes: If the computational feedback value is greater than the feedback value threshold, then the initial media prediction bitrate is determined as the media prediction bitrate of the multimedia data.

11. A data processing apparatus, characterized in that, include: The acquisition module is used to receive a playback request for multimedia data sent by the terminal device, and acquire the real-time network layer data and real-time application layer data of the multimedia data based on the playback request. The comparison module is used to obtain the media duration of the multimedia data and compare the media duration with a duration threshold. The training module is used to train the bitrate prediction enhancement model online based on the real-time network layer data and the real-time application layer data if the media duration exceeds the duration threshold, so as to obtain the target bitrate prediction enhancement model; the bitrate prediction enhancement model refers to the model obtained by expanding the depth and width of the network structure of the bitrate prediction model that has been trained offline. The prediction module is used to determine the media prediction bitrate of the multimedia data using the target bitrate prediction enhancement model, the real-time network layer data, and the real-time application layer data. A transcoding module is used to transcode the multimedia data based on the predicted bitrate of the media.

12. A computer device, characterized in that, include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide network communication functions, the memory is used to store computer programs, and the processor is used to call the computer programs to cause the computer device to execute the method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and to execute the method according to any one of claims 1-10.

Images