Data processing method and device and readable storage medium

Abstract

The embodiment of the invention discloses a data processing method and device and a readable storage medium, and the method comprises the steps: obtaining a target frame in a business application, and generating an image quality value of the target frame according to a frame evaluation parameter of the target frame; obtaining an image quality standard parameter matched with the service application, and generating a recommended adjustment code rate for the next frame of the target frame according to the frame evaluation parameter, the image quality value of the target frame and the image quality standard parameter; the recommended adjustment code rate is used for controlling the image quality value corresponding to the next frame to be matched with the image quality standard parameter. By adopting the method and the device, the overall image quality can be ensured to be constant in a mode of dynamically adjusting the code rate.

Description

Technical Field

[0001] The present invention relates to the field of computer science and technology, and in particular to a data processing method, device and readable storage medium. Background Art

[0002] The data processing method is widely used in digital audio processing. By calculating the bit rate of the picture frame in different business scenarios, the method can optimize the encoding performance while the bit rate of the output video meets the restriction requirements.

[0003] The data processing method using a constant bit rate can be implemented through the following steps: first, a fixed channel transmission bandwidth or a fixed bit rate is set, and then each video frame is encoded with the fixed bandwidth or bit rate, thereby quickly obtaining a video with a constant bit rate.

[0004] It can be seen that the video quality is unstable when using the constant bit rate method, and the complexity of the video content is not taken into account, which easily leads to insufficient bit rate and low picture quality in complex scenes. Summary of the invention

[0005] The embodiments of the present application provide a data processing method, device, and readable storage medium, which can ensure the overall image quality to be constant by dynamically adjusting the bit rate.

[0006] On the one hand, an embodiment of the present application provides a data processing method, including:

[0007] Acquire a target frame in a service application, and acquire a frame evaluation parameter of the target frame, wherein the frame evaluation parameter includes any one or more of a quality evaluation index or coding information;

[0008] Generate an image quality value of the target frame according to the frame evaluation parameter of the target frame;

[0009] Obtaining image quality standard parameters that match the business application, and generating a recommended adjustment bit rate for the next frame of the target frame according to the frame evaluation parameters, the image quality value of the target frame, and the image quality standard parameters;

[0010] It is recommended to adjust the bit rate to control the image quality value corresponding to the next frame to match the image quality standard parameters.

[0011] An embodiment of the present application provides a data processing device, including:

[0012] A first acquisition module is used to acquire a target frame in a business application and acquire a frame evaluation parameter of the target frame, where the frame evaluation parameter includes any one or more of a quality evaluation index or coding information;

[0013] A first generating module, used for generating an image quality value of a target frame according to a frame evaluation parameter of the target frame;

[0014] The second generation module is used to obtain a picture quality standard parameter that matches the business application, and generate a recommended adjustment bit rate for the next frame of the target frame according to the frame evaluation parameter, the picture quality value of the target frame and the picture quality standard parameter;

[0015] It is recommended to adjust the bit rate to control the image quality value corresponding to the next frame to match the image quality standard parameter.

[0016] Wherein, the first generation module includes:

[0017] A first input unit, used to input a frame evaluation parameter of a target frame into the image quality evaluation model;

[0018] A first feature extraction unit is used to perform feature extraction processing on the frame evaluation parameters of the target frame in the feature extraction layer of the image quality evaluation model to obtain a target feature vector corresponding to the frame evaluation parameters of the target frame;

[0019] A first feature fusion unit is used to perform convolution calculation on the target feature vector and the convolution kernel in the feature fusion layer of the image quality assessment model to obtain a feature vector matrix;

[0020] The first prediction unit is used to perform a weighted sum operation on the feature information and the weight in the feature vector matrix in the fully connected layer of the image quality assessment model to predict the image quality value of the target frame.

[0021] Wherein, the second generation module includes:

[0022] A first acquisition unit, used to acquire image quality standard parameters matching the business application;

[0023] A second input unit, used to input a frame evaluation parameter of a target frame, a picture quality value of the target frame, and a picture quality standard parameter into the bit rate recommendation model;

[0024] A second feature extraction unit is used to perform feature extraction processing on the frame evaluation parameter of the target frame, the image quality value of the target frame, and the image quality standard parameter in the feature extraction layer of the bit rate recommendation network, respectively, to obtain a first feature vector corresponding to the frame evaluation parameter of the target frame, a second feature vector corresponding to the image quality value of the target frame, and a third feature vector corresponding to the image quality standard parameter;

[0025] A second feature fusion unit is used to perform feature fusion on the first feature vector, the second feature vector and the third feature vector in a feature fusion layer of the bit rate recommendation network to obtain a fused feature vector;

[0026] The second prediction unit is used to identify the fused feature vector in the fully connected layer of the bit rate recommendation network, and predict the recommended adjusted bit rate.

[0027] Among them, it also includes:

[0028] A second acquisition module is used to acquire a picture quality standard parameter fluctuation range corresponding to the picture quality standard parameter;

[0029] A first setting module, configured to set the bit rate of the next frame of the target frame to the bit rate of the target frame when the image quality value of the target frame is within the fluctuation range of the image quality standard parameter;

[0030] The second setting module is used to adjust the bit rate of the next frame of the target frame according to the code control rule and the recommendation to adjust the bit rate when the image quality value of the target frame is not within the fluctuation range of the image quality standard parameter.

[0031] Among them, the second setting module is specifically used to determine the bit rate upper limit for the business application as the bit rate threshold; if the recommended adjusted bit rate is less than the bit rate threshold, the bit rate of the next frame of the target frame is determined as the recommended adjusted bit rate; if the recommended adjusted bit rate is greater than or equal to the bit rate threshold, the bit rate of the next frame of the target frame is determined as the bit rate threshold.

[0032] Among them, it also includes:

[0033] A third acquisition module is used to acquire a first sample target frame for a sample application; the first sample target frame is annotated with a picture quality value label; the application type of the sample application is the same as the application type of the business application, and the picture quality value label is obtained based on the picture quality evaluation standard of the sample application;

[0034] The first training module is used to input the sample frame evaluation parameters corresponding to the sample target frame into the initial image quality evaluation model, and output the predicted image quality value corresponding to the sample target frame through the initial image quality evaluation model;

[0035] The first training module is further used to generate a picture quality assessment loss value according to the predicted picture quality value and the picture quality value label;

[0036] The first training module is also used to correct the model parameters in the initial image quality assessment model through the image quality assessment loss function to obtain the image quality assessment model.

[0037] Among them, it also includes:

[0038] A fourth acquisition module is used to acquire a second sample target frame for the sample application; the second sample target frame is annotated with a recommended adjustment bit rate label; the recommended adjustment bit rate label is annotated based on a recommended adjustment bit rate evaluation standard for the sample application;

[0039] The second training module is used to input the sample frame evaluation parameter corresponding to the second sample target frame, the sample image quality value corresponding to the second sample target frame, and the image quality standard parameter corresponding to the second sample target frame into the initial bit rate recommendation model, and output the predicted recommended adjustment bit rate corresponding to the sample target frame through the initial bit rate recommendation model;

[0040] The second training module is further used to generate a bit rate recommendation loss value according to the predicted recommended bit rate adjustment and the recommended bit rate adjustment label;

[0041] The second training module is also used to correct the model parameters in the initial bit rate recommendation model through the bit rate recommendation loss value to obtain the bit rate recommendation model.

[0042] On the one hand, an embodiment of the present application provides a computer device, including: a processor, a memory, and a network interface;

[0043] The above-mentioned processor is connected to the above-mentioned memory and the above-mentioned network interface, wherein the above-mentioned network interface is used to provide a data communication function, the above-mentioned memory is used to store a computer program, and the above-mentioned processor is used to call the above-mentioned computer program to execute the method in the embodiment of the present application.

[0044] On one hand, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded by a processor and executing the method in the embodiment of the present application.

[0045] On the one hand, an embodiment of the present application provides a computer program product or a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method in the embodiment of the present application.

[0046] It can be seen that the embodiment of the present application can generate a picture quality value of the target frame according to the frame evaluation parameters of the target frame, and then generate a recommended adjustment bit rate for the next frame of the target frame according to the frame evaluation parameters, the picture quality value of the target frame and the picture quality standard parameters; the recommended adjustment bit rate can be used to control the picture quality value corresponding to the next frame to match the picture quality standard parameters, so that the overall picture quality can be kept constant by dynamically adjusting the bit rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0049] Figure 2 It is a scenario schematic diagram of a data processing method provided in an embodiment of the present application;

[0050] Figure 3 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0051] Figure 4 It is a flowchart of another data processing method provided in an embodiment of the present application;

[0052] Figure 5 It is a flowchart of another data processing method provided in an embodiment of the present application;

[0053] Figure 6 is a structural diagram of a picture quality assessment model provided in an embodiment of the present application;

[0054] Figure 7 It is a structural diagram of a bit rate recommendation model provided in an embodiment of the present application;

[0055] Figure 8 It is a training diagram of a bit rate recommendation model provided in an embodiment of the present application;

[0056] Fig. 9 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0057] Fig.10 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0060] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0061] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes data processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0062] Rate control (RC) is a very important technical module in video encoders. Rate control is used for various application scenarios. By controlling the quantization parameter (QP) of each frame or even each coding unit, the coding performance of the video, including coding efficiency and subjective quality, is optimized as much as possible while its bit rate meets various specific restrictions.

[0063] The solution provided in the embodiments of the present application involves artificial intelligence computer vision technology, deep learning, bit rate control and other technologies, and the specific process is explained through the following embodiments.

[0064] See also Figure 1 , Figure 1Schematic diagram of a system architecture provided by an embodiment of the present application. Figure 1 As shown, the system architecture may include a server 10a and a terminal cluster, and the terminal cluster may include: terminal device 10b, terminal device 10c, ..., terminal device 10d, wherein there may be a communication connection between the terminal clusters, for example, there is a communication connection between terminal device 10b and terminal device 10c, and there is a communication connection between terminal device 10b and terminal device 10d. At the same time, any terminal device in the terminal cluster may have a communication connection with the server 10a, for example, there is a communication connection between terminal device 10b and server 10a, wherein the above communication connection does not limit the connection method, and may be directly or indirectly connected by wired communication, or directly or indirectly connected by wireless communication, or by other methods, which are not limited in this application.

[0065] It should be understood that Figure 1 Each terminal device in the terminal cluster shown in the figure can be installed with an application client. When the application client runs on each terminal device, it can be respectively connected to the above Figure 1Data interaction is performed between the servers 10a shown. Among them, the application client can be an entertainment client (for example, a game client), a multimedia client (for example, a video client), a social client, an educational client, a live broadcast client, etc., which has a frame sequence (for example, a frame animation sequence) loading and playing function. Among them, the application client can be an independent client, or it can be an embedded sub-client integrated in a client (for example, a multimedia client, a social client, an educational client, etc.), which is not limited here. The server 10a provides services for the terminal cluster through the communication function. When the terminal device (which can be a terminal device 10b, a terminal device 10c or a terminal device 10d) obtains the video frame A and needs to process the video frame A, for example, detecting the picture quality of the video frame A and adjusting the bit rate for the next frame of the video frame A, the terminal device can send the video frame A to the server 10a through the above-mentioned application client. It can be understood that when the terminal device plays the video online, the video frame A is transmitted to the server for processing by the method of transmitting the picture frame in real time. After receiving the video frame A sent by the terminal device, the server 10a obtains the frame evaluation parameter B (such as including encoding information and quality evaluation index) of the video frame A according to the video frame A, and the server 10a can identify the image quality value C of the video frame A according to the frame evaluation parameter B. Further, the server 10a can perform bit rate prediction based on the obtained frame evaluation parameter B, the image quality value C and the set image quality standard parameter D, and obtain the recommended adjustment bit rate E for the next frame of the video frame A. The server 10a can adjust the bit rate for the next frame of the video frame A according to the recommended adjustment bit rate E and the set bit rate threshold, so as to return the adjusted video frame (i.e., the next frame with the adjusted bit rate) to the application client of the terminal device in real time. After receiving the adjusted video frame sent by the server 10a, the application client of the terminal device can display the video frame with the bit rate adjusted by the server 10a on its corresponding screen. Among them, the frame evaluation parameter B can include one or more other parameters that can evaluate the image quality value of the target frame, and only the encoding information and the quality evaluation index are used as examples for explanation here. In addition, the above-mentioned image quality value C may vary due to different evaluation criteria for model fitting.

[0066] It is understandable that the method provided in the embodiment of the present application can be performed by a computer device, and the computer device includes but is not limited to a terminal device or a server. The server 10a in the embodiment of the present application can be a computer device. The above-mentioned server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The above-mentioned terminal device can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a PDA, a mobile Internet device (mobile internet device, MID), a wearable device (such as a smart watch, a smart bracelet, etc.), a smart TV, a smart car, etc., which can run the above-mentioned application client. The number of terminal devices and servers is not limited, and the two can be directly or indirectly connected by wired or wireless communication, and this application is not limited here.

[0067] It can be understood that the present application can be applied to the picture frame data processing scheme in various forms. The following is an example of processing the real-time picture frame of a cloud game. Please refer to Figure 2 , is a scenario diagram of a data processing method provided in an embodiment of the present application. Figure 2 As shown, the computer device implementing the data processing scenario may include modules such as frame evaluation parameter acquisition, image quality value generation, and recommended adjustment bit rate generation. The implementation process of these modules can be as follows: Figure 1 The execution is performed in the server 10a shown in the figure. The present application does not limit the execution subject, and only takes the server as the execution subject as an example for explanation. Figure 2As shown, the terminal user sends the real-time picture frame of the cloud game to the server 20b through the terminal device 20a, where the real-time picture frame of the cloud game can be used as the target frame 20c. First, the terminal device 20a can fix the picture quality level 20f of the picture frame according to the application, for example, the full score is 100 points, and the picture quality level 20f can be set to 90 points, or different picture quality levels 20f for users to choose from can be displayed on the screen, and users can choose the picture quality level 20f that suits them. For example, the user can be directly provided with a choice of picture quality levels, such as 60 points, 90 points, and 100 points; the frame rate level can also be provided to the user, and the corresponding picture quality level can be set in the application according to different frame rate levels. The quality score indirectly provides the user with a choice of image quality level 20f. For example, when the user selects 30 frames, the server may consider that the user selects an image quality level of 60 points. When the user selects 60 frames, the server may consider that the user selects an image quality level of 90 points. It should be noted that the value of the above-mentioned image quality level 20f is equal to the image quality standard parameter. After the user selects the image quality level, the server will process the picture frames around the image quality level. Optionally, the image quality level may also be determined by the default setting of the server.

[0068] After receiving the processing request sent by the terminal device 20a, the server 20b will call the relevant application to identify the frame evaluation parameter 20d of the target frame 20c. The above-mentioned relevant application can be an application that can identify the encoding information of the picture frame or video, and the embodiment of the present application is not limited to this. For example, the frame evaluation parameter 20d may include at least one of SATD (sum of absolute transformed difference), QP (quantizer parameter), MV (motion vector), FL (frame length, the length of the current frame), RES (resolution) or BR (bitrate). It should be noted that the above-mentioned frame evaluation information can be any combination of one or more parameter information, and the present application is not limited to this. Then call the relevant interface (such as a network interface, and can also be in other forms, which are not limited by the embodiment of the present application) to transmit the target frame 20c to the processing model. Among them, the processing model may include a picture quality evaluation model 20e and a bit rate recommendation model 20g. Specifically, the image quality assessment model 20e first performs data preprocessing on the frame assessment parameter 20d of the incoming target frame 20c to obtain an initial feature vector, and then determines the high-order terms of the obtained initial feature vector as the first target feature vector. Finally, the first target feature vector is predicted by the image quality assessment function in the image quality assessment model 20e to obtain the image quality value of the target frame 20c. The bit rate recommendation model 20g first uses the image quality value predicted by the image quality assessment model 20e, the frame assessment parameter 20d of the target frame 20c, and the image quality standard parameter obtained according to the image quality gear 20f as the input of the bit rate recommendation model 20g. After data preprocessing and feature conversion, the second target feature vector is obtained. Finally, the bit rate recommendation function in the bit rate recommendation model 20g predicts the second target feature vector to obtain the recommended adjusted bit rate for the next frame of the target frame 20c.

[0069] Further, the server 20b can adjust the bit rate of the next frame of the target frame 20c according to the above-mentioned recommended bit rate adjustment and code control rules, and obtain the processed next frame 20h for the target frame 20c. The code control rules can be set according to the actual application scenario. The embodiment of the present application only takes a possible code control rule as an example. The specific process can be: according to the above-mentioned set picture quality standard parameters, obtain the picture quality quality standard parameter fluctuation range that fluctuates around the picture quality quality standard parameters. For example: if the set picture quality quality standard parameter is 90, the picture quality quality standard parameter fluctuation range can be [89.5, 90.5], or it can be other reasonable values ​​set according to the business application scenario. The set picture quality quality standard parameter can be regarded as an upper limit value, a lower limit value, an intermediate value, etc., and the present application does not impose any restrictions on this. If the image quality value of the target frame 20c is within the fluctuation range of the image quality standard parameter, the bit rate of the next frame of the target frame 20c does not need to be adjusted and remains consistent with the bit rate of the current target frame 20c. For example, if the image quality value of the target frame 20c is 90 and the bit rate is 2.5Mbps, the bit rate of the next frame of the target frame 20c is still 2.5Mbps; if the image quality value of the target frame 20c is not within the fluctuation range of the image quality standard parameter, the bit rate of the next frame of the target frame 20c is adjusted according to the recommended adjustment bit rate predicted by the bit rate recommendation model 20g. In actual application scenarios, a bit rate threshold of the picture frame can also be set to prevent the bit rate from being too high. When the recommended adjustment bit rate is higher than the bit rate threshold, the bit rate of the next frame of the target frame 20c is adjusted to the bit rate threshold. For example, if the recommended adjustment bit rate is 5Mbps and the bit rate threshold is 4Mbps, the bit rate of the next frame of the target frame 20c should be adjusted to 4Mbps. It should be noted that the data shown in the above embodiments are all fictitious data. In actual application, the actual data should be taken as the standard.

[0070] For further information, see Figure 3 , is a flow chart of another data processing method provided in an embodiment of the present application. The data processing method can be Figure 1 The method may be executed by a terminal device or a server, or may be executed by a terminal device and a server together. In the embodiment of the present application, the method is described by taking the method executed by a server as an example. Figure 3 As shown, the method may include the following steps:

[0071] Step S101, obtaining a target frame in a service application, and obtaining a frame evaluation parameter of the target frame, where the frame evaluation parameter includes any one or more of a quality evaluation index or coding information;

[0072] Specifically, the terminal user transmits the video clip to be processed to the server through the service application in the terminal device, and the server can obtain the video clip to be processed. The above-mentioned video clip to be processed may belong to a video stored locally or a video obtained in real time, such as a real-time game screen or a video clip shot in real time. The server can use a video editor or a video editing algorithm to decode the video clip to be processed, for example, through Adobe Premiere Pro (a commonly used video editing software), Fast Forward Mpeg (FFmpeg for short, a set of open source computer programs that can be used to record, convert digital audio and video, and convert them into streams) and Open CV (a cross-platform computer vision and machine learning software library that can run on multiple operating systems) to decode the video clip to be processed. Finally, the server can obtain each frame of the video clip to be processed and obtain the target frame in each frame of the video clip to be processed. Optionally, if the terminal user is in a scene such as live viewing, video playback, cloud game use, etc., the terminal device can also transmit the picture frame to the server in real time, and the server determines the received picture frame as the target frame. The corresponding frame evaluation parameters of the target frame can be obtained through the above-mentioned target frame. The frame evaluation parameters may include quality evaluation indicators or coding information, such as PSNR (peak signal to noise ratio), SSIM (structural similarity index) and other quality evaluation indicators or SATD, QP, MV, FL, FR, Res and other coding information. The frame evaluation parameters of the target frame may include one or more of them.

[0073] Step S102, generating an image quality value of the target frame according to the frame evaluation parameters of the target frame;

[0074] Specifically, the server can use the frame evaluation parameters obtained in the above step S101 as the input of the image quality evaluation model, and output the image quality value of the target frame through the image quality evaluation model. The image quality evaluation model can use a lightweight fitting algorithm, such as polynomial regression or fully connected network algorithms, to fit the image quality evaluation standard and obtain an image quality evaluation model that meets the business application scenario. The image quality evaluation standard can change according to different business application scenarios. For example, the average subjective opinion score MOS of the user on the image quality can be used as the image quality evaluation standard, and then the image quality evaluation model that best meets the feelings of the current user group can be obtained.

[0075] Step S103, obtaining a picture quality standard parameter that matches the business application, and generating a recommended adjustment bit rate for the next frame of the target frame according to the frame evaluation parameter, the picture quality value of the target frame, and the picture quality standard parameter; the recommended adjustment bit rate is used to control the picture quality value corresponding to the next frame to match the picture quality standard parameter;

[0076] Specifically, after the above step S101, one or more frame evaluation parameters for the target frame can be obtained, and after the above step S102, the image quality value of the target frame can be obtained. Different image quality standard parameters are set according to different business application scenario requirements, and this application does not limit this. The above image quality standard parameter is a constant target quality score gear, which is used to evaluate whether the target frame image quality value predicted by the image quality evaluation model meets the standard, and as one of the inputs of the bit rate recommendation model. It can be understood that the recommended adjustment bit rate for the next frame of the target frame generated by the bit rate recommendation model needs to use the above frame evaluation parameters, the image quality value of the target frame and the image quality standard parameters as the input of the bit rate recommendation model, and after processing by the bit rate recommendation model, the recommended adjustment bit rate for the next frame of the target frame is predicted. The recommended adjustment bit rate of the above target frame can make the image quality value of the next frame of the target frame match the image quality standard parameters when the recommended adjustment bit rate is set. It should be noted that the above-mentioned image quality assessment model and bit rate recommendation model can be the same network model or two independent network models, and both can use lightweight fitting algorithms. Such algorithm models can quickly derive prediction values ​​based on the input to ensure the speed of picture frame processing. This application does not impose any restrictions on this.

[0077] The following is an example of how to process the real-time scene of a cloud game. Figure 4 , is a flow chart of another data processing method provided by an embodiment of the present application. Figure 4 As shown, the data processing method can be applied to the real-time scene picture processing solution of cloud games, and the method can specifically include the following steps:

[0078] Step S1, setting the image quality standard parameters, setting the image quality standard parameter fluctuation range, and setting the bit rate threshold of the picture frame;

[0079] Step S2, the terminal device transmits the real-time picture frame of the cloud game to the server, and determines the real-time picture frame as the target frame;

[0080] Step S3, the server obtains frame evaluation parameters of the target frame;

[0081] Step S4, the server inputs the frame evaluation parameter into the image quality evaluation model, and outputs the image quality value of the target frame through the image quality evaluation model;

[0082] Step S5, the server inputs the image quality standard parameter obtained in step S1, the frame evaluation parameter obtained in step S3, and the image quality value obtained in step S4 into a bit rate recommendation model, and outputs a recommended adjusted bit rate through the bit rate recommendation model;

[0083] Step S6, the server determines whether the image quality value obtained in step S4 is within the fluctuation range of the image quality standard parameter obtained in step S1. If it is within the fluctuation range of the image quality standard parameter, the bit rate of the next frame of the target frame is consistent with the bit rate of the target frame. If it is not within the fluctuation range of the image quality standard parameter, the bit rate of the next frame of the current picture frame is adjusted according to the recommended adjustment bit rate obtained in step S5, and the highest bit rate does not exceed the bit rate threshold.

[0084] In step S7, the server inputs the adjusted next frame for the target frame as a new target frame and re-enters the above process until the terminal no longer transmits a new real-time picture frame.

[0085] The method provided in the embodiment of the present application can generate a picture quality value of the target frame according to the frame evaluation parameters of the target frame, and then generate a recommended adjustment bit rate for the next frame of the target frame according to the frame evaluation parameters, the picture quality value of the target frame and the picture quality standard parameters; the picture quality value corresponding to the next frame can be controlled to match the picture quality standard parameters through the recommended adjustment bit rate, so that the overall picture quality can be kept constant by dynamically adjusting the bit rate.

[0086] Furthermore, the above-mentioned cloud game real-time scene screen processing process is further elaborated. Figure 5 , is a flow chart of another data processing method provided in an embodiment of the present application. The data processing method can be Figure 1 The method can be executed by a terminal device or a server, or by a terminal device and a server together. In the embodiment of the present application, the method is described by taking the method executed by a server as an example. The model used in the data processing method can be a lightweight fitting algorithm model, and the method can include the following steps:

[0087] Step S501, obtaining a target frame in a service application, and obtaining a frame evaluation parameter of the target frame, wherein the frame evaluation parameter includes any one or more of a quality evaluation index or coding information;

[0088] For details, please refer to Figure 6 , is a schematic diagram of the structure of a picture quality assessment model provided by an embodiment of the present invention, Figure 6In the example, the picture frame sent by the terminal device to the server 30b can be used as the target frame 30a. Optionally, the terminal device can also send a video clip to the server 30b. The server 30b extracts the picture frames in the video clip after processing, and uses the first picture frame of the video clip as the target frame 30a. The server 30b first extracts the frame evaluation parameters of the target frame, such as the encoding information or quality evaluation index of the target frame, which are respectively the frame evaluation parameters C 1 , frame evaluation parameter C 2 , frame evaluation parameter C 3 , ..., frame evaluation parameter C i .

[0089] Step S502, obtaining image quality standard parameters matching the business application;

[0090] Specifically, the server may obtain the image quality standard parameters of the target frame. The image quality standard parameters may be set in the terminal device or may be uniformly set by the business scenario application. This application does not impose any restrictions on this.

[0091] Step S503, inputting the frame evaluation parameters of the target frame into the image quality evaluation model;

[0092] For details, please see Figure 6 , the server 30b obtains the frame evaluation parameter C of the target frame 1 , frame evaluation parameter C 2 , frame evaluation parameter C 3 , ..., frame evaluation parameter C i Then, the frame evaluation parameter C of the target frame is input into the image quality evaluation model 30c. 1 , frame evaluation parameter C 2 , frame evaluation parameter C 3 , ..., frame evaluation parameter C i .

[0093] Step S504, in the feature extraction layer of the image quality assessment model, performing feature extraction processing on the frame assessment parameters of the target frame to obtain a target feature vector corresponding to the frame assessment parameters of the target frame;

[0094] For details, please see Figure 6 The image quality assessment model 30c receives the frame assessment parameter C transmitted by the server 30b. 1 , frame evaluation parameter C 2 , frame evaluation parameter C 3 , ..., frame evaluation parameter C i Afterwards, feature extraction processing is performed on the frame evaluation parameters in the included feature extraction layer 301c to obtain the corresponding target feature vector [D1, D2, D3, ..., Di].

[0095] Step S505, in the feature fusion layer of the image quality assessment model, performing convolution calculation on the target feature vector and the convolution kernel to obtain a feature vector matrix;

[0096] For details, please see Figure 6 In the feature fusion layer 302c included in the image quality assessment model 30c, the above-mentioned target feature vector [D1, D2, D3, ..., Di] is convolved with the convolution kernel set according to the actual application scenario to obtain the feature vector matrix E.

[0097] Step S506, in the fully connected layer of the image quality assessment model, performing a weighted sum operation on the feature information and the weights in the feature vector matrix to predict the image quality value of the target frame;

[0098] For details, please see Figure 6 In the fully connected layer 303c included in the image quality assessment model 30c, the feature information in the feature vector matrix E is mapped and connected with the weights, and a weighted sum operation is performed on each item to obtain the image quality value of the target frame.

[0099] Optionally, the above-mentioned image quality assessment model can also be a polynomial regression model. Specifically, the server inputs the frame assessment parameters of the target frame into the image quality assessment model; in the data preprocessing module of the image quality assessment model, the frame assessment parameters of the target frame are preprocessed to obtain the initial feature vector corresponding to the frame assessment parameters of the target frame; in the feature conversion module of the image quality assessment model, the initial feature vector is feature converted to determine the high-order terms of the initial feature vector as the target feature vector; in the prediction module of the image quality assessment model, the target feature vector is predicted by the image quality assessment function to obtain the image quality value of the target frame. It can be understood that the image quality assessment model can also be other lightweight fitting models, and this application does not limit this.

[0100] It should be noted that before using the image quality assessment model and the bit rate recommendation model in actual business application scenarios, it is necessary to train these models with enough sample data so that the output of the model is consistent with the expected value. Specifically, taking the image quality assessment model as an example, a large number of sample frames must be collected first, and the frame evaluation parameters of each sample frame must be obtained as input for model training, and an image quality value label must be marked in each sample frame as a reference label for model training to participate in model convergence, where the frame evaluation parameters can be commonly used quality assessment indicators such as PSNR and SSIM, or coding information such as SATD, QP, MV, FL, FR, and Res, which are not limited in this embodiment of the present application, and the image quality value label can be the average subjective opinion score of the user on the image quality, which is not limited in this embodiment of the present application. After the frame evaluation parameters of the sample frame are input into the initial image quality evaluation model, data preprocessing, feature vector conversion, vector fitting and other operations are first performed to obtain the predicted image quality value, and then the loss function value is calculated based on the predicted image quality value and the above-mentioned image quality value label. The above-mentioned loss function value is then used to adjust the model parameters of the above-mentioned initial image quality evaluation model. The above steps are repeated until the model converges to obtain the image quality evaluation model.

[0101] Step S507, inputting the frame evaluation parameter of the target frame, the image quality value of the target frame and the image quality standard parameter into the bit rate recommendation model;

[0102] For details, see Figure 7 , is a schematic diagram of the structure of a bit rate recommendation model provided in an embodiment of the present application, Figure 7 In the example, the server may determine the frame evaluation parameters, picture quality value and picture quality standard parameters of the target frame as inputs to the bit rate recommendation model 40d, and the bit rate recommendation model 40d inputs the above frame evaluation parameters, picture quality value and picture quality standard parameters.

[0103] Step S508, in the feature extraction layer of the bit rate recommendation model, feature extraction processing is performed on the frame evaluation parameter of the target frame, the image quality value of the target frame, and the image quality standard parameter, respectively, to obtain a first feature vector corresponding to the frame evaluation parameter of the target frame, a second feature vector corresponding to the image quality value of the target frame, and a third feature vector corresponding to the image quality standard parameter;

[0104] For details, please see Figure 7 In the feature extraction layer 401d of the bit rate recommendation model 40d, feature extraction is performed on the frame evaluation parameters of the target frame to obtain a first feature vector, feature extraction is performed on the image quality value of the target frame to obtain a second feature vector, and feature extraction is performed on the image quality standard parameter to obtain a third feature vector.

[0105] Step S509, in the feature fusion layer of the bit rate recommendation model, feature fusion is performed on the first feature vector, the second feature vector and the third feature vector to obtain a fused feature vector;

[0106] For details, please see Figure 7 The first feature vector, the second feature vector and the third feature vector can be passed into the feature fusion layer 402d of the bit rate recommendation model 40d. In the feature fusion layer 402d, the first feature vector, the second feature vector and the third feature vector can be fused to obtain a fused feature. The fused feature is then convolved with a convolution kernel set according to the actual application scenario to obtain a feature vector matrix H.

[0107] Step S510, identifying the fused feature vector in the fully connected layer of the bit rate recommendation model, and predicting the recommended adjusted bit rate;

[0108] For details, please see Figure 7 , the feature vector matrix H is passed into the fully connected layer 403d of the bit rate recommendation model 40d, in which the feature information in the feature vector matrix H is mapped and linked with the weights, and a weighted sum operation is performed on each item to obtain the recommended adjusted bit rate of the target frame.

[0109] Optionally, the above-mentioned rate recommendation model can also be a polynomial regression model. Specifically, the server inputs the frame evaluation parameters of the target frame, the image quality value of the target frame, and the image quality standard parameters of the target frame into the rate recommendation model; in the data preprocessing module of the rate recommendation model, the frame evaluation parameters of the target frame, the image quality value of the target frame, and the image quality standard parameters are respectively preprocessed to obtain the first initial vector corresponding to the frame evaluation parameters of the target frame, the second initial vector corresponding to the image quality value of the target frame, and the third initial vector corresponding to the image quality standard parameters; in the feature conversion module of the rate recommendation model, the first initial vector, the second initial vector, and the third initial vector are feature converted, and the corresponding high-order terms are respectively determined as the corresponding first feature vector, the second feature vector, and the third feature vector; in the prediction module of the rate recommendation model, the first feature vector, the second feature vector, and the third feature vector are calculated using the rate recommendation function to predict the recommended adjustment bit rate for the next frame of the target frame. It can be understood that the rate recommendation model can also be other lightweight fitting models, and this application does not limit this.

[0110] It should be noted that the bitrate recommendation model also needs to be trained before it is used. For details, see Figure 8 , is a schematic diagram of a training of a bit rate recommendation model provided by an embodiment of the present invention. The training process of the bit rate recommendation model may be: firstly, a large number of sample frames (such as Figure 8 ), obtain the frame evaluation parameters, image quality value and image quality standard parameters of each sample frame and use them as the input of the bitrate recommendation model, annotate the recommended adjusted bitrate label in each sample frame as the reference label for model training and participate in model convergence. It should be noted that the above sample frames can use the same sample frames as the image quality evaluation model, or different sample frames, as long as the various parameters used as input can be obtained in the sample frames used. For example, Figure 8 In the example, sample frame 1 corresponds to picture quality standard parameter 1a and recommended adjustment bit rate label 1b; sample frame 1 may also correspond to picture quality standard parameter 1c and recommended adjustment bit rate label 1d; sample frame 1 may also correspond to picture quality standard parameter 1e and recommended adjustment bit rate label 1f; sample frame 2 corresponds to picture quality standard parameter 2a and recommended adjustment bit rate label 2b; sample frame 2 may also correspond to picture quality standard parameter 2c and recommended adjustment bit rate label 2d; sample frame 2 may also correspond to picture quality standard parameter 2e and recommended adjustment bit rate label 2f. This application does not limit the number of recommended adjustment bit rate labels marked on each sample frame. Figure 8 In the process, the frame evaluation parameters, picture quality value and picture quality standard parameters 1a of the above sample frame 1 can be input into the initial bit rate recommendation model. The initial bit rate recommendation model will perform data preprocessing, feature vector conversion, vector fitting and other operations on the frame evaluation parameters, picture quality value and picture quality standard parameters 1a of the sample frame 1 to obtain the predicted recommended adjusted bit rate, and then calculate the loss function value according to the predicted recommended adjusted bit rate and the recommended adjusted bit rate label 1b. Finally, the parameters of the above prediction function are adjusted using the above loss function value. Similarly, the above steps can be repeated through the picture quality standard parameters 1c and the recommended adjusted bit rate label 1d corresponding to the sample frame 1, and the above steps can also be repeated through the picture quality standard parameters 2a and the recommended adjusted bit rate label 2b corresponding to the sample frame 2 until the model converges to obtain the bit rate recommendation model.

[0111] Step S511, determining the bit rate upper limit value for the service application as the bit rate threshold;

[0112] Specifically, different upper limit values of the bit rate are set according to different business application scenarios, and the upper limit value of the bit rate is used as the bit rate threshold. The bit rate threshold can be set by the server or by the user, and this application does not limit this. The server further obtains the fluctuation range of the picture quality standard parameters corresponding to the picture quality standard parameters. If the picture quality value of the target frame is within the fluctuation range of the picture quality standard parameters, the bit rate of the next frame of the target frame is set to the bit rate of the target frame. The server can set an appropriate picture quality fluctuation range value according to the requirements of the business application scenario. Since this value needs to adapt to different business application scenario requirements, this application does not limit this here. Combining this range value with the above picture quality standard parameters to obtain the fluctuation range of the picture quality standard parameters. The fluctuation range of the picture quality standard parameters can have the picture quality standard parameters as the lower limit and the sum of the picture quality standard parameters and the picture quality fluctuation range value as the upper limit, or the difference between the picture quality standard parameters and the picture quality fluctuation range value as the lower limit and the picture quality standard parameters as the upper limit, or the difference between the picture quality standard parameters and the picture quality fluctuation range value as the lower limit and the sum of the picture quality standard parameters and the picture quality fluctuation range value as the upper limit. Different schemes can be selected according to different business application scenarios, and this application does not limit this here. For example, if the picture quality standard parameter is S target , and the picture quality fluctuation range value is x, the fluctuation range of the picture quality standard parameters can be [S target , S target + x], or [S target - x, S target , or [S target - x, S target + x].

[0113] Step S512, if the recommended adjusted bit rate is less than the bit rate threshold, the bit rate of the next frame of the target frame is determined as the recommended adjusted bit rate;

[0114] Specifically, if the server detects that the picture quality value of the target frame is not within the fluctuation range of the picture quality standard parameters, it further determines the relationship between the recommended adjusted bit rate and the bit rate threshold. If it is determined that the recommended adjusted bit rate is less than the bit rate threshold, according to the rate control rule, the bit rate of the next frame of the target frame is set to the recommended adjusted bit rate. For example, if the set bit rate threshold is A, the recommended adjusted bit rate is B, and the final bit rate of the next frame of the target frame should be set to C, then when A < B, C = A.

[0115] Step S513, if the recommended adjusted bit rate is greater than or equal to the bit rate threshold, the bit rate of the next frame of the target frame is determined as the bit rate threshold;

[0116] Specifically, if the server detects that the image quality value of the target frame is not within the fluctuation range of the image quality standard parameter, it further determines the relationship between the recommended adjustment bit rate and the bit rate threshold. If the judgment result that the recommended adjustment bit rate is greater than or equal to the bit rate threshold is obtained, the bit rate of the next frame of the target frame 40a is set to the bit rate threshold according to the code control rule. For example, if the set bit rate threshold is A, the recommended adjustment bit rate is B, and the bit rate of the next frame of the target frame should eventually be set to C, then when A≥B, C=B.

[0117] It should be noted that the data shown in the above embodiments are all fictitious data. In actual application, the actual numbers should be used as the basis. The code control rules can be set according to the actual application scenario. The embodiment of this application only takes a possible code control rule as an example, and this application does not impose any restrictions on this.

[0118] The method provided in the embodiment of the present application can generate a picture quality value of the target frame according to the frame evaluation parameters of the target frame, and then generate a recommended adjustment bit rate for the next frame of the target frame according to the frame evaluation parameters, the picture quality value of the target frame and the picture quality standard parameters; the picture quality value corresponding to the next frame can be controlled to match the picture quality standard parameters through the recommended adjustment bit rate, so that the overall picture quality can be kept constant by dynamically adjusting the bit rate; while quickly predicting the picture quality value of the target frame through a lightweight picture quality evaluation model, the recommended adjustment bit rate can also be quickly predicted through a lightweight bit rate recommendation model, so that the timeliness of the picture frame data processing can be guaranteed; the smoothness of the picture can be guaranteed through the code control rules, and the bit rate of the target frame can be guaranteed to be within a reasonable range.

[0119] See also Fig. 9 , is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. The above-mentioned data processing device may be a computer program (including program code) running on a computer device, for example, the data processing device is an application software; the device may be used to execute the corresponding steps in the method provided in an embodiment of the present application. Fig. 9 As shown, the data processing device 1 may include: a first acquisition module 101, a first generation module 102, and a second generation module 103;

[0120] A first acquisition module 101 is used to acquire a target frame in a service application and acquire a frame evaluation parameter of the target frame, where the frame evaluation parameter includes any one or more of a quality evaluation index or coding information;

[0121] A first generating module 102, configured to generate an image quality value of the target frame according to a frame evaluation parameter of the target frame;

[0122] The first generation module 102 is specifically used to predict the image quality value of the target frame through the image quality evaluation model after obtaining the frame evaluation parameters of the target frame, and the predicted image quality value is used as one of the inputs of the model used by the subsequent second generation module 103;

[0123] The second generating module 103 is used to obtain a picture quality standard parameter matching the service application, and generate a recommended adjustment bit rate for the next frame of the target frame according to the frame evaluation parameter, the picture quality value of the target frame and the picture quality standard parameter;

[0124] The above-mentioned second generation module 103 is specifically used to take the above-mentioned frame evaluation parameters, the image quality value of the target frame and the image quality standard parameters as input, and predict the recommended adjustment bit rate through the bit rate recommendation model. The image quality standard parameters are used to obtain the fluctuation range of the image quality standard parameters, allowing the picture frame quality to fluctuate within a certain range to ensure the smoothness of the picture. The recommended adjustment bit rate is used to adjust the bit rate of the next frame for the target frame so that it can meet the image quality standard.

[0125] The specific functional implementation of the first acquisition module 101 can be found in the above Figure 3 The corresponding step S101 in the embodiment, or, can refer to the above Figure 4 The corresponding steps S2 and S3 in the embodiment, or, refer to the above Figure 5 The specific functional implementation of the first generation module 102 can refer to the above Figure 3 The corresponding step S102 in the embodiment, or, can refer to the above Figure 4 The corresponding step S4 in the embodiment, or, can refer to the above Figure 5 The corresponding steps S503, S504, S505 and S506 in the embodiment, or, refer to the above Figure 6 The specific implementation of the second generation module 103 can refer to the above Figure 3 The corresponding step S103 in the embodiment, or, can refer to the above Figure 4 The corresponding steps S1 and S5 in the embodiment, or, can refer to the above Figure 5 Steps S502, S507, S508, S509 and S510 in the corresponding embodiment, or, refer to the above Figure 7 The corresponding embodiments will not be described in detail here.

[0126] See also Fig. 9 The above data processing device 1 may further include: a second acquisition module 104, a first setting module 105, and a second setting module 106;

[0127] The second acquisition module 104 is used to acquire the picture quality standard parameter fluctuation range corresponding to the picture quality standard parameter;

[0128] A first setting module 105 is used to set the bit rate of the next frame of the target frame to the bit rate of the target frame when the image quality value of the target frame is within the fluctuation range of the image quality standard parameter;

[0129] The second setting module 106 is used to adjust the bit rate of the next frame of the target frame according to the bit rate control rule and the recommendation when the image quality value of the target frame is not within the fluctuation range of the image quality standard parameter.

[0130] The above-mentioned second setting module 106 is specifically used to determine the bit rate upper limit for the business application as the bit rate threshold; if the recommended adjusted bit rate is less than the bit rate threshold, the bit rate of the next frame of the target frame is determined as the recommended adjusted bit rate; if the recommended adjusted bit rate is greater than or equal to the bit rate threshold, the bit rate of the next frame of the target frame is determined as the bit rate threshold.

[0131] The specific functional implementation of the second acquisition module 104 can be found in the above Figure 4 The corresponding step S1 in the embodiment, or, can refer to the above Figure 5 The specific description of the code control rules in the corresponding embodiment; the specific functional implementation of the first setting module 105 can be found in the above Figure 4 The corresponding step S6 in the embodiment, or, can refer to the above Figure 5 The specific description of the code control rules in the corresponding embodiment; the specific functional implementation of the second setting module 106 can be found in the above Figure 4 The corresponding step S6 in the embodiment, or, can refer to the above Figure 5 Steps S511, S512 and S513 in the corresponding embodiment are not described in detail here.

[0132] See also Fig. 9 , the above-mentioned data processing device 1 may further include: a third acquisition module 107, a first training module 108;

[0133] The third acquisition module 107 is used to acquire a first sample target frame for the sample application; the first sample target frame is annotated with a picture quality value label; the application type of the sample application is the same as the application type of the business application, and the picture quality value label is obtained based on the picture quality evaluation standard of the sample application;

[0134] The first training module 108 is used to input the sample frame evaluation parameters corresponding to the sample target frame into the initial image quality evaluation model, and output the predicted image quality value corresponding to the sample target frame through the initial image quality evaluation model;

[0135] The above-mentioned first training module 108 is specifically used to generate a picture quality assessment loss value based on the predicted picture quality value and the picture quality value label; and to correct the model parameters in the initial picture quality assessment model through the picture quality assessment loss function to obtain the picture quality assessment model.

[0136] The specific functional implementation of the third acquisition module 107 and the first training module 108 can refer to the above Figure 5 The specific description of the image quality assessment model in the corresponding embodiment will not be repeated here.

[0137] See also Fig. 9 , the above-mentioned data processing device 1 may further include: a fourth acquisition module 109, a second training module 110;

[0138] The fourth acquisition module 109 is used to acquire a second sample target frame for the sample application; the second sample target frame is annotated with a recommended adjustment bit rate label; the recommended adjustment bit rate label is annotated based on a recommended adjustment bit rate evaluation standard for the sample application;

[0139] The second training module 110 is used to input the sample frame evaluation parameter corresponding to the second sample target frame, the sample image quality value corresponding to the second sample target frame, and the image quality standard parameter corresponding to the second sample target frame into the initial bit rate recommendation model, and output the predicted recommended adjusted bit rate corresponding to the sample target frame through the initial bit rate recommendation model;

[0140] The second training module 110 is further specifically used to generate a rate recommendation loss value according to the predicted recommended adjusted rate and the recommended adjusted rate label; and to correct the model parameters in the initial rate recommendation model by the rate recommendation loss value to obtain the rate recommendation model.

[0141] The specific functional implementation of the fourth acquisition module 109 and the second training module 110 can be found in the above Figure 5 For the specific description of the bit rate recommendation model in the corresponding embodiment, or, please refer to the above Figure 8 The corresponding embodiments will not be described in detail here.

[0142] See also Fig. 9 , the first generation module 102 may include: a first input unit 1021, a first feature extraction unit 1022, a first feature fusion unit 1023, and a first prediction unit 1024;

[0143] A first input unit 1021, used to input frame evaluation parameters of a target frame into the image quality evaluation model;

[0144] A first feature extraction unit 1022 is used to perform feature extraction processing on the frame evaluation parameters of the target frame in the feature extraction layer of the image quality evaluation model to obtain a target feature vector corresponding to the frame evaluation parameters of the target frame;

[0145] A first feature fusion unit 1023 is used to perform convolution calculation on the target feature vector and the convolution kernel in the feature fusion layer of the image quality assessment model to obtain a feature vector matrix;

[0146] The first prediction unit 1024 is used to perform a weighted sum operation on the feature information and the weights in the feature vector matrix in the fully connected layer of the image quality assessment model to predict the image quality value of the target frame.

[0147] The specific function implementation of the first input unit 1021 can refer to the above Figure 5 The specific functional implementation of the first feature extraction unit 1022 can refer to the above step S503 in the corresponding embodiment. Figure 5 The specific functional implementation of the first feature fusion unit 1023 can refer to the above step S504 in the corresponding embodiment. Figure 5 The specific functional implementation of the first prediction unit 1024 can refer to the above Figure 5 Step S505 in the corresponding embodiment will not be described in detail here.

[0148] See also Fig. 9 , the second generation module 103 may include: a first acquisition unit 1031, a second input unit 1032, a second feature extraction unit 1033, a second feature fusion unit 1034, and a second prediction unit 1035;

[0149] The first acquisition unit 1031 is used to acquire image quality standard parameters matching the service application;

[0150] The second input unit 1032 is used to input the frame evaluation parameter of the target frame, the image quality value of the target frame and the image quality standard parameter into the bit rate recommendation model;

[0151] The second feature extraction unit 1033 is used to perform feature extraction processing on the frame evaluation parameter of the target frame, the image quality value of the target frame, and the image quality standard parameter in the feature extraction layer of the bit rate recommendation network, respectively, to obtain a first feature vector corresponding to the frame evaluation parameter of the target frame, a second feature vector corresponding to the image quality value of the target frame, and a third feature vector corresponding to the image quality standard parameter;

[0152] A second feature fusion unit 1034 is used to perform feature fusion on the first feature vector, the second feature vector and the third feature vector in a feature fusion layer of the bit rate recommendation network to obtain a fused feature vector;

[0153] The second prediction unit 1035 is used to identify the fused feature vector in the fully connected layer of the bit rate recommendation network, and predict the recommended adjusted bit rate.

[0154] The specific functional implementation of the first acquisition unit 1031 can be found in the above Figure 5 The specific function implementation of the second input unit 1032 can refer to the above Figure 5 The specific functional implementation of the second feature extraction unit 1033 can refer to the above Figure 5 The specific functional implementation of the second feature fusion unit 1034 can refer to the above Figure 5 The specific functional implementation of the second prediction unit 1035 can refer to the above Figure 5 Step S510 in the corresponding embodiment will not be described in detail here.

[0155] The embodiment of the present application can generate a picture quality value of the target frame according to the frame evaluation parameters of the target frame, and then generate a recommended adjustment bit rate for the next frame of the target frame according to the frame evaluation parameters, the picture quality value of the target frame and the picture quality standard parameters; the picture quality value corresponding to the next frame can be controlled to match the picture quality standard parameters through the recommended adjustment bit rate, so that the overall picture quality can be kept constant by dynamically adjusting the bit rate.

[0156] See also Fig.10 , is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.10As shown, the computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the above-mentioned computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. Among them, the user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally also be at least one storage device located away from the aforementioned processor 1001. As Fig.10 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0157] In such Fig.10 In the computer device 1000 shown, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0158] Acquire a target frame in a service application, and acquire a frame evaluation parameter of the target frame, wherein the frame evaluation parameter includes any one or more of a quality evaluation index or coding information;

[0159] Generate an image quality value of the target frame according to the frame evaluation parameter of the target frame;

[0160] Obtain image quality standard parameters that match the business application, and generate a recommended adjustment bit rate for the next frame of the target frame based on the frame evaluation parameters, the image quality value of the target frame, and the image quality standard parameters. The recommended adjustment bit rate is used to control the image quality value corresponding to the next frame to match the image quality standard parameters.

[0161] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 2 , Figure 3 , Figure 4 , Figure 5 The description of the data processing method in any corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated here.

[0162] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the data processing device 1 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, the computer program can execute the above-mentioned data processing device 1. Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 The description of the above data processing method in any corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0163] The computer-readable storage medium may be the data processing device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0164] In addition, it should be noted that: the embodiment of the present application also provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 Any method provided by the corresponding embodiment.

[0165] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0166] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that: include: Acquire a target frame in a service application, and acquire a frame evaluation parameter of the target frame, wherein the frame evaluation parameter includes any one or more of a quality evaluation index or coding information; Generating an image quality value of the target frame according to a frame evaluation parameter of the target frame; Acquire a picture quality standard parameter matching the service application, and generate a recommended adjustment bit rate for a next frame of the target frame according to the frame evaluation parameter, the picture quality value of the target frame, and the picture quality standard parameter; The recommended adjusted bit rate is used to control the image quality value corresponding to the next frame to match the image quality standard parameter.

2. The method according to claim 1, characterized in that Generating the image quality value of the target frame according to the frame evaluation parameter of the target frame includes: Inputting frame evaluation parameters of the target frame into an image quality evaluation model; In the feature extraction layer of the image quality assessment model, feature extraction processing is performed on the frame assessment parameters of the target frame to obtain a target feature vector corresponding to the frame assessment parameters of the target frame; In the feature fusion layer of the image quality assessment model, convolution calculation is performed on the target feature vector and the convolution kernel to obtain a feature vector matrix; In the fully connected layer of the image quality assessment model, the feature information and weights in the feature vector matrix are weighted and operated to predict the image quality value of the target frame.

3. The method according to claim 1, characterized in that The acquiring of the picture quality standard parameter matching the service application, and generating a recommended adjustment bit rate for the next frame of the target frame according to the frame evaluation parameter, the picture quality value of the target frame and the picture quality standard parameter, comprises: Acquire image quality standard parameters matching the business application; Inputting the frame evaluation parameter of the target frame, the image quality value of the target frame and the image quality standard parameter into the bit rate recommendation model; In the feature extraction layer of the bit rate recommendation model, feature extraction processing is performed on the frame evaluation parameter of the target frame, the image quality value of the target frame, and the image quality standard parameter, respectively, to obtain a first feature vector corresponding to the frame evaluation parameter of the target frame, a second feature vector corresponding to the image quality value of the target frame, and a third feature vector corresponding to the image quality standard parameter; In the feature fusion layer of the bit rate recommendation model, feature fusion is performed on the first feature vector, the second feature vector and the third feature vector to obtain a fused feature vector; In the fully connected layer of the bit rate recommendation model, the fused feature vector is identified and the recommended adjusted bit rate is predicted.

4. The method according to claim 1, characterized in that: Also includes: Obtaining a picture quality standard parameter fluctuation range corresponding to the picture quality standard parameter; If the image quality value of the target frame is within the image quality standard parameter fluctuation range, setting the bit rate of the next frame of the target frame to the bit rate of the target frame; If the image quality value of the target frame is not within the image quality standard parameter fluctuation range, the bit rate of the next frame of the target frame is adjusted according to the code control rule and the recommended bit rate adjustment.

5. The method according to claim 4, characterized in that The adjusting the bit rate of the next frame of the target frame according to the bit control rule and the recommended bit rate adjustment includes: Determine the bit rate upper limit value for the business application as the bit rate threshold; If the recommended adjusted bit rate is less than the bit rate threshold, determining the bit rate of the next frame of the target frame as the recommended adjusted bit rate; If the recommended adjusted bit rate is greater than or equal to the bit rate threshold, the bit rate of the next frame of the target frame is determined as the bit rate threshold.

6. The method according to claim 2, characterized in that Also includes: Acquire a first sample target frame for a sample application; The first sample target frame is marked with an image quality value label; The application type of the sample application is the same as the application type of the business application, and the image quality value label is obtained based on the image quality evaluation standard of the sample application; Inputting the sample frame evaluation parameters corresponding to the sample target frame into an initial image quality evaluation model, and outputting the predicted image quality value corresponding to the sample target frame through the initial image quality evaluation model; Generate a picture quality assessment loss value according to the predicted picture quality value and the picture quality value label; The model parameters in the initial image quality assessment model are corrected by using the image quality assessment loss function to obtain the image quality assessment model.

7. The method according to claim 3, characterized in that Also includes: acquiring a second sample target frame for the sample application; The second sample target frame is marked with a recommended adjusted bit rate label; The recommended adjusted bitrate label is obtained based on the recommended adjusted bitrate evaluation standard of the sample application; Inputting a sample frame evaluation parameter corresponding to the second sample target frame, a sample image quality value corresponding to the second sample target frame, and an image quality standard parameter corresponding to the second sample target frame into an initial bit rate recommendation model, and outputting a predicted recommended adjusted bit rate corresponding to the sample target frame through the initial bit rate recommendation model; Generate a recommended bitrate loss value according to the predicted recommended adjusted bitrate and the recommended adjusted bitrate label; The model parameters in the initial bit rate recommendation model are corrected according to the bit rate recommendation loss value to obtain a bit rate recommendation model.

8. A data processing device, characterized in that: include: A first acquisition module, used to acquire a target frame in a business application, and acquire a frame evaluation parameter of the target frame, wherein the frame evaluation parameter includes any one or more of a quality evaluation index or coding information; A first generating module, configured to generate an image quality value of the target frame according to a frame evaluation parameter of the target frame; A second generating module is used to obtain a picture quality standard parameter matching the business application, and generate a recommended adjustment bit rate for a next frame of the target frame according to the frame evaluation parameter, the picture quality value of the target frame and the picture quality standard parameter; The recommended adjusted bit rate is used to control the image quality value corresponding to the next frame to match the image quality standard parameter.

9. A computer device, characterized in that: include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide a data communication function, the memory is used to store program code, and the processor is used to call the program code to execute the method described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the method according to any one of claims 1 to 8.

Images