Data processing method and related device

Abstract

The invention provides a data processing method in the field of artificial intelligence and a related device, which are used for acquiring a picture to be subjected to frame reduction based on application information of an application video to be subjected to frame reduction and carrying out frame reduction processing on the picture to be subjected to frame reduction based on the content of the picture to be subjected to frame reduction so as to dynamically adjust a frame rate. The method comprises the following steps: acquiring application information of a first application on the terminal equipment; n frames of pictures in a first time period are obtained, the N frames of pictures indicate the content displayed by operation of the first application in the first time period, and N is a positive integer; according to the application information of the first application and the content of the N frames of pictures, the N frames of pictures are subjected to frame reduction, n frames of pictures are obtained, n is smaller than N, and n is a positive integer; and storing or sending the n frames of pictures.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular, to a data processing method and related devices. Background Art

[0002] With the increasing development and improvement of mobile phone operating systems and functions, they are also involved in more and more application scenarios in daily life. Among them, application scenarios such as casting the mobile phone screen to a car infotainment system, a large screen, a computer screen, etc. are becoming more and more common. However, in different screen casting scenarios, the user's usage purposes are different. For example, in the case of casting to a car infotainment system, users mainly have relatively frequent needs for navigation and music playback; in the case of casting to a large screen, users mainly have relatively frequent needs for office software and video playback.

[0003] Currently, screen casting is mainly divided into two major modes: push screen casting and mirror screen casting. Among them, the mirror screen casting mode records the screen or takes screenshots on the mobile phone while sending them to the TV, and the TV shows the data after receiving it to form a same-screen display. Since the mobile phone has to record the screen and send data at the same time, and the data sent is usually the RGB values of each pixel on each page of the picture, the amount of data sent is extremely large. The mobile phone will continuously work in a large data volume transmission scenario, resulting in extremely high power consumption on the mobile phone side, which will have negative impacts such as the mobile phone getting hot and the battery draining faster.

[0004] Existing solutions have addressed the above problems by adjusting the data acquisition time through the frame data interval time, thereby achieving dynamic frame rate adjustment. However, only considering the interval time, when the interval time remains unchanged, the acquisition frame rate will also be fixed. Collecting frame data according to the fixed frame rate may cause frame data that should be deleted to be retained, thus failing to achieve an effective frame reduction effect. Summary of the Invention

[0005] Embodiments of this application provide a data processing method and related devices, which are used to obtain pictures to be frame-reduced based on the application information of the application video to be frame-reduced, and perform frame reduction processing on the pictures to be frame-reduced based on the content of the pictures to be frame-reduced, thereby dynamically adjusting the frame rate.

[0006] In view of this, in a first aspect, this application provides a data processing method. In this method, when the terminal device performs screen casting, data transmission will be carried out between the terminal device and the receiving device. The terminal device can perform frame reduction processing on the obtained pictures to be screen-cast, and then transmit the frame-reduced pictures to the receiving device. The receiving device receives the pictures after frame reduction is completed and renders and displays them.

[0007] First, obtain the application information of the first application on the terminal device, where the application information includes one or more of categories or scenarios; obtain N frames of pictures within the first time period, where the N frames of pictures indicate the content displayed during the operation of the first application within the first time period, and the N frames of pictures are the picture frames to be frame-dropped; after obtaining the application information of the first application and the N frames of pictures to be frame-dropped, the N frames of pictures can be frame-dropped according to the application information and the content of the N frames of pictures to obtain n frames of pictures, where n is less than N; after obtaining the n frames of pictures after frame-dropping, the n frames of pictures can be stored or sent.

[0008] In the embodiments of the present application, pictures can be frame-dropped according to the application information of the application and the content of the pictures to obtain the pictures after frame-dropping, and the number of frames of the pictures after frame-dropping is less than the number of frames of the obtained pictures. Frame-dropping according to the application information of the application and the content of the pictures takes into account the characteristics of the application and the relevance of the picture content, and thus can dynamically adjust the output picture frames, thereby dynamically adjusting the frame rate.

[0009] In a possible implementation manner, before obtaining the N frames of pictures within the first time period, it may further include: obtaining M frames of pictures within the second time period, where the M frames of pictures indicate the content displayed during the operation of the first application within the second time period; storing or sending the M frames of pictures.

[0010] In the implementation manner of the present application, since frame-dropping calculation takes a certain amount of time, when starting screen mirroring, there is no reserved time for frame-dropping calculation. To avoid delaying data transmission, the pictures obtained in the first round are not frame-dropped, and before obtaining the pictures to be frame-dropped, the M frames of pictures obtained in the first round can be directly stored or sent.

[0011] In a possible implementation manner, the foregoing obtaining M frames of pictures within the second time period may include: performing screen recording on the first application at the first screen recording frame rate to obtain M frames of pictures.

[0012] In a possible implementation manner, the foregoing obtaining N frames of pictures within the first time period may include: performing screen recording on the first application at the second screen recording frame rate to obtain N frames of pictures, where the second screen recording frame rate is less than the first screen recording frame rate.

[0013] In a possible implementation manner, before performing screen recording on the first application at the second screen recording frame rate to obtain N frames of pictures, the method further includes: determining the second screen recording frame rate according to the application information of the first application and the first screen recording frame rate, and the second screen recording frame rates of the first applications with different application information are different.

[0014] In the embodiments of the present application, the screen recording frame rate can be adjusted according to the application information of the first application. The application information includes one or more of the category or the scenario. The screen recording frame rate can be adjusted according to different scenarios of the application. For example, when the navigation application switches from the vehicle driving scenario to the traffic light waiting scenario, the second screen recording frame rate can be adjusted based on the first screen recording frame rate. The second screen recording frame rate is less than the first screen recording frame rate, and the number of picture frames obtained based on the second screen recording frame rate is less than the number of picture frames obtained based on the first screen recording frame rate, reducing the occupied cache space, and thus can save energy consumption and memory.

[0015] In a possible implementation manner, the foregoing downsampling the N picture frames to obtain n picture frames according to the application information of the first application and the content of the N picture frames may include: downsampling the N picture frames through a retina perception model according to the application information of the first application and the content of the N picture frames to obtain n picture frames, and the similarity between the n picture frames is less than a preset threshold, and the preset thresholds of the first applications with different application information are different.

[0016] In the embodiments of the present application, the picture frames can be downsampled through a retina perception model, and the obtained picture frames are compared pairwise. When the similarity between the contents of two picture frames is higher than the threshold, any one of the two picture frames is deleted, so as to reduce the number of picture frames and complete the downsampling. And because the deleted pictures are pictures with high similarity, it will not affect the user experience and can ensure video smoothness. Different categories of applications have different requirements for judging the similarity between picture frames, and different thresholds can be set according to the basic information of the first application. For example, for a navigation application, the content difference between picture frames may not be large, so a higher threshold needs to be set to distinguish the difference between two picture frames. In addition, the downsampled pictures can be used for screen mirroring, reducing the amount of data transmitted from the terminal device to the receiving device, saving the power consumption of the terminal device, and can complete the downsampling on the terminal device, avoiding relying on the cooperation of the receiving end for downsampling.

[0017] In a possible implementation manner, if the first application switches to the second application, and the first application and the second application have different categories, the method further includes: obtaining X picture frames within a third time period, where the X picture frames are obtained by screen recording the second application at a third screen recording frame rate; storing or sending the X picture frames.

[0018] In a possible implementation manner, after storing or sending the X picture frames, the method further includes: obtaining the application information of the second application; obtaining Y picture frames within a fourth time period, where the Y picture frames are obtained by screen recording the second application at a fourth screen recording frame rate, Y is a positive integer, and the fourth screen recording frame rate is less than the third screen recording frame rate; downsampling the Y picture frames according to the application information of the second application and the content of the Y picture frames to obtain y picture frames, where y is less than Y and y is a positive integer; storing or sending the y picture frames.

[0019] In a second aspect, the present application provides a method for screening downsampled pictures, including:

[0020] Obtain the application information of a first application and a first screen recording frame rate;

[0021] Determine a second screen recording frame rate according to the application information of the first application and the first screen recording frame rate;

[0022] Obtain N pictures to be downsampled according to the second screen recording frame rate.

[0023] In a possible implementation manner, the foregoing determining the second screen recording frame rate according to the application information of the first application and the first screen recording frame rate may include: classifying the application information of the first application through a classification model to obtain a classification result of the first application; obtaining the weight of the first application according to the classification result; and determining the second screen recording frame rate according to the weight of the first application and the first screen recording frame rate.

[0024] In a possible implementation manner, the foregoing obtaining N pictures to be downsampled according to the second screen recording frame rate may include: performing screen recording on the first application at the second screen recording frame rate to obtain N pictures to be downsampled.

[0025] In a third aspect, the present application provides a downsampling method, including:

[0026] Obtain the application information of a first application and N pictures to be downsampled;

[0027] Downsample the N pictures according to the application information of the first application and the content of the N pictures to obtain n pictures, where n is less than N.

[0028] In a possible implementation manner, the foregoing downsampling the N pictures according to the application information of the first application and the content of the N pictures to obtain n pictures may include: determining a preset threshold according to the application information of the first application; and downsampling the N pictures through a retinal perception model according to the preset threshold to obtain n pictures, and the similarity between the n pictures is less than the preset threshold.

[0029] In a fourth aspect, the present application provides a data processing device, including:

[0030] An obtaining module, configured to obtain the application information of a first application on a terminal device;

[0031] The foregoing obtaining module is further configured to obtain N pictures within a first time period, where the N pictures indicate the content displayed during the operation of the first application within the first time period, and N is a positive integer;

[0032] A frame - down module, configured to perform frame - down on N frames of pictures according to the application information of the first application and the content of the N frames of pictures, to obtain n frames of pictures, where n is less than N and n is a positive integer;

[0033] A processing module, configured to store or send the n frames of pictures.

[0034] In a possible implementation manner, before obtaining N frames of pictures within the first time period, the apparatus further includes:

[0035] The above - mentioned acquisition module is further configured to obtain M frames of pictures within the second time period, where the M frames of pictures indicate the content displayed during the operation of the first application within the second time period;

[0036] The above - mentioned processing module is further configured to store or send the M frames of pictures.

[0037] In a possible implementation manner, the above - mentioned acquisition module is specifically configured to: perform screen recording on the first application at the first screen - recording frame rate to obtain M frames of pictures.

[0038] In a possible implementation manner, the above - mentioned acquisition module is specifically configured to: perform screen recording on the first application at the second screen - recording frame rate to obtain N frames of pictures, where the second screen - recording frame rate is less than the first screen - recording frame rate.

[0039] In a possible implementation manner, the application information includes at least one of category and scenario. After performing screen recording on the first application at the second screen - recording frame rate to obtain N frames of pictures, the apparatus further includes:

[0040] A determination module, configured to determine the second screen - recording frame rate according to the application information of the first application and the first screen - recording frame rate, and the second screen - recording frame rates of the first applications with different application information are different.

[0041] In a possible implementation manner, the above - mentioned frame - down module is specifically configured to: perform frame - down on the N frames of pictures through a retinal perception model according to the application information of the first application and the content of the N frames of pictures, to obtain n frames of pictures, where the similarity between the n frames of pictures is less than a preset threshold, and the preset thresholds of the first applications with different application information are different.

[0042] In a possible implementation manner, if the first application switches to the second application, and the categories of the first application and the second application are different, the apparatus further includes: the above - mentioned acquisition module is further configured to obtain X frames of pictures within the third time period, where the X frames of pictures are obtained by performing screen recording on the second application at the third screen - recording frame rate; the above - mentioned processing module is further configured to store or send the X frames of pictures.

[0043] In a possible implementation, after storing or sending the X-frame pictures, the device further includes: the above-mentioned acquisition module, which is further used to acquire the application information of the second application; the above-mentioned acquisition module, which is further used to acquire Y-frame pictures within a fourth time period, where the Y-frame pictures are obtained by recording the screen of the second application at a fourth screen recording frame rate, Y is a positive integer, and the fourth screen recording frame rate is less than the third screen recording frame rate; the frame reduction module, which is further used to reduce the frame rate of the Y-frame pictures according to the application information of the second application and the content of the Y-frame pictures to obtain y-frame pictures, where y is less than Y and y is a positive integer; the processing module, which is further used to store or send the y-frame pictures.

[0044] In a fifth aspect, the present application provides a frame reduction picture screening device, including:

[0045] An acquisition module, configured to acquire the application information of the first application and the first screen recording frame rate;

[0046] A determination module, configured to determine a second screen recording frame rate according to the application information of the first application and the first screen recording frame rate;

[0047] A processing module, configured to obtain N-frame pictures to be frame-reduced according to the second screen recording frame rate.

[0048] In a possible implementation, the above-mentioned determination module is specifically configured to: classify the application information of the first application through a classification model to obtain the classification result of the first application; obtain the weight of the first application according to the classification result; determine the second screen recording frame rate according to the weight of the first application and the first screen recording frame rate.

[0049] In a possible implementation, the above-mentioned processing module is specifically configured to: record the screen of the first application at the second screen recording frame rate to obtain N-frame pictures to be frame-reduced.

[0050] In a sixth aspect, the present application provides a frame reduction device, including:

[0051] An acquisition module, configured to acquire the application information of the first application and the N-frame pictures to be frame-reduced;

[0052] A frame reduction module, configured to reduce the frame rate of the N-frame pictures according to the application information of the first application and the content of the N-frame pictures to obtain n-frame pictures, where n is less than N.

[0053] In a possible implementation, the above-mentioned frame reduction module is specifically configured to: determine a preset threshold according to the application information of the first application; reduce the frame rate of the N-frame pictures through a retinal perception model according to the preset threshold to obtain n-frame pictures, and the similarity between the n-frame pictures is less than the preset threshold.

[0054] Seventh aspect, the present application provides a terminal device, which includes: a processor, a memory, an input / output device, and a bus; computer instructions are stored in the memory; when the processor executes the computer instructions in the memory, computer instructions are stored in the memory; when the processor executes the computer instructions in the memory, it is used to implement any implementation manner in the first aspect, the second aspect, or the third aspect.

[0055] Eighth aspect, an embodiment of the present application provides a chip system, which includes a processor and an input / output port. The processor is used to implement the processing functions involved in the method described in the first aspect, the second aspect, or the third aspect above, and the input / output port is used to implement the transceiver functions involved in the method described in any possible implementation manner in the first aspect, the second aspect, or the third aspect above.

[0056] In a possible design, the chip system further includes a memory, and the memory is used to store program instructions and data for implementing the functions involved in the method described in any possible implementation manner in the first aspect, the second aspect, or the third aspect above.

[0057] The chip system can be composed of chips or can include chips and other discrete devices.

[0058] Ninth aspect, an embodiment of the present application provides a computer-readable storage medium. Computer instructions are stored in the computer-readable storage medium; when the computer instructions run on a computer, the computer is caused to execute the method described in any possible implementation manner in the first aspect, the second aspect, or the third aspect.

[0059] Tenth aspect, an embodiment of the present application provides a computer program product. The computer program product includes a computer program or instructions, and when the computer program or instructions run on a computer, the computer is caused to execute the method described in any possible implementation manner in the first aspect, the second aspect, or the third aspect. Description of the Drawings

[0060] Figure 1 It is a schematic diagram of a complete process of mobile phone screen mirroring provided by the present application;

[0061] Figure 2 It is a schematic diagram of a system framework provided by the present application;

[0062] Figure 3 It is a schematic diagram of a process of a data processing method provided by the present application;

[0063] Figure 4 It is a schematic diagram of a process of another data processing method provided by the present application;

[0064] Figure 5 A flowchart showing a method for screening dropped-frame pictures provided by this application;

[0065] Figure 6 A flowchart showing a method for dropping frames provided by this application;

[0066] Figure 7 A schematic structural diagram of a data processing device provided by this application;

[0067] Figure 8 A schematic structural diagram of a dropped-frame picture screening device provided by this application;

[0068] Figure 9 A schematic structural diagram of a dropped-frame device provided by this application;

[0069] Figure 10 A schematic structural diagram of another data processing device provided by this application;

[0070] Figure 11 A schematic structural diagram of another dropped-frame picture screening device provided by this application;

[0071] Figure 12 A schematic structural diagram of another dropped-frame device provided by this application;

[0072] Figure 13 A schematic structural diagram of a chip provided by this application. Detailed implementation manners

[0073] Next, the technical solutions in the embodiments of this application will be described with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only some embodiments of this application, rather than all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of this application.

[0074] The method provided by this application can be applied to artificial intelligence (AI) scenarios. AI uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, sense the environment, acquire knowledge, and use knowledge to obtain the best results in terms of theory, method, technology, and application systems. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines to enable machines to have functions of perception, reasoning, and decision-making. Research in the field of artificial intelligence includes robots, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, AI basic theory, etc.

[0075] First, the overall workflow of the artificial intelligence system is described. Below, the above-mentioned artificial intelligence theme framework is elaborated from two dimensions: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be the general processes of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes the refinement process of "data - information - knowledge - wisdom". The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of artificial intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system.

[0076] (1) Infrastructure

[0077] The infrastructure provides computing power support for the artificial intelligence system, enables communication with the external world, and is supported through the basic platform. It communicates with the external world through sensors; the computing power is provided by intelligent chips (hardware acceleration chips such as CPU, NPU, GPU, ASIC, FPGA, etc.); the basic platform includes relevant platform guarantees and supports such as distributed computing frameworks and networks, and can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the external world to obtain data, and these data are provided to the intelligent chips in the distributed computing system provided by the basic platform for computing.

[0078] (2) Data

[0079] The data at the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voices, texts, and also involves the Internet of Things data of traditional devices, including the business data of existing systems and the sensed data such as force, displacement, liquid level, temperature, humidity, etc.

[0080] (3) Data Processing

[0081] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0082] Among them, machine learning and deep learning can perform symbolic and formal intelligent information modeling, extraction, preprocessing, training, etc. on data.

[0083] Reasoning refers to the process of simulating the intelligent reasoning method of humans in a computer or intelligent system, and using formal information to perform machine thinking and solve problems according to the reasoning control strategy. The typical function is search and matching.

[0084] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, prediction, etc.

[0085] (4) General capabilities

[0086] After the data is processed through the above-mentioned data processing, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system. For example, it can be translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0087] (5) Intelligent products and industry applications

[0088] Intelligent products and industry applications refer to the products and applications of artificial intelligence systems in various fields. It is the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes the landing application. Its application fields mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0089] The embodiments of this application involve related applications of neural networks. To better understand the solutions of the embodiments of this application, the following first introduces the related terms and concepts of neural networks that may be involved in the embodiments of this application and the related terms and concepts that may be involved in the embodiments of this application.

[0090] (1) Neural network

[0091] A neural network can be composed of neural units. A neural unit can refer to an operation unit that takes xs and intercept 1 as inputs. The output of this operation unit can be as shown in formula (1-1):

[0092]

[0093] Among them, s = 1, 2,..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit (activation functions), which is used to introduce non-linear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting multiple such single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of several neural units.

[0094] (2) Deep neural network

[0095] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple intermediate layers. Dividing the DNN according to the positions of different layers, the neural networks inside the DNN can be divided into three categories: the input layer, the intermediate layer, and the output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the intermediate layers are all intermediate layers, or are called hidden layers. The layers are fully connected between each other, that is to say, any neuron in the i-th layer must be connected to any neuron in the (i + 1)-th layer.

[0096] Although the DNN seems very complex, each layer can be expressed as a linear relationship expression: Among them, is the input vector, is the output vector, is the offset vector or is called the bias parameter, w is the weight matrix (also called the coefficient), and α() is the activation function. Each layer is just to perform such a simple operation on the input vector to obtain the output vector Due to the large number of layers in the DNN, the number of coefficients W and the offset vector is also relatively large. The definitions of these parameters in the DNN are as follows: Taking the coefficient w as an example: Suppose in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer where the coefficient W is located, and the subscripts correspond to the index 2 of the output third layer and the index 4 of the input second layer.

[0097] To sum up, the coefficient from the k-th neuron in the (L - 1)-th layer to the j-th neuron in the L-th layer is defined as

[0098] It should be noted that there is no W parameter in the input layer. In a deep neural network, more intermediate layers enable the network to better depict complex situations in the real world. Theoretically speaking, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is also the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrices of all layers of the trained deep neural network (the weight matrix formed by vectors W of many layers).

[0099] (3) Convolutional neural network

[0100] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract features is independent of position. Convolution kernels can be formalized as matrices of random size, and convolution kernels can obtain reasonable weights through learning during the training process of convolutional neural networks. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0101] (4) Graph Convolutional Network (GCN)

[0102] Graph neural network is a deep learning model that models and processes non-Euclidean spatial data (such as graph data). Its principle is to use pairwise message passing so that graph nodes iteratively update their corresponding representations by exchanging information with their neighbors.

[0103] GCN is similar to CNN, but the difference is that the input of CNN is usually two-dimensional structured data, while the input of GCN is usually graph structured data. GCN has cleverly designed a method to extract features from graph data, so that these features can be used to perform node classification, graph classification, link prediction, and graph embedding.

[0104] (5) Loss Function

[0105] During the process of training a deep neural network, since we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two. (Of course, there is usually a process of parameterization before the first update, that is, configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is too high, we adjust the weight vector to make it predict lower, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function. They are important equations for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. Then the training of the deep neural network becomes a process of minimizing this loss as much as possible. The loss function usually can include mean squared error, cross entropy, logarithm, exponential and other loss functions. For example, mean squared error can be used as the loss function, defined as Specifically, the specific loss function can be selected according to the actual application scenario.

[0106] (6) Backpropagation algorithm

[0107] An algorithm for calculating the gradient of model parameters according to the loss function and updating the model parameters. The neural network can use the error backpropagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, forward propagating the input signal until the output will generate an error loss, and updating the parameters in the initial neural network model by backpropagating the error loss information, so that the error loss converges. The backpropagation algorithm is a reverse propagation movement dominated by the error loss, aiming to obtain the parameters of the optimal neural network model, such as the weight matrix.

[0108] (7) Retinal perception model

[0109] The retinal perception model is a model that explains how visual information is processed and interpreted. The retinal perception model can be used to compare two frames of pictures. Since there are many factors causing differences between pictures, multiple factors can be considered comprehensively, such as color, brightness, contrast, texture, edges, and shapes. Thus, the comparison result between the pictures can be obtained through the retinal perception model.

[0110] (8) Frame rate

[0111] Frame rate is a unit of measurement used to express the number of frames updated per second in a computer graphics or motion capture system. It is also called frame frequency or frames per second (fps). Frame rate is expressed in Hertz (Hz). A high frame rate can produce smoother and more realistic animation or action. Generally speaking, the higher the frame rate of a computer graphics or motion capture system, the smoother the action or picture will be displayed.

[0112] (9) Frames

[0113] Frame rate is the abbreviation of the number of frames generated. Each frame is a still image. Displaying frames in rapid succession creates the illusion of motion. A high frame rate can produce smoother and more realistic animations.

[0114] The following introduces the location and scenario of the deployment of this application solution in the system.

[0115] See also Figure 1 , Figure 1 This is a complete process diagram of mobile phone screen projection. When the screen projection function is turned on, the mobile phone will record or capture the screen to obtain the picture to be projected. After encoding and packaging the picture, it will be sent to the receiving device via WiFi, such as a car computer, computer and other devices. After receiving the data, the receiving device will unpack and decode the data to obtain the projected data, and then render it on the screen for display. In the traditional screen projection process, when the frame rate of the video to be projected is high, the mobile phone will record or capture the screen, encode, package and other steps at high speed, causing the mobile phone CPU / GPU to perform a lot of calculations and consume power.

[0116] For the mobile phone screen projection process, the present application solution can be arranged before or after the video and audio encoding. The addition of the present application solution can complete the frame reduction before the screen projection, and the process can be completed only on the mobile phone side (i.e., the screen projection sending side). The picture received by the receiving device is already a picture after the frame reduction, and there is no need to rely on the receiving device to cooperate with the algorithm for frame reduction. After the frame reduction calculation module of the present application solution, the number of picture frames obtained is less than the number of picture frames to be projected, which reduces the amount of data that needs to be transmitted for the screen projection, thereby saving the power consumption of the mobile phone side (i.e., the screen projection sending side), and also saving the power consumption of the receiving device.

[0117] The following introduces the system framework provided by the embodiments of the present application.

[0118] See also Figure 2, an embodiment of the present application provides a system framework. The data source 210 can be used to obtain the pictures to be screen-cast and the application information of the application to be screen-cast. After the data source 210 obtains the foregoing data, these data are stored in the cache buffer 201. The chip / CPU / hardware accelerator 202 can perform frame rate reduction calculation based on the obtained pictures to be screen-cast and the application information of the application, and output the pictures after frame rate reduction for screen-casting. The controller 203 can be used to control the size of the cache buffer 201, that is, it can control the number of frames of the pictures to be screen-cast read by the cache buffer 201 based on the screen-casting frame rate calculated by the chip / CPU / hardware accelerator 202. That is, when the screen-casting frame rate is calculated for each round, the buffer 201 obtains the number of frames of the pictures for frame rate reduction based on the screen-casting frame rate, so that the cache size can be controlled according to the size of the screen-casting frame rate. When the screen-casting frame rate is small, there is no need to occupy too much cache, thus saving energy consumption and memory.

[0119] It should be noted that Figure 2 is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 2 , the cache buffer 201 is an internal memory relative to the frame rate reduction calculation module 220. In other cases, the cache buffer 201 can also be placed outside the frame rate reduction calculation module 220.

[0120] Such as Figure 2 shown, the chip / CPU / hardware accelerator 202 can include the retina perception model in the present application in the embodiment of the present application. Through the chip / CPU / hardware accelerator 202, the number of frames of the pictures in the present application and the pictures after frame rate reduction can be obtained.

[0121] Next, in combination with the foregoing system architecture and application scenarios, the method flow provided by the present application will be introduced.

[0122] Refer to Figure 3 , a schematic flow chart of a data processing method provided by the present application is as follows.

[0123] 301. Obtain the application information of the first application;

[0124] Among them, the application information of the first application includes one or more of the category or scenario of the first application. The category can be a navigation application, a food delivery application, a chat application, or a video playback application. Specifically, it is not limited here. Then the scenario can be a navigation scenario, a food delivery scenario, a video call scenario, or a drama-watching scenario, etc. Specifically, it is not limited here.

[0125] 302. Obtain N frames of pictures within the first time period;

[0126] Generally, the frame dropping calculation takes a certain amount of time. Therefore, when the terminal device performs data processing in the first round, no frame dropping operation is performed. Thus, before obtaining N frames of pictures to be frame-dropped, M frames of pictures can be directly stored or sent.

[0127] Optionally, before obtaining N frames of pictures in the first time period, M frames of pictures can also be obtained. These M frames of pictures indicate the content displayed by the first application during the second time period, and the M frames of pictures can be stored or sent.

[0128] Optionally, the M frames of pictures can be obtained by screen recording the first application at the first screen recording frame rate.

[0129] Specifically, the first application can be screen recorded at the first screen recording frame rate during the second time period to obtain a first video; subsequently, picture extraction is performed on the first video to obtain M frames of pictures.

[0130] After completing the first-round data processing, the second screen recording frame rate can be determined according to the application information of the first application and the first screen recording frame rate, and then the first application is screen recorded at the second screen recording frame rate to obtain N frames of pictures.

[0131] Optionally, the application information of the first application can be classified through a classification model to obtain a classification result of the first application. The classification result includes a high threshold and a low threshold; according to the classification result of the first application, the weight corresponding to the first application can be obtained; subsequently, the first screen recording frame rate can be adjusted according to this weight to obtain the second screen recording frame rate.

[0132] Among them, applications of different categories in different classification results correspond to different weights. For example, the weight corresponding to an application with a high threshold can be 70%, or it can also be 80%; the weight corresponding to an application with a low threshold can be 10%, or it can also be 20%.

[0133] In the embodiments of the present application, the first screen recording frame rate is adjusted according to the weight of the first application to obtain the second screen recording frame rate, and the second screen recording frame rate is less than the first screen recording frame rate. When the terminal device obtains N frames of pictures to be frame-dropped at the second screen recording frame rate in the second round, since the receiving end device will receive M frames of pictures and M is greater than N, the time for the receiving end to receive M frames of pictures will be greater than the time for the terminal device to obtain N frames of pictures. Therefore, this period of time can be used for subsequent frame dropping processing.

[0134] 303. Frame-drop the N frames of pictures according to the application information of the first application and the content of the N frames of pictures to obtain n frames of pictures;

[0135] After obtaining N frames of images, the N frames of images can be downscaled through a retinal perception model to obtain n frames of images. The N frames of images can also be downscaled through a neural network model, and the specific details are not limited here.

[0136] Optionally, based on the application information of the first application and the content of N frames of pictures, the N frames of pictures can be compared through a retinal perception model, by comparing the similarity of two adjacent frames of pictures, or by comparing the similarity between three adjacent frames of pictures. If the similarity is higher than a preset threshold, any one of the two frames of pictures is deleted, thereby completing the frame reduction of the N frames of pictures. Among them, the number of picture frames for similarity comparison can be selected according to the application category of the first application. For picture frames with smaller differences in picture content, more adjacent frames can be selected for comparison. For picture frames with larger differences in picture content, pairwise comparisons can be performed. In addition, the preset threshold can be set according to the application information of the first application, and different categories of applications can set different preset thresholds.

[0137] Among them, the embodiment of the present application can reduce the frame rate according to the application information of the application and the content of the picture. Therefore, the number of picture frames n obtained after the frame reduction process is not fixed, and can be dynamically adjusted according to the category of the application and the content of the picture, thereby realizing dynamic adjustment of the frame rate, and the obtained number of picture frames n is less than N, which reduces the number of picture frames, thereby reducing the amount of data transmitted during the sending process and reducing the power consumption of the terminal device.

[0138] Optionally, when the first application is switched to the second application, and the categories of the first application and the second application are different, for example, switching from a navigation application to a food delivery application. Since the application information of the application changes suddenly, frame reduction processing cannot be performed immediately, and therefore, frame reduction processing is still not performed for the first round of data processing after the application is switched, and the second application in the third period is recorded at a third recording frame rate to obtain X frames of images to be processed, and the X frames of images can be directly stored or sent.

[0139] Optionally, after storing or sending the X-frame picture, the Y-frame picture can be downgraded according to the application information of the second application and the content of the Y-frame picture to be processed to obtain the Y-frame picture; then, the Y-frame picture can be stored or sent. The Y-frame picture can be obtained by recording the second application at a fourth screen recording frame rate, the fourth screen recording frame rate is less than the third screen recording frame rate, and the method for determining the fourth screen recording frame rate is similar to the method for determining the second screen recording frame rate, which will not be repeated here.

[0140] 304. Store or send n frames of pictures.

[0141] After obtaining n frames of pictures, the n frames of pictures can be stored to achieve video compression after frame rate reduction processing, or the n frames of pictures can be sent to the receiving device for screen mirroring or storage. Specifically, it is not limited here.

[0142] Among them, sending the pictures after frame rate reduction reduces the amount of data transmitted during the sending process, thereby reducing the power consumption of the sending end, i.e., the terminal device, and also reducing the power consumption of the receiving device.

[0143] Refer to Figure 4 the schematic flowchart of another data processing method provided by this application, which is described as follows.

[0144] 401. Obtain the first video to be screen mirrored, the application information of the first application to be screen mirrored, and the first screen recording frame rate.

[0145] Among them, the first video to be screen mirrored can be obtained by screen recording the first application on the device to be screen mirrored at the first screen recording frame rate. The application information of the first application is as described above and will not be elaborated here.

[0146] 402. Extract pictures from the first video and screen mirror the obtained M frames of pictures.

[0147] After obtaining the first video to be screen mirrored, pictures can be extracted from the first video to obtain a series of pictures arranged in sequence, that is, the M frames of pictures to be screen mirrored.

[0148] Since screen mirroring delays are to be avoided during the first round of screen mirroring, the obtained pictures to be screen mirrored are not subject to frame rate reduction. Therefore, the obtained M frames of pictures can be directly screen mirrored.

[0149] 403. Determine the second screen recording frame rate according to the application information of the first application and the first screen recording frame rate.

[0150] Among them, different types of applications have different second screen recording frame rates obtained according to the first screen recording frame rate. The first screen recording frame rate can be adjusted according to the weights corresponding to different applications to obtain the second screen recording frame rate. The specific process of determining the second screen recording frame rate according to the application information of the first application and the first screen recording frame rate is similar to the process of determining the second screen recording frame rate in step 302 as described above and will not be elaborated here. Figure 3 in step 302 as described above and will not be elaborated here.

[0151] 404. Extract pictures from the obtained second video to obtain N frames of pictures.

[0152] After obtaining the second screen recording frame rate, the first application can be screen recorded at the second screen recording frame rate to obtain the second video, which includes the content displayed during the operation of the second application; pictures can be extracted from the second video to obtain N frames of pictures to be subject to frame rate reduction.

[0153] 405. Downsample the N-frame images according to the application information of the first application and the content of the N-frame images to obtain n-frame images;

[0154] Among them, according to the application information of the first application and the image content, the N-frame images can be downsampled through a retinal perception model to obtain the downsampled images.

[0155] Generally, the application information of the first application may include the category or the scenario. When the application information is the category, the scenario of the application can be inferred according to the category. Different scenarios have different impacts on image downsampling. For example, the navigation scenario pays more attention to the changes in the road, and the content of the images in different frames changes less. For the drama-watching scenario, the content of the images in different frames changes more. Therefore, according to the application information of different applications, different preset thresholds can be set for the image similarity of different applications to provide a screening basis for subsequent downsampling.

[0156] Specifically, the retinal perception model can compare the obtained N-frame images pairwise in the arrangement order of the N consecutive images. By comparing the similarity of two frames of images, according to the specific content of the images and the correlation between the images, any one of the two frames of images with a similarity higher than the preset threshold is deleted, and then the downsampled n-frame images are obtained. Among them, for the navigation scenario, a higher preset threshold can be set, and for the drama-watching scenario, a lower preset threshold can be set.

[0157] Generally, the videos of different applications at different times are different, so the images to be cast are also different. Downsampling is performed according to the content of the images and the correlation between the images, realizing dynamic downsampling according to the image content. And by deleting any one of the two frames of images with a similarity higher than the preset threshold through the retinal perception model, it will not affect the user experience. While ensuring the smoothness and integrity of the video information composed of the images received by the screen-casting receiving end, downsampling is achieved.

[0158] 406. When the first application switches to the second application, cast the X-frame images to be cast;

[0159] During the screen-casting process, the application to be cast may change. When the application switches, both the application information and the video content of the application will change. At this time, the X-frame images to be cast can be re-obtained. The X-frame images can be obtained by recording the second application at the third screen recording frame rate, or can be obtained by taking a screenshot of the screen by the device to be cast. The specific method is not limited here.

[0160] When an application sends a switch, it is the same as when casting the screen for the first time. Therefore, for the first round of screen casting after the application switch, for the obtained X-frame pictures, no frame reduction processing is performed, and the obtained X-frame pictures can be directly used for screen casting.

[0161] 407. Obtain the application information of the second application to be screen-cast and the Y-frame pictures to be frame-reduced;

[0162] Among them, the Y-frame pictures to be frame-reduced can be obtained by recording the second application at the fourth screen recording frame rate. The method for determining the fourth screen recording frame rate is similar to that in step 303 described above, and will not be elaborated here specifically. Figure 3 The steps for determining the fourth screen recording frame rate are similar to those in step 303 as described above, and will not be elaborated here specifically.

[0163] 408. Frame-reduce the Y-frame pictures according to the application information of the second application and the content of the Y-frame pictures to obtain y-frame pictures.

[0164] After obtaining the Y-frame pictures to be frame-reduced, frame reduction can be performed through a retina perception model or a neural network model. The specific method is not limited here, so as to obtain y-frame pictures, where y is less than Y, thus completing frame reduction, and then casting the frame-reduced pictures, thereby saving the power consumption of the screen-casting device. The process of specifically performing frame reduction using the retina perception model has been described above and will not be elaborated here.

[0165] In the embodiments of the present application, according to the application information of the application to be screen-cast and the first video frame rate, N-frame pictures for frame reduction calculation are obtained. Subsequently, according to the application information of the application to be screen-cast and the content of the N-frame pictures, the N-frame pictures are frame-reduced, and any one of the two frames of pictures with a similarity higher than the preset threshold is deleted, thereby obtaining n-frame pictures, where n is less than N. The obtained n-frame pictures are used as the pictures to be screen-cast for screen casting. This reduces the amount of picture data transmitted from the terminal device to the receiving device, thereby saving the power consumption of the terminal device and the receiving device, and can perform frame reduction calculation according to the application information of the application and the content of the pictures, taking into account the characteristics of the application, and can dynamically adjust the output picture frames, thereby dynamically adjusting the frame rate.

[0166] For ease of understanding, below, by way of example, taking some specific application scenarios of some applications as examples, the effects of the data processing method provided by the present application will be introduced.

[0167] In the embodiments of the present application, the applicable scenarios may include any one of a navigation scenario, a food delivery scenario, a video call scenario, or a TV drama watching scenario. Taking the navigation scenario and the video call scenario as examples below, the method provided by the embodiments of the present application is used to perform frame dropping calculation on the acquired original video to obtain the pictures after frame dropping, and combine them into a new video. The duration of the new video is the same as that of the original video. Furthermore, the frame rates of the original video and the new video can be compared (i.e., the total number of pictures / duration), and then the degree of frame rate reduction can be determined.

[0168] The results are shown in Table 1, where the cut value is the threshold parameter setting in the frame dropping calculation.

[0169] Table 1

[0170]

[0171] Taking the screen mirroring scenario of Tencent Map in the navigation scenario as an example, the original frame rate can be reduced by 28% to 34% while ensuring the user experience (i.e., the visual fluency effect is still maintained after frame rate reduction). Taking the screen mirroring scenario of WeChat in the video call scenario as an example, the original frame rate can be reduced by 11% to 20% while ensuring the user experience (i.e., the visual fluency effect is still maintained after frame rate reduction), thereby reducing the power consumption of the mobile phone and improving the user experience.

[0172] Based on the foregoing data processing method, the frame dropping picture screening process and the frame dropping process in the data processing are introduced below.

[0173] Refer to Figure 5 , the schematic flowchart of a frame dropping picture screening method provided by the present application is as follows.

[0174] 501. Classify the application information of the first application through a classification model to obtain the classification result of the first application;

[0175] Generally, since the frame dropping calculation takes a certain amount of time, after the first round of screen mirroring, the first screen recording frame rate can be adjusted according to the application information of the first application. When obtaining the pictures to be frame dropped in the second round, a screen recording frame rate smaller than the first screen recording frame rate is used. At this time, the duration for the terminal device to transmit N pictures to be frame dropped will be less than the duration for the receiving device to receive the M non-frame dropped pictures sent in the first round, thus reserving time for the frame dropping calculation.

[0176] Among them, the classification model can be a neural network model, a decision tree model, or a support vector machine model. Specifically, it is not limited here. After obtaining the application information of the first application, the application information of the first application can be classified through the classification model to obtain the classification result of the first application.

[0177] 502. Obtain the weight of the first application according to the classification result;

[0178] Among them, the classification results include two major categories: high threshold and low threshold. Each classification result includes multiple categories of applications, and the weights corresponding to different categories of applications are different. Therefore, after obtaining the classification result of the first application, the weight of the first application can be determined according to the corresponding classification result of the first application.

[0179] 503. Obtain the second screen recording frame rate according to the weight of the first application and the first screen recording frame rate;

[0180] After obtaining the weight corresponding to the first application, the first screen recording frame rate can be adjusted to obtain the second screen recording frame rate, and the second screen recording frame rate is less than the first screen recording frame rate.

[0181] In addition, the frame reduction ratio I can also be calculated according to the n frame pictures after frame reduction and the N frame pictures to be frame reduced. The frame reduction ratio satisfies the formula I = n / N. Then, the weight corresponding to the application can be adjusted according to the frame reduction ratio. For example, at this time, the weight corresponding to the first application is 80%, the first screen recording frame rate is 75 frames per second, and the determined second screen recording frame rate is 60 frames per second. Use this second screen recording frame rate to obtain 60 frame pictures to be frame reduced, and 36 frame pictures are obtained after frame reduction. At this time, the frame reduction ratio is 60%. From the pictures after frame reduction, it can be known that there are more pictures with high similarity in the pictures to be frame reduced. From this, it can be deduced that in the scene where the application is located, the picture content changes little and a high screen recording frame rate is not required. Therefore, the weight corresponding to the application can be adjusted downwards according to the frame reduction ratio, and the weight can be adjusted to 70% to reduce the second screen recording frame rate.

[0182] In the embodiments of the present application, not only can the screen recording frame rate be adjusted according to the weights corresponding to different categories of applications, but also the weights can be corrected according to the ratio of the number of frames of the pictures after frame reduction and the number of frames of the pictures to be frame reduced, so that the determined screen recording frame rate is more suitable for the current application scenario.

[0183] 504. Obtain the N frame pictures to be frame reduced according to the second screen recording frame rate.

[0184] Among them, the determined second screen recording frame rate can be used to record the screen of the first application, so as to obtain the N frame pictures to be frame reduced. Furthermore, the screening of the frame-reduced pictures is completed by adjusting the screen recording frame rate.

[0185] Refer to Figure 6 , the flowchart of a frame reduction method provided by the present application is as follows.

[0186] 601. Obtain the application information of the first application and the N frame pictures to be frame reduced;

[0187] Among them, for different categories of applications, the changes in their picture content are different. Therefore, for different categories of applications, different preset thresholds for picture similarity are set. The application information of the first application and N pictures to be frame-dropped can be obtained, so as to frame-drop the N pictures according to the application information of the first application subsequently.

[0188] 602. Determine the preset threshold for the picture similarity corresponding to the first application according to the application information of the first application;

[0189] Among them, the application information includes at least one of the category or the scenario. The preset threshold for the picture similarity of the first application can be determined according to the category or the scenario of the first application. For applications with large changes in picture content, the corresponding preset threshold is large, such as the drama-watching application; for applications with small changes in picture content, the corresponding preset threshold is small, such as the navigation application, the voice application, etc.

[0190] 603. Perform frame-dropping processing on the N pictures according to the preset threshold to obtain n pictures.

[0191] After obtaining the preset threshold and the N pictures to be frame-dropped, the retinal perception model can be used to perform frame-dropping processing on the N pictures, and the neural network model can also be used to perform frame-dropping processing on the N pictures. The specific method is not limited here.

[0192] Specifically, when performing frame-dropping processing on the N pictures through the retinal perception model, two pictures or multiple pictures can be sequentially selected according to the arrangement order of the N pictures, or two pictures or multiple pictures can be randomly selected from the N pictures; by comparing the similarity between the content of two pictures or multiple pictures, any one of the two pictures with a similarity higher than the preset threshold can be deleted, or other pictures in the multiple pictures can be deleted and only one picture is retained, so as to obtain the n pictures after frame-dropping.

[0193] In the embodiment of the present application, the characteristics of the application and the relevance between the picture contents are considered, and frame-dropping can be performed according to the similarity between the picture contents, which not only ensures the smoothness and coherence of the pictures forming a video but also realizes frame-dropping.

[0194] The foregoing introduced the method flow provided by the present application. Next, based on the foregoing method flow, the device provided by the present application will be introduced.

[0195] Refer to Figure 7 , the structural schematic diagram of a data processing device provided by the present application is as follows.

[0196] The acquisition module 701 is used to acquire the application information of the first application on the terminal device;

[0197] The above-mentioned acquisition module 701 is further configured to acquire N frames of pictures within a first time period, where the N frames of pictures indicate the content displayed during the operation of a first application within the first time period, and N is a positive integer;

[0198] The frame reduction module 702 is configured to reduce the frame rate of the N frames of pictures according to the application information of the first application and the content of the N frames of pictures to obtain n frames of pictures, where n is less than N and n is a positive integer;

[0199] The processing module 703 is configured to store or send the n frames of pictures.

[0200] In a possible implementation manner, before acquiring the N frames of pictures within the first time period, the apparatus further includes:

[0201] The above-mentioned acquisition module 701 is further configured to acquire M frames of pictures within a second time period, where the M frames of pictures indicate the content displayed during the operation of the first application within the second time period;

[0202] The above-mentioned processing module 703 is further configured to store or send the M frames of pictures.

[0203] In a possible implementation manner, the above-mentioned acquisition module 701 is specifically configured to: perform screen recording on the first application at a first screen recording frame rate to obtain M frames of pictures.

[0204] In a possible implementation manner, the above-mentioned acquisition module 701 is specifically configured to: perform screen recording on the first application at a second screen recording frame rate to obtain N frames of pictures, where the second screen recording frame rate is less than the first screen recording frame rate.

[0205] In a possible implementation manner, the application information includes at least one of category and scenario. After performing screen recording on the first application at the second screen recording frame rate to obtain N frames of pictures, the apparatus further includes:

[0206] The determination module 704 is configured to determine the second screen recording frame rate according to the application information of the first application and the first screen recording frame rate, and the second screen recording frame rates of the first applications with different application information are different.

[0207] In a possible implementation manner, the above-mentioned frame reduction module 702 is specifically configured to: reduce the frame rate of the N frames of pictures through a retinal perception model according to the application information of the first application and the content of the N frames of pictures to obtain n frames of pictures, where the similarity between the n frames of pictures is less than a preset threshold, and the preset thresholds of the first applications with different application information are different.

[0208] In a possible implementation manner, if the first application switches to the second application, and the categories of the first application and the second application are different, the device further includes: the obtaining module 701, further configured to obtain X frames of pictures within a third time period, where the X frames of pictures are obtained by recording the second application at a third screen recording frame rate; the processing module 703, further configured to store or send the X frames of pictures.

[0209] In a possible implementation manner, after storing or sending the X frames of pictures, the device further includes: the obtaining module 701, further configured to obtain the application information of the second application; the obtaining module 701, further configured to obtain Y frames of pictures within a fourth time period, where the Y frames of pictures are obtained by recording the second application at a fourth screen recording frame rate, Y is a positive integer, and the fourth screen recording frame rate is less than the third screen recording frame rate; the frame rate reduction module 702, further configured to reduce the frame rate of the Y frames of pictures according to the application information of the second application and the content of the Y frames of pictures to obtain y frames of pictures, where y is less than Y and y is a positive integer; the processing module 703, further configured to store or send the y frames of pictures.

[0210] Refer to Figure 8 , the structural schematic diagram of a frame rate reduction picture screening device provided by the present application is as follows.

[0211] The obtaining module 801 is configured to obtain the application information of the first application and the first screen recording frame rate;

[0212] The determining module 802 is configured to determine the second screen recording frame rate according to the application information of the first application and the first screen recording frame rate;

[0213] The processing module 803 is configured to obtain N frames of pictures to be frame rate reduced according to the second screen recording frame rate.

[0214] In a possible implementation manner, the determining module 802 is specifically configured to: classify the application information of the first application through a classification model to obtain the classification result of the first application; obtain the weight of the first application according to the classification result; determine the second screen recording frame rate according to the weight of the first application and the first screen recording frame rate.

[0215] In a possible implementation manner, the processing module 803 is specifically configured to: record the first application at the second screen recording frame rate to obtain N frames of pictures to be frame rate reduced.

[0216] Refer to Figure 9 , the structural schematic diagram of a frame rate reduction device provided by the present application is as follows.

[0217] The obtaining module 901 is configured to obtain the application information of the first application and the N frames of pictures to be frame rate reduced;

[0218] The frame dropping module 902 is configured to drop frames of N pictures according to the application information of the first application and the content of the N pictures, so as to obtain n pictures, where n is less than N.

[0219] In a possible implementation manner, the foregoing frame dropping module 902 is specifically configured to: determine a preset threshold according to the application information of the first application; and drop frames of the N pictures through a retinal perception model according to the preset threshold, so as to obtain n pictures, and the similarity between the n pictures is less than the preset threshold.

[0220] Refer to Figure 10 , a schematic structural diagram of another data processing device provided by this application is described as follows.

[0221] The data processing device may include a processor 1001 and a memory 1002. The processor 1001 and the memory 1002 are interconnected by a line. Among them, program instructions and data are stored in the memory 1002.

[0222] The memory 1002 stores the program instructions and data corresponding to the steps in the foregoing Figure 3 and Figure 4 .

[0223] The processor 1001 is configured to execute the method steps executed by the data processing device shown in any one of the foregoing Figure 3 and Figure 4 .

[0224] Optionally, the data processing device may further include a transceiver 1003, configured to receive or send data.

[0225] In an embodiment of this application, a computer-readable storage medium is further provided. When a program stored in the computer-readable storage medium runs on a computer, the computer is enabled to execute the steps in the method described in the foregoing Figure 3 and Figure 4 .

[0226] Refer to Figure 11 , a schematic structural diagram of another frame dropping picture screening device provided by this application is described as follows.

[0227] The frame dropping picture screening device may include a processor 1101 and a memory 1102. The processor 1101 and the memory 1102 are interconnected by a line. Among them, program instructions and data are stored in the memory 1102.

[0228] The memory 1102 stores the program instructions and data corresponding to the steps in the foregoing Figure 5 .

[0229] The processor 1101 is configured to execute the foregoing Figure 5The method steps executed by the dropped-frame picture screening device shown in any of the embodiments.

[0230] Optionally, the dropped-frame picture screening device may further include a transceiver 1103 for receiving or sending data.

[0231] An embodiment of the present application also provides a computer-readable storage medium storing a program that, when running on a computer, causes the computer to execute the steps in the method described in the foregoing Figure 5 shown embodiments.

[0232] Refer to Figure 12 , a structural schematic diagram of another dropped-frame device provided by the present application is described as follows.

[0233] The dropped-frame device may include a processor 1201 and a memory 1202. The processor 1201 and the memory 1202 are interconnected by a line. Among them, the memory 1202 stores program instructions and data.

[0234] The memory 1202 stores the program instructions and data corresponding to the steps in the foregoing Figure 6 .

[0235] The processor 1201 is configured to execute the method steps executed by the dropped-frame device shown in any of the foregoing Figure 6 embodiments.

[0236] Optionally, the dropped-frame device may further include a transceiver 1203 for receiving or sending data.

[0237] An embodiment of the present application also provides a computer-readable storage medium storing a program that, when running on a computer, causes the computer to execute the steps in the method described in the foregoing Figure 6 shown embodiments.

[0238] Optionally, the foregoing Figure 13 data processing device is a chip.

[0239] An embodiment of the present application also provides a data processing device, a dropped-frame picture screening device, and a dropped-frame device. The data processing device, the dropped-frame picture screening device, and the dropped-frame device may also be referred to as a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is configured to execute the method steps executed by the data processing device, the dropped-frame picture screening device, and the dropped-frame device shown in any of the foregoing Figure 3 , Figure 4 , Figure 5 and Figure 6 embodiments.

[0240] The embodiments of the present application further provide a digital processing chip. Circuits for implementing the functions of the foregoing processors 1001, 1101, and 1201 and one or more interfaces are integrated in the digital processing chip. When a memory is integrated in the digital processing chip, the digital processing chip can complete the method steps of any one or more of the foregoing embodiments. When a memory is not integrated in the digital processing chip, it can be connected to an external memory through a communication interface. The digital processing chip implements the actions performed by the data processing device, the dropped-frame picture screening device, and the dropped-frame device in the foregoing embodiments according to the program code stored in the external memory.

[0241] The embodiments of the present application further provide a computer program product. When it runs on a computer, it causes the computer to execute as described in the foregoing Figure 3 、 Figure 4 、 Figure 5 and Figure 6 the method steps described in the illustrated embodiments.

[0242] The data processing device, the dropped-frame picture screening device, and the dropped-frame device provided in the embodiments of the present application may be chips. The chips include: a processing unit and a communication unit. The processing unit may be a processor, for example, and the communication unit may be an input / output interface, a pin, a circuit, or the like. The processing unit can execute the computer execution instructions stored in the storage unit, so that the chips in the server execute the methods described in the foregoing Figure 3 、 Figure 4 、 Figure 5 and Figure 6 illustrated embodiments. Optionally, the storage unit is a storage unit inside the chip, such as a register, a cache, etc. The storage unit may also be a storage unit outside the chip in the radio access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0243] Specifically, the aforementioned processing unit or processor may be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0244] Exemplarily, please refer to Figure 13 , Figure 13 , which is a schematic structural diagram of a chip provided by an embodiment of the present application. The chip may be embodied as a neural network processor NPU 1300. The NPU 1300 is mounted on the main CPU (Host CPU) as a coprocessor, and tasks are assigned by the Host CPU. The core part of the NPU is the arithmetic circuit 1303. The arithmetic circuit 1303 is controlled by the controller 1304 to extract matrix data from the memory and perform multiplication operations.

[0245] In some implementations, the arithmetic circuit 1303 includes multiple processing units (process engine, PE) internally. In some implementations, the arithmetic circuit 1303 is a two-dimensional systolic array. The arithmetic circuit 1303 may also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1303 is a general matrix processor.

[0246] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit fetches the corresponding data of matrix B from the weight memory 1302 and caches it on each PE in the arithmetic circuit. The arithmetic circuit fetches the data of matrix A from the input memory 1301 and performs matrix operations with matrix B. The partial results or final results of the obtained matrix are stored in the accumulator 1308.

[0247] The unified memory 1306 is used to store input data and output data. The weight data directly passes through the direct memory access controller (DMAC) 1305, and the DMAC transfers it to the weight memory 1302. The input data is also transferred to the unified memory 1306 through the DMAC.

[0248] The bus interface unit (BIU) 1310 is used for the interaction between the AXI bus, the DMAC, and the instruction fetch buffer (IFB) 1309.

[0249] The bus interface unit 1310 (bus interface unit, BIU) is used for the instruction fetch buffer 1309 to obtain instructions from the external memory, and also for the storage unit access controller 1305 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0250] The DMAC is mainly used to transfer the input data in the external memory DDR to the unified memory 1306, or transfer the weight data to the weight memory 1302, or transfer the input data to the input memory 1301.

[0251] The vector calculation unit 1307 includes multiple arithmetic processing units, which, if necessary, further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolution / full connection layer network calculations in neural networks, such as batch normalization, pixel-level summation, upsampling of the feature plane, etc.

[0252] In some implementations, the vector calculation unit 1307 can store the processed output vector in the unified memory 1306. For example, the vector calculation unit 1307 can apply a linear function and / or a non-linear function to the output of the arithmetic circuit 1303, such as performing linear interpolation on the feature plane extracted by the convolutional layer, or, for another example, a vector of accumulated values, to generate activation values. In some implementations, the vector calculation unit 1307 generates normalized values, pixel-level summation values, or both. In some implementations, the processed output vector can be used as the activation input to the arithmetic circuit 1303, such as for use in subsequent layers in a neural network.

[0253] The instruction fetch buffer 1309 connected to the controller 1304 is used to store the instructions used by the controller 904;

[0254] The unified memory 1306, the input memory 1301, the weight memory 1302, and the fetch memory 1309 are all On-Chip memories. The external memory is private to the NPU hardware architecture.

[0255] Wherein, the processor mentioned anywhere above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the above Figure 3 、 Figure 4 、 Figure 5 and Figure 6 programs of the method.

[0256] In addition, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the attached drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be specifically implemented as one or more communication buses or signal lines.

[0257] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0258] In several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be in electrical, mechanical or other forms.

[0259] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0260] In addition, each functional unit in the various embodiments of the present application may be integrated into one processing unit, may exist physically as individual units, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0261] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, may be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0262] The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims, and the above-mentioned drawings of the present application are used to distinguish similar objects and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such used data may be interchanged under appropriate circumstances so that the embodiments described herein can be implemented in an order different from that illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0263] Finally, it should be noted that the above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application.

Claims

1. A data processing method, applied to a terminal device, characterized in that, Including: Obtain the application information of the first application on the terminal device; Obtain N frames of pictures within a first time period, where the N frames of pictures indicate the content displayed during the operation of the first application within the first time period, and N is a positive integer; Downsample the N frames of pictures according to the application information of the first application and the content of the N frames of pictures to obtain n frames of pictures, where n is less than N and n is a positive integer; Store or send the n frames of pictures.

2. The method according to claim 1, characterized in that, Before obtaining the N frames of pictures within the first time period, it further includes: Obtain M frames of pictures within a second time period, where the M frames of pictures indicate the content displayed during the operation of the first application within the second time period; Store or send the M frames of pictures.

3. The method according to claim 2, wherein The obtaining of the M frames of pictures within the second time period includes: Perform screen recording on the first application at a first screen recording frame rate to obtain the M frames of pictures.

4. The method according to any one of claims 1 to 3, characterized in that, The obtaining of the N frames of pictures within the first time period includes: Perform screen recording on the first application at a second screen recording frame rate to obtain the N frames of pictures, where the second screen recording frame rate is less than the first screen recording frame rate.

5. The method according to claim 4, characterized in that, The application information includes at least one of category and scenario. Before performing screen recording on the first application at the second screen recording frame rate to obtain the N frames of pictures, it further includes: Determine the second screen recording frame rate according to the application information of the first application and the first screen recording frame rate, and the second screen recording frame rate of the first application with different application information is different.

6. The method according to any one of claims 1 to 5, characterized in that, The downsampling of the N frames of pictures according to the application information of the first application and the content of the N frames of pictures to obtain n frames of pictures includes: Perform the downsampling on the N frames of pictures through a retina perception model according to the application information of the first application and the content of the N frames of pictures to obtain the n frames of pictures, where the similarity between the n frames of pictures is less than a preset threshold, and the preset threshold of the first application with different application information is different.

7. The method according to any one of claims 1 to 6, characterized in that If the first application switches to a second application and the categories of the first application and the second application are different, the method further includes: Obtain X frames of pictures within a third time period, where the X frames of pictures are obtained by performing screen recording on the second application at a third screen recording frame rate; Store or send the X frames of pictures.

8. The method according to claim 7, wherein After storing or sending the X frames of pictures, the method further includes: Obtain the application information of the second application; Obtain Y frames of pictures within a fourth time period, where the Y frames of pictures are obtained by performing screen recording on the second application at a fourth screen recording frame rate, Y is a positive integer, and the fourth screen recording frame rate is less than the third screen recording frame rate; Downsample the Y frames of pictures according to the application information of the second application and the content of the Y frames of pictures to obtain y frames of pictures, where y is less than Y and y is a positive integer; Store or send the y frames of pictures.

9. A data processing device, applied to a terminal device, characterized in that, Including: An obtaining module for obtaining the application information of the first application on the terminal device; The obtaining module is further configured to obtain N frames of pictures within a first time period, where the N frames of pictures indicate the content displayed during the operation of the first application within the first time period, and N is a positive integer; A frame - dropping module, configured to drop frames of the N frames of pictures according to the application information of the first application and the content of the N frames of pictures, to obtain n frames of pictures, where n is less than N, and n is a positive integer; A processing module, configured to store or send the n frames of pictures.

10. The device according to claim 9, characterized in that, Before obtaining the N frames of pictures within the first time period, the apparatus further includes: The obtaining module is further configured to obtain M frames of pictures within a second time period, where the M frames of pictures indicate the content displayed during the operation of the first application within the second time period; The processing module is further configured to store or send the M frames of pictures.

11. The device according to claim 10, characterized in that, The obtaining module is specifically configured to: Perform screen recording on the first application at a first screen - recording frame rate to obtain the M frames of pictures.

12. The device according to any one of claims 9 to 11, characterized in that, The obtaining module is specifically configured to: Perform screen recording on the first application at a second screen - recording frame rate to obtain the N frames of pictures, where the second screen - recording frame rate is less than the first screen - recording frame rate.

13. The device according to claim 12, characterized in that, The application information includes at least one of a category and a scenario. After performing screen recording on the first application at the second screen - recording frame rate to obtain the N frames of pictures, the apparatus further includes: A determining module, configured to determine the second screen - recording frame rate according to the application information of the first application and the first screen - recording frame rate, and the second screen - recording frame rate of the first application with different application information is different.

14. The device according to any one of claims 9 to 13, characterized in that The frame - dropping module is specifically configured to: Drop frames of the N frames of pictures through a retina perception model according to the application information of the first application and the content of the N frames of pictures, to obtain the n frames of pictures, where the similarity between the n frames of pictures is less than a preset threshold, and the preset threshold of the first application with different application information is different.

15. The device according to any one of claims 9 to 14, characterized in that If the first application switches to a second application, and the categories of the first application and the second application are different, the apparatus further includes: The obtaining module is further configured to obtain X frames of pictures within a third time period, where the X frames of pictures are obtained by performing screen recording on the second application at a third screen - recording frame rate; The processing module is further configured to store or send the X frames of pictures.

16. The device according to claim 15, characterized in that, After storing or sending the X frames of pictures, the apparatus further includes: The obtaining module is further configured to obtain the application information of the second application; The obtaining module is further configured to obtain Y frames of pictures within a fourth time period, where the Y frames of pictures are obtained by performing screen recording on the second application at a fourth screen - recording frame rate, Y is a positive integer, and the fourth screen - recording frame rate is less than the third screen - recording frame rate; The frame - dropping module is further configured to drop frames of the Y frames of pictures according to the application information of the second application and the content of the Y frames of pictures, to obtain y frames of pictures, where y is less than Y, and y is a positive integer; The processing module is further configured to store or send the y frames of pictures.

17. A terminal device, characterized in that, It includes: A processor and a memory, where the processor is coupled to the memory; The memory is used to store programs; The processor is configured to execute the programs in the memory, so as to perform the method according to any one of claims 1 to 8.

18. A computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the method according to claims 1 to 8.

19. A computer program product containing instructions that, when executed on a computer, cause the computer to perform the method according to claims 1 to 8.

Images