Scene change based gop scheduling method, system, terminal device and medium

Abstract

The application discloses a scene change-based GOP scheduling method and system, a terminal device and a computer storage medium. The scene change-based GOP scheduling method performs scene detection on streaming media data to determine whether a video picture scene changes; when detecting that the video picture scene changes, reconstructs a GOP structure of an original GOP in the streaming media data; and for the streaming media data after the reconstructed GOP structure, calls a preset machine learning model to allocate corresponding decoding devices to each GOP one by one, so as to schedule the streaming media data for decoding in a GOP unit. The technical scheme can achieve a good balance between image quality and transmission and decoding efficiency of the streaming media data, thereby improving the coding and transmission efficiency of the streaming media data.

Description

Technical Field

[0001] This invention relates to the field of streaming media data technology, and in particular to a GOP scheduling method, system, terminal device, and computer storage medium based on scene changes. Background Technology

[0002] Currently, transcoding of streaming media data involved in live video services involves directly delivering the entire streaming media data to a single device for decoding via an encoding / decoding system. During the decoding process, fixed lengths and composition types are used to divide the streaming media data into GOPs (Group of Pictures, a group of consecutive frames. A GOP can be independently decoded based on an image designated as an "I-frame").

[0003] However, the composition of a Group of Pictures (GOP) varies in different scenarios. For example, in parking lot surveillance videos, in addition to scenes showing cars or passengers moving, there are also many scenes that remain static over time. While reducing the number of I-frames and adding more P-frames or B-frames to a GOP can improve the encoding and transmission efficiency of media data, it is still necessary to insert more information-rich I-frames into the GOP to ensure basic image quality.

[0004] Therefore, achieving a balance between image quality and encoding efficiency has become a critical issue that urgently needs to be addressed in the encoding and decoding of streaming media data. Summary of the Invention

[0005] The main objective of this invention is to provide a GOP scheduling method, system, terminal device, and computer storage medium based on scene changes. The aim is to reconstruct the GOP structure of streaming media data based on scene changes, and to push streaming media data by using balanced scheduling on the reconstructed GOP structure at the GOP granularity, thereby achieving a balance between video image quality and streaming media data transmission decoding efficiency.

[0006] To achieve the above objectives, the present invention provides a GOP scheduling method based on scene changes, the GOP scheduling method based on scene changes comprising:

[0007] Scene detection is performed based on streaming media data to determine whether the scene in the video has changed;

[0008] When a change in the video scene is detected, the GOP structure of the original GOP in the streaming media data is reconstructed;

[0009] For the streaming media data after the GOP structure has been reconstructed, a preset machine learning model is invoked to allocate corresponding decoding devices to each GOP, so as to schedule the decoding of the streaming media data according to the GOP.

[0010] Preferably, the step of performing scene detection based on streaming media data to determine whether the video scene has changed includes:

[0011] Detect the first scene change rate between two consecutive video frames in the streaming media data;

[0012] If the first scene change rate exceeds a preset first scene change rate threshold, then it is determined that the video scene has undergone a drastic change.

[0013] or,

[0014] Detect the second scene change rate between any two consecutive video frames in the streaming media data;

[0015] If the change rate of each of the second scenes is greater than the preset change rate threshold of the second scene, then it is determined that the scene of the video frame has undergone a gradual change, wherein the change rate threshold of the second scene is less than the change rate threshold of the first scene.

[0016] Preferably, the step of reconstructing the GOP structure of the streaming media data includes:

[0017] For streaming media data where the scene of the video frame changes drastically, detect continuous video frames that meet preset conditions, wherein the preset conditions are: the number of frames of the continuous video frame exceeds a preset frame number threshold, and the first scene change rate between each two consecutive video frames in the continuous video frame exceeds a preset first scene change rate threshold.

[0018] If it is determined that there is no I-frame in one or more original GOPs to which the continuous video frames belong, then the I-frame is inserted after the last video frame of the continuous video frames;

[0019] The GOP length is set according to the inserted I-frame to reconstruct one or more of the original GOPs.

[0020] Preferably, the step of setting the GOP length based on the inserted I-frame includes:

[0021] The frame sequence is determined based on the inserted I-frame, and the cumulative error of the frame sequence is calculated;

[0022] The length of the frame sequence when the cumulative error is minimized is set as the GOP length.

[0023] Preferably, the step of reconstructing the GOP structure of the streaming media data includes:

[0024] For streaming media data where the scene of the video frame changes smoothly, determine whether there is an I-frame in one or more original GOPs to which the consecutive multiple video frames that have changed, wherein the second scene change rate between each consecutive two video frames is greater than a preset second scene change rate threshold.

[0025] If so, the I-frames in one or more of the original GOPs are removed.

[0026] Preferably, the step of calling a preset machine learning model to allocate corresponding decoding devices to each GOP includes:

[0027] The decoding impact parameters of each GOP in the streaming media data and the real-time resource status of the scheduling device to be allocated are sequentially converted into model feature vectors.

[0028] The model feature vectors are input one by one into the machine learning model to obtain the corresponding classification learning results;

[0029] Based on the classification learning results, each GOP is assigned a corresponding decoding device in sequence.

[0030] Preferably, the decoding device includes a central processing unit and a graphics processing unit;

[0031] The step of assigning corresponding decoding devices to each GOP according to the logistic regression results includes:

[0032] If the classification learning result is the first identifier mapped by the central processing unit, then the first GOP mapped by the model feature vector in each GOP is assigned to the central processing unit;

[0033] or,

[0034] If the classification learning result is the second identifier mapped by the graphics processor, then the second GOP mapped by the model feature vector in each GOP is assigned to the graphics processor.

[0035] Furthermore, to achieve the above objectives, the present invention also provides a scene-change-based GOP scheduling system, the scene-change-based GOP scheduling system comprising:

[0036] The scene detection module is used to perform scene detection based on streaming media data to determine whether the scene in the video frame has changed;

[0037] The GOP reconstruction module is used to reconstruct the GOP structure of the original GOP in the streaming media data when a change in the video scene is detected.

[0038] The scheduling module is used to call a preset machine learning model to allocate corresponding decoding devices to each GOP for the streaming media data after the GOP structure is reconstructed, so as to schedule the streaming media data for decoding according to GOP.

[0039] In this invention, each functional module of the scene-change-based GOP scheduling system implements the steps of the scene-change-based GOP scheduling method described above during runtime.

[0040] In addition, to achieve the above objectives, the present invention also provides a terminal device, the terminal device comprising: a memory, a processor, and a scene-change-based GOP scheduler stored in the memory and executable on the processor, wherein the scene-change-based GOP scheduler, when executed by the processor, implements the steps of the scene-change-based GOP scheduling method as described above.

[0041] In addition, to achieve the above objectives, the present invention also provides a computer storage medium storing a scene-changing GOP scheduler, which, when executed by a processor, implements the steps of the scene-changing GOP scheduling method as described above.

[0042] Furthermore, to achieve the above objectives, the present invention also provides a computer storage medium storing a scene-changing GOP scheduler, which, when executed by a processor, implements the steps of the scene-changing GOP scheduling method described above.

[0043] This invention provides a GOP scheduling method, system, terminal device, computer storage medium, and computer storage medium based on scene changes. The method involves determining whether the video scene has changed by performing scene detection based on streaming media data; when a scene change is detected, the GOP structure of the original GOPs in the streaming media data is reconstructed; and for the streaming media data with the reconstructed GOP structure, a preset machine learning model is invoked to allocate corresponding decoding devices to each GOP, thereby scheduling the streaming media data for decoding on a GOP-by-GOP basis.

[0044] In the process of encoding and decoding streaming media data, this invention detects video scene changes based on the streaming media data to determine whether the real-time scene has changed. When a change in the video scene is detected, the original GOP in the streaming media data is reconstructed. Then, for the streaming media data with the reconstructed GOP structure, a preset machine learning model is invoked to allocate corresponding decoding devices to each GOP in the streaming media data, thereby decoding the streaming media data according to GOP as the scheduling unit.

[0045] Compared to the traditional method of decoding streaming media data by dividing it into fixed GOP structures, this invention dynamically reconstructs GOPs based on the specific changes in video scene conditions within the streaming media data. Then, by calling a pre-trained machine learning model based on real-time information such as video resolution, frame rate, bit rate, and device resources, it allocates decoding devices at the GOP granularity to balance GOP scheduling for data push. In this way, a good balance is achieved between image quality and transmission decoding efficiency in streaming media data. This not only greatly improves the encoding and transmission efficiency of streaming media data but also effectively reduces data encoding time and facilitates the deployment and implementation of data scheduling schemes. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the device structure of the terminal device hardware operating environment involved in the embodiments of the present invention;

[0047] Figure 2 This is a flowchart illustrating an embodiment of the GOP scheduling method based on scene changes according to the present invention.

[0048] Figure 3 This is a schematic diagram illustrating an application scenario of an embodiment of the GOP scheduling system based on scene changes according to the present invention.

[0049] Figure 4 This is a schematic diagram of the functional modules of an embodiment of the scene-changing GOP scheduling system of the present invention.

[0050] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0051] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0052] Reference Figure 1 , Figure 1 This is a schematic diagram of the hardware operating environment of the terminal device involved in the embodiments of the present invention.

[0053] The terminal device in this embodiment of the invention can be a device for encoding and decoding streaming media data. Specifically, the terminal device can be a server, smartphone, PC (Personal Computer), tablet computer, portable computer, etc.

[0054] like Figure 1 As shown, the terminal device may include: a processor 1001, such as a CPU; a communication bus 1002; a user interface 1003; a network interface 1004; and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0055] Those skilled in the art will understand that Figure 1 The terminal device structure shown does not constitute a limitation on the terminal device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0056] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a GOP scheduler based on scene changes.

[0057] exist Figure 1 In the terminal shown, network interface 1004 is mainly used to connect to the backend server and communicate data with it; user interface 1003 is mainly used to connect to the client and communicate data with it; while processor 1001 can be used to call the scene-change-based GOP scheduler stored in memory 1005 and execute the following steps:

[0058] Scene detection is performed based on streaming media data to determine whether the scene in the video has changed;

[0059] When a change in the video scene is detected, the GOP structure of the original GOP in the streaming media data is reconstructed;

[0060] For the streaming media data after the GOP structure has been reconstructed, a preset machine learning model is invoked to allocate corresponding decoding devices to each GOP, so as to schedule the decoding of the streaming media data according to the GOP.

[0061] Preferably, the processor 1001 can be used to call the scene-change-based GOP scheduler stored in the memory 1005, and further perform the following steps:

[0062] Detect the first scene change rate between two consecutive video frames in the streaming media data;

[0063] If the first scene change rate exceeds a preset first scene change rate threshold, then it is determined that the video scene has undergone a drastic change.

[0064] or,

[0065] Detect the second scene change rate between any two consecutive video frames in the streaming media data;

[0066] If the change rate of each of the second scenes is greater than the preset change rate threshold of the second scene, then it is determined that the scene of the video frame has undergone a gradual change, wherein the change rate threshold of the second scene is less than the change rate threshold of the first scene.

[0067] Preferably, the processor 1001 can be used to call the scene-change-based GOP scheduler stored in the memory 1005, and further perform the following steps:

[0068] For streaming media data where the scene of the video frame changes drastically, detect continuous video frames that meet preset conditions, wherein the preset conditions are: the number of frames of the continuous video frame exceeds a preset frame number threshold, and the first scene change rate between each two consecutive video frames in the continuous video frame exceeds a preset first scene change rate threshold.

[0069] If it is determined that there is no I-frame in one or more original GOPs to which the continuous video frames belong, then the I-frame is inserted after the last video frame of the continuous video frames;

[0070] The GOP length is set according to the inserted I-frame to reconstruct one or more of the original GOPs.

[0071] Preferably, the processor 1001 can be used to call the scene-change-based GOP scheduler stored in the memory 1005, and further perform the following steps:

[0072] The frame sequence is determined based on the inserted I-frame, and the cumulative error of the frame sequence is calculated;

[0073] The length of the frame sequence when the cumulative error is minimized is set as the GOP length.

[0074] Preferably, the processor 1001 can be used to call the scene-change-based GOP scheduler stored in the memory 1005, and further perform the following steps:

[0075] For streaming media data where the scene of the video frame changes smoothly, determine whether there is an I-frame in one or more original GOPs to which the consecutive multiple video frames that have changed, wherein the second scene change rate between each consecutive two video frames is greater than a preset second scene change rate threshold.

[0076] If so, the I-frames in one or more of the original GOPs are removed.

[0077] Preferably, the processor 1001 can be used to call the scene-change-based GOP scheduler stored in the memory 1005, and further perform the following steps:

[0078] The decoding impact parameters of each GOP in the streaming media data and the real-time resource status of the scheduling device to be allocated are sequentially converted into model feature vectors.

[0079] The model feature vectors are input one by one into the machine learning model to obtain the corresponding classification learning results;

[0080] Based on the classification learning results, each GOP is assigned a corresponding decoding device in sequence.

[0081] Preferably, the decoding device includes a central processing unit and a graphics processing unit. The processor 1001 can be used to call a scene-change-based GOP scheduler stored in the memory 1005, and also perform the following steps:

[0082] If the classification learning result is the first identifier mapped by the central processing unit, then the first GOP mapped by the model feature vector in each GOP is assigned to the central processing unit;

[0083] or,

[0084] If the classification learning result is the second identifier mapped by the graphics processor, then the second GOP mapped by the model feature vector in each GOP is assigned to the graphics processor.

[0085] Based on the above hardware structure, various embodiments of the GOP scheduling method based on scene changes of the present invention are proposed.

[0086] It should be noted that in the various embodiments of the GOP scheduling method based on scene changes in this invention, since the transcoding of streaming media data involved in live video services is currently done by directly delivering the streaming media data as a whole to a single device for decoding through an encoding and decoding system, and during the decoding process, a fixed length and composition type are used to divide the streaming media data into GOPs (Group of Pictures, a group of consecutive frames. A GOP can be independently decoded based on an image designated as an "I-frame").

[0087] However, the composition of a Group of Pictures (GOP) varies in different scenarios. For example, in parking lot surveillance videos, in addition to scenes showing cars or passengers moving, there are also many scenes that remain static over time. While reducing the number of I-frames and adding more P-frames or B-frames to a GOP can improve the encoding and transmission efficiency of media data, it is still necessary to insert more information-rich I-frames into the GOP to ensure basic image quality.

[0088] Therefore, achieving a balance between image quality and encoding efficiency has become a critical issue that urgently needs to be addressed in the encoding and decoding of streaming media data.

[0089] To address the above issues, this invention provides a GOP scheduling method based on scene changes. For example... Figure 2 In the application scenario shown, after the terminal device retrieves the streaming media data to be encoded and decoded via the puller, it detects the video scene based on the streaming media data to determine whether the real-time scene has changed. If the video scene has changed, the original GOP in the streaming media data is reconstructed. Then, for the streaming media data with the reconstructed GOP structure, a preset machine learning model is called to allocate corresponding decoding devices to each GOP in the streaming media data one by one. Thus, the streaming media data is decoded in a balanced manner according to GOP as the scheduling unit. Finally, the decoded streaming media data is pushed out via the pusher for video playback and display.

[0090] Compared to the traditional method of decoding streaming media data by dividing it into fixed GOP structures, this invention dynamically reconstructs GOPs based on the specific changes in video scene conditions within the streaming media data. Then, by calling a pre-trained machine learning model based on real-time information such as video resolution, frame rate, bit rate, and device resources, it allocates decoding devices at the GOP granularity to balance GOP scheduling for data push. In this way, a good balance is achieved between image quality and transmission decoding efficiency in streaming media data. This not only greatly improves the encoding and transmission efficiency of streaming media data but also effectively reduces data encoding time and facilitates the deployment and implementation of data scheduling schemes.

[0091] Please refer to Figure 3 , Figure 3 This is a flowchart illustrating the first embodiment of the scene-change-based GOP scheduling method of the present invention. In this embodiment, the scene-change-based GOP scheduling method of the present invention is applied to the aforementioned terminal device. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0092] In this embodiment, the GOP scheduling method based on scene changes of the present invention includes:

[0093] Step S10: Perform scene detection based on streaming media data to determine whether the scene in the video frame has changed;

[0094] In this embodiment, during the encoding and decoding process of the retrieved streaming media data, the terminal device first detects the video scene in the streaming media data to determine whether the video scene has changed.

[0095] As a possible implementation method, the terminal device can determine whether the video scene has changed by detecting whether there is a difference between the Nth frame and the (N-1)th frame of the streaming media data.

[0096] It should be noted that, in this embodiment, the changes in the video scene can be divided into two categories: drastic changes and gradual changes. Specifically, drastic changes can be the rapid changes that occur in the video scene, which take only one frame to change. Gradual changes, on the other hand, can be the slow and gradual changes in the scene that take several or even dozens of frames to complete.

[0097] Step S20: When a change in the video scene is detected, the GOP structure of the original GOP in the streaming media data is reconstructed;

[0098] In this embodiment, when the terminal device determines that the video scene of the current streaming media data has changed during scene detection, it immediately performs GOP structure reconstruction processing on one or more original GOPs in the streaming media data that meet the reconstruction conditions.

[0099] It should be noted that, in this embodiment, the reconstruction conditions include: in one or more original GOPs where the video scene changes drastically, there is no I-frame containing rich information for independent decoding of the GOP; and in one or more original GOPs where the video scene changes smoothly, there is an I-frame.

[0100] In addition, as a feasible implementation method, the above-mentioned reconstruction conditions may also include: the existence of I-frames in one or more original GOPs where the video scene has not changed.

[0101] Step S30: For the streaming media data after reconstructing the GOP structure, a preset machine learning model is invoked to allocate corresponding decoding devices to each GOP, so as to schedule the streaming media data for decoding according to GOP.

[0102] In this embodiment, after the terminal device reconstructs the GOP structure of the original GOP in the streaming media data, it further calls a preset machine learning model to allocate corresponding decoding devices to each GOP in the streaming media data after the GOP reconstruction, so as to schedule and decode the streaming media data at the GOP granularity with GOP as the scheduling unit.

[0103] It should be noted that, in this embodiment, the preset machine learning model can be a pre-trained convolutional neural network model or a logistic regression model for the terminal device. Specifically, the terminal device can use the convolutional neural network module or the logistic regression model to allocate a GPU (graphics processing unit, also known as a display core, visual processor, or display chip) or a CPU (central processing unit) as the corresponding decoding device for each GOP in the streaming media data after GOP reconstruction.

[0104] It should be understood that, based on different design needs of actual applications, in any other feasible implementation, the terminal device may of course use other locally available device resources as the above-mentioned decoding devices for scheduling GOPs in streaming media data. The GOP scheduling method based on scene changes in this invention does not limit the specific types of local device resources used.

[0105] In this embodiment, the GOP scheduling method based on scene change of the present invention detects the scene of the video frame according to the streaming media data to determine whether the scene of the real-time frame has changed. When it is determined that the scene of the video frame has changed, the original GOP in the streaming media data is reconstructed. Then, for the streaming media data after the GOP structure has been reconstructed, a preset machine learning model is called to allocate the corresponding decoding device to each GOP in the streaming media data one by one, so as to decode the streaming media data according to GOP as the scheduling unit.

[0106] Compared to the traditional method of decoding streaming media data by dividing it into fixed GOP structures, this invention dynamically reconstructs GOPs based on the specific changes in video scene conditions within the streaming media data. Then, by calling a pre-trained machine learning model based on real-time information such as video resolution, frame rate, bit rate, and device resources, it allocates devices at the GOP granularity to balance GOP scheduling for data push. In this way, a good balance is achieved between image quality and transmission decoding efficiency in streaming media data. This not only greatly improves the encoding and transmission efficiency of streaming media data but also effectively reduces data encoding time and facilitates the deployment and implementation of data scheduling schemes.

[0107] Furthermore, based on the first embodiment of the scene-change-based GOP scheduling method of the present invention described above, a second embodiment of the scene-change-based GOP scheduling method of the present invention is proposed.

[0108] In this embodiment, step 10 above, which involves scene detection based on streaming media data to determine whether the video scene has changed, may include:

[0109] Detect the first scene change rate between two consecutive video frames in the streaming media data;

[0110] If the first scene change rate exceeds a preset first scene change rate threshold, then it is determined that the video scene has undergone a drastic change.

[0111] In this embodiment, during the process of detecting video scene in streaming media data, the terminal device first detects the first scene change rate between two consecutive video frames in the streaming media data, and then detects whether the first scene change rate exceeds a preset first scene change rate threshold. Thus, if the first scene change rate is detected to exceed the first scene change rate threshold, it is determined that a drastic change in the video scene has occurred between the two consecutive video frames in the streaming media data.

[0112] It should be noted that, in this embodiment, the terminal device can specifically define the rate of change between two consecutive video frames by the ratio of their respective SATD (Standardized Aspect Ratio of Video Residual Signal) values. To reduce complexity and improve the time efficiency of the terminal device in encoding and decoding the streaming media data, in a feasible real-time manner, the terminal device can specifically use the block used to calculate each video frame during the encoding of the streaming media data to calculate the SATD value of each video frame.

[0113] Furthermore, the first scene change rate threshold can be a change rate empirical value set by the terminal device based on the actual application design scenario and stored locally. For example, after calculating the first scene change rate between two consecutive video frames (such as the Nth frame and the (N-1)th frame) in the streaming media data, the terminal device immediately reads the pre-set first scene change rate threshold from the local storage space to detect whether the first scene change rate is greater than the first scene change rate threshold. If it is detected, it is determined that a drastic change in the video scene has occurred between the two consecutive video frames in the streaming media data.

[0114] It should be understood that, based on different design needs of practical applications, the first scene change rate threshold configured by the terminal device can of course be any different empirical value of change rate in different feasible implementation methods. The GOP scheduling method based on scene change in this invention does not limit the specific size of the first scene change rate threshold.

[0115] Furthermore, in this embodiment, step 10 above, which involves performing scene detection based on streaming media data to determine whether the video scene has changed, may further include:

[0116] Detect the second scene change rate between any two consecutive video frames in the streaming media data;

[0117] If the change rate of each of the second scenes is greater than the preset change rate threshold of the second scene, then it is determined that the scene of the video image has undergone a gradual change.

[0118] In this embodiment, during the detection of video scene in streaming media data, the terminal device, in addition to detecting only the first scene change rate between two consecutive video frames in the streaming media data, also synchronously or asynchronously detects the second scene change rate between each pair of consecutive video frames in multiple consecutive video frames in the streaming media data. Then, the terminal device further detects whether each of the multiple second scene change rates exceeds a preset second scene change rate threshold. Thus, when it is detected that each of the multiple second scene change rates exceeds the second scene change rate threshold, the terminal device determines that a gradual change in video scene has occurred in the streaming media data between the multiple consecutive video frames.

[0119] It should be noted that in this embodiment, the second scene change rate threshold is smaller than the first scene change rate threshold mentioned above. Similarly, the second scene change rate threshold can also be an empirical value of change rate set and stored locally by the terminal device based on the actual application's design scenario. Furthermore, based on different design needs of the actual application, the second scene change rate threshold configured by the terminal device can of course be any different size of empirical value of change rate in different feasible implementations. Therefore, the GOP scheduling method based on scene changes in this invention does not limit the specific size of the second scene change rate threshold.

[0120] For example, after calculating the rate of change of each second scene between every two consecutive video frames in a series of consecutive video frames (such as the Nth to the Mth video frames) in the streaming media data, the terminal device immediately reads a pre-set threshold for the rate of change of the second scene from the local storage space to detect whether the rate of change of the second scene is greater than the threshold. Thus, when the terminal device detects that each threshold for the rate of change of the second scene is greater than the threshold, it determines that a smooth change in the scene of the video frame has occurred between the consecutive video frames in the streaming media data.

[0121] It should be noted that, in this embodiment, the gradual change of video scene in streaming media data can be divided into two stages: a fading stage and a convergence stage. In the fading stage, the rate of change (the ratio of the SATD values ​​of the two video frames) between the current frame and the last frame with a changed scene changes, and this rate of change is greater than the aforementioned second scene change rate threshold. In the convergence stage, although the SATD value of the video frame slows its growth trend, causing the second scene change rate of two consecutive frames in the later video frames to enter the convergence stage, even so, this second scene change rate is still greater than the aforementioned second scene change rate threshold. Furthermore, since the gradual change of video scene is completed in the convergence stage, when the terminal device detects that the second scene change rate between two consecutive video frames is less than or equal to the aforementioned second scene change rate threshold, the terminal device determines that the gradual change has ended.

[0122] Furthermore, based on the first and / or second embodiments of the scene-change-based GOP scheduling method of the present invention described above, a third embodiment of the scene-change-based GOP scheduling method of the present invention is proposed.

[0123] In this embodiment, the step of "reconstructing the GOP structure of the streaming media data" in step S20 above may include:

[0124] For streaming media data where the video scene undergoes drastic changes, detect continuous video frames that meet preset conditions;

[0125] If it is determined that there is no I-frame in one or more original GOPs to which the continuous video frames belong, then the I-frame is inserted after the last video frame of the continuous video frames;

[0126] The GOP length is set according to the inserted I-frame to reconstruct one or more of the original GOPs.

[0127] In this embodiment, when the terminal device detects and determines that a change has occurred in the video scene of the streaming media data, and thus reconstructs one or more original GOPs in the streaming media data, if the terminal device detects a drastic change in the video scene, the terminal device first detects consecutive video frames in the streaming media data that meet preset conditions. Then, if further detection determines that there is no I-frame in one or more original GOPs to which the consecutive video frames belong, an information-rich I-frame is immediately inserted after the last frame of the consecutive video frames. Afterwards, the terminal device sets the GOP length based on the inserted I-frame to complete the reconstruction process for the one or more original GOPs.

[0128] It should be noted that, in this embodiment, the preset conditions are: the number of frames in a continuous video frame exceeds a preset frame rate threshold, and the rate of change of the first scene between any two consecutive frames in the continuous video frame exceeds a preset first scene rate of change threshold. Furthermore, the preset frame rate threshold is set based on the video frame rate corresponding to the encoding and decoding of streaming media data by the terminal device. For example, if the video frame rate is generally 25fps, the terminal device can specifically set the preset frame rate threshold to 25.

[0129] For example, assuming that the duration of a video scene in the streaming media data is no less than 25 frames, and that the terminal device detects that the first scene change rate (the ratio of the SATD values ​​of the two consecutive video frames) of the 25 consecutive video frames in the streaming media data is greater than the aforementioned first scene change rate threshold, the terminal device can set an I-frame at this point. Then, the terminal device further sets the GOP length based on the distance between the inserted I-frame and another I-frame (which can be an I-frame in the original GOP or another newly inserted one), thereby completing the GOP structure reconstruction process for one or more original GOPs to which the current 25 video frames belong.

[0130] Furthermore, in a feasible embodiment, the step of "setting the GOP length according to the inserted I-frame" described above may include:

[0131] The frame sequence is determined based on the inserted I-frame, and the cumulative error of the frame sequence is calculated;

[0132] The length of the frame sequence when the cumulative error is minimized is set as the GOP length.

[0133] In this embodiment, when the terminal device is reconstructing the original GOP in the streaming media data, it first determines the frame sequence based on the distance between the inserted I-frame and another I-frame when setting the GOP length based on the I-frame. Then, it calculates the cumulative error of the frame sequence. Thus, the terminal device sets the length of the frame sequence as the GOP length when the cumulative error is minimized.

[0134] For example, in this embodiment, the GOP length set by the terminal device can specifically be the distance between two adjacent I-frames in the streaming media data. Thus, for any frame sequence in the streaming media data, the terminal device can obtain a SATD value sequence {S1, ..., S...} based on the ratio of the SATD values ​​between two adjacent frames. m(This SATD value sequence reflects the changing state of video scenes in streaming media data. Specifically, when the S value in the SATD value sequence exceeds the first scene change rate threshold, it can be determined that a drastic change has occurred between the two consecutive video frames corresponding to that S value; or, when multiple consecutive S values ​​in the SATD value sequence exceed the second scene change rate threshold, it can be determined that a gradual change has occurred between the multiple video frames corresponding to those multiple S values.) Subsequently, the terminal device uses the SATD value sequence {S1, ..., S...} to... m} Perform calculations to find a set {S1, ..., S} m The time n with the highest expected value is determined by the length of the frame sequence corresponding to that time n with the smallest cumulative error, which is the length of the GOP to be set.

[0135] It should be noted that, in this embodiment, it is assumed that the SATD value of each video frame in any frame sequence follows a normal distribution, and thus the probability density function of the SATD value is known. Therefore, the terminal device defines the cumulative error Y of the frame sequence. n for:

[0136]

[0137] Here, y is a non-negative number, representing the initial error. Furthermore, the terminal device defines T as the maximum tolerable error of the frame sequence, thus W(Y) exists. n ,T) so that:

[0138]

[0139] Therefore, the terminal device can transform the above computational problem into finding a time t* such that at time t*, the expected value E{W(Y)} is... t* The maximum value is found in |Y1}. The specific value at time t* can be obtained by combining the probability density function of the frame sequence distribution described above.

[0140] Furthermore, in a feasible embodiment, the step of "setting the GOP length according to the inserted I-frame" may further include:

[0141] For streaming media data where the scene of the video frame changes smoothly, determine whether there is an I-frame in one or more original GOPs to which the consecutive multiple video frames that have changed, wherein the second scene change rate between each consecutive two video frames is greater than a preset second scene change rate threshold.

[0142] If so, the I-frames in one or more of the original GOPs are removed.

[0143] In this embodiment, when the terminal device detects and determines that the video scene in the streaming media data has changed, and then reconstructs one or more original GOPs in the streaming media data, if the terminal device currently detects that the video scene has changed gradually, the terminal device first detects and determines several consecutive video frames that have changed in the streaming media data, that is, several consecutive video frames where the second scene change rate between two consecutive video frames is greater than the second scene change rate threshold. Then, when it further detects and determines that there is an I-frame in one or more original GOPs to which the consecutive video frames belong, the I-frame is immediately removed to perform reconstruction processing for one or more original GOPs.

[0144] Furthermore, based on the first, second, and / or third embodiments of the scene-change-based GOP scheduling method of the present invention described above, a fourth embodiment of the scene-change-based GOP scheduling method of the present invention is proposed.

[0145] In this embodiment, the step S30 above, "calling a preset machine learning model to allocate corresponding decoding devices to each GOP," may include:

[0146] The decoding impact parameters of each GOP in the streaming media data and the real-time resource status of the scheduling device to be allocated are sequentially converted into model feature vectors.

[0147] The model feature vectors are input one by one into the machine learning model to obtain the corresponding classification learning results;

[0148] Based on the classification learning results, each GOP is assigned a corresponding decoding device in sequence.

[0149] In this embodiment, when the terminal device calls a pre-trained machine learning model to allocate available decoding devices for a Group of Pictures (GOPs) in the streaming media data, the terminal device converts the various decoding influence parameters for decoding the GOP and the real-time resource status of the currently available scheduling devices into a model feature vector for the machine learning model. Then, the terminal device inputs this model feature vector corresponding to the GOP into the currently invoked machine learning model to obtain a classification learning result after model training. Based on this, the terminal device can allocate decoding devices to the current GOP according to the type of the classification learning result. Therefore, the terminal device can sequentially allocate decoding devices for each GOP in the streaming media data.

[0150] It should be noted that, in this embodiment, the various decoding parameters that affect the terminal device when decoding GOPs in streaming media data include, but are not limited to, video resolution, frame rate, and bit rate, while the real-time resource status of the terminal device in the currently available scheduling device includes, but is not limited to, device idle rate, occupancy rate, or idle state.

[0151] Furthermore, in a feasible embodiment, the aforementioned decoding device includes a central processing unit (CPU) and a graphics processing unit (GPU). Based on this, the step of "assigning corresponding decoding devices to each GOP sequentially according to the classification learning results" may specifically include:

[0152] If the classification learning result is the first identifier mapped by the central processing unit, then the first GOP mapped by the model feature vector in each GOP is assigned to the central processing unit;

[0153] or,

[0154] If the classification learning result is the second identifier mapped by the graphics processor, then the second GOP mapped by the model feature vector in each GOP is assigned to the graphics processor.

[0155] In this embodiment, during the process of the terminal device calling a pre-trained machine learning model to allocate decoding devices to each Group of Pictures (GOPs) in the streaming media data, after obtaining the classification learning result through model training, the terminal device immediately detects whether the machine learning result is a first identifier mapped by the CPU or a second identifier mapped by the GPU. Thus, if the terminal device detects that the classification learning result is a first identifier mapped by the CPU, it can allocate the first GOP, mapped by the model feature vector trained by the current machine learning model, to the CPU for subsequent decoding processing; or, if the terminal device detects that the classification learning result is a second identifier mapped by the GPU, it can allocate the second GOP, mapped by the model feature vector trained by the current machine learning model, to the GPU for subsequent decoding processing.

[0156] For example, assume that the terminal device pre-sets the first identifier for the CPU mapping to 1 and the second identifier for the GPU mapping to 0. Thus, during the prediction phase where the terminal device calls a machine learning model to train on the aforementioned model feature vectors to predict whether the GOP mapped by the model feature vectors should be allocated to the CPU for decoding or to the GPU for decoding, after inputting the model feature vector of the GOP mapping to be scheduled into the machine learning model to obtain the classification learning result output by the machine learning model, if the classification learning result is 0, the terminal device can schedule and allocate the GOP to the GPU for decoding; conversely, if the classification learning result is 1, the terminal device can schedule and allocate the GOP to the CPU for decoding.

[0157] Furthermore, in a feasible embodiment, if the machine learning model used by the terminal device is a logistic regression model, the terminal device can pre-train and construct the logistic regression model based on the decoding impact parameters of each GOP in the streaming media data, the real-time resource status of the scheduling device, and the scheduling device corresponding to each GOP. Based on this, the GOP scheduling method based on scene changes of the present invention may further include:

[0158] Sample data is constructed based on the decoding impact parameters of each GOP in the streaming media data, the real-time resource status of the scheduling device, and the scheduling device corresponding to each GOP;

[0159] Logistic regression training is performed based on the sample data to obtain a logistic regression model that assigns a corresponding scheduling device to each GOP in the streaming media data.

[0160] In this embodiment, before allocating available scheduling devices to each GOP in the streaming media data, the terminal device can construct sample data for logistic regression training by reading the decoding impact parameters such as video resolution, frame rate, and bit rate when decoding the GOP in the streaming media data, the real-time resource status of the scheduling device specifically allocated for scheduling the GOP, such as the device idle rate, and the device identification information of the scheduling device. Then, the terminal device performs logistic regression training locally based on the sample data to obtain the logistic regression model for allocating corresponding scheduling devices to each GOP in the streaming media data.

[0161] In addition, as a feasible implementation method, the terminal device can also directly call the logistic regression model obtained by other terminals based on sample data of the same nature to allocate scheduling devices for GOPs in streaming media data.

[0162] In addition, as a feasible implementation method, after the terminal device calls the logistic regression model to allocate a scheduling device for a certain GOP in the streaming media data, it can also combine the decoding impact parameters such as video resolution, frame rate, and bit rate of the GOP decoded by the terminal device, the real-time resource status such as the idle rate of the scheduling device, and the device identifier of the scheduling device to construct new sample data for subsequent updates to the logistic regression model, or for training other new logistic regression models.

[0163] Furthermore, this invention also provides a GOP scheduling system based on scene changes. Please refer to... Figure 4 , Figure 4 This is a functional module diagram of an embodiment of the GOP scheduling system based on scene changes according to the present invention. Figure 4 As shown, the GOP scheduling system based on scene changes of the present invention includes:

[0164] Scene detection module 10 is used to perform scene detection based on streaming media data to determine whether the scene of the video frame has changed;

[0165] GOP reconstruction module 20 is used to reconstruct the GOP structure of the original GOP in the streaming media data when a change in the video scene is detected.

[0166] The scheduling module 30 is used to call a preset machine learning model to allocate corresponding decoding devices to each GOP for the streaming media data after the GOP structure is reconstructed, so as to schedule the streaming media data for decoding according to GOP.

[0167] Furthermore, the scene detection module 10 includes:

[0168] The first detection unit is used to detect the first scene change rate between two consecutive video frames in the streaming media data;

[0169] The first change determination unit is used to determine that the video scene scene has undergone a drastic change if the first scene change rate exceeds a preset first scene change rate threshold.

[0170] The second detection unit is used to detect the second scene change rate between each two consecutive video frames in the continuous multi-frame video frames in the streaming media data.

[0171] The second change determination unit is used to determine that the video scene has undergone a gradual change if each of the second scene change rates is greater than a preset second scene change rate threshold, wherein the second scene change rate threshold is less than the first scene change rate threshold.

[0172] Furthermore, the GOP refactoring module 20 includes:

[0173] The third detection unit is used to detect continuous video frames that meet preset conditions for the streaming media data where the video scene changes drastically. The preset conditions are: the number of frames of the continuous video frames exceeds a preset frame number threshold, and the first scene change rate between each two consecutive video frames in the continuous video frames exceeds a preset first scene change rate threshold.

[0174] The I-frame insertion unit is used to insert the I-frame after the last video frame of the continuous video frames if it is determined that there is no I-frame in one or more original GOPs to which the continuous video frames belong.

[0175] The GOP length setting unit is used to set the GOP length according to the inserted I-frame in order to reconstruct one or more of the original GOPs.

[0176] Furthermore, the GOP length setting unit includes:

[0177] A calculation subunit is used to determine a frame sequence based on the inserted I-frame and to calculate the cumulative error of the frame sequence;

[0178] A subunit is set to set the length of the frame sequence when the cumulative error is minimized as the GOP length.

[0179] Furthermore, the GOP refactoring module 20 also includes:

[0180] The determining unit is used to determine whether there is an I-frame in one or more original GOPs to which the video scene of the streaming media undergoes a gradual change, wherein the second scene change rate between any two consecutive video frames is greater than a preset second scene change rate threshold.

[0181] The I-frame removal unit is used to remove one or more I-frames from the original GOP.

[0182] Furthermore, the scheduling module 30 includes:

[0183] The conversion unit is used to sequentially convert the decoding impact parameters of each GOP in the streaming media data and the real-time resource status of the scheduling device to be allocated into model feature vectors.

[0184] The model computation unit is used to input the model feature vectors one by one into the machine learning model to obtain the corresponding classification learning results;

[0185] The scheduling and allocation unit is used to allocate corresponding decoding devices to each GOP in sequence according to the classification learning results.

[0186] Furthermore, the scheduling and allocation unit is also configured to allocate the first GOP of the model feature vector mapping in each GOP to the central processing unit if the classification learning result is the first identifier mapped by the central processing unit; or, if the classification learning result is the second identifier mapped by the graphics processing unit, allocate the second GOP of the model feature vector mapping in each GOP to the graphics processing unit.

[0187] Furthermore, the GOP scheduling system based on scene changes of the present invention also includes:

[0188] The sample construction module is used to construct sample data based on the decoding impact parameters of each GOP in the streaming media data, the real-time resource status of the scheduling device, and the scheduling device corresponding to each GOP.

[0189] The model training module is used to perform logistic regression training based on the sample data to obtain a logistic regression model for allocating corresponding scheduling devices to each GOP in the streaming media data.

[0190] The functions of each module in the above-mentioned GOP scheduling system based on scene changes correspond to the steps in the above-mentioned GOP scheduling method embodiment based on scene changes, and their functions and implementation processes will not be described in detail here.

[0191] The present invention also provides a computer storage medium comprising a computer program, which, when executed by a processor, implements the steps of the scene-change-based GOP scheduling method as described in any of the above embodiments.

[0192] The specific embodiments of the computer storage medium of the present invention are basically the same as the embodiments of the GOP scheduling method based on scene changes described above, and will not be repeated here.

[0193] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0194] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0195] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0196] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A GOP scheduling method based on scene changes, characterized in that, The scene-change-based GOP scheduling method includes: Scene detection is performed based on streaming media data to determine whether the scene in the video has changed; When a change in the video scene is detected, the GOP structure of the original GOP in the streaming media data is reconstructed; For the streaming media data after the GOP structure is reconstructed, a preset machine learning model is invoked to allocate corresponding decoding devices to each GOP, so as to schedule the decoding of the streaming media data according to the GOP. The step of reconstructing the GOP structure of the original GOP in the streaming media data includes: For streaming media data where the scene of the video frame changes smoothly, determine whether there is an I-frame in one or more original GOPs to which the consecutive multiple video frames that have changed, wherein the second scene change rate between each consecutive two video frames is greater than a preset second scene change rate threshold. If so, the I-frames in one or more of the original GOPs are removed.

2. The GOP scheduling method based on scene changes as described in claim 1, characterized in that, The step of performing scene detection based on streaming media data to determine whether the video scene has changed includes: Detect the first scene change rate between two consecutive video frames in the streaming media data; If the first scene change rate exceeds a preset first scene change rate threshold, then it is determined that the video scene has undergone a drastic change. or, Detect the second scene change rate between any two consecutive video frames in the streaming media data; If the change rate of each of the second scenes is greater than the preset change rate threshold of the second scene, then it is determined that the scene of the video frame has undergone a gradual change, wherein the change rate threshold of the second scene is less than the change rate threshold of the first scene.

3. The GOP scheduling method based on scene changes as described in claim 1 or 2, characterized in that, The step of reconstructing the GOP structure of the streaming media data includes: For streaming media data where the scene of the video frame changes drastically, detect continuous video frames that meet preset conditions, wherein the preset conditions are: the number of frames of the continuous video frame exceeds a preset frame number threshold, and the first scene change rate between each two consecutive video frames in the continuous video frame exceeds a preset first scene change rate threshold. If it is determined that there is no I-frame in one or more original GOPs to which the continuous video frames belong, then the I-frame is inserted after the last video frame of the continuous video frames; The GOP length is set according to the inserted I-frame to reconstruct one or more of the original GOPs.

4. The GOP scheduling method based on scene changes as described in claim 3, characterized in that, The step of setting the GOP length based on the inserted I-frame includes: The frame sequence is determined based on the inserted I-frame, and the cumulative error of the frame sequence is calculated; The length of the frame sequence when the cumulative error is minimized is set as the GOP length.

5. The GOP scheduling method based on scene changes as described in claim 1, characterized in that, The step of using a preset machine learning model to assign a corresponding decoding device to each GOP includes: The decoding impact parameters of each GOP in the streaming media data and the real-time resource status of the scheduling device to be allocated are sequentially converted into model feature vectors. The model feature vectors are input one by one into the machine learning model to obtain the corresponding classification learning results; Based on the classification learning results, each GOP is assigned a corresponding decoding device in sequence.

6. The GOP scheduling method based on scene changes as described in claim 5, characterized in that, The decoding device includes a central processing unit and a graphics processing unit; The step of assigning corresponding decoding devices to each GOP according to the classification learning results includes: If the classification learning result is the first identifier mapped by the central processing unit, then the first GOP mapped by the model feature vector in each GOP is assigned to the central processing unit; or, If the classification learning result is the second identifier mapped by the graphics processor, then the second GOP mapped by the model feature vector in each GOP is assigned to the graphics processor.

7. A GOP scheduling system based on scene changes, characterized in that, The scene-change-based GOP scheduling system includes: The scene detection module is used to perform scene detection based on streaming media data to determine whether the scene in the video frame has changed; The GOP reconstruction module is used to reconstruct the GOP structure of the original GOP in the streaming media data when a change in the video scene is detected. The scheduling module is used to call a preset machine learning model to allocate corresponding decoding devices to each GOP for the streaming media data after the GOP structure is reconstructed, so as to schedule the streaming media data for decoding according to GOP. The GOP reconstruction module also includes: The determining unit is used to determine whether there is an I-frame in one or more original GOPs to which the video scene of the streaming media undergoes a gradual change, wherein the second scene change rate between any two consecutive video frames is greater than a preset second scene change rate threshold. The I-frame removal unit is used to remove one or more I-frames from the original GOP.

8. A terminal device, characterized in that, The terminal device includes: a memory, a processor, and a scene-change-based GOP scheduler stored in the memory and executable on the processor. When the scene-change-based GOP scheduler is executed by the processor, it implements the steps of the scene-change-based GOP scheduling method as described in any one of claims 1 to 6.

9. A computer storage medium, characterized in that, The computer storage medium stores a scene-change-based GOP scheduler, which, when executed by the processor, implements the steps of the scene-change-based GOP scheduling method as described in any one of claims 1 to 6.

Images