Frame rate adjustment method and device based on video coding and storage medium

Abstract

The invention discloses a frame rate adjustment method and device based on video coding and a storage medium, and the method comprises the steps: selecting a target frame rate from preset frame rates for collection according to a detection result of an interested target in a current scene, and obtaining a target video frame; in response to the fact that the target frame rate is different from the initial frame rate, modifying an initial code stream protocol corresponding to the initial frame rate to obtain a target code stream protocol; and coding and packaging the target video frame according to the target code stream protocol to obtain a target code stream. According to the scheme, the video frame rate can be dynamically adjusted, the code rate is effectively saved, and the playing effect during decoding playing is ensured.

Description

Technical Field

[0001] This application relates to the field of video encoding and decoding technology, and in particular to a frame rate adjustment method, device, and storage medium based on video encoding. Background Technology

[0002] With the rapid development of artificial intelligence technology, it is now common to see its application in target detection in daily life.

[0003] Currently, the main task of object detection is to find target objects and / or target events of interest in the current scene, and object detection technology based on image recognition and video detection is a core issue in the field of computer vision.

[0004] Video detection technology can detect target objects and / or target events in a current scene. This involves video encoding technology to encode video data collected by the front-end device and encapsulate it into a bitstream according to a bitstream protocol before transmitting it to the back-end device for decoding and playback. However, such methods place high demands on network transmission capabilities and bitstream storage capacity. Summary of the Invention

[0005] This application provides at least one frame rate adjustment method, apparatus, device, and computer-readable storage medium based on video coding.

[0006] The first aspect of this application provides a frame rate adjustment method based on video coding, comprising: selecting a target frame rate from a preset frame rate for acquisition based on the detection result of an interesting target in the current scene to obtain a target video frame; modifying the initial bitstream protocol corresponding to the initial frame rate in response to the difference between the target frame rate and the initial frame rate to obtain a target bitstream protocol; and encoding and encapsulating the target video frame according to the target bitstream protocol to obtain a target bitstream.

[0007] In one embodiment, modifying the initial bitstream protocol corresponding to the initial frame rate to obtain the target bitstream protocol includes: obtaining the video playback type corresponding to the target video frame; and modifying the initial bitstream protocol according to the video playback type to obtain the target bitstream protocol.

[0008] In one embodiment, the video playback type includes frame rate playback and timestamp playback. Modifying the initial bitstream protocol according to the video playback type to obtain the target bitstream protocol includes: in response to the video playback type being frame rate playback, modifying the frame rate information in the initial bitstream protocol according to the target frame rate to obtain the target bitstream protocol; and in response to the video playback type being timestamp playback, modifying the timestamp information in the initial bitstream protocol to obtain the target bitstream protocol.

[0009] In one embodiment, the initial bitstream protocol includes a standard protocol and a proprietary protocol. Modifying the frame rate information in the initial bitstream protocol according to the target frame rate to obtain the target bitstream protocol includes: modifying the frame header frame rate information in the standard protocol according to the target frame rate to obtain a modified standard protocol, and inserting keyframes according to the frame header frame rate information in the modified standard protocol; modifying the frame rate information in the proprietary protocol according to the target frame rate, and modifying the timestamp information in the proprietary protocol according to the obtained target timestamp to obtain a modified proprietary protocol; and determining the modified standard protocol and the modified proprietary protocol as the target bitstream protocol.

[0010] In one embodiment, the preset frame rate includes a first frame rate and a second frame rate, wherein the first frame rate is less than the second frame rate. The step of selecting a target frame rate from the preset frame rate for acquisition based on the detection result of a target of interest in the current scene to obtain a target video frame includes: in response to the detection result indicating that the target of interest does not exist in the current scene, determining the first frame rate as the target frame rate for acquisition to obtain the target video frame; and in response to the detection result indicating that the target of interest exists in the current scene, determining the second frame rate as the target frame rate for acquisition to obtain the target video frame.

[0011] In one embodiment, before encoding and encapsulating the target video frame according to the target bitstream protocol to obtain the target bitstream, the method further includes: obtaining the time difference between the current video frame and adjacent historical video frames; and performing time correction processing based on the time difference.

[0012] In one embodiment, the time correction process based on the time difference includes: obtaining the frame interval time between the current video frame and the historical video frames; determining the intermediate frame rate corresponding to the frame rate switching process based on the frame interval time; and correcting the current frame rate of the current video frame based on the intermediate frame rate.

[0013] In one embodiment, the time correction processing based on the time difference includes: obtaining the timestamp difference between the first video frame and the second video frame; obtaining the timestamp difference between the current video frame and the historical video frames; determining whether the current video frame is ahead or behind based on the timestamp difference; if so, performing correction processing on the current timestamp of the current video frame.

[0014] A second aspect of this application provides a frame rate adjustment device based on video encoding, comprising: a frame rate selection module, configured to select a target frame rate from a preset frame rate for acquisition based on the detection result of a target of interest in the current scene, thereby obtaining a target video frame; a protocol modification module, configured to modify the initial bitstream protocol corresponding to the initial frame rate in response to the difference between the target frame rate and the initial frame rate, thereby obtaining a target bitstream protocol; and a video encoding module, configured to encode and encapsulate the target video frame according to the target bitstream protocol, thereby obtaining a target bitstream.

[0015] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the above-described video coding-based frame rate adjustment method.

[0016] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described video coding-based frame rate adjustment method.

[0017] The above scheme, by detecting the target of interest in the current scene, selects the target frame rate corresponding to the detection result from a preset frame rate for image and video acquisition, thus obtaining the target video frame. This allows for dynamic adjustment of the video frame rate based on the current scene. If the frame rate switches during image and video acquisition, in response to the difference between the target frame rate and the initial frame rate, the initial bitstream protocol corresponding to the initial frame rate is modified to obtain the target bitstream protocol. The target video frame is then encoded and encapsulated according to the target bitstream protocol to obtain the target bitstream. While ensuring no impact on encoding quality and decoding playback, this scheme more effectively saves bitrate transmission bandwidth and data storage capacity, resulting in smooth playback without stuttering when the frame rate changes and a more natural transition during frame rate switching.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0020] Figure 1 This is a flowchart illustrating an exemplary embodiment of the video coding-based frame rate adjustment method of this application;

[0021] Figure 2 This is a schematic diagram illustrating the effects of forced I-frames and non-forced I-frames in the video coding-based frame rate adjustment method of this application.

[0022] Figure 3This is an exemplary overall flowchart of the video coding-based frame rate adjustment method of this application;

[0023] Figure 4 This is an exemplary bitrate saving effect diagram of the frame rate adjustment method based on video coding in this application;

[0024] Figure 5 This is a block diagram illustrating a video-encoded frame rate adjustment device in an exemplary embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0026] Figure 7 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0027] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0028] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0029] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0030] Please see Figure 1 , Figure 1 This is a flowchart illustrating an exemplary embodiment of the video coding-based frame rate adjustment method of this application. Specifically, it may include the following steps:

[0031] Step S110: Based on the detection results of the target of interest in the current scene, select the target frame rate from the preset frame rate for acquisition to obtain the target video frame.

[0032] The target of interest may include, but is not limited to, objects of interest and / or events of interest. For example, an object of interest is detected in the current scene, or an event of interest occurs (regardless of the type of object involved in the event), or an event of interest occurs on an object of interest (regardless of the type of object involved in the event), etc.

[0033] It should be noted that the target detection method for an object of interest in the current scene in this application may include, but is not limited to, image detection, video detection, infrared detection, and radar detection. For example, for image detection, an image of the current scene can be acquired, and an image-based target detection algorithm can be used on the acquired image data to obtain the detection result. For infrared detection, a detection device equipped with an infrared sensor can be set up in the current scene, and the presence of an object can be determined by changes in the infrared signal. For radar detection, a detection device equipped with a radar sensor (such as millimeter-wave radar, ultrasonic radar, microwave radar, etc.) can be set up in the current scene, and the presence of an object can be determined by changes in the radar signal.

[0034] For example, technologies such as Radio Frequency Identification (RFID), ZigBee, GPS, Bluetooth, and Wi-Fi positioning can also be used to detect targets of interest. For instance, a target carrying an RFID tag or a target without an RFID tag can be identified as a target of interest. Another example is the identification of targets of interest via Bluetooth signals detected by Bluetooth beacons or Wi-Fi signals detected by Wi-Fi network infrastructure; these will not be elaborated upon here.

[0035] It should also be noted that the preset frame rate in this application may include one or more. The following examples will primarily illustrate two preset frame rates. Specifically, it may include a high frame rate R and a low frame rate L, where R > L. For example, the high frame rate R may be the maximum frame rate supported by the video capture process, while the low frame rate L may be a frame rate reduced relative to the maximum frame rate.

[0036] Exemplarily, the detection result can indicate the presence or absence of a target of interest in the current scene. If there is a target of interest in the current scene, a high frame rate can be selected as the target frame rate for image and video acquisition. If there is no target of interest in the current scene, a low frame rate can be selected as the target frame rate for image and video acquisition. Additionally, the frame rate can be dynamically adjusted according to the number of targets of interest in the current scene. The frame rate can be positively correlated with the number of targets of interest. When the number of targets of interest is greater than or equal to a threshold, the preset maximum frame rate is used for acquisition. Thus, the corresponding frame rate can be adaptively selected for acquisition based on the presence or absence of a target of interest in the current scene to obtain target video frames. The detection result of the present application can be determined based on a certain video frame or based on video frames within a certain period, which is not limited herein.

[0037] In a specific application scenario, the image acquisition device may be in a certain operating state, and the device has an initial frame rate (or real-time frame rate). The target frame rate selected from the preset frame rates may be the same as or different from the initial frame rate. For example, the image acquisition device defaults to video acquisition of the current scene at a low frame rate; performs target detection on the acquired images; if the detection result indicates that there is no target of interest in the current scene, continues to acquire the current scene at the low frame rate; if the detection result indicates that there is a target of interest in the current scene, selects and switches to the high frame rate for video acquisition of the current scene.

[0038] Another exemplification is that when the target detection algorithm detects that there is no target of interest in the current scene for a period of time, the frame rate of the video encoder can be switched from the high frame rate value R to the low frame rate value L, and the GOP (Group of Pictures) value F of the encoder remains unchanged. Here, GOP defines a series of consecutive frames or groups of images in a video sequence. A GOP usually contains an I-frame (key frame) and several P-frames (predicted frames), and these frames are arranged in a specific order to enable quick access to any part of the video when needed. Since the frame rate decreases from R to L, the time interval of the I-frame changes from F / R to F / L. And because R > L, then F / R < F / L, that is, the interval time of the I-frame becomes longer, resulting in less I-frame data in the same time, thus reducing the overall bit rate.

[0039] If the target detection algorithm detects the presence of a target object of interest (such as animals, motor vehicles, non-motor vehicles, etc.) and / or the occurrence of an event of interest (such as triggering tripwire detection, perimeter detection, etc.) in the current scene, the input frame rate of the encoder can be switched to the high frame rate value R to include the target of interest in the video data acquired at the high frame rate.

[0040] Step S120: In response to the difference between the target frame rate and the initial frame rate, modify the initial bitstream protocol corresponding to the initial frame rate to obtain the target bitstream protocol.

[0041] The initial frame rate can be an unadjusted frame rate (such as the default frame rate of the acquisition device) or an adjusted frame rate. This is not limited here, but is only for the purpose of easy distinction.

[0042] Based on the steps outlined above, in specific application scenarios, the frame rate may or may not be adjusted depending on the detection results. If the frame rate is not adjusted (i.e., the target frame rate is the same as the initial frame rate), one can choose to maintain the previous relevant acquisition parameters for image and video acquisition, or one can choose to make certain adjustments to the relevant acquisition parameters; no limitation is made here.

[0043] If the frame rate is adjusted (i.e., the target frame rate differs from the initial frame rate), the initial bitstream protocol corresponding to the initial frame rate is modified to obtain the target bitstream protocol. The initial bitstream protocol refers to the streaming media protocol used when capturing images and videos at the initial frame rate. By modifying the protocol parameters in the initial bitstream protocol according to the target frame rate, the streaming media protocol required for capturing images and videos at the target frame rate can be obtained. It should be noted that the bitstream protocol in this application can include standard protocols and / or proprietary protocols, such as H.264, H.265, Onvif access protocol, national standard access protocol, RTSP streaming protocol, etc., which will not be elaborated here.

[0044] For example, methods for modifying the protocol may include, but are not limited to, making corresponding adjustments to frame rate information and / or timestamp information in the protocol.

[0045] Step S130: Encode and encapsulate the target video frame according to the target bitstream protocol to obtain the target bitstream.

[0046] Based on the steps described above, after obtaining the target bitstream protocol, the bitstream can be processed according to the target bitstream protocol and existing encoding and encapsulation methods to obtain the target bitstream.

[0047] As can be seen, this application, by detecting the target of interest in the current scene, can select the target frame rate corresponding to the detection result from the preset frame rate for image and video acquisition, thereby obtaining the target video frame. This allows for dynamic adjustment of the video frame rate according to the current scene. If the frame rate switches during image and video acquisition, in response to the difference between the target frame rate and the initial frame rate, the initial bitstream protocol corresponding to the initial frame rate is modified to obtain the target bitstream protocol. The target video frame is then encoded and encapsulated according to the target bitstream protocol to obtain the target bitstream. While ensuring that the encoding effect and decoding playback are not affected, this application more effectively saves bitrate transmission bandwidth and data storage capacity, resulting in smooth playback when the frame rate changes and a more natural transition when the frame rate switches.

[0048] Based on the above embodiments, this application embodiment describes the steps for modifying the initial bitstream protocol corresponding to the initial frame rate to obtain the target bitstream protocol. Specifically, the method of this embodiment includes the following steps:

[0049] Obtain the video playback type corresponding to the target video frame; modify the initial bitstream protocol according to the video playback type to obtain the target bitstream protocol.

[0050] The video playback type can be preset or obtained from the video decoder (or video playback terminal). To maintain consistency in the encoding and decoding process, a playback type extension field can be added to the bitstream protocol used by the video encoding segment and the video decoder to characterize the video playback type at the decoder (e.g., adding an extension field to a proprietary protocol). Thus, the video encoding segment can obtain the current video playback strategy (video playback type) from the video decoder through application-layer communication. Common video playback types can be distinguished based on the corresponding decoding playback strategy as either frame-per-second (FPS) playback or presentation-time-stamp (PTS) playback. In other words, the video encoding segment can distinguish whether the current video playback type is frame-per-second or presentation-time-stamp based on the extension field during network communication.

[0051] For example, after determining the video playback type at the decoding end, the relevant parameters in the bitstream protocol can be modified accordingly to adapt to the encoding and decoding process after frame rate switching, ensuring that the video bitstream can be transcoded and played normally at the decoding end, thus optimizing the video playback effect.

[0052] Based on the above embodiments, this application embodiment describes the steps of modifying the initial bitstream protocol according to the video playback type to obtain the target bitstream protocol. The video playback type includes frame rate playback and timestamp playback. Specifically, the method of this embodiment includes the following steps:

[0053] In response to the video playback type being frame rate playback, the frame rate information in the initial bitstream protocol is modified according to the target frame rate to obtain the target bitstream protocol; in response to the video playback type being timestamp playback, the timestamp information in the initial bitstream protocol is modified to obtain the target bitstream protocol.

[0054] Referring to the foregoing embodiments, in the prior art, if the frame rate within a GOP changes and the frame rate value is not fixed, but playback is performed according to the frame rate information of I-frames, it will cause a slow-motion effect in the video playback. Therefore, if the video playback type is frame rate playback, this application can solve the above problem by modifying the initial bitstream protocol according to the target frame rate after switching the frame rate. Similarly, if the frame rate within a GOP changes and the frame rate value is not fixed, but playback is performed according to the PTS timestamp, this application can modify the initial bitstream protocol according to the obtained timestamp information after switching the frame rate.

[0055] Based on the above embodiments, this application embodiment describes the steps of modifying the frame rate information in the initial bitstream protocol according to the target frame rate to obtain the target bitstream protocol. The initial bitstream protocol includes standard protocols and proprietary protocols. Specifically, the method of this embodiment includes the following steps:

[0056] The frame rate information in the frame header of the standard protocol is modified according to the target frame rate to obtain the modified standard protocol, and keyframes are inserted according to the frame rate information in the frame header of the modified standard protocol. The frame rate information in the private protocol is modified according to the target frame rate, and the timestamp information in the private protocol is modified according to the obtained target timestamp to obtain the modified private protocol. The modified standard protocol and the modified private protocol are determined as the target bitstream protocol.

[0057] It should be noted that when switching frame rates, if the video playback type is frame rate playback, the frame rate information in the VUI (Video Usability Information) of the SPS (Sequence Parameter Set) frame header of the standard encoding protocol is modified to the switched frame rate (target frame rate, equivalent to R or L in the aforementioned embodiments), resulting in the modified standard protocol. This ensures that even when the frame rate is parsed and played according to the frame rate field in the raw bitstream frame header protocol, it can still play normally at the actual frame rate.

[0058] Since video frames before and after a frame rate change are difficult to reference each other, this application can forcibly insert a special key frame (IDR frame) to ensure the stability of the decoding process and error recovery capability. This can immediately refresh the reference frame list in the decoder, ensuring that the decoding process restarts from the key frame, thereby preventing error propagation.

[0059] For example, since the SPS frame header appears only before the I-frame, this application can ensure that the changed frame rate is correctly read immediately in the next frame (I-frame) by forcing I-frame processing, thus ensuring that the frame rate is normal for some players that play based on the raw bitstream frame rate information. For instance, this application can perform forced I-frame processing during frame rate switching, inserting IDR keyframes at the video frame position of the bitstream frame rate switching according to the frame header frame rate information in the modified standard protocol.

[0060] Furthermore, if a proprietary protocol exists in the specific application scenario, this application will also modify the frame rate information in the proprietary protocol according to the target frame rate, and modify the timestamp information in the proprietary protocol according to the obtained target timestamp, to obtain a modified proprietary protocol. The modified standard protocol and the modified proprietary protocol are then determined as the target bitstream protocol. This application does not limit the execution order of frame rate switching and protocol modification.

[0061] For example, modify the frame rate in the private protocol to the target frame rate, and modify the PTS of the private protocol and the PTS filled in the RTSP protocol according to the timestamp (PTS) recorded for each video frame captured after the frame rate switch.

[0062] By modifying the bitstream protocol as described above, modifying the VUI frame rate and forcing I-frames in the standard protocol ensures that third-party devices can play the raw bitstream normally at the switched frame rate after decoding. Modifying the frame rate and timestamp in proprietary protocols ensures normal transcoding and playback of proprietary protocols, Onvif access protocols, national standard access protocols, RTSP streaming protocols, and other protocols.

[0063] It should also be noted that if the video playback type is timestamp playback when switching frame rates, the frame rate issue does not need to be considered. Only the PTS information for each frame in the proprietary protocol needs to be correctly filled in, and I-frames are not required, achieving a more bitrate-efficient effect. The PTS information can be obtained from the device clock, frame acquisition time, or by referring to other existing technologies, which will not be elaborated here. For example, refer to... Figure 2 As shown, Figure 2 This is a schematic diagram illustrating the effects of forced I-frames and non-forced I-frames in the video encoding-based frame rate adjustment method of this application. As can be seen from the foregoing embodiments, the video playback type (whether played according to FPS frame rate information or PTS timestamp) can be determined by the extended fields of the proprietary protocol, thereby controlling whether to forcibly insert an I-frame during frame rate switching. This application does not limit the order in which protocol modification and forced I-frame insertion are executed.

[0064] Based on the above embodiments, this application embodiment describes the steps of selecting a target frame rate from a preset frame rate for acquisition based on the detection result of a target of interest in the current scene to obtain a target video frame. The preset frame rate includes a first frame rate and a second frame rate, where the first frame rate is less than the second frame rate. Specifically, the method of this embodiment includes the following steps:

[0065] In response to the detection result indicating that there is no target of interest in the current scene, the first frame rate is determined as the target frame rate for acquisition, and the target video frame is obtained; in response to the detection result indicating that there is a target of interest in the current scene, the second frame rate is determined as the target frame rate for acquisition, and the target video frame is obtained.

[0066] Referring to the foregoing embodiments, the first frame rate corresponds to the low frame rate L in the foregoing embodiments, and the second frame rate corresponds to the high frame rate R in the foregoing embodiments. Therefore, if the detection result indicates that there is no target of interest in the current scene, the first frame rate is determined as the target frame rate for acquisition, resulting in a target video frame. If the detection result indicates that there is a target of interest in the current scene, the second frame rate is determined as the target frame rate for acquisition, resulting in a target video frame.

[0067] Based on the above embodiments, this application embodiment describes the steps prior to encoding and encapsulating the target video frame according to the target bitstream protocol to obtain the target bitstream. Specifically, the method of this embodiment includes the following steps:

[0068] Obtain the time difference between the current video frame and the adjacent historical video frames; perform time correction processing based on the time difference.

[0069] The historical video frame (historical frame) can be the previous video frame adjacent to the current video frame (current frame). The current video frame and the historical video frame can be I-frames or other types of video frames; there are no restrictions here. The time difference includes the frame interval time and / or the timestamp difference.

[0070] It should be noted that, due to the internal stabilization adjustment time and encoding buffer of the encoder during frame rate switching, the output video frame rate after the frame rate switch cannot guarantee that its frame interval is exactly the same as the frame interval corresponding to the target frame rate. Therefore, in order to ensure the smoothness of frame rate switching and the subsequent video playback effect, this application performs time correction processing based on the frame interval time ΔT (in ms) between the current video frame and the historical video frames and / or the difference in the acquisition timestamps of these two frames ΔPTS (in ms). The time correction processing can include frame rate correction and timestamp correction.

[0071] Based on the above embodiments, this application embodiment describes the steps of time correction processing based on time difference. Specifically, the method of this embodiment includes the following steps:

[0072] Obtain the frame interval time between the current video frame and historical video frames; determine the intermediate frame rate corresponding to the frame rate switching process based on the frame interval time; correct the current frame rate of the current video frame based on the intermediate frame rate.

[0073] This embodiment describes the frame rate correction process in conjunction with the aforementioned embodiments. The frame interval between the current video frame and historical video frames is ΔT, and the difference in acquisition timestamps is ΔPTS. The intermediate frame rate TempFPS between the current frame and historical frames can be calculated as TempFPS = 1000 / ΔT (or 1000 / ΔPTS). Frame rate correction based on the intermediate frame rate makes the playback smoother during frame rate switching, eliminating the noticeable stuttering caused by frame rate changes.

[0074] Specifically, the current frame rate of the current frame is compared with the historical frame rate of the previous frame and the target frame rate. If the TempFPS is close to the historical frame rate of the previous frame (i.e., the frame rate before the switch, such as the initial frame rate), the current frame rate of the current frame is assigned to the frame rate before the switch in the bitstream protocol. If the TempFPS is close to the frame rate after the switch (such as the target frame rate), the current frame rate is assigned to the frame rate after the switch in the protocol. Furthermore, it can be determined whether the current video frame is an I-frame. If it is an I-frame, the current frame rate can be directly assigned to the frame rate after the switch in the protocol to ensure that the subsequent GOP frame rates are parsed according to the frame rate after the switch.

[0075] Based on the above embodiments, this application embodiment describes the steps of time correction processing based on time difference. Specifically, the method of this embodiment includes the following steps:

[0076] Obtain the timestamp difference between the current video frame and historical video frames; determine whether the current video frame is ahead or behind based on the timestamp difference; if so, correct the current timestamp of the current video frame.

[0077] In conjunction with the foregoing embodiments, this embodiment describes the process of correcting timestamps.

[0078] For example, calculate the frame interval time ΔTfloat = 1000 / CurrentFPS (current frame rate, i.e., the frame rate after switching, such as the target frame rate). Calculate the adjusted frame interval time ΔTfloat = ΔTfloat + TimeLeft (TimeLeft is the decimal part left over from the last timestamp correction; its specific calculation method can be found in TimeLeft = ΔTfloat - ΔTint, where ΔTint = (rounded)ΔTfloat, i.e., rounding ΔTfloat to obtain ΔTint). Current frame TimePts = Historical frame TimePts + ΔTint (the current frame's PTS is obtained by adding ΔTint to the timestamp of the historical frame).

[0079] Furthermore, the time stamp difference between the current frame and historical frames can be used to determine whether the current frame's time stamp duration (PTS) is ahead or behind. For example, if the difference between the current frame and the previous frame (historical frame) is greater than twice the time stamp duration (ΔTint), the current frame is behind. If the difference between the previous frame (historical frame) and the current frame (current frame) is greater than ΔTint / 2, the current frame is ahead. The specific criteria can be flexibly adjusted according to actual needs, and will not be elaborated here.

[0080] If the PTS lags, the adjustment is the frame time ΔTint, which is the actual timestamp of the current frame TimePts = current timestamp TimePts + ΔTint. If the PTS leads, the adjustment is ΔTint / 2, which is the actual timestamp of the current frame TimePts = current timestamp TimePts - ΔTint / 2.

[0081] It should be noted that the frame rate correction in this application mainly corrects the video frames during the frame rate switching process, meaning that after the frame rate switch is completed, the video can be parsed according to the frame rate specified in the protocol. Timestamp correction, on the other hand, can correct each frame in which the frame rate switch occurs.

[0082] In summary, the overall process of the video coding-based frame rate adjustment method of this application can be referred to as follows: Figure 3 As shown, Figure 3 This is an exemplary overall flowchart of the video coding-based frame rate adjustment method of this application. A detection algorithm can be used to detect whether a target of interest exists in the current scene, and then the capture frame rate of the image / video acquisition device is selected based on the detection result. If the frame rate changes, the encoding protocol is modified. It is determined whether the video playback type is frame-rate or timestamp-based, and then whether to perform forced I-frame processing is selected. Subsequently, the protocol frame rate and timestamp of the bitstream encapsulation can be corrected, and then the video frames can be encoded and encapsulated to obtain the target bitstream.

[0083] Alternatively, in addition to adjusting the frame rate of the image and video acquisition device based on the presence of a target of interest in the current scene, the method of this application can also similarly adjust other acquisition quality parameters of the image and video acquisition device (such as acquisition resolution, compression format, device power consumption, etc., which are not limited here) based on the presence of a target of interest in the current scene. For example, when there is no target of interest in the current scene, acquisition can be performed according to a first resolution, and when there is a target of interest in the current scene, acquisition can be performed according to a second resolution, where the first resolution is smaller than the second resolution. This ensures that the image and video quality when there is no target of interest in the current scene is lower than the image and video quality when there is a target of interest in the current scene, and the specific details are not elaborated here.

[0084] Figure 4 This is an exemplary bitrate saving effect diagram of the video coding-based frame rate adjustment method of this application. The method described above can effectively save bitrate when there is no target of interest in the current scene.

[0085] It should be further noted that the execution entity of the video coding-based frame rate adjustment method can be a video coding-based frame rate adjustment device. For example, the video coding-based frame rate adjustment method can be executed by a terminal device, a server, or other processing devices. The terminal device can be a user equipment (UE), computer, mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, this video coding-based frame rate adjustment method can be implemented by a processor calling computer-readable instructions stored in memory.

[0086] Figure 5 This is a block diagram illustrating a video encoding-based frame rate adjustment device, as shown in an exemplary embodiment of this application. Figure 5 As shown, the exemplary video encoding-based frame rate adjustment device 500 includes: a frame rate selection module 510, a protocol modification module 520, and a video encoding module 530. Specifically:

[0087] The frame rate selection module 510 is used to select the target frame rate from the preset frame rate for acquisition based on the detection results of the target of interest in the current scene, so as to obtain the target video frame.

[0088] The protocol modification module 520 is used to modify the initial bitstream protocol corresponding to the initial frame rate in response to the difference between the target frame rate and the initial frame rate, so as to obtain the target bitstream protocol.

[0089] The video encoding module 530 is used to encode and encapsulate the target video frames according to the target bitstream protocol to obtain the target bitstream.

[0090] In this exemplary video coding-based frame rate adjustment device, by detecting the target of interest in the current scene, the target frame rate corresponding to the detection result can be selected from the preset frame rate for image and video acquisition to obtain the target video frame. This allows for dynamic adjustment of the video frame rate according to the current scene. If the frame rate switches during image and video acquisition, in response to the difference between the target frame rate and the initial frame rate, the initial bitstream protocol corresponding to the initial frame rate is modified to obtain the target bitstream protocol. The target video frame is then encoded and encapsulated according to the target bitstream protocol to obtain the target bitstream. While ensuring that the encoding effect and decoding playback are not affected, this device more effectively saves bitrate transmission bandwidth and data storage capacity, resulting in smooth playback without stuttering when the frame rate changes and a more natural transition when the frame rate switches.

[0091] It should be noted that the apparatus and method provided in the above embodiments belong to the same concept, and the specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the apparatus provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation.

[0092] The functions of each module can be found in the implementation example of the frame rate adjustment method based on video encoding, and will not be repeated here.

[0093] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 100 includes a memory 101 and a processor 102. The processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in any of the above embodiments of the video encoding-based frame rate adjustment method. In a specific implementation scenario, the electronic device 100 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 100 may also include mobile devices such as laptops and tablets, which are not limited here.

[0094] Specifically, processor 102 controls itself and memory 101 to implement the steps in any of the above-described embodiments of the video coding-based frame rate adjustment method. Processor 102 may also be referred to as a CPU (Central Processing Unit). Processor 102 may be an integrated circuit chip with signal processing capabilities. Processor 102 may also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 102 may be implemented using integrated circuit chips.

[0095] In this exemplary electronic device, by detecting the target of interest in the current scene, the target frame rate corresponding to the detection result can be selected from a preset frame rate for image and video acquisition to obtain the target video frame. This allows for dynamic adjustment of the video frame rate according to the current scene. If the frame rate switches during image and video acquisition, in response to the difference between the target frame rate and the initial frame rate, the initial bitstream protocol corresponding to the initial frame rate is modified to obtain the target bitstream protocol. The target video frame is then encoded and encapsulated according to the target bitstream protocol to obtain the target bitstream. While ensuring that the encoding effect and decoding playback are not affected, this method more effectively saves bitrate transmission bandwidth and data storage capacity, resulting in smooth playback when the frame rate changes and a more natural transition when the frame rate switches.

[0096] Please see Figure 7 , Figure 7 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor. The program instructions 111 are used to implement the steps in any of the above embodiments of the video coding-based frame rate adjustment method.

[0097] In this exemplary storage medium, by running the program instructions within the storage medium, the detection results of the target of interest in the current scene can be selected from a preset frame rate to capture images and videos at the target frame rate corresponding to the detection results, thereby obtaining the target video frame. This allows for dynamic adjustment of the video frame rate according to the current scene. If the frame rate switches during image and video capture, in response to the difference between the target frame rate and the initial frame rate, the initial bitstream protocol corresponding to the initial frame rate is modified to obtain the target bitstream protocol. The target video frame is then encoded and encapsulated according to the target bitstream protocol to obtain the target bitstream. While ensuring that the encoding effect and decoding playback are not affected, this method more effectively saves bitrate transmission bandwidth and data storage capacity, resulting in smooth playback when the frame rate changes and a more natural transition when the frame rate switches.

[0098] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0099] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0100] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0101] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A frame rate adjustment method based on video coding, characterized in that, The method includes: Based on the detection results of the target of interest in the current scene, the target frame rate is selected from the preset frame rate for acquisition to obtain the target video frame; In response to the difference between the target frame rate and the initial frame rate, the initial bitstream protocol corresponding to the initial frame rate is modified to obtain the target bitstream protocol; The target video frame is encoded and encapsulated according to the target bitstream protocol to obtain the target bitstream.

2. The method according to claim 1, characterized in that, The step of modifying the initial bitstream protocol corresponding to the initial frame rate to obtain the target bitstream protocol includes: Obtain the video playback type corresponding to the target video frame; The initial bitstream protocol is modified according to the video playback type to obtain the target bitstream protocol.

3. The method according to claim 2, characterized in that, The video playback type includes frame rate playback and timestamp playback. Modifying the initial bitstream protocol according to the video playback type to obtain the target bitstream protocol includes: In response to the video playback type being the frame rate playback, the frame rate information in the initial bitstream protocol is modified according to the target frame rate to obtain the target bitstream protocol; In response to the video playback type being timestamp playback, the timestamp information in the initial bitstream protocol is modified to obtain the target bitstream protocol.

4. The method according to claim 3, characterized in that, The initial bitstream protocol includes a standard protocol and a proprietary protocol. Modifying the frame rate information in the initial bitstream protocol according to the target frame rate to obtain the target bitstream protocol includes: The frame header frame rate information in the standard protocol is modified according to the target frame rate to obtain the modified standard protocol, and a keyframe is inserted according to the frame header frame rate information in the modified standard protocol. The frame rate information in the private protocol is modified according to the target frame rate, and the timestamp information in the private protocol is modified according to the obtained target timestamp to obtain the modified private protocol. The modified standard protocol and the modified proprietary protocol are determined as the target bitstream protocol.

5. The method according to claim 1, characterized in that, The preset frame rate includes a first frame rate and a second frame rate, wherein the first frame rate is less than the second frame rate. The step of selecting a target frame rate from the preset frame rate for acquisition based on the detection results of the target of interest in the current scene to obtain the target video frame includes: In response to the detection result indicating that the target of interest does not exist in the current scene, the first frame rate is determined as the target frame rate and collected to obtain the target video frame; In response to the detection result indicating the presence of the target of interest in the current scene, the second frame rate is determined as the target frame rate and collected to obtain the target video frame.

6. The method according to claim 1, characterized in that, Before encoding and encapsulating the target video frame according to the target bitstream protocol to obtain the target bitstream, the method further includes: Obtain the time difference between the current video frame and the adjacent historical video frames; Time correction is performed based on the time difference.

7. The method according to claim 6, characterized in that, The time correction process based on the time difference includes: Obtain the frame interval time between the current video frame and the historical video frames; The intermediate frame rate corresponding to the frame rate switching process is determined based on the frame interval time. The current frame rate of the current video frame is corrected based on the intermediate frame rate.

8. The method according to claim 6, characterized in that, Time correction processing is performed based on the time difference, including: Obtain the timestamp difference between the current video frame and the historical video frame; Determine whether the current video frame is ahead or behind based on the timestamp difference; If so, the current timestamp of the current video frame is corrected.

9. An electronic device, characterized in that, The method includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the method described in any one of claims 1 to 8.

Images