Video frame rate control method and device, electronic equipment and readable storage medium

Abstract

The invention provides a video frame rate control method and device, electronic equipment and a readable storage medium, and relates to the technical field of video processing. In view of the problems of tedious traditional pre-configuration of frame rate parameters, unstable output quality and configuration errors, the video frame rate control method provided by the invention comprises the following steps: generating a sequence of output frame identifiers, and circularly broadcasting each output frame identifier in the sequence in sequence according to the period of the output frame rate; for an input channel corresponding to a video source, after a video frame is received, obtaining a current output frame identifier; based on a comparison result of the current output frame identifier and an output frame identifier obtained last time, determining whether to write the received video frame into a cache; and based on the output frame identifier, performing video output on the video frame in the cache according to the output frame rate. According to the embodiment of the invention, the video output quality, stability and dynamic adaptability can be improved.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video frame rate control method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] In video processing systems, it is often necessary to integrate and output video streams from different sources (such as cameras, video files, and network streams). These input video streams often have different frame rates (e.g., 30fps, 60fps). Traditional frame rate conversion methods typically require pre-configuring the input and output frame rate parameters. However, in scenarios where the input frame rate is unknown, has slight deviations, or involves a mixture of multiple different frame rates, pre-configuration is not only cumbersome but also difficult to adapt to dynamic changes, leading to unstable output quality (e.g., stuttering, frame skipping) or an increased risk of configuration errors. Summary of the Invention

[0003] According to various embodiments of this application, a video frame rate control method, apparatus, electronic device, and readable storage medium are provided; which can improve video output quality and stability.

[0004] In a first aspect, this application provides a video frame rate control method, the method comprising: A sequence of output frame identifiers is generated, and each output frame identifier in the sequence is broadcast cyclically according to the period of the output frame rate. For the input channel corresponding to the video source, after receiving a video frame, the current output frame identifier is obtained. Based on the comparison result between the current output frame identifier and the previously obtained output frame identifier, it is determined whether to write the received video frame into the buffer. Based on the output frame identifier, the video frames in the buffer are output according to the output frame rate.

[0005] By generating a sequence and using the sequentially broadcast output frame identifiers as a timing reference for each received video frame, the above method eliminates the need for pre-configured frame rate parameters. It is applicable to scenarios with dynamically changing video source frame rates and multiple video sources with different frame rates, reducing configuration complexity. Furthermore, the output frame identifiers obtained for each video frame provide a basis for subsequent decisions on whether to write to the buffer. This allows for frame dropping or insertion by controlling whether to write to the buffer, ensuring orderly data processing, adapting to dynamic frame rate changes, ensuring stable video output, reducing video stuttering and frame skipping caused by improper frame rate configuration, avoiding configuration errors, and improving output quality. It boasts strong usability and practicality.

[0006] In one possible implementation of the first aspect, the sequence for generating the output frame identifier includes: The sequence of output frame identifiers is determined based on the multiple relationship between the expected maximum output frame rate and the expected minimum input frame rate.

[0007] In one possible implementation of the first aspect, determining whether to write the currently received video frame to the buffer based on a comparison between the current output frame identifier and the previously acquired output frame identifier includes: If the current output frame identifier is equal to the previously acquired output frame identifier, the received video frame will not be written to the buffer. If the current output frame identifier is not equal to the previously acquired output frame identifier, the received video frame is written to the buffer.

[0008] In one possible implementation of the first aspect, the step of writing the received video frame into the buffer if the current output frame identifier is not equal to the previously acquired output frame identifier includes: If the current output frame identifier is not equal to the previously acquired output frame identifier, the write start position of the cache is determined based on the current output frame identifier; The frame information of the received video frame is written from the starting position to be written and the subsequent preset number of positions; the preset number is less than the number of output frame identifiers in the sequence.

[0009] In one possible implementation of the first aspect, based on the output frame identifier, the video frames in the buffer are output as video at the output frame rate, including: Based on the output frame identifier, determine the virtual frame identifier of the video frame in the buffer; Based on the virtual frame identifier, the video frames are read sequentially, and the video is output according to the output frame rate; The output frame identifier corresponds one-to-one with the virtual frame identifier, and the virtual frame identifier is used to indicate the location where the video frame is stored.

[0010] In one possible implementation of the first aspect, the step of outputting the video frames in the buffer according to the output frame rate based on the output frame identifier includes: Based on the output frame identifier, the video frames from multiple video sources in the buffer are spliced ​​together and output according to the output frame rate.

[0011] In one possible implementation of the first aspect, the step of concatenating the video frames from multiple video sources in the buffer based on the output frame identifier includes: Based on the same output frame identifier, the video frames from multiple video sources are identified as having the same virtual frame identifier in the buffer; Based on the same virtual frame identifier, the video frames are read from the cache corresponding to each video source, and the read video frames are stitched together into one frame.

[0012] Secondly, this application provides a video frame rate control device, comprising: The identification broadcast unit is used to generate a sequence of output frame identifiers and to cyclically broadcast each of the output frame identifiers in the sequence according to the period of the output frame rate. The receiving unit is used to obtain the current output frame identifier after receiving a video frame from the input channel corresponding to the video source. The processing unit is used to determine whether to write the received video frame into the buffer based on the comparison result between the current output frame identifier and the previously acquired output frame identifier; The output unit is used to output the video frames in the buffer according to the output frame rate based on the output frame identifier.

[0013] Thirdly, this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in any one of the first aspects.

[0014] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any one of the first aspects.

[0015] Fifthly, this application provides a computer program product that, when run on a device, causes the device to perform the method described in any one of the first aspects above.

[0016] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram illustrating the application scenario architecture of the video frame rate control method provided in the embodiments of this application. Figure 2 A schematic diagram illustrating the implementation process of the video frame rate control method provided in this application embodiment; Figure 3 A schematic diagram illustrating the update of the cache mapping table for the frame interpolation control process provided in an embodiment of this application; Figure 4 A schematic diagram of video frame interpolation control provided in an embodiment of this application; Figure 5 A schematic diagram illustrating video frame dropping control provided in an embodiment of this application; Figure 6 A schematic diagram of adaptive frame interpolation control provided in an embodiment of this application; Figure 7 A schematic diagram of adaptive frame dropping control provided in an embodiment of this application; Figure 8 This is a schematic diagram of the spliced ​​screen display provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of the video frame rate control device provided in the embodiments of this application; Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0021] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0024] In video processing systems, video streams from different sources often have different frame rates, and the frame rate of the same video stream may change dynamically or have slight deviations at different times. Traditional frame rate conversion methods (such as frame duplication, frame dropping, and frame interpolation) usually require pre-configuring the input and output frame rate parameters. However, in scenarios where the input frame rate is unknown, the preset frame rate has slight deviations from the actual frame rate, or there are multiple mixed inputs with different frame rates, pre-configuring the frame rate parameters is not only cumbersome but also difficult to adapt to dynamically changing scenarios, leading to unstable output quality or an increased risk of configuration errors.

[0025] To address the above technical issues, this application provides a video frame rate control method. This method introduces a globally synchronized, periodically looping output frame identifier and broadcasts it to the input channel corresponding to the video source. The input channel can perform local sampling and decision-making based on this output frame identifier. It can adaptively drop frames by comparing whether two consecutively acquired output frame identifiers are the same, or pre-write video frames into multiple buffer locations corresponding to future output frame identifiers for adaptive frame interpolation. The output channel reads video frames from the buffer based on the currently broadcast output frame identifier and outputs the video. By using the global output frame identifier as a timing reference for sampling, no configuration of any input frame rate parameters is required, enabling stable output frame rate control in scenarios with unknown input frame rates, input frame rate deviations, and mixed input frame rates.

[0026] The following examples illustrate the application scenario architecture of this video frame rate control method.

[0027] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating the application scenario architecture of the video frame rate control method provided in the embodiments of this application. Figure 1 As shown, video streams from different video sources are input through their respective input channels, such as input channel 1 for video source 1, input channel 2 for video source 2, and input channel 3 for video source 3. The identifier broadcasting unit in the processor generates and periodically broadcasts the output frame identifier. For each input channel, upon receiving a video frame, the processor determines whether to write the video frame to the buffer based on whether the output frame identifiers obtained in two consecutive transactions are the same, and adaptively performs frame dropping or frame insertion. The output channel reads the video from the buffer according to the broadcast output frame identifier and outputs the video to the display at a set output frame rate for image display.

[0028] Based on the above application scenario architecture, the specific implementation process of this video frame rate control method will be described below through an example.

[0029] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating the implementation flow of the video frame rate control method provided in this application embodiment. The execution subject of this method can include, for example, […]. Figure 1 The electronic device shown includes an input channel, a processing unit, and an output channel. This electronic device is a system for performing video processing, and may further include, for example... Figure 1 The display shown; as Figure 2 As shown, the method may include the following steps: S201, generate a sequence of output frame identifiers, and broadcast each output frame identifier in the sequence in a cyclic manner according to the period of the output frame rate.

[0030] In some embodiments, the electronic device generates and maintains a global, periodically cyclic sequence of output frame identifiers. This sequence can be an increasing sequence of natural numbers, such as 0, 1, 2, ..., N-1, where N is an integer greater than 1. The output frame identifiers are periodically updated according to a specified output frame rate. The updated output frame identifiers are then periodically and cyclically broadcast to the input channels via a communication channel (such as a bus, internal messages, etc.), for example, broadcasting periodically and cyclically from 0 to N-1. When multiple video sources exist, each updated output frame identifier is periodically and cyclically broadcast to the input channels corresponding to all video sources; for example, simultaneously broadcasting to... Figure 1 The input channels 1, 2, and 3 are shown.

[0031] For example, the broadcast trigger signal can be driven by a timer for the output frame rate; the broadcast period is calculated based on the output frame rate. For instance, if the output frame rate is 30 FPS, the broadcast period is 1 / 30 ≈ 33.33 milliseconds, meaning the output frame identifier is updated and broadcast every 33.33 milliseconds. The output frame rate can be adjusted based on actual application requirements.

[0032] In some embodiments, generating a sequence of output frame identifiers includes: The sequence of output frame identifiers is determined based on the multiple relationship between the expected maximum output frame rate and the expected minimum input frame rate.

[0033] For example, the maximum value in the sequence of output frame identifiers, such as the value of N, can be determined based on the multiple relationship between the expected maximum output frame rate and the expected minimum input frame rate. The expected maximum output frame rate and the expected minimum input frame rate can be determined based on the fastest and slowest frame rates supported by the transmission protocol or standard specification, or based on the coding rate and packet loss rate of the network transmission model (such as when the network is poor), or based on the frame rate distribution in historical data, or based on the network and environmental constraints in the actual application scenario.

[0034] Accordingly, the maximum value of the sequence is determined based on the maximum ratio of the expected maximum output frame rate to the expected minimum input frame rate. To ensure the continuity of video frames written to the buffer, the maximum interpolation multiple must not exceed N-2, that is, the minimum value of N = ceil(expected maximum output frame rate / expected minimum input frame rate) + 2, where ceil() is rounded up, thus determining the value of N.

[0035] For example, the input channel writes a frame of data when the current output frame identifier = X. During the write buffer, since the position corresponding to output frame identifier = X is being read by the output channel, the writing needs to start from the position of the next frame, i.e., the starting position of the write is X+1. During writing, multiple subsequent positions can be written consecutively, such as M positions, covering positions X+1, X+2, ..., X+M. If the written M positions exceed N-1 positions, the loop wraps back to position 0 and continues moving until position X-1 is reached. Therefore, the maximum interpolation multiple can also be N-1.

[0036] Furthermore, when the theoretical interpolation rate differs from the actual instantaneous interpolation rate (i.e., the actual instantaneous interpolation rate is greater than the theoretical interpolation rate), such as when the instantaneous input frame rate is smaller than the theoretical input frame rate, and the total buffer size N is 2 frames larger than the theoretical minimum number of consecutive positions required (one of which is the frame being read and is prohibited from being written to), and the actual instantaneous capability can handle a maximum of N-1 times the demand (i.e., the actual hardware can handle the multiplier), the theoretical value of the maximum interpolation rate can still be N-2, reserving a margin of one frame to ensure that even when the instantaneous demand slightly exceeds the theoretical value, the system will not experience output interruptions. This avoids the small error between the theoretical and actual input frame rates affecting the interpolation, making the system more stable and reliable, and better able to cope with instantaneous fluctuations.

[0037] For example, if the expected maximum output frame rate is 30 FPS and the expected minimum input frame rate is 10 FPS, the required maximum interpolation ratio is 30 / 10 = 3 times, and the theoretical minimum is N = 3 + 2 = 5. In practice, N = 8 can also be chosen, resulting in an actual interpolation performance of 8 - 2 = 6 times. Although there is performance redundancy in this case, the interpolation effect still takes effect at the required maximum ratio of 3 times, and choosing N = 8 facilitates the implementation of binary counting loops.

[0038] In addition, to maintain a safe distance of at least one frame between read and write operations, besides prohibiting writing to the current read frame position X, its preceding position X-1 is also set as a protection frame and prohibited from writing, ensuring that a safe distance of at least one frame is always maintained between the write frame position and the read frame position. In application scenarios facing clock domain differences and communication delays in cross-platform or distributed systems, the frame position X-1 protection frame provides additional buffer margin, ensuring that even if read and write operations experience a brief desynchronization due to delay, the write frame position will not occupy the currently read frame position X, fundamentally preventing read-write conflicts. Because its preceding position X-1 is also set as a protection frame and prohibited from writing, the maximum interpolation multiple after correction is N-2.

[0039] S202: For the input channel corresponding to the video source, after receiving a video frame, obtain the current output frame identifier.

[0040] In some embodiments, after the input channel has received a frame of video data, the identifier of the output frame currently being broadcast by the system is immediately read. This read operation is an instantaneous, synchronous hardware operation. Based on the currently read output frame identifier, timing parameters are provided for subsequent decisions (such as dropping frames or writing to the buffer).

[0041] Accordingly, for scenarios with multiple video sources, the same operation is performed on the input channel corresponding to each video source. For example, if the output frame rate is 30 FPS, the output frame identifier is updated and broadcast every 1 / 30 ms, the input frame rate corresponding to video source 1 is 60 FPS (faster), and the input frame rate corresponding to video source 2 is 15 FPS (slower). For input channel 1 corresponding to video source 1, the current output frame identifier is read once every 1 / 60 ms when a video frame is received. For input channel 2 corresponding to video source 2, the current output frame identifier is read once every 1 / 15 ms when a video frame is received. Thus, multiple input sources with different, unknown, or dynamically changing frame rates can be uniformly processed based on the output frame identifier. A unified output frame identifier is assigned to the video frames of each channel and used as a timing reference. No prior configuration of the input frame rate is required, significantly reducing the system configuration complexity.

[0042] S203, based on the comparison between the current output frame identifier and the previously acquired output frame identifier, determine whether to write the received video frame into the buffer.

[0043] In some embodiments, each time the input channel receives a video frame, it reads the current output frame identifier and compares it with the output frame identifier read when the last video frame was received, to preliminarily determine the speed of the input frame rate relative to the output frame rate. The comparison result includes whether the previously read output frame identifier and the currently read output frame identifier are equal or unequal.

[0044] Accordingly, based on the comparison results, it is determined whether to perform frame dropping or frame insertion operations. If frame dropping is performed, the currently received video frame is not written to the buffer. If frame insertion is performed, the currently received video frame is written to the buffer. Thus, in cases where the specific input frame rate is uncertain, there is no need to configure parameters such as the input frame rate and conversion ratio. The received video frame is adjusted accordingly based on the output frame identifier. Furthermore, it can respond instantly when the input frame rate changes, achieving real-time adaptive adjustment of the frame rate.

[0045] In addition, for multiple input channels, each input channel performs independent judgment and frame processing based on a unified output frame identifier and the same implementation principle.

[0046] In some embodiments, determining whether to write the currently received video frame to the buffer based on a comparison between the current output frame identifier and the previously acquired output frame identifier includes: If the current output frame identifier is equal to the previously acquired output frame identifier, the received video frame will not be written to the buffer; if the current output frame identifier is not equal to the previously acquired output frame identifier, the received video frame will be written to the buffer.

[0047] For example, if the last output frame identifier lastID is equal to the current output frame identifier currentID, it means that the input frame rate of the video frame is faster than the broadcast frequency, that is, the input frame rate is greater than the specified output frame rate. In order to keep in line with the output, the currently received video frame is discarded and not written to the buffer to prevent backlog.

[0048] For example, if the output frame identifier read last time is not equal to the output frame identifier read now, it may be because the input frame rate is less than the output frame rate or the input frame rate is greater than the output frame rate. In the case where the input frame rate is less than the output frame rate, the output frame identifier read each time a video frame is received is different from the previous one. For example, when the first video frame is received, the output frame identifier read is 2. When the second video frame, which is consecutive to the first video frame, is received, the output frame identifier read is 4, which is different from the previous one. Then the second video frame is written to the buffer.

[0049] In cases where the input frame rate is greater than the output frame rate, for example, when receiving consecutive first, second, and third video frames, if the output frame identifiers read from the second and third video frames are the same as those read from the first video frame, then the second and third video frames are discarded and not written to the buffer. When receiving a consecutive fourth video frame, the system broadcasts a new output frame identifier. If the output frame identifier read from the fourth video frame is different from that of the previous frame, then the fourth video frame is written to the buffer.

[0050] In some embodiments, if the current output frame identifier is not equal to the previously acquired output frame identifier, the received video frame is written to the buffer, including: If the current output frame identifier is not equal to the previously acquired output frame identifier, determine the write start position of the buffer based on the current output frame identifier; write the frame information of the received video frame from the write start position and the subsequent preset number of positions; the preset number is less than the number of output frame identifiers in the sequence.

[0051] For example, since the output frame identifier is consistent with the cached virtual frame identifier, in order to ensure that the write and read operations do not conflict (for example, when the output frame identifier broadcast by the system is 1, the output channel is reading the video frame at the virtual frame 1 position), after reading the current output frame identifier, if it is different from the previous one, the position of the next virtual frame identifier corresponding to the current output frame identifier is taken as the writing start position; and when frame interpolation is required, the received video frame can be pre-written to the writing start position and multiple subsequent consecutive preset positions; when a new video frame needs to be written, based on the determined writing start position, the previously occupied cache position can be overwritten.

[0052] The preset number is determined by the interpolation factor, which is based on the ratio of the output frame rate to the input frame rate. When the output frame rate is known, the maximum interpolation factor can be derived by estimating the minimum input frame rate, thus determining the upper limit of the number of consecutive writable positions. To ensure the availability of virtual frame identifiers, this number of consecutive positions must be strictly less than the total number of virtual frame identifiers (i.e., the length of the output frame identifier sequence). The mapping relationship between virtual frame identifiers and physical frame identifiers is recorded and maintained by a cache mapping table, such as... Figure 3 As shown.

[0053] For example, writing a received video frame to a buffer may include writing the frame information of the video frame to the buffer, which may include the image size of the video frame and an address pointer in the physical frame buffer; writing a received video frame to a buffer may also include writing the actual image data of the video frame to the physical frame buffer.

[0054] For example, when processing video frame drops, the frame information of the currently received video frame is not written to the buffer, and the actual image data of the received video frame is written to the physical frame buffer. The buffer does not establish a mapping relationship with the physical frame buffer. When a new video frame is received in the future, since the previous video frame has not established a mapping relationship, the new video frame can overwrite the previous video frame, and the actual image data of the video frame is written to the physical buffer.

[0055] For example, when performing video frame interpolation, the actual image data of the video frame is written to the physical frame buffer once, and the frame information of the video frame (such as the same physical address in the physical frame buffer) is written to multiple consecutive virtual frame positions in the buffer.

[0056] Because each video frame contains a large amount of data, the actual image data of the video frame is processed using a streaming mechanism that receives and writes simultaneously. Once a frame of data is written, the output frame identifier is immediately read, and the corresponding physical frame identifier is written to the buffer mapping table. This write operation overwrites the mapping units for multiple consecutive frames starting from the next virtual frame. For example... Figure 3 The 2x interpolation mapping table uses the horizontal axis index to represent the virtual frame identifier, and each row of the mapping table stores the physical frame identifier (which can be converted into the actual storage address); the vertical axis represents the timing change of the output frame identifier, and the contents of the mapping table are dynamically updated with the timing of the output frames. Figure 3 The shaded area marks the virtual frame currently being read, and the content inside is the physical frame being read.

[0057] For example, when the input frame rate is less than or equal to the output frame rate, frame interpolation is required; such as... Figure 3 As shown, taking 8 virtual frames (virtual frame identifier (ID) = 0~7), 4 physical frames (physical frame identifier (ID) = 0~3), and an output frame rate twice the input frame rate as an example; each row represents a mapping table at a certain frame moment, and the mapping table is constantly updated as the output frame identifier changes; for the output frame 0, when video frame 0 is received, video frame 0 is written starting from the next virtual frame position of the currently read virtual frame 0, and written to multiple consecutive positions, such as positions 1 to 6 of virtual frames, and the output channel reads the actual image data corresponding to virtual frame 0 (no data at the first startup); for the output frame 1, no new video frame is received, and at this time the output channel reads the pre-inserted frame video data corresponding to virtual frame 1, that is, the actual data content of video frame 0.

[0058] For output frame 2, upon receiving video frame 1, the current output frame identifier is read as 2. Unlike the previous step, video frame 1 is written starting from the next virtual frame position after the currently read virtual frame 2, and this process continues for multiple consecutive positions, such as from virtual frame 3 to position 0. At this point, the output channel reads the pre-interpolated frame video data corresponding to virtual frame 2, which is still the actual data content of video frame 0. Similarly, for output frame 3, since no new video frame has been received, the output channel reads the pre-interpolated frame video data corresponding to virtual frame 3, which is the actual data content of video frame 1. For subsequent output frames 4 to 7, and so on, the same frame interpolation process is performed based on the same principle.

[0059] like Figure 4 As shown, when the input frame rate is less than or equal to the output frame rate, and Figure 3 Following the same principle, when physical frame 0 ends, if the output frame identifier is read as 0, then physical frame 0 is written to positions 1 to 6 of virtual frames. When the output frame identifier is 1, if no new video frame is received, then the previously pre-inserted physical frame 0 is read directly. When physical frame 1 ends, if the output frame identifier is read as 2, unlike the previous time, physical frame 1 is written to positions 3 to 0 of virtual frames. When the output frame identifier is 3, if no new video frame is received, then the previously pre-inserted physical frame 1 is read directly. When physical frame 2 ends, if the output frame identifier is read as 4, unlike the previous time, physical frame 2 is written to positions 5 to 2 of virtual frames, overwriting the positions 5 to 0 of virtual frames written by the previous video frame.

[0060] For example, when the input frame rate is greater than the output frame rate, frame dropping is performed; such as... Figure 5 As shown, when physical frame 0 is received, the current output frame identifier is read as 0, and physical frame 0 is written to positions 1 to 6 of virtual frames. When the current physical frame 1 is received, the output frame identifier read is still 0, the same as the previous one, so the current physical frame 1 is discarded. When physical frame 1 is received again (after the previous frame was discarded, the physical frame identifier remains unchanged at 1), the output frame identifier read is 2, different from the previous one, so the second physical frame 1 is written to positions 2 to 7 of virtual frames, overwriting the virtual frames 2 to 6 written in the previous frame. When the first physical frame 2 is received, the output frame identifier read is the same as the previous one, so the first physical frame 2 is discarded. The same principle is used to perform the corresponding frame discarding and writing operations for subsequent physical frames.

[0061] In some embodiments, when the larger frame rate and the smaller frame rate in the input and output frame rates satisfy a non-integer multiple relationship, the operation of dropping and inserting video frames can be dynamically adjusted based on the currently read output frame identifier to adapt to the dynamically changing number of dropped or inserted frames.

[0062] like Figure 6 In the adaptive frame interpolation scenario shown, the buffer corresponds to frame positions 0 to N-1, and there are N-2 possible interpolation positions (the maximum interpolation ratio derived from determining the size of N above can be N-2; for example, when N is 8, the corresponding maximum interpolation ratio is 6). When a video frame is received, the current output frame identifier is read, and the starting position to be written and the subsequent preset number of frame positions are determined based on the output frame identifier; when the ratio of the output frame rate to the input frame rate is not an integer, such as... Figure 6 When the interpolation frame rate is between 2.0 and 3.0, after inserting N-2 frames into the input video frame, some of the frames will be effective as 2 frames, and the other part will be effective as 3 frames. The ineffective frames will be automatically overwritten by the newly input video frame.

[0063] like Figure 7 In the adaptive frame dropping scenario shown, when the ratio of the input frame rate to the output frame rate is not an integer, such as when the frame dropping ratio is between 2.0 and 3.0, some input video frames are processed by dropping 1 frame every 2 frames, while the other part is processed by dropping 1 frame every 3 frames; thus, it can cope with different ratios.

[0064] S204, based on the output frame identifier, outputs the video frames in the buffer according to the output frame rate.

[0065] In some embodiments, the output frame identifier corresponds to the same virtual frame identifier in the buffer, and the virtual frame identifier in the buffer is associated with the physical frame identifier in the physical frame buffer. This allows output rules to be set based on the correspondence between the output frame identifier and the virtual frame identifier, thus determining the video frame to be output. For example, the output channel reads the video frame at the same virtual frame identifier position as the current output frame identifier and outputs the video; or the output channel reads the video frame at the same virtual frame identifier position as the previous (or previous several) output frame identifiers and outputs the video.

[0066] For example, based on the output frame identifier of the current broadcast and based on the preset output rules, the corresponding virtual frame identifier is determined, the address of the video frame, such as the physical frame identifier, is read from the buffer location corresponding to the virtual frame identifier, and the actual data content of the video frame is read from the physical frame buffer based on the address and then output.

[0067] For example, in one scenario, broadcasting and video output can be synchronous and consistent. For instance, if the current broadcast output frame identifier is 3, the output video frame is the video frame stored at the buffer location corresponding to output frame identifier 3 (such as the location of virtual frame 3). In another scenario, video output and the broadcast sequence of output frame identifiers can be asynchronous. For instance, video frames are written to the buffer based on the output frame identifier, generating a video frame buffer queue. The video output is then delayed for a certain period of time (such as a delay of one frame). That is, if the current broadcast output frame identifier is 3, the output video frame is the video frame stored at the buffer location corresponding to output frame identifier 2 (i.e., the previous output frame identifier) ​​(such as the location of virtual frame 2).

[0068] In some embodiments, based on the output frame identifier, the buffered video frames are output as video at the output frame rate, including: Based on the output frame identifier, the virtual frame identifier of the video frame in the buffer is determined; based on the virtual frame identifier, the video frames are read sequentially and the video is output according to the output frame rate; wherein, the output frame identifier and the virtual frame identifier correspond one-to-one, and the virtual frame identifier is used to indicate the location where the video frame is stored.

[0069] For example, a virtual frame identifier is an index that stores the frame information of a video frame in the buffer. There is a one-to-one correspondence between the output frame identifier and the virtual frame identifier, and the virtual frame identifier is associated with the physical frame identifier. Multiple virtual frame identifiers may be mapped to the same physical frame identifier, such as virtual frame identifiers 1 to 3 being associated with physical frame 2 (in the case of frame interpolation).

[0070] For example, if the currently read output frame identifier is 3, the corresponding virtual frame identifier is determined to be 3, and the frame information at virtual frame 3 is read; based on the mapping relationship, physical frame 2 is determined, and based on the frame information, the address for storing the actual data content is determined; based on physical frame 2 and the address, the actual data content of the video frame is read.

[0071] In some embodiments, based on the output frame identifier, the buffered video frames are output as video at the output frame rate, including: Based on the output frame identifier, video frames from multiple video sources in the buffer are spliced ​​together and output according to the output frame rate.

[0072] like Figure 1 As shown, the received video frames can be video data from multiple video sources, each of which is input based on its corresponding input channel. By using a unified output frame identifier as a timing reference, video frames from different video sources under the same output frame identifier are read simultaneously, spliced ​​together, and then the spliced ​​video is output.

[0073] In some embodiments, based on the output frame identifier, video frames from multiple video sources in the buffer are concatenated, including: Based on the same output frame identifier, determine the same virtual frame identifier for video frames from multiple video sources in the buffer; based on the same virtual frame identifier, read video frames from the buffer corresponding to each video source, and stitch the multiple read video frames into a single frame.

[0074] For example, different video sources can correspond to different input frame rates. Each input channel samples and stores the video frames input by the video source based on the timing indicated by the globally unified output frame identifier. The output channel determines the virtual frame identifier in the buffer corresponding to each video source based on the broadcast output frame identifier. Based on the same virtual frame identifier, it determines the associated mapped physical frame and the address for storing the actual data content, reads the corresponding actual data content, and splices the actual data content corresponding to each video source into a frame to be displayed according to the preset desktop display rules.

[0075] like Figure 8 As shown, the main screen corresponds to one video source, the anchor's camera corresponds to one video source, the guest's camera corresponds to one video source, and the video clips / images / titles played correspond to one video source. Different video sources correspond to different input frame rates. For example, the input frame rate of the first video source corresponding to the main screen may be higher (e.g., greater than the specified output frame rate), the input frame rate of the second video source corresponding to the anchor's camera and the third video source corresponding to the guest's camera may be the same as or less than the specified output frame rate, and the input frame rate of the fourth video source, such as the video clips played, may be less than the specified output frame rate.

[0076] Accordingly, frame dropping is performed on the first video source, no frame dropping or frame interpolation is performed on the second and third video sources (when the frame rates are the same), and frame interpolation is performed on the fourth video source. Based on the globally unified output frame identifier, the virtual frame identifier of each video source is determined, and then the corresponding video data is read, spliced, and output. For example, when the output frame identifier is 3, the virtual frame identifier corresponding to each video source is determined to be 3. For the first video source, virtual frame 3 is associated with its physical frame 4. For the second and third video sources, virtual frame 3 corresponds to its physical frame 3. For the fourth video source, virtual frame 3 corresponds to its physical frame 2. Then, the actual data content of the physical frame 4 of the first video source, the actual data content of the physical frames 3 of the second and third video sources, and the actual data content of the physical frame 2 of the fourth video source are spliced ​​and output as a single frame.

[0077] Accordingly, when the output frame identifier is updated from 3 to 4, the virtual frame identifier corresponding to each video source is determined to be 4. For the first video source, the virtual frame 4 is associated with its physical frame 6. For the second and third video sources, the virtual frame 4 corresponds to its physical frame 4. For the fourth video source, the virtual frame 4 corresponds to its physical frame 2. Then, the actual data content of the physical frame 6 of the first video source, the actual data content of the physical frames 4 of the second and third video sources, and the actual data content of the physical frame 2 of the fourth video source are spliced ​​together and output as a single frame.

[0078] In this embodiment, when the input frame rate is less than the output frame rate, frame interpolation is required. By pre-writing a preset number of M frame information (M is greater than or equal to the ratio of the output to the input frame rate), the output channel can read the information of the same input frame for at most M consecutive output cycles, thus achieving automatic frame copying and interpolation. When the input frame rate equals the output frame rate, no frame rate conversion is needed. The newly written frame information can overwrite the redundant frames in the pre-written frame information (i.e., subsequent writing will overwrite the previously pre-written copies of frame information that have not yet been output), and ultimately each input frame corresponds to only one output frame, ensuring that the frame rate remains unchanged. When the input frame rate is greater than the output frame rate, frame dropping is required. By judging whether the output frame identifier currentID == lastID, overspeeding input frames are accurately identified and discarded, ensuring a stable output frame rate.

[0079] This application embodiment achieves real-time synchronous coordination of multiple input sources with different, unknown, or dynamically changing frame rates through adaptive hybrid frame rate processing. It requires no pre-configuration of input frame rates or frame rate parameters (such as conversion rates). Only the system's output frame rate is set. The input channels dynamically decide whether to copy (pre-write the positions of M virtual frames) or discard video frames based on the read global output frame identifier. This completely eliminates the output instability problems caused by configuration errors or input frame rate changes in traditional solutions, significantly reducing the complexity and difficulty of system deployment and use. Through a globally unified, periodically updated output frame identifier and a mechanism for timed broadcasting, the output channels are guaranteed to output strictly according to the specified frame rate, achieving precise control of the output frame rate. Input channels independently decide and process based on locally sampled output frame identifiers, while output channels only rely on broadcast output frame identifiers for output. The loosely coupled architecture of each channel results in clear processing logic that is easy to implement and extend.

[0080] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0081] Corresponding to the video frame rate control method provided in the above embodiments, such as Figure 9The diagram shown is a schematic representation of the video frame rate control device provided in this application embodiment. For ease of explanation, only the parts related to this application embodiment are shown.

[0082] The video frame rate control device includes: The identification broadcast unit 91 is used to generate a sequence of output frame identifiers and to sequentially broadcast each of the output frame identifiers in the sequence according to the period of the output frame rate. The receiving unit 92 is used to obtain the current output frame identifier after receiving a video frame for the input channel corresponding to the video source; Processing unit 93 is used to determine whether to write the received video frame into the buffer based on the comparison result between the current output frame identifier and the previously acquired output frame identifier; The output unit 94 is used to output the video frames in the buffer according to the output frame rate based on the output frame identifier.

[0083] In one possible implementation, the identification broadcast unit 91 is further configured to determine the sequence of output frame identifiers based on the multiple relationship between the expected maximum output frame rate and the expected minimum input frame rate.

[0084] In one possible implementation, the processing unit 93 is further configured to: if the current output frame identifier is equal to the previously acquired output frame identifier, then not write the received video frame into the buffer; if the current output frame identifier is not equal to the previously acquired output frame identifier, then write the received video frame into the buffer.

[0085] In one possible implementation, the processing unit 93 is further configured to determine the write start position of the buffer based on the current output frame identifier if the current output frame identifier is not equal to the previously acquired output frame identifier; and write the frame information of the received video frame from the write start position and a subsequent preset number of positions; the preset number is less than the number of output frame identifiers in the sequence.

[0086] In one possible implementation, the output unit 94 is further configured to determine, based on the output frame identifier, the virtual frame identifier of the video frame in the buffer; based on the virtual frame identifier, sequentially read the video frames and output the video according to the output frame rate; wherein, the output frame identifier corresponds one-to-one with the virtual frame identifier, and the virtual frame identifier is used to indicate the location where the video frame is stored.

[0087] In one possible implementation, the output unit 94 is further configured to stitch together the video frames from multiple video sources in the buffer based on the output frame identifier, and output the video according to the output frame rate.

[0088] In one possible implementation, the output unit 94 is further configured to determine the same virtual frame identifier in the buffer for the video frames of multiple video sources based on the same output frame identifier; read the video frame from the buffer corresponding to each video source based on the same virtual frame identifier; and stitch the multiple video frames read into a single frame.

[0089] Figure 10 A schematic diagram of the hardware structure of the electronic device 10 is shown.

[0090] like Figure 10 As shown, the electronic device 10 of this embodiment includes: at least one processor 101 ( Figure 10 (Only one is shown in the image) A memory 102 stores a computer program 103 that can run on the processor 101. When the processor 101 executes the computer program 103, it implements the steps in the above method embodiments, for example... Figure 2 S201 to S204 are shown. Alternatively, when the processor 101 executes the computer program 103, it implements the functions of each module / unit in the above-described device embodiments.

[0091] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 10. In other embodiments of this application, the electronic device 10 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0092] The electronic device 10 may include, but is not limited to, a processor 101 and a memory 102. Those skilled in the art will understand that... Figure 10 This is merely an example of electronic device 10 and does not constitute a limitation on electronic device 10. It may include more or fewer components than shown, or combine certain components, or different components. For example, the server may also include input sending devices, network access devices, buses, etc.

[0093] The processor 101 mentioned above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0094] The processor 101 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 101 is a cache memory. This memory can store instructions or data that the processor 101 has just used or that are used repeatedly. If the processor 101 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 101, and thus improves the efficiency of the system.

[0095] In some embodiments, the aforementioned memory 102 may be an internal storage unit of the electronic device 10, such as a hard disk or memory of the electronic device 10. The memory 102 may also be an external storage device of the electronic device 10, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 10. Furthermore, the memory 102 may include both internal and external storage units of the electronic device 10. The memory 102 is used to store operating systems, applications, bootloaders, data, and other programs, such as program code for computer programs. The memory 102 can also be used to temporarily store data that has been sent or will be sent.

[0096] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0097] It should be noted that the structure of the above-mentioned electronic device is only illustrative and may include other physical structures depending on the application scenario. The physical structure of the electronic device is not limited here.

[0098] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0099] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0100] This application provides a computer program product that, when run on a server, enables the server to execute the steps described in the above-described method embodiments.

[0101] If the integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0102] The electronic devices, computer storage media, and computer program products provided in the embodiments of this application are all used to execute the methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects corresponding to the methods provided above, and will not be repeated here.

[0103] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. It should be understood that the above description is merely to help those skilled in the art better understand the embodiments of this application, and is not intended to limit the scope of the embodiments of this application. Based on the examples given above, those skilled in the art can obviously make various equivalent modifications or changes. For example, some steps in the various embodiments of the above detection method may be unnecessary, or new steps may be added. Alternatively, any combination of two or more of the above embodiments may be used. Such modifications, changes, or combinations also fall within the scope of the embodiments of this application.

[0104] It should also be understood that the methods, situations, categories, and classifications of embodiments in this application are for the convenience of description only and should not constitute a special limitation. Various methods, categories, situations, and features in embodiments can be combined without contradiction.

[0105] It should also be understood that, in the various embodiments of this application, unless otherwise specified or in case of logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced by each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0106] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0107] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0108] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0109] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

[0110] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video frame rate control method, characterized in that, The method includes: Generate a sequence of output frame identifiers, and broadcast each of the output frame identifiers in the sequence in a cyclic manner according to the period of the output frame rate; For the input channel corresponding to the video source, after receiving a video frame, obtain the current output frame identifier; Based on the comparison between the current output frame identifier and the previously acquired output frame identifier, determine whether to write the received video frame into the buffer; Based on the output frame identifier, the video frames in the buffer are output as video at the output frame rate.

2. The method according to claim 1, characterized in that, The sequence for generating the output frame identifier includes: The sequence of output frame identifiers is determined based on the multiple relationship between the expected maximum output frame rate and the expected minimum input frame rate.

3. The method according to claim 1, characterized in that, The step of determining whether to write the received video frame to the buffer based on the comparison between the current output frame identifier and the previously acquired output frame identifier includes: If the current output frame identifier is equal to the previously acquired output frame identifier, the received video frame will not be written to the buffer. If the current output frame identifier is not equal to the previously acquired output frame identifier, the received video frame is written to the buffer.

4. The method according to claim 3, characterized in that, The step of writing the received video frame into the buffer if the current output frame identifier is not equal to the previously acquired output frame identifier includes: If the current output frame identifier is not equal to the previously acquired output frame identifier, the write start position of the cache is determined based on the current output frame identifier; The frame information of the received video frame is written from the starting position to be written and the subsequent preset number of positions; the preset number is less than the number of output frame identifiers in the sequence.

5. The method according to claim 1, characterized in that, Based on the output frame identifier, the video frames in the buffer are output at the output frame rate, including: Based on the output frame identifier, determine the virtual frame identifier of the video frame in the buffer; Based on the virtual frame identifier, the video frames are read sequentially, and the video is output according to the output frame rate; The output frame identifier corresponds one-to-one with the virtual frame identifier, and the virtual frame identifier is used to indicate the location where the video frame is stored.

6. The method according to any one of claims 1 to 5, characterized in that, The step of outputting the video frames in the buffer according to the output frame rate based on the output frame identifier includes: Based on the output frame identifier, the video frames from multiple video sources in the buffer are spliced ​​together and output according to the output frame rate.

7. The method according to claim 6, characterized in that, The step of concatenating video frames from multiple video sources in the buffer based on the output frame identifier includes: Based on the same output frame identifier, the video frames from multiple video sources are identified as having the same virtual frame identifier in the buffer; Based on the same virtual frame identifier, the video frames are read from the cache corresponding to each video source, and the read video frames are stitched together into one frame.

8. A video frame rate control device, characterized in that, The device includes: The identification broadcast unit is used to generate a sequence of output frame identifiers and to cyclically broadcast each of the output frame identifiers in the sequence according to the period of the output frame rate. The receiving unit is used to obtain the current output frame identifier after receiving a video frame from the input channel corresponding to the video source. The processing unit is used to determine whether to write the received video frame into the buffer based on the comparison result between the current output frame identifier and the previously acquired output frame identifier; The output unit is used to output the video frames in the buffer according to the output frame rate based on the output frame identifier.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, implements the method of any one of claims 1 to 7.

Images