Qt-based multi-video rendering method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a Qt-based multi-video rendering method and device, computer equipment, a readable storage medium and a program product. The method comprises the steps of determining current visual data in a visual area based on offset information of a content layer in a rolling layer, full-amount layout information corresponding to current data of the content layer and the size of the visual area after a visual area data change event is responded, and displaying the current visual data in the visual area according to a subscript of the current data and a sub-control of the current visual data. And the visual area can be rearranged, so that multi-video rendering is realized.

Description

Technical Field

[0001] This application relates to the field of multi-video rendering technology, and in particular to a Qt-based multi-video rendering method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of real-time communication scenarios such as remote work, online education, and audio / video conferencing, conferencing software needs to render video streams from a large number of participants simultaneously. Real-time performance, stability, and scalability of the graphical interface have become critical requirements in these scenarios. Especially in scenarios where hundreds or tens of thousands of people simultaneously open their videos, the system needs strong control capabilities and an efficient video rendering mechanism to ensure a smooth and stable overall interactive experience.

[0003] Currently, there are no controls with similar functionality and support for complex user interfaces on the cross-platform Qt (Qt Toolkit). Therefore, in traditional techniques, multi-user video rendering is often implemented based on QWidget (Qt Widget, the base class for Qt components / controls) and QVideoWidget (Qt Video Widget, a Qt video component). However, the QWidget-based rendering method cannot be integrated with other Qt elements, has insufficient extensibility, and the QVideoWidget-based rendering method is underperforming when handling large video streams or complex UI (User Interface) scenarios. Summary of the Invention

[0004] Therefore, it is necessary to provide a Qt-based multi-video rendering method, apparatus, computer device, computer-readable storage medium, and computer program product that can achieve cross-platform implementation of the above-mentioned technical problems.

[0005] Firstly, this application provides a Qt-based multi-video rendering method, including:

[0006] In response to visible area data change events, obtain the offset information of the content layer in the scroll layer and the current data of the content layer; among which, visible area data change events include mouse scroll events and data addition / deletion events;

[0007] Retrieve historical data from the content layer prior to responding to events that change data in the visible area;

[0008] Update the index of the current data based on the historical and current data of the content layer;

[0009] Obtain the size of the visible area and the full layout information of the current data;

[0010] The current visible data is determined based on the full layout information, offset information, and the size of the visible area;

[0011] Based on the index of the current data and the child controls of the currently visible data, the visible area is rearranged; where the child controls are Qt-type controls.

[0012] In one embodiment, the step of updating the index of the current data based on the historical data and the current data of the content layer includes:

[0013] Based on historical and current data in the content layer, determine which data to add, remove, and remain unchanged;

[0014] Delete the index of removed data, add the index of added data, and update the index of unmodified data.

[0015] In one embodiment, the step of rearranging the visible area based on the index of the current data and the child controls of the currently visible data includes:

[0016] From the visible area, reclaim the child controls whose visible data has been removed to the buffer area; wherein the buffer area is located before the visible area, and / or, is located after the visible area, and the removed visible data is the data removed from the visible area;

[0017] If the number of child controls in the buffer area is zero, create child controls in the buffer area; during the creation of child controls, determine the coordinate position of the child controls.

[0018] Retrieve child controls from the buffer area to the visible area and bind corresponding visual data; where adding visual data refers to adding data to the visible area.

[0019] The current data are arranged in ascending order of subscript, and the visible area is rearranged based on the coordinate positions of the child controls whose visible data has been added and the coordinate positions of the child controls whose visible data has not been modified.

[0020] In one embodiment, when the visible area data change event is a data addition / deletion event, the steps of deleting the index of the removed data, adding the index of the added data, and updating the index of the unmodified data include:

[0021] If the index of the data to be removed is greater than or equal to the current minimum index value of the visible area, and less than the current maximum index value of the visible area, then delete the index of the data to be removed.

[0022] If the index of the data to be removed is less than the current minimum index value of the visible area, remove the index of the first currently visible data in the visible area;

[0023] Shift the index of the first target data one position forward; where the first target data is the data in the unmodified data whose index is greater than the index of the removed data.

[0024] Remove the index of the last currently visible data in the visible area;

[0025] If the index of the data to be added is less than the current maximum index value of the visible area, remove the index of the last currently visible data in the visible area;

[0026] Shift the index of the second target data one position to the right; where the second target data is the data in the unmodified data whose index is greater than the index of the added data;

[0027] Remove the index of the first currently visible data in the visible area.

[0028] In one embodiment, the above-described Qt-based multi-video rendering method further includes:

[0029] The first buffered data entering the buffer area is prepared; the data preparation includes data notification and network requests for the first buffered data.

[0030] The first visible data entering the visible area is rendered; the data rendering process includes data notification, video retrieval, video decoding, and video rendering of the first visible data.

[0031] The data stopping process is performed on the second visible data that leaves the visible area; the data stopping process includes data notification of the second visible data, stopping video fetching, stopping video decoding, and stopping video rendering;

[0032] The second buffered data that has left the buffer area is reclaimed; the data reclamation process includes data notification of the first buffered data and network shutdown.

[0033] In one embodiment, prior to the steps of obtaining the offset information of the content layer in the scrolling layer and the current data of the content layer, the above-described Qt-based multi-video rendering method further includes:

[0034] Based on the current data and layout type of the content layer, a logical coordinate system is generated, and the initial layout of the visible area is performed according to the logical coordinate system; where the logical coordinate system is the layout mapping rule for each current data.

[0035] Secondly, this application also provides a Qt-based multi-video rendering device, comprising:

[0036] The response module is used to respond to visible area data change events, obtain the offset information of the content layer in the scroll layer and the current data of the content layer; among which, visible area data change events include mouse scroll events and data addition / deletion events;

[0037] The first acquisition module is used to acquire historical data of the content layer before responding to events that change data in the visible area.

[0038] The index update module is used to update the index of the current data based on the historical data and the current data in the content layer;

[0039] The second acquisition module is used to acquire the size of the visible area and the full layout information of the current data.

[0040] The current visible data determination module is used to determine the current visible data based on the full layout information, offset information, and the size of the visible area;

[0041] The Relayout module is used to relayout the visible area based on the index of the current data and the child controls of the currently visible data; where the child controls are Qt-type controls.

[0042] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any of the steps in the Qt-based multi-video rendering method described above.

[0043] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the steps in the Qt-based multi-video rendering method described above.

[0044] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the steps in the Qt-based multi-video rendering method described above.

[0045] The aforementioned Qt-based multi-video rendering method, apparatus, computer device, computer-readable storage medium, and computer program product, wherein the Qt-based multi-video rendering method includes: after responding to a visible area data change event, determining the current visible data in the visible area based on the content layer's offset information in the scroll layer, the full layout information corresponding to the current data in the content layer, and the size of the visible area; and rearranging the visible area according to the index of the current data and the child controls of the current visible data, thereby realizing the rendering of multiple videos. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is an application environment diagram of a Qt-based multi-video rendering method in one embodiment;

[0048] Figure 2 This is a flowchart illustrating a Qt-based multi-video rendering method in one embodiment;

[0049] Figure 3 This is a schematic diagram of the display page corresponding to the Qt-based multi-video rendering method in one embodiment;

[0050] Figure 4 This is a schematic diagram illustrating the scrolling of the display page corresponding to a Qt-based multi-video rendering method in one embodiment;

[0051] Figure 5 This is a logical coordinate correspondence diagram of a Qt-based multi-video rendering method in one embodiment;

[0052] Figure 6 This is a block diagram of a Qt-based multi-video rendering method in one embodiment.

[0053] Figure 7 This is a structural block diagram of a Qt-based multi-video rendering device in one embodiment;

[0054] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] It should be noted that the terms "first," "second," etc., used in this application may be used to describe various elements or data, but these elements or data are not limited by these terms. These terms are only used to distinguish the first element or data from the second element or data. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0057] Traditional multi-video rendering methods require creating independent controls for each video stream, leading to a linear increase in system resource consumption with the number of videos and resulting in significant resource waste. The control architecture of traditional multi-video rendering methods struggles to support complex interfaces, dynamic video additions and removals, layer blending, and other graphics processing requirements, exhibiting poor scalability. Furthermore, traditional multi-video rendering methods suffer from frequent black screens, increased latency, and severe stuttering, reducing overall interface rendering efficiency. The management methods of system APIs (Application Programming Interfaces) and GPUs (Graphics Processing Units) differ significantly across platforms, necessitating complex adaptation of traditional multi-video rendering methods to multiple platforms.

[0058] The Qt-based multi-video rendering method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. In response to a visible area data change event from terminal 102, server 104 obtains the offset information of the content layer in the scroll layer and the current data of the content layer; after obtaining the historical data of the content layer before responding to the visible area data change event, it updates the index of the current data based on the historical data and the current data; then it obtains the size of the visible area and the full layout information of the current data; based on the full layout information, offset information, and the size of the visible area, it determines the current visible data; and based on the index of the current data and the sub-controls of the current visible data, it performs a rearrangement of the visible area displayed on the terminal 102's page. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart TVs, smart in-vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Headset devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0059] In one exemplary embodiment, such as Figure 2 As shown, a multi-video rendering method based on Qt is provided, which is then applied to... Figure 1 Taking server 104 as an example, the explanation includes:

[0060] S202, in response to the visible area data change event, obtain the offset information of the content layer in the scroll layer and the current data of the content layer; wherein, the visible area data change event includes mouse scroll event and data addition / deletion event.

[0061] like Figure 3 As shown, the scroll layer is the direct parent container of the content layer, and the content layer is embedded inside the scroll layer as a child element. The size of the scroll layer is fixed. When the size of the content layer exceeds the size of the scroll layer, the scrolling mechanism is triggered, causing a scrollbar to appear in the scroll layer. The size of the scrollbar is determined based on the size of the content layer, and the real-time position of the scrollbar is determined based on the offset information of the content layer within the scroll layer.

[0062] After a mouse scroll event is triggered by the mouse wheel or a touch swipe, the offset of the content layer within the scroll layer changes. Initially, the offset of the content layer within the scroll layer is 0.

[0063] Furthermore, after a window resize event triggered by a user zoom operation, the position and size of the content layer within the scroll layer's offset information will change. This position and size refers to the position and size of the content layer within the scroll layer.

[0064] S204, retrieve historical data of the content layer prior to the response to a data change event in the visible area.

[0065] like Figure 4 As shown, when the visible area data change event is a mouse scroll event, the historical data of the content layer is consistent with the current data. However, because the content layer has scrolled, the historical visible data and the current visible data are inconsistent, as are the historical buffered data and the current buffered data. Specifically, the historical visible data refers to the data in the visible area before responding to the visible area data change event; the current visible data refers to the data in the visible area after responding to the visible area data change event; the historical buffered data refers to the data in the buffered area before responding to the visible area data change event; and the current buffered data refers to the data in the buffered area after responding to the visible area data change event.

[0066] When the visible area data change event is a data addition / deletion event, the historical data in the content layer is inconsistent with the current data because data has been added and / or deleted. Therefore, historical visible data and current visible data are inconsistent, as are historical buffered data and current buffered data.

[0067] S206, Update the index of the current data based on the historical data and current data of the content layer.

[0068] like Figure 5As shown, each current data point corresponds to an index. For example, the index of the first current data point is 0, the index of the second current data point is 1, the index of the third current data point is 2, and so on, until the index of the last current data point is obtained.

[0069] S208: Obtain the size of the visible area and the full layout information of the current data. The full layout information includes the position coordinates of each current data point.

[0070] S210: Determine the current visible data based on the full layout information, offset information, and the size of the visible area.

[0071] S212, based on the index of the current data and the child controls of the currently visible data, rearrange the visible area; where the child controls are Qt-type controls.

[0072] After the layout is rearranged, the global layout view is updated accordingly to achieve multi-video rendering. Child controls are used to subscribe to and render videos. These child controls can be either `QGraphicsWidget` (Qt graphics widget) or `QGraphicsVideoItem` (Qt graphics video item). `QGraphicsWidget` is a high-level control class under the `QGraphics View` framework, inheriting from `QGraphicsItem` (Qt graphics item). It supports embedding controls in `QGraphicsScene` (Qt graphics scene) and is often used in games and other scenarios to achieve flexible 2D graphics management. Compared to `QWidget`, `QGraphicsWidget` is more suitable for complex layouts and custom drawing, and is suitable for high-concurrency graphical interface scenarios. `QGraphicsVideoItem` is a graphics item class in the Qt multimedia module used for video playback, inheriting from `QGraphicsObject` (Qt graphics object), and can be embedded in `QGraphicsScene`. `QGraphicsVideoItem` is used to render the decoded video content as a graphics item into Qt's graphics view system.

[0073] Qt is an open-source, cross-platform C++ (The C++ Programming Language) application development framework widely used for graphical user interface, multimedia, and network application development. It supports major operating systems such as Windows, Linux, macOS, Android, and iOS. Therefore, implementing a Qt-based multi-video rendering method using Qt class controls enables cross-platform applications. Cross-platform means that the Qt-based multi-video rendering method and the system running it can run on multiple operating system platforms while maintaining core functionality consistency. Thus, leveraging Qt's cross-platform characteristics, the same video rendering logic can be deployed on platforms such as Windows, macOS, and Linux, exhibiting good portability and scalability.

[0074] The content layer is used to obtain the corresponding physical range based on the actual size of the current data content, and in conjunction with the view window, calculates and determines the visible area and the corresponding buffer area. The content layer is also used to add child controls and dynamically adjust their position coordinates and size. This Qt-based multi-video rendering method can be determined based on QGraphicsScene, which supports child controls for subscribing to and rendering videos, as well as other custom controls, offering good extensibility.

[0075] The Qt-based multi-video rendering method's controls mainly consist of three parts: a scroll layer, a content layer, and child controls (i.e., renderers). The bottom canvas, QGraphicsScene, allows you to add controls used by this Qt-based multi-video rendering method, and can also be used in conjunction with other native controls.

[0076] You can create QGraphicsView (Q Graphics View) and QGraphicsScene. QGraphicsView can serve as the visualization window of QGraphicsScene, rendering the graphic items within it onto the screen. You can add child controls to QGraphicsScene to listen for size change events and dynamically adjust the view size of the child controls.

[0077] The Qt-based multi-video rendering method described above, after responding to a data change event in the visible area, can determine the current visible data in the visible area based on the offset information of the content layer in the scroll layer, the full layout information corresponding to the current data in the content layer, and the size of the visible area. Based on the index of the current data and the child controls of the current visible data, the visible area can be rearranged to achieve multi-video rendering.

[0078] The Qt-based multi-video rendering method is suitable for complex scenarios in video conferencing software where multiple users simultaneously open video calls, especially for real-time video calls and collaborations involving thousands or even tens of thousands of people. By constructing lightweight video sub-controls, high-performance, low-latency, and low-resource-consumption multi-video rendering can be achieved. Furthermore, when QGraphicsWidget works in conjunction with the underlying video decoder, it ensures that systems using this Qt-based multi-video rendering method maintain a smooth user experience even with extremely high concurrency video windows. The underlying video decoder can be FFmpeg (Fast Forward Moving Picture Experts Group) or a system hardware decoder. Moreover, the Qt-based multi-video rendering method supports cross-platform multi-video rendering and is suitable for real-time audio and video communication scenarios on PCs (Personal Computers) where a large number of users simultaneously open video calls.

[0079] Therefore, this Qt-based multi-video rendering method not only provides efficient video scheduling and rendering capabilities, but also makes full use of Qt's graphical view architecture, possessing good scalability and flexibility. It can adapt to complex graphical interface requirements such as multi-user dynamic layout, frequent changes in video windows, layer overlay, animation effects, and interactive annotations, and is suitable for various multimedia interactive scenarios such as remote conferencing, online education, and security monitoring, and is especially applicable in the embedded field.

[0080] In an exemplary embodiment, the step of updating the index of the current data based on the historical data and the current data of the content layer includes:

[0081] Based on historical and current data in the content layer, determine which data to add, remove, and remain unchanged.

[0082] Delete the index of removed data, add the index of added data, and update the index of unmodified data.

[0083] If new data is added, then the new data is considered added data. If there is unmodified data in the current data, then the index of the added data and the index of the current data corresponding to the updated unmodified data are consecutive.

[0084] If old data is removed, then that old data is removed data. If there is unmodified data in the current data, then delete the index of the removed data and update the index of the unmodified data so that the indices of the current data corresponding to the unmodified data are continuous.

[0085] During data initialization, each current data item generates full layout information to determine the current buffered data in the buffer area and the current visible data in the visible area on the initial page. Upon responding to a visible area data change event, both the current buffered data in the buffer area and the current visible data in the visible area change. For example... Figure 4 As shown, before responding to the visible area data change event, the current buffered data in the buffer area is the data corresponding to indices 3, 4, and 5, and the current visible data in the visible area is the data corresponding to indices 6, 7, 8, 9, 10, and 11. After responding to the visible area data change event, the data corresponding to indices 3, 4, and 5 become historical buffered data, the data corresponding to indices 6, 7, and 8 become historical buffered data, the data corresponding to indices 6, 7, 8, 9, 10, and 11 become historical visible data, and the data corresponding to indices 9, 10, 11, 12, 13, and 14 become historical visible data.

[0086] In an exemplary embodiment, the step of rearranging the visible area based on the index of the current data and the child controls of the currently visible data includes:

[0087] From the visible area, reclaim the child controls whose visible data has been removed to the buffer area; wherein the buffer area is located before the visible area, and / or, is located after the visible area, and the removed visible data is the data removed from the visible area.

[0088] Determine the buffer area based on its size.

[0089] like Figure 3 and Figure 4 As shown, when the offset of the content layer in the scroll layer is 0, the buffer area is located after the visible area; when the offset of the content layer in the scroll layer is greater than 0 and less than the maximum offset, the buffer area is located before and after the visible area; when the offset of the content layer in the scroll layer is equal to the maximum offset, the buffer area is located before the visible area.

[0090] Each currently visible data area corresponds to a child control. Data in the buffer area and other areas does not have corresponding child controls.

[0091] If the number of child controls in the buffer area is zero, create child controls in the buffer area; during the creation of child controls, determine the coordinate position of the child controls.

[0092] Remove the child control from the buffer area to the visible area and bind the corresponding added visual data; where the added visual data is the data added to the visible area.

[0093] If the number of child controls in the buffer area is not zero, there is no need to create new child controls in the buffer area; these child controls can be used directly. Therefore, after removing a child control containing visual data, the same number of child controls will be added to the buffer area to await the binding requirements of child controls containing new visual data. By reusing child controls, system resources for this Qt-based multi-video rendering method can be saved.

[0094] Child controls with the same visual data are not recycled. For example, such as... Figure 4 As shown, before responding to the visible area data change event, the data corresponding to the same visible data 9, 10, and 11 is removed, the data corresponding to visible data 6, 7, and 8 is removed, and the data corresponding to visible data 12, 13, and 14 is added. Based on this, the child controls corresponding to the data 6, 7, and 8 are retrieved from the visible area to the buffer area, and the child controls are extracted from the buffer area to the visible area to match the data corresponding to 12, 13, and 14. The child controls corresponding to the data 6, 7, and 8 are not modified.

[0095] The current data are arranged in ascending order of subscript, and the visible area is rearranged based on the coordinate positions of the child controls whose visible data has been added and the coordinate positions of the child controls whose visible data has not been modified.

[0096] In an exemplary embodiment, when the visible area data change event is a data addition / deletion event, the steps of deleting the index of the removed data, adding the index of the added data, and updating the index of the unmodified data include:

[0097] If the index of the data to be removed is greater than or equal to the current minimum index value of the visible area, and less than the current maximum index value of the visible area, then delete the index of the data to be removed.

[0098] If the index of the data to be removed is less than the current minimum index value of the visible area, remove the index of the first currently visible data in the visible area.

[0099] Shift the index of the first target data one position forward; where the first target data is the data in the unmodified data whose index is greater than the index of the removed data.

[0100] Remove the index of the last currently visible data in the visible area.

[0101] If the index of the removed data is greater than the current maximum index value of the visible area, there is no impact on the data in the visible area, so the index is not modified.

[0102] If the index of the data to be removed is greater than or equal to the current minimum index value of the visible area, and less than the current maximum index value of the visible area, then the data to be removed is the visible data within the visible area. This visible data had a corresponding child control before removal, but no corresponding child control after removal. Since the child controls in the visible area correspond one-to-one with the indices of the data to be removed, it is necessary to delete the index of the removed data and return the corresponding child control to the buffer area. At this point, the indices of the data in the visible area are not consecutive. Therefore, starting from the index of the next data to be removed, the indices of the first target data are shifted forward by one position to form consecutive indices. After modifying the index of the first target data, the last currently visible data in the visible area is not bound to a corresponding child control. Therefore, the index of the last currently visible data without a binding relationship is removed so that each currently visible data in the visible area corresponds to a child control, thus enabling the rendering of the corresponding video.

[0103] When the index of the removed data is less than the current minimum index value of the visible area, the indices of the data in the visible area and the areas preceding it are not continuous. Therefore, the index of the first currently visible data in the visible area is removed, and the index of the first target data is shifted forward one position to make the indices of the data in the visible area and the areas preceding it continuous. In this case, it is equivalent to shifting the data in the visible area forward one position. Thus, before the index shift, the first currently visible data in the visible area is moved out of the visible area after the shift. Therefore, after the index shift, the index of this first currently visible data needs to be removed, and the corresponding child control needs to be reclaimed. After the index shift, the last currently visible data in the visible area is not bound to a corresponding child control. Therefore, the index of the last currently visible data without a binding relationship is removed so that each currently visible data in the visible area corresponds to a child control, enabling the rendering of the corresponding video.

[0104] If the index of the data to be added is less than the current maximum index value of the visible area, remove the index of the last currently visible data in the visible area.

[0105] Shift the index of the second target data one position to the right; where the second target data is the data whose index in the unmodified data is greater than the index of the added data.

[0106] Remove the index of the first currently visible data in the visible area.

[0107] If the index of the added data is greater than the current maximum index value of the visible area, it will have no effect on the data in the visible area, so the index will not be modified.

[0108] When the index of the added data is less than the current maximum index value of the visible area, it affects the data in the visible area. Specifically, when the index of the added data is less than the current maximum index value of the visible area, the visible area and the area before it need to reserve a data position for the added data. Therefore, it is necessary to remove the index of the last currently visible data in the visible area, and correspondingly reclaim the child control of the last currently visible data, while increasing the index of the added data. At this time, the indices of the data in the visible area and the area before it are not continuous. The index of the second target data is shifted one position to the right to form continuous indices. In this case, it is equivalent to shifting all the data in the visible area one position to the right. That is to say, the first currently visible data in the visible area is not bound to a corresponding child control. Therefore, the index of the last currently visible data without a binding relationship is removed so that each currently visible data in the visible area corresponds to a child control, so as to realize the rendering of the corresponding video.

[0109] Therefore, in a video conference, video rendering after a user leaves can be achieved based on the removal steps described above, and video rendering after a user joins can be achieved based on the addition steps described above. By comparing the indices of the old and new data, a new index group and the corresponding bound child controls can be obtained, thereby achieving dynamic video layout.

[0110] In an exemplary embodiment, the Qt-based multi-video rendering method described above further includes:

[0111] The process involves preparing the first batch of buffered data entering the buffer area. This preparation includes data notification and network requests for the first batch of buffered data. The first batch of buffered data refers to the current data newly entering the buffer area.

[0112] The system performs data rendering on the first visible data entering the visible area; this rendering includes data notification of the first visible data, video retrieval, video decoding, and video rendering. The first visible data refers to the current data newly entering the visible area.

[0113] The process involves stopping the data processing of second visible data that is about to leave the visible area. This stopping includes notifying the user of the second visible data, stopping video fetching, stopping video decoding, and stopping video rendering. The second visible data refers to the current data that is about to leave the visible area.

[0114] The second buffered data leaving the buffer area undergoes data reclamation; this reclamation includes notifying the first buffered data and disabling network access. The second buffered data is the current data about to leave the buffer area.

[0115] The purpose of the data notification is to load and close videos. A fixed number of videos are displayed on the user interface at most. When scrolling, child controls need to be reused. By loading and closing video resources, a fixed number of child controls can be rendered to render hundreds or thousands of videos at the same time.

[0116] The first buffered data entering the buffer undergoes data preparation work before video fetching. This preparation work for the first buffered data does not include video fetching; that is, no video rendering occurs within the buffer to conserve computing resources. The data preparation work includes all preparatory tasks other than video fetching to ensure sufficient preparation for video rendering.

[0117] The second buffered data that has left the buffer area is recycled in preparation for the next connection.

[0118] In an exemplary embodiment, prior to the steps of obtaining the offset information of the content layer in the scroll layer and the current data of the content layer, the method further includes:

[0119] Based on the current data and layout type of the content layer, a logical coordinate system is generated, and the initial layout of the visible area is performed according to the logical coordinate system; where the logical coordinate system is the layout mapping rule for each current data.

[0120] like Figure 5 As shown, each current data point is mapped to a unique area coordinate system. During this process, no corresponding child controls are bound to each current data point; the logical coordinate system only records the layout and mapping rules. Only after the data enters the visible area does the currently visible data correspond to and match a child control.

[0121] like Figure 6 As shown, in a system using Qt-based multi-video rendering methods, the system can be divided into a data module, a layout module, a control module, a cache module, and a notification module.

[0122] The data module manages the current data. Taking a multi-user video conference as an example, the data module stores detailed information such as each user's name, role, and video data stream. A user joining the conference adds data to the current data, and a user leaving the conference removes data. The data module can also update the index of the current data based on historical data from the content layer and sort the data. Furthermore, users can adjust their positions, which corresponds to adjusting the position of the current data. The data source for the current data is separate from the actual child controls. A one-to-one correspondence between child controls only exists when the current data is currently visible. When the current data leaves the visible area, the binding relationship of the corresponding child control is released, and the child control returns to its original position.

[0123] The layout module manages the position coordinates and view size of each current data element. The layout module allows you to set a default layout type or customize the layout to your needs. Layouts include the distribution style of multiple current data elements and the scrolling direction of content layers.

[0124] The control module is the core module, used for state control, position calculation, and visible area management of each sub-control in the system. Its core functions include: calculating the currently visible data and other hidden data in real time based on the visible range, scroll position, and zoom level of the corresponding control in the content layer; creating a position area index structure to accelerate the calculation of the intersection of the currently visible data and the visible area; and interacting with the caching module to create an appropriate number of sub-controls in the cache area as needed, and dynamically extracting or reclaiming sub-controls from the cache area based on the current information from the data and layout modules.

[0125] The caching module is used to cache and create child controls. When the number of child controls in the buffer area is zero, it creates and adds child controls to the buffer area.

[0126] The notification module is divided into two types: buffered data notification and visible data notification. Buffered data notification is used to negotiate media signaling in advance. Buffered data notification does not fetch data but is used to accelerate rendering. Visible data notification is used to fetch video data and perform the actual rendering. Once the visible data leaves the visible area, the fetching of video data must also be stopped accordingly.

[0127] Specifically, a logical coordinate system is generated based on the total amount of data in the data module and the current layout type in the layout module, and then passed to the control module.

[0128] Therefore, the Qt-based multi-video rendering method described above uses lightweight QGraphicsWidget objects to replace traditional QWidget or QVideoWidget. Each video's child control is a QGraphicsWidget or QGraphicsVideoItem. These child controls can directly participate in the QGraphics View's graphics rendering system, achieving more efficient graphics resource management and layer control.

[0129] The aforementioned Qt-based multi-video rendering method supports dynamic creation and object pooling of video rendering sub-controls. This lifecycle management and resource reuse avoids system jitter and memory overhead caused by frequent creation or destruction of sub-controls. Compared to creating each control individually, these sub-controls are drawn by the developer, significantly reducing system resource consumption. Through efficient control reuse, the number of users can be increased from a maximum of 500 to tens of thousands simultaneously opening videos.

[0130] The system, equipped with a Qt-based multi-video rendering method, has changed its overall response speed from linear to constant, improving frame rate stability and significantly reducing issues such as black screens and stuttering. This makes it particularly suitable for dynamic scenes involving frequent scrolling, fast scrolling, and entering or exiting content. Fast scrolling includes dragging the scrollbar for each video.

[0131] A preloading strategy is introduced to preload the child controls in the buffer area, thereby establishing connections in advance. The video is then pulled when the current data becomes the current visible data, ensuring a smooth and seamless video rendering process.

[0132] The Qt-based multi-video rendering method also supports features such as free arrangement of child controls on QGraphicsWidget, layer overlay, batch selection, and real-time annotation to enhance the user's interactive experience.

[0133] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0134] Based on the same inventive concept, this application also provides a Qt-based multi-video rendering apparatus for implementing the Qt-based multi-video rendering method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more Qt-based multi-video rendering apparatus embodiments provided below can be found in the limitations of the Qt-based multi-video rendering method described above, and will not be repeated here.

[0135] In one exemplary embodiment, such as Figure 7 As shown, a Qt-based multi-video rendering device 700 is provided, including: a response module 702, a first acquisition module 704, an index update module 706, a second acquisition module 708, a current visible data determination module 710, and a rearrangement module 712, wherein:

[0136] The response module 702 is used to respond to visible area data change events, obtain the offset information of the content layer in the scroll layer and the current data of the content layer; wherein, visible area data change events include mouse scroll events and data addition / deletion events.

[0137] The first acquisition module 704 is used to acquire historical data of the content layer before responding to data change events in the visible area.

[0138] The subscript update module 706 is used to update the subscript of the current data based on the historical data and the current data of the content layer.

[0139] The second acquisition module 708 is used to acquire the size of the visible area and the full layout information of the current data.

[0140] The current visible data determination module 710 is used to determine the current visible data based on the full layout information, offset information, and the size of the visible area.

[0141] The rearrangement module 712 is used to rearrange the visible area based on the index of the current data and the child controls of the currently visible data; where the child controls are Qt-type controls.

[0142] In an exemplary embodiment, the subscript update module 706 includes a data determination module and a subscript update submodule.

[0143] The data determination module is used to determine whether to add, remove, or leave unchanged data based on historical and current data from the content layer.

[0144] The index update submodule is used to delete the index of removed data, add the index of added data, and update the index of unmodified data.

[0145] In an exemplary embodiment, the above-mentioned rearrangement module 712 includes: a sub-control recycling module, a sub-control creation module, a sub-control binding module, and a rearrangement sub-module.

[0146] The child control recycling module is used to recycle child controls from the visible area where visible data has been removed to a buffer area; wherein the buffer area is located before the visible area, and / or after the visible area, and the removed visible data is the data removed from the visible area.

[0147] The child control creation module is used to create child controls in the buffer area when the number of child controls in the buffer area is zero; during the creation process of the child control, the coordinate position of the child control is determined.

[0148] The child control binding module is used to retrieve child controls from the buffer area to the visible area and bind corresponding visual data; whereby adding visual data refers to adding data in the visible area.

[0149] The Relayout submodule is used to rearrange the visible area by sorting the current data in ascending order of subscripts and by considering the coordinate positions of the child controls whose visible data has been added and the coordinate positions of the child controls whose visible data has not been modified.

[0150] In an exemplary embodiment, the above-mentioned subscript update submodule includes: a first subscript removal module, a second subscript removal module, a subscript forward shift module, a third subscript removal module, a fourth subscript removal module, a subscript backward shift module, and a fifth subscript removal module.

[0151] The first index removal module is used to delete the index of the data to be removed when the index of the data to be removed is greater than or equal to the current minimum index value of the visible area and less than the current maximum index value of the visible area.

[0152] The second index removal module is used to remove the index of the first currently visible data in the visible area when the index of the data to be removed is less than the current minimum index value of the visible area.

[0153] The index forward shift module is used to shift the index of the first target data forward by one position; wherein, the first target data is the data in the unmodified data whose index is greater than the index of the removed data.

[0154] The third subscript removal module is used to remove the subscript of the last currently visible data in the visible area.

[0155] The fourth subscript removal module is used to remove the subscript of the last currently visible data in the visible area when the subscript of the data to be added is less than the current maximum subscript value of the visible area.

[0156] The subscript shift module is used to shift the subscript of the second target data one position to the right; wherein, the second target data is the data in the unmodified data whose subscript is greater than the subscript of the added data.

[0157] The fifth subscript removal module is used to remove the subscript of the first currently visible data in the visible area.

[0158] In an exemplary embodiment, the Qt-based multi-video rendering apparatus 700 further includes: a data preparation module, a data rendering module, a data stopping module, and a data recycling module.

[0159] The data preparation module is used to prepare the first buffered data entering the buffer area; the data preparation includes data notification and network requests for the first buffered data.

[0160] The data rendering module is used to perform data rendering on the first visible data entering the visible area; the data rendering work includes data notification of the first visible data, video retrieval, video decoding, and video rendering.

[0161] The data stop module is used to stop the data operation of second visual data that has left the visible area; the data stop operation includes data notification of the second visual data, video stop pulling, video stop decoding, and video stop rendering.

[0162] The data reclamation module is used to reclaim second buffered data that has left the buffer area; the data reclamation process includes data notification of the first buffered data and network shutdown.

[0163] In an exemplary embodiment, the Qt-based multi-video rendering apparatus 700 further includes a logical coordinate system generation module.

[0164] The logical coordinate system generation module is used to generate a logical coordinate system based on the current data and current layout type of the content layer, and to perform the initial layout of the visible area based on the logical coordinate system; wherein, the logical coordinate system is the layout mapping rule for each current data.

[0165] The modules in the aforementioned Qt-based multi-video rendering device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0166] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores the data required for a Qt-based multi-video rendering method. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a Qt-based multi-video rendering method.

[0167] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0168] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above for the Qt-based multi-video rendering method.

[0169] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above for the Qt-based multi-video rendering method.

[0170] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of any of the methods described above in the Qt-based multi-video rendering method.

[0171] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0172] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0173] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0174] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multi-video rendering method based on Qt, characterized in that, The method includes: In response to a visible area data change event, the offset information of the content layer in the scroll layer and the current data of the content layer are obtained; wherein, the visible area data change event includes mouse scroll event and data addition / deletion event; Obtain historical data of the content layer prior to responding to the visible area data change event; Update the index of the current data based on the historical data of the content layer and the current data; Obtain the size of the visible area and the full layout information of the current data; The current visible data is determined based on the full layout information, the offset information, and the size of the visible area; Based on the index of the current data and the child controls of the current visible data, the visible area is rearranged; wherein, the child controls are Qt-type controls.

2. The method according to claim 1, characterized in that, The step of updating the index of the current data based on the historical data of the content layer and the current data includes: Based on the historical data and current data of the content layer, determine the data to be added, removed, and unchanged; Delete the index of the removed data, add the index of the added data, and update the index of the unmodified data.

3. The method according to claim 2, characterized in that, The step of rearranging the visible area based on the index of the current data and the child controls of the current visible data includes: From the visible area, the child controls whose visible data has been removed are moved to a buffer area; wherein the buffer area is located before the visible area and / or after the visible area, and the removed visible data is the data removed from the visible area; When the number of child controls in the buffer area is zero, the child control is created in the buffer area; wherein, during the creation of the child control, the coordinate position of the child control is determined; The child control is retrieved from the buffer area and placed into the visible area, and corresponding visual data is added and bound; wherein, the added visual data is the data added to the visible area; The current data are arranged in ascending order of subscript, and the visible area is rearranged according to the coordinate positions of the sub-controls whose visible data has been added and the coordinate positions of the sub-controls whose visible data has not been modified.

4. The method according to claim 2, characterized in that, When the visible area data change event is the data addition / deletion event, the steps of deleting the index of the removed data, increasing the index of the added data, and updating the index of the unmodified data include: If the index of the data to be removed is greater than or equal to the current minimum index value of the visible area, and less than the current maximum index value of the visible area, then delete the index of the data to be removed. If the index of the data to be removed is less than the current minimum index value of the visible area, remove the index of the first currently visible data in the visible area; Shift the index of the first target data forward by one position; wherein, the first target data is the data in the unmodified data whose index is greater than the index of the removed data; Remove the index of the last currently visible data in the visible area; If the index of the data to be added is less than the current maximum index value of the visible area, remove the index of the last currently visible data in the visible area; Shift the index of the second target data one position to the right; wherein, the second target data is the data in the unmodified data whose index is greater than the index of the added data; Remove the index of the first currently visible data in the visible area.

5. The method according to any one of claims 1 to 4, characterized in that, Also includes: The first buffered data entering the buffer area is prepared; wherein, the data preparation includes data notification and network request for the first buffered data; The first visible data entering the visible area is subjected to data rendering; wherein, the data rendering includes data notification, video retrieval, video decoding, and video rendering of the first visible data; The data is stopped when the second visible data leaves the visible area; wherein, the data stopping includes data notification of the second visible data, stopping video fetching, stopping video decoding, and stopping video rendering; The second buffered data that leaves the buffer area is reclaimed; wherein, the data reclamation includes data notification of the first buffered data and network shutdown.

6. The method according to claim 1, characterized in that, Before the step of obtaining the offset information of the content layer in the scroll layer and the current data of the content layer, the method further includes: Based on the current data and current layout type of the content layer, a logical coordinate system is generated, and the initial layout of the visible area is performed according to the logical coordinate system; wherein, the logical coordinate system is the layout mapping rule for each of the current data.

7. A Qt-based multi-video rendering device, characterized in that, The device includes: The response module is used to respond to visible area data change events, obtain the offset information of the content layer in the scroll layer and the current data of the content layer; wherein, the visible area data change events include mouse scroll events and data addition / deletion events; The first acquisition module is used to acquire historical data of the content layer prior to responding to the visible area data change event; The index update module is used to update the index of the current data based on the historical data of the content layer and the current data; The second acquisition module is used to acquire the size of the visible area and the full layout information of the current data; The current visible data determination module is used to determine the current visible data based on the full layout information, the offset information, and the size of the visible area; The rearrangement module is used to rearrange the visible area based on the index of the current data and the child controls of the current visible data; wherein the child controls are Qt-type controls.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images