RREF-based Video Layout Method and System

Abstract

The present invention relates to a video layout method and system based on RREF, belonging to the field of computer technology. The method includes: structuring the data of the picture layout to generate a layout tree; defining the DIS vector to be solved, generating a constraint matrix according to its constraint relationship, performing RREF processing on it, and solving to obtain the DIS vector that meets the requirements; searching the layout tree, determining the parameters of each DIS node according to the solved DIS vector, and substituting the parameters into the layout tree to output the json file of the layout. This method can achieve more efficient automatic generation of the sizes and positions of different picture layouts, can reduce the workload of product designers and R & D and testing personnel while improving the layout generation efficiency, and realize user-defined layouts.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and particularly relates to a video layout method and system based on RREF. Background Art

[0002] In the past design and implementation of video layouts, designers would design sketches and then hand them over to R & D personnel for implementation. In fixed video scenarios, the workload was not large. However, in current mainstream video conferencing software on the market, to meet user needs, various layouts are provided for users to choose from. Taking Tianyi Cloud Conference as an example, in advanced conference control, Tianyi Cloud Conference provides dozens of conference layouts for users, including: Picture layout 1 + N mode: 1 large picture + multiple small pictures; Picture layout 2 + N mode: 2 large pictures + multiple small pictures; Picture layout 3 + N mode: 3 large pictures + multiple small pictures; Picture layout equal division mode.

[0003] Currently, for such a large number of layouts, the main method is still to manually draw and then write layout codes one by one. The disadvantages of this method are obvious: firstly, the workload is too large when there are many layout modes; secondly, there are many specific size and position relationships in the layout, and these relationships may not be unique, which leads to errors between the implemented layout and the design drawing. Summary of the Invention

[0004] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art, and provide a video layout method and system based on RREF, which can be used in fields such as remote conferences, video live broadcasts, and video monitoring. Through this method, more efficient automatic generation of different picture layout sizes and positions can be achieved, which can reduce the workload of product designers and R & D and testing personnel while improving the layout generation efficiency and realizing user-defined layouts.

[0005] According to one aspect of the present invention, the present invention provides a video layout method based on RREF, and the method includes the following steps:

[0006] S1: Structurize the data of the picture layout to generate a layout tree;

[0007] S2: Define the DIS vector to be solved, generate a constraint matrix according to its constraint relationship, perform RREF processing on it, and solve to obtain the DIS vector that meets the requirements;

[0008] S3: Search the layout tree, determine the parameters of each DIS node according to the solved DIS vector, and substitute the parameters into the layout tree to output the json file of the layout.

[0009] Preferably, the generating of the layout tree includes: generating a layout tree according to a layout tree generation function, where the layout tree includes four types of nodes: ROOT, ROW, COL, and DIS; ROOT is the outermost canvas node, and the attributes of the canvas need to be marked; ROW and COL are two types of container nodes used to describe the nesting relationship of nodes; the actually displayed nodes are DIS nodes.

[0010] Preferably, the generating of the layout tree according to the layout tree generation function includes:

[0011] Generating a layout tree according to the following functions: init function, addRow function, addCol function, and addDis function.

[0012] Preferably, the generating of the constraint matrix according to the constraint relationship includes:

[0013] Determining the constraint relationship between layout blocks according to the constraint conditions, where the constraint conditions include: width constraint, height constraint, mutual constraint, and self-constraint.

[0014] Preferably, the substituting the parameters into the layout tree and outputting a layout json file includes:

[0015] ① Access the current node, query the length childLen of the child nodes of the current node and the index index of the next child node to be accessed; if index is not equal to childLen, access the child node at index in the child node array and execute ②; if index is equal to childLen, if the current node is a ROW node, take the height of the first child node as the height height of the ROW, and set the top of the parent node of the current ROW to top + ROW.height; if the current node is a COL node, take the width of the first child node as the width width of the COL, and set the left of the parent node of the current COL to left + COL.width, and then return;

[0016] ② If the current node is not a DIS node, execute ①, otherwise obtain the left distance and top distance of the parent node of the current node. If the parent node is a ROW node, it means that the current DIS node is arranged horizontally, and update the parent node left = DIS.width + left; if the parent node is a COL node, it means that the current DIS node is arranged vertically, and update the parent node top = DIS.height + top, and then return;

[0017] ③ Repeat executing ① and ② until the ROOT root node returns.

[0018] According to another aspect of the present invention, the present invention also provides a video layout system based on RREF, and the system includes:

[0019] A generation module for structuring data of the screen layout and generating a layout tree;

[0020] A solution module for defining a DIS vector to be solved, generating a constraint matrix according to its constraint relationships, performing RREF processing on it, and solving to obtain a DIS vector that meets the requirements;

[0021] An output module for searching the layout tree, determining the parameters of each DIS node according to the solved DIS vector, and substituting the parameters into the layout tree to output a JSON file of the layout.

[0022] Preferably, the generation of the layout tree by the generation module includes: generating a layout tree according to a layout tree generation function, where the layout tree includes four types of nodes: ROOT, ROW, COL, and DIS; ROOT is the outermost canvas node, and the attributes of the canvas need to be marked; ROW and COL are two types of container nodes for describing the nesting relationship of the nodes; what is actually displayed is the DIS node.

[0023] Preferably, the generation of the layout tree according to the layout tree generation function includes:

[0024] Generating a layout tree according to the following functions: init function, addRow function, addCol function, addDis function.

[0025] Preferably, the solution module generating a constraint matrix according to the constraint relationships includes:

[0026] Determining the constraint relationships between layout blocks according to the constraint conditions, where the constraint conditions include: width constraint, height constraint, mutual constraint, and self - constraint.

[0027] Preferably, the output module substituting the parameters into the layout tree to output a JSON file of the layout includes:

[0028] ① Access the current node, query the length childLen of the child nodes of the current node and the index index of the next child node to be accessed; if index is not equal to childLen, access the child node at index in the child node array and execute ②; if index is equal to childLen, if the current node is a ROW node, take the height of the first child node as the height height of the ROW, and set the top of the parent node of the current ROW to top + ROW.height; if the current node is a COL node, take the width of the first child node as the width width of the COL, and set the left of the parent node of the current COL to left + COL.width, and then return;

[0029] ② If the current node is not a DIS node, execute ①; otherwise, obtain the left distance and top distance of the parent node of the current node. If the parent node is a ROW node, it means the current DIS node is horizontally arranged, and update the parent node's left = DIS.width + left; if the parent node is a COL node, it means the current DIS node is vertically arranged, and update the parent node's top = DIS.height + top, then return.

[0030] ③ Repeat executing ① and ② until the ROOT root node returns.

[0031] Beneficial effects: By structuring the data of the screen layout, generating a layout tree, then solving the layout tree to obtain the parameters that meet the requirements, and substituting the parameters into the layout tree to output a JSON file of the layout, the present invention can more efficiently automatically generate the sizes and positions of different screen layouts, can reduce the workload of product designers and R & D and testing personnel while improving the layout generation efficiency, and realize user-defined layouts.

[0032] The features and advantages of the present invention will become clear by referring to the following drawings and the detailed description of the specific embodiments of the present invention. Description of the Drawings

[0033] Figure 1 is a flowchart of the video layout method based on RREF;

[0034] Figure 2 is a schematic diagram of a video layout;

[0035] Figure 3 is Figure 2 a schematic diagram of the tree structure corresponding to the video layout in

[0036] Figure 4 is a schematic diagram of the system structure of the video layout based on RREF. Specific Embodiments

[0037] Next, in combination with the drawings in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0038] Embodiment 1

[0039] Figure 1 is a flowchart of the video layout method based on RREF. As Figure 1 shown, the present invention provides a video layout method based on RREF, and the method includes the following steps:

[0040] S1: Structurize the data of the screen layout to generate a layout tree.

[0041] Specifically, as Figure 2 , which is a schematic diagram of video layout, Figure 3 is Figure 2 the schematic diagram of the tree structure corresponding to the video layout in . The purpose of structurizing the layout data is to quickly generate a JSON format or a picture form of a layout according to DFS.

[0042] Preferably, the generating of the layout tree includes: generating a layout tree according to a layout tree generation function, and the layout tree includes four types of nodes: ROOT, ROW, COL, and DIS; ROOT is the outermost canvas node, and the attributes of the canvas need to be marked; ROW and COL are two types of container nodes, which are used to describe the nesting relationship of the nodes; what is actually displayed is the DIS node.

[0043] Refer to Figure 3 , in order to accurately express the structure of a layout, four types of nodes are introduced in this embodiment: ROOT, ROW, COL, and DIS; among them, ROOT is the outermost canvas node, and the attributes of the canvas need to be marked (such as whether the overall screen is 16:9 or 4:3); ROW and COL nodes are two types of container nodes, which are used to describe the nesting relationship of the nodes; finally, what is actually displayed is the DIS node. It should be noted that in this embodiment, the filled black background will also be regarded as a DIS node to participate in the calculation.

[0044] S2: Define the DIS vector to be solved, generate a constraint matrix according to its constraint relationship, and perform RREF processing on it to solve for the DIS vector that meets the requirements.

[0045] Specifically, taking the layout shown in Figure 2 as an example, when the constraints are not enough in this layout, the solutions that meet the layout may not be unique. Since there are ten unknowns, the vector to be solved is directly defined as follows:

[0046]

[0047] Then, the constraint matrix is generated according to its constraint relationship as follows:

[0048]

[0049] Under the condition that the external canvas is 16:9, only the equation needs to be processed by RREF (reduced row echelon form) and then solved. The solution results are as follows:

[0050] 1 2 3 4 5 6 7 8 9 10 11 1 0 -0 -0 -0 0 0 0 0 -0.75 300 0 1 0 0 0 0 0 0 0 -1.3333 533.33 0 0 1 0 0 0 0 0 0 0.375 300 0 0 0 1 0 0 0 0 0 0.66667 533.33 0 0 0 0 1 0 0 0 0 -0.75 300 0 0 0 0 0 1 0 0 0 0.66667 533.33 0 0 0 0 0 0 1 0 0 0.375 300 0 0 0 0 0 0 0 1 0 -1 0 0 0 0 0 0 0 0 0 1 0.375 300 0 0 0 0 0 0 0 0 0 0 0

[0051] It can be seen that there is a degree of freedom, which means that there is more than one layout that conforms to this form when all constraints are satisfied. The designer only needs to add more constraints to further obtain the unique layout that best meets the requirements.

[0052] Preferably, generating the layout tree according to the layout tree generation function includes:

[0053] Generating the layout tree according to the following functions: init function, addRow function, addCol function, addDis function.

[0054] Specifically, the role of the init function is to generate a ROOT node and return a pointer to this node. It can specify two parameters, namely the aspect ratio of the outermost layout container. By default, the aspect ratio is 16:9, and other actual parameters such as 4:3 can also be passed according to needs.

[0055] The role of the addRow function is to generate an empty ROW node and add it to the children array of its parent node. The objects that call the addRow function are the ROOT node and the COL node. In other words, the DIS node and the ROW node cannot use addRow. This is to ensure that container nodes do not nest indefinitely. If unrestricted nesting is allowed, the semantics of the layout will be lost. For example, it is logically impossible for a specific display node DIS to be added to a container node. At the same time, if a ROW container node can directly nest a ROW node, then in fact these are two ROW nodes at the same level, and this kind of nesting is a redundant semantic logical expression.

[0056] The role of the addCol function is to generate an empty COL node and add it to the children array of its parent node. The objects that call the addCol function are the ROOT node and the ROW node. In other words, the DIS node and the COL node cannot use addCol. The reason is similar to the limitation of the addRow function.

[0057] The role of the addDis function is to generate an empty DIS node and add it to the children array of its parent node. The objects that call the addDis function are the ROW node and the COL node. In other words, the ROOT node and the DIS node cannot use addDis. It is easy to understand that the DIS node cannot nest the DIS node. The reason why the ROOT node cannot be directly added to the DIS node is that the ROOT node does not know whether the DIS node should be arranged in rows or columns.

[0058] Preferably, generating the constraint matrix according to the constraint relationship includes:

[0059] Determine the constraint relationships between layout blocks according to the constraint conditions, where the constraint conditions include: width constraint, height constraint, mutual constraint, and self constraint.

[0060] Specifically, the width constraint is naturally established (i.e., automatically generated) under general conditions. Specifically: the width of the ROW node should be equal to the width of its parent node. The sum of the widths of all nodes in the children array of the ROW node should be equal to the width of the ROW node.

[0061] The height constraint is also naturally established (i.e., automatically generated) under general conditions. Specifically: the height of the COL node should be equal to the height of its parent node. The sum of the heights of all nodes in the children array of the COL node should be equal to the width of the COL node.

[0062] The mutual constraint is a constraint on the mutual width and height relationships between DIS nodes. In fact, DIS nodes have different categories. For example, in the screen layout, there are three types of actually displayed screens: large screen, small screen, and background filling screen. That is to say, in this example, DIS nodes have three categories. Sometimes we require that the width of a large screen is equal to the width of three small screens, which means this is a new mutual constraint.

[0063] The self constraint is a requirement for the width and height of the DIS node itself. Most commonly, we require that the video display screen is also 16:9, which means that this type of DIS node adds a self constraint.

[0064] Among them, what the user needs to manually supplement are only the mutual constraint and the self constraint, that is, the following two functions are required: The function of the addMutualCons function is to add a mutual constraint for two types of DIS nodes. The underlying algorithm will parse it into a row vector in the matrix and add it to the matrix. The function of the addSelfCons function is to add a self constraint for one type of DIS node. The underlying algorithm will parse it into a row vector in the matrix and add it to the matrix.

[0065] S3: Search the layout tree, determine the parameters of each DIS node according to the solved DIS vector, and substitute the parameters into the layout tree to output the layout json file.

[0066] Preferably, the substituting the parameters into the layout tree to output the layout json file includes:

[0067] ① Access the current node, query the length of the child nodes of the current node childLen and the index of the next child node to be accessed index; if index is not equal to childLen, access the child node at index in the child node array and execute ②; if index is equal to childLen, if the current node is a ROW node, take the height of the first child node as the height height of the ROW, and set the top of the parent node of the current ROW to top + ROW.height; if the current node is a COL node, take the width of the first child node as the width width of the COL, and set the left of the parent node of the current COL to left + COL.width, then return.

[0068] ② If the current node is not a DIS node, execute ①, otherwise obtain the left distance and top distance of the parent node of the current node. If the parent node is a ROW node, it means that the current DIS node is arranged horizontally, and update the parent node left = DIS.width + left; if the parent node is a COL node, it means that the current DIS node is arranged vertically, and update the parent node top = DIS.height + top, then return.

[0069] ③ Repeat executing ① and ② until the ROOT root node returns.

[0070] Specifically, the function of the solve function is to automatically fill the data of the formal layout tree in a depth-first traversal manner after solving through the RREF algorithm. After the filling is completed, a layout tree in json format is output for the user to call. It should be noted that after performing RREF on the constraint matrix, there may be cases of no solution or no unique solution. This means that there are contradictions in the user's constraint conditions or there are free variables due to fewer constraints. If it is a case of no solution, directly output an empty formal layout tree. If it is a case of no unique solution, the user needs to be prompted to add constraint conditions.

[0071] In this embodiment, by structuring the data of the screen layout, generating a layout tree, then solving the layout tree to obtain the parameters that meet the requirements, and substituting the parameters into the layout tree to output a json file of the layout, it can achieve more efficient automatic generation of different screen layout sizes and positions, can reduce the workload of product designers and R & D and testing personnel while improving the layout generation efficiency, and realize user-defined layout.

[0072] Embodiment 2

[0073] Figure 4 is a schematic structural diagram of a video layout system based on RREF. As Figure 4 shown, the present invention also provides a video layout system based on RREF, and the system includes:

[0074] A generation module 401 for structuring data of a screen layout to generate a layout tree;

[0075] A solving module 402 for defining a DIS vector to be solved, generating a constraint matrix according to its constraint relationships, and performing RREF processing on it to solve for a DIS vector that meets the requirements;

[0076] An output module 403 for searching the layout tree, determining the parameters of each DIS node according to the solved DIS vector, and substituting the parameters into the layout tree to output a JSON file of the layout.

[0077] Preferably, the generation module 401 generating a layout tree includes: generating a layout tree according to a layout tree generation function, where the layout tree includes four types of nodes: ROOT, ROW, COL, and DIS; ROOT is the outermost canvas node and needs to mark the attributes of the canvas; ROW and COL are two types of container nodes used to describe the nesting relationship of nodes; what is actually displayed is the DIS node.

[0078] Preferably, the generating a layout tree according to a layout tree generation function includes:

[0079] Generating a layout tree according to the following functions: init function, addRow function, addCol function, addDis function.

[0080] Preferably, the solving module 402 generating a constraint matrix according to constraint relationships includes:

[0081] Determining the constraint relationships between layout blocks according to constraint conditions, where the constraint conditions include: width constraint, height constraint, mutual constraint, and self - constraint.

[0082] Preferably, the output module 403 substituting the parameters into the layout tree to output a JSON file of the layout includes:

[0083] ① Access the current node, query the length childLen of the child nodes of the current node and the index index of the next child node to be accessed; if index is not equal to childLen, access the child node at index in the child node array and execute ②; if index is equal to childLen, if the current node is a ROW node, take the height of the first child node as the height height of the ROW, and set the top of the parent node of the current ROW to top + ROW.height; if the current node is a COL node, take the width of the first child node as the width width of the COL, and set the left of the parent node of the current COL to left + COL.width, and then return;

[0084] ② If the current node is not a DIS node, execute ①. Otherwise, obtain the left distance and top distance of the parent node of the current node. If the parent node is a ROW node, it indicates that the current DIS node is arranged horizontally. Update the parent node's left = DIS.width + left. If the parent node is a COL node, it indicates that the current DIS node is arranged vertically. Update the parent node's top = DIS.height + top, and then return.

[0085] ③ Repeat executing ① and ② until the ROOT root node returns.

[0086] The specific implementation processes of the functions implemented by each module in this Embodiment 2 are the same as those in Embodiment 1 and will not be elaborated here.

[0087] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structural transformation made under the concept of the present invention by using the content of the specification and drawings of the present invention, or any direct / indirect application in other related technical fields, is included in the patent protection scope of the present invention.

Claims

1. A video layout method based on RREF, characterized in that, The method includes the following steps: S1: Structure the data of the screen layout to generate a layout tree; S2: Define the DIS vector to be solved, generate a constraint matrix according to its constraint relationships, and perform RREF processing on it to solve for the DIS vector that meets the requirements; S3: Search the layout tree, determine the parameters of each DIS node according to the solved DIS vector, and substitute the parameters into the layout tree to output a JSON file of the layout; The generation of the layout tree includes: generating a layout tree according to a layout tree generation function, where the layout tree includes four types of nodes: ROOT, ROW, COL, and DIS; ROOT is the outermost canvas node, and the attributes of the canvas need to be marked; ROW and COL are two types of container nodes used to describe the nesting relationship of the nodes; what is actually displayed is the DIS node; The substitution of the parameters into the layout tree to output a JSON file of the layout includes: ① Access the current node, query the length childLen of the child nodes of the current node and the index index of the next child node to be accessed; if index is not equal to childLen, access the child node at index in the child node array and execute ②; if index is equal to childLen, if the current node is a ROW node, take the height of the first child node as the height height of the ROW, and set the top of the parent node of the current ROW to top + ROW.height; if the current node is a COL node, take the width of the first child node as the width width of the COL, and set the left of the parent node of the current COL to left + COL.width, and then return; ② If the current node is not a DIS node, execute ①, otherwise obtain the left distance and top distance of the parent node of the current node. If the parent node is a ROW node, it means that the current DIS node is arranged horizontally, and update the parent node left = DIS.width + left; if the parent node is a COL node, it means that the current DIS node is arranged vertically, and update the parent node top = DIS.height + top, and then return; ③ Repeat the execution of ① and ② until the ROOT root node returns.

2. The method according to claim 1, characterized in that, The generation of the layout tree according to the layout tree generation function includes: Generating a layout tree according to the following functions: init function, addRow function, addCol function, addDis function.

3. The method according to claim 1, characterized in that, The generation of the constraint matrix according to the constraint relationships includes: Determining the constraint relationships between layout blocks according to the constraint conditions, where the constraint conditions include: width constraint, height constraint, mutual constraint, and self - constraint.

4. A video layout system based on RREF, characterized in that The system includes: A generation module for structuring the data of the screen layout to generate a layout tree; A solution module for defining the DIS vector to be solved, generating a constraint matrix according to its constraint relationships, and performing RREF processing on it to solve for the DIS vector that meets the requirements; An output module, which is used to search the layout tree, determine the parameters of each DIS node according to the solved DIS vector, and substitute the parameters into the layout tree to output a JSON file of the layout; The generation module generates a layout tree, including: generating a layout tree according to a layout tree generation function, where the layout tree includes four types of nodes: ROOT, ROW, COL, and DIS; ROOT is the outermost canvas node, and the attributes of the canvas need to be marked; ROW and COL are two types of container nodes, which are used to describe the nesting relationship of nodes; what is actually displayed is the DIS node; The output module substitutes the parameters into the layout tree to output a JSON file of the layout, including: ① Access the current node, query the length of the child nodes of the current node childLen and the index of the next child node to be accessed index; if index is not equal to childLen, access the child node at index in the child node array, and execute ②; if index is equal to childLen, if the current node is a ROW node, take the height of the first child node as the height height of the ROW, and set the top of the parent node of the current ROW to top + ROW.height; if the current node is a COL node, take the width of the first child node as the width width of the COL, and set the left of the parent node of the current COL to left + COL.width, and then return; ② If the current node is not a DIS node, execute ①, otherwise obtain the left distance and top distance of the parent node of the current node. If the parent node is a ROW node, it means that the current DIS node is arranged horizontally, and update the parent node left = DIS.width + left; if the parent node is a COL node, it means that the current DIS node is arranged vertically, and update the parent node top = DIS.height + top, and then return; ③ Repeat executing ① and ② until the ROOT root node returns.

5. The system according to claim 4, characterized in that, The generation of the layout tree according to the layout tree generation function includes: Generating a layout tree according to the following functions: init function, addRow function, addCol function, addDis function.

6. The system according to claim 4, wherein The solution module generates a constraint matrix according to the constraint relationship, including: Determining the constraint relationship between layout blocks according to the constraint conditions, where the constraint conditions include: width constraint, height constraint, mutual constraint, and self-constraint.

Images