Video aggregation display system and video aggregation display method

Abstract

To provide a video aggregation display function at low cost and in a space-saving manner. [Solution] A video aggregation display system (100) includes video aggregation devices (130a-130d) equipped with a display control unit (137) that receives multiple pieces of video information via a network line, aggregates the received video information, and displays the video information on a assigned display device based on assigned display device information (22) that indicates which assigned display device among the multi-displays (160) the video information should be displayed on, and a video aggregation management device (140). The video aggregation management device (140) includes a storage unit (149) that stores layout state information (21) that indicates which position on the multi-display (160) a video window related to each piece of video information should be displayed on, and a synchronization unit (148) that synchronizes the layout state information with the video aggregation devices (130a-130c). The video aggregation devices (130a-130d) include a communication unit (138) that transmits and receives data to and from the video aggregation management device (140).

Description

[Technical Field]

[0001] The present invention relates to a monitoring system that aggregates and displays multiple videos in a control room or a monitoring room, allowing staff to share information. [Background technology]

[0002] In command and monitoring rooms installed in the communications control room of prefectural police headquarters, fire department headquarters, and water purification plant monitoring rooms, multiple images are displayed on a large multi-display, allowing staff to gather together and share information while carrying out monitoring duties. Patent Document 1 describes an invention that allows a plurality of surveillance videos in different formats to be freely displayed in a display area. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-081039 Summary of the Invention [Problem to be solved by the invention]

[0004] When carrying out surveillance work, a multi-display is constructed by combining multiple LCD displays to form a huge display surface, on which multiple images are displayed in parallel. Generally, it is required to be able to display around 20 to 30 images simultaneously. The hardware that allows such images to be freely laid out across multiple displays is called a "multi-vision processor," and depending on the scale of the configuration, it is an expensive device costing in the tens of millions of yen range.

[0005] Previously, displaying multiple images on the same display required dedicated hardware such as a screen combiner or multi-vision processor. With a screen combiner, only a small number of pre-set layouts could be selected, making flexible arrangements impossible. On the other hand, multi-vision processors are extremely expensive and have a hardware restriction of only being able to display a maximum of four images per display. Furthermore, current multi-vision processors have a resolution limit, so when attempting to build a large-scale multi-display system, the resolution per display must be reduced.

[0006] In the prefectural police communications control room, it is common to display a map in the background and overlay traffic control footage and in-vehicle footage on top of it. However, due to the aforementioned four-image limit, only three images can be displayed simultaneously, excluding the background map. To consolidate and display video on a large multi-display, it is necessary to install a large number of expensive, large-scale dedicated devices such as matrix switchers and multi-vision processors, and there is a lack of installation space in small conference rooms. Also, when staff from related departments working on different floors need to share information while viewing the same video, they must gather in a control room equipped with a large multi-display, which can lead to delays in information sharing and transmission in emergencies. Therefore, there is a need for a video consolidation display function that can share multiple videos with staff in remote locations in real time, enabling information sharing without having to gather in the same space.

[0007] Therefore, an object of the present invention is to realize an image aggregation display function at low cost and in a space-saving manner. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, the video aggregation display system of the present invention is configured as follows. That is, the video aggregation display system of the present invention receives a plurality of pieces of video information via a network line, Layout state information indicating at which positions on the multi-display a plurality of video windows generated from the plurality of pieces of video information are to be arranged when displayed on the multi-display, and layout state information indicating at which positions on the multi-display a plurality of video windows are to be arranged on the multi-display. Which of the multiple display devices Reni Based on the responsible display device information indicating whether to display connected to itself On the display device Any of the plurality of video windows display When the plurality of pieces of video information are displayed on a single display, a plurality of video windows are generated from the plurality of pieces of video information, and the plurality of video windows are displayed in accordance with layout state information indicating positions on the single display where the plurality of video windows are to be arranged. It aggregates the received video information and The single display device in charge of displaying the aggregated image in which the plurality of image windows are aggregated is Display control unit 、 and a video aggregation management device, The aforementioned The video aggregation device has a memory unit that stores layout state information and a synchronization unit that synchronizes the layout state information between the first and second video aggregation devices, and the first video aggregation device has a first communication unit that transmits and receives data between the first and second video aggregation devices, and the second video aggregation device has a second communication unit that transmits and receives data between the first and second video aggregation devices.

[0009] The video aggregation and display method of the present invention receives a plurality of pieces of video information via a network line. a step of displaying on a multi-display, a display control unit of the first or second video aggregation device displaying any of the multiple video windows generated from the multiple video information on a display device connected to the first or second video aggregation device based on layout state information indicating at which positions on the multi-display the multiple video windows should be placed and on which of the multiple display devices constituting the multi-display the multiple video windows should be displayed; and a step of displaying on a single display, a display control unit of the first or second video aggregation device generating multiple video windows from the multiple video information, aggregating the received video information according to the layout state information indicating at which positions on the single display the multiple video windows should be placed, and displaying an aggregated image in which the multiple video windows are aggregated on the assigned display device which is the single display. The first video aggregation device position Change depending on , The aforementioned Video window The multi-display or the single display a step of modifying layout state information indicating the position at which the image is to be displayed; and a first communication unit of the first video aggregation device: The aforementioned Correct Tare outputting layout status information to the video centralization management device; The aforementioned a synchronization unit of the video central management device having a storage unit that stores layout state information, Fixed a step of synchronizing layout state information with the second video aggregation device; and Between the video centralized management device Synchronized The above correction A video window based on layout state information A display device connected to itself among multiple display devices that make up a multi-display, or a single display and a step of displaying the information on a responsible display device. Other means will be described in the detailed description of the invention. [Effects of the Invention]

[0010] According to the present invention, an image aggregation display function can be realized at low cost and in a space-saving manner. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a configuration diagram of a synchronized video aggregation display system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram illustrating a logical configuration of a synchronized video aggregation display system. [Figure 3] FIG. 10 is a diagram illustrating buttons displayed on a load-distribution multi-display. [Figure 4] FIG. 10 is a diagram illustrating a load-distribution type multi-display using a plurality of video aggregation devices. [Figure 5] FIG. 10 is a diagram showing the structure of window information. [Figure 6] FIG. 10 is a diagram illustrating windows displayed on a load-distribution multi-display. [Figure 7] FIG. 2 is a diagram illustrating a layer structure of a window. [Figure 8] FIG. 10 is a sequence diagram of a synchronization operation of window information. [Figure 9] FIG. 2 is a diagram showing the configuration of drawing information. [Figure 10] FIG. 10 is a sequence diagram of a synchronization operation of drawing information. [Figure 11] FIG. 10 is a diagram showing the configuration of user call information. [Figure 12] FIG. 2 is a diagram showing the configuration of task information. [Figure 13] FIG. 10 is a sequence diagram of a synchronization operation of call information. [Figure 14] FIG. 10 is a sequence diagram of a synchronization operation of call information. DETAILED DESCRIPTION OF THE INVENTION

[0012] The following describes in detail the embodiments of the present invention with reference to the accompanying drawings. The present invention realizes the functions of a multi-vision processor through software, enabling it to be used in place of multiple general-purpose gaming PCs. It also adds an electronic whiteboard function not available in conventional multi-vision processors.

[0013] FIG. 1 is a configuration diagram of an image aggregation display system 100 according to this embodiment. The video aggregation display system 100 includes multi-displays 160a-160c and a display 160d installed at each monitoring station, and video aggregation devices 130a-130d corresponding to each of these display devices. The multi-display 160a forms a display surface by combining individual displays 161a-161d in a tiled pattern, and displays multiple videos supplied from the video aggregation device 130a. Similarly, the other multi-displays 160b, 160c, and the display 160d receive video signals from their corresponding video aggregation devices 130b-130d. The video aggregation devices 130b-130d receive multiple pieces of video information 150a-150n via a network line, aggregate the received video information 150a-150n, and include a display control unit 131 that displays the video on a assigned display device based on assigned display device information indicating which of the displays 161a-161d constituting the multi-display 160a is to display the video on that assigned display device.

[0014] The video aggregation device 130a incorporates a display control unit 131, a communication unit 132, a call execution unit 133, and a drawing execution unit 134. The display control unit 131 outputs video to the multi-display 160a. The communication unit 132 communicates with an external device and synchronizes various pieces of information by transmitting and receiving various pieces of information to and from a synchronization unit 141 of the video aggregation management device 140 (described later).

[0015] The call execution unit 133 controls voice calls between locations. The drawing execution unit 134 performs freehand drawing on the aggregated video to superimpose annotations and figures. The video aggregation devices 130b to 130d also have similar functional configurations, and perform video aggregation at each location, calls between each location, and synchronization of drawing information between each location.

[0016] Each of the video aggregation devices 130a to 130d is connected to a hub 180 via a network 170 represented by a thick line in the figure. The hub 180 receives input of multiple types of video information 150a to 150n transmitted from external cameras, drones, etc., and is also connected to a video aggregation management device 140. The video aggregation management device 140 includes a synchronization unit 141 and a call status management unit 142, and aggregates control information and video streams generated and input at each location, monitors them in real time, and performs overall control. The synchronization unit 141 synchronizes layout state information of video windows to be distributed to all locations. The synchronization unit 141 synchronizes drawing information to be distributed to all locations. The synchronization unit 141 synchronizes various pieces of information between the video aggregation devices 130a to 130d. The call status management unit 142 performs connection and priority control of calls between locations.

[0017] With the above configuration, the video aggregation and display system 100 can aggregate and share a large number of videos, audio, and annotations across multiple locations in real time without using expensive multi-vision processors, enabling efficient monitoring and command operations.

[0018] FIG. 2 is a diagram showing the logical configuration of the video aggregation display system 100. As shown in FIG. 2 is a block diagram of the video aggregation display system 100 shown in FIG. 1, focusing on the connection configuration between the video aggregation devices 130a-130d and the video aggregation management device 140, external video information 150a-150d, and the multi-display 160. Multiple types of video information 150a-150d sent from external cameras and sensors are input to the video aggregation devices 130a-130d located at each site via network devices (e.g., LAN (Local Area Network) routers and VPN (Virtual Private Network) routers) shown on the left side of the figure. Each of these video aggregation devices 130 outputs the received video to the multi-display 160 as a video window.

[0019] The video aggregation device 130d is provided with a storage unit 139, a communication unit 132, and a display control unit 131, and the storage unit 139 stores layout state information 21 indicating the window arrangement and video switching state on the multi-display 160, and assigned display device information 22 indicating the display device assigned to the video aggregation device 130d. The layout state information 21 indicates at which position on the multi-display 160 a video window related to each piece of video information is to be displayed. The assigned display device information 22 indicates on which assigned display device of the multi-display 160 the video information is to be displayed.

[0020] Correspondingly, video aggregation management device 140 installed in the center is provided with storage unit 149 and synchronization unit 141, and holds layout state information 21 and assigned display device information 22 in the same format. Synchronization unit 141 synchronizes this layout state information 21 and assigned display device information 22 between video aggregation devices 130a to 130d.

[0021] As indicated by the double arrows, layout status information 21 and assigned display device information 22 are transmitted and received between the video aggregation device 130d and the video aggregation management device 140, and the video display status at each location is synchronized in real time. Based on the collected status of each location, the video aggregation management device 140 performs layout integration and authority management for the entire system, and provides feedback of control instructions to each video aggregation device 130 as necessary.

[0022] As described above, Figure 2 shows the video transmission path from video information 150a to 150d to multi-display 160, as well as the storage structure and sharing structure of layout status information 21 and responsible display device information 22 in video aggregation devices 130a to 130d and video aggregation management device 140, and allows one to understand that the present invention has an information synchronization function for smoothly carrying out remote collaborative work while maintaining a consistent display layout between bases.

[0023] FIG. 3 is a diagram illustrating buttons displayed on a load-distribution type multi-display. Screen 110 is a rectangular display area divided into four sub-windows by vertical and horizontal boundaries, allowing any surveillance video to be displayed in each sub-window. Arranged vertically on the left edge of the display area are a "Video Selection" button 116 for selecting a video, a "Rectangle Frame" button 117 for drawing a rectangular frame, a "Freehand" button 118 for drawing freehand lines, an "Eraser" button 119 for partially erasing previously drawn lines, and a "Clear All" button 120 for erasing all annotations at once. Operators can use these tools to add or delete annotations by drawing freehand on each sub-window. Located in the lower left corner is an "Operation Mode Switch" button 121 for switching input devices and processing modes.

[0024] Near the bottom center of the screen are provided side by side "Personal" button 122a, which switches to a display exclusive to the operator's location, and "Shared" button 122b, which switches to a display shared with all locations, allowing the operator to instantly specify the scope of disclosure for each sub-window. Meanwhile, along the right edge of screen 110, are arranged vertically "Pattern Switch" buttons 114a-114d, which change the layout for each sub-window, and "Update" buttons 115a-115c, which redraw the video content to bring it up to date. The bottommost sub-window does not require real-time updating, so the corresponding update button 115d is omitted.

[0025] In this way, the screen 110 of this embodiment enables operations such as video selection, drawing, layout changes, and sharing range settings to be performed intuitively on a single screen, thereby realizing fast and flexible video sharing among multiple multi-displays, including those at remote locations.

[0026] FIG. 4 is a diagram illustrating a load-distribution type multi-display using a plurality of video aggregation devices. At the top of Figure 4, a central control node, video aggregation management device 140, is located, and layout state information 21 indicating the window arrangement and video switching status on multi-displays 160a-160c is stored therein. Video aggregation management device 140 is connected to multiple video aggregation devices 130a, 130b, 130c, and 130d via a network, and transmits control instructions to each of the video aggregation devices 130a, 130b, 130c, and 130d. Additionally, video aggregation management device 140 receives and constantly updates the latest layout state information 21 reported from each of the video aggregation devices 130a, 130b, 130c, and 130d. This synchronizes the layout state information 21 between the video aggregation devices 130a, 130b, 130c, and 130d.

[0027] Video aggregation devices 130a-130d are installed corresponding to the display systems of the respective opposing sites, and store layout state information 21 of the same format in their internal device memories. In the lower part of the figure, multi-display 160a to which video aggregation device 130a outputs video is shown as the assigned display device. Similarly, multi-display 160b to which video aggregation device 130b outputs video is shown as the assigned display device. Multi-display 160c to which video aggregation device 130c outputs video is shown as the assigned display device. And single display 160d to which video aggregation device 130d outputs video is shown as the assigned display device. On each of multi-displays 160a-160c, an example arrangement of multiple video windows is depicted using diagonal lines or shading. These video windows are generated based on layout state information 21 of video aggregation devices 130a to 130d, and are synchronized in real time with information held by central video aggregation and management device 140.

[0028] As such, Figure 4 shows the functional relationship in which the video aggregation management device 140 acts as a hub to centrally distribute layout state information 21 from each video aggregation device 130 and uniformly control the arrangement of video windows on multi-displays 160a to 160c and display 160d, enabling flexible reconfiguration while maintaining a consistent video window layout across all locations.

[0029] FIG. 5 is a diagram showing the structure of the window information 23. As shown in FIG. Fig. 5 is a diagram showing a typical data structure of window information 23 that stores detailed parameters related to each video window, as an example of the components of layout state information 21 shown in Fig. 4. Window information 23 includes window number 231 that uniquely identifies each video window arranged on multi-display 160, and manages which video information is being displayed by associating this video window with video number 232 that indicates the video stream assigned to it.

[0030] Furthermore, an audio presence / absence flag 233 is provided, which indicates whether audio is to be accompanied by the corresponding video information. In the illustrated example, this is set to "False," which indicates that audio is not being output from this video window. Additionally, in order to define the display position and size of the video window, upper left coordinates 234 and lower right coordinates 235 based on the screen coordinate system are stored. These coordinate values ​​are configured to represent the entire surface of the multi-display 160 as normalized coordinates ranging from -1.0 to +1.0, thereby enabling the layout to be reproduced independently of the resolution of the multi-display 160. In this embodiment, the video aggregation devices 130a to 130d and the video aggregation management device 140 mutually transmit and receive layout state information 21, including window information 23, for synchronization, thereby enabling the accurate synchronization of the video window layouts between the locations.

[0031] FIG. 6 is a diagram illustrating a video window displayed on the load-distribution type multi-display 160. As shown in FIG. Figure 6 is a conceptual diagram illustrating the geometric relationship of video windows 31 arranged on the display screen 3 based on a normalized coordinate system. In the figure, the outer frame surrounded by dashed lines represents the entire display surface, and a virtual Cartesian coordinate system is set with the lower left corner at (-1.00, 1.00) and the upper right corner at (+1.00, -1.00). The dotted line running vertically through the center indicates the x=0 axis, which roughly divides the screen into two areas, left and right. The dotted rectangle represents the shared video area that is already being displayed, and its boundary is marked with black vertex markers indicating the four corners.

[0032] A video window 31 is superimposed on the left side of the lower half, with the coordinates of its upper left vertex indicated as (-0.89, 0.50). The bottom center and bottom right corner of the shared video area are assigned coordinates of (0.00, 1.00) and (0.89, 1.00), respectively, indicating that the location and size of video window 31 can be uniquely identified regardless of display resolution. These coordinates correspond to the top-left coordinate 234 and bottom-right coordinate 235 stored in window information 23. By exchanging these coordinates between video aggregation devices 130a-130d and video aggregation management device 140, the same layout can be accurately reproduced between locations. The sequence shown in Figure 6 manages window positions using a normalized coordinate system, enabling flexible layout synchronization even between display devices with different resolutions.

[0033] FIG. 7 is a diagram illustrating the layer structure of the video window 31. As shown in FIG. Figure 7 shows an enlarged example of the video window 31 shown in Figure 6, and illustrates that the video window 31 is composed of three parts: a background layer 311, a video display layer 312, and a user interface layer 313. The background layer 311 is the base layer that displays shared video, maps, and the like. The video display layer 312 is placed on top of the background layer 311. The video display layer 312 displays live video, such as footage of the inside of a control vehicle. The user interface layer 313, which is placed on top of the background layer 311, includes a "small" resize button, a "medium" resize button, a "large" resize button, a "full screen" resize button, a close button, a back button, and a speaker button. The resize button switches the window size between "small," "medium," "large," and "full screen." The "back button" sends the user interface layer 313 to the back. The speaker button toggles between audio output and mute. The close button closes the window. The identification name of the video, "Control Video A," is displayed at the bottom of the user interface layer 313. The operator can intuitively operate the window by operating the user interface layer 313 while visually checking the content of the video on the video display layer 312.

[0034] FIG. 8 is a sequence diagram of the window information synchronization operation. 8 shows a timeline of layout synchronization processing performed between video aggregation devices 130a-130d at each location and video aggregation management device 140 when a video layout change occurs. First, an operator changes the layout of the video windows via the user interface of video aggregation device 130d. Video aggregation device 130d generates post-change layout state information 21 and sets it in its own device in step S10. Next, in step S11, video aggregation device 130d transmits the generated layout state information 21 to video aggregation management device 140 via the network.

[0035] When the video aggregation and management device 140 acquires the received information, it first stores the changes in volatile memory in step S12, and simultaneously writes them to a permanent database in step S13 to update the history. This makes it possible to restore the latest state even in the event of a system failure. Next, the video aggregation and management device 140, via the synchronization unit 141, proceeds to step S14, where it broadcasts the integrated layout state information 21 to the video aggregation devices 130a, 130b, and 130c for synchronization. These video aggregation devices 130a, 130b, and 130c reflect the received layout state information 21 within their own devices and reproduce the same layout on the multi-displays 160a-160c that they control, thereby instantly coordinating displays across all locations.

[0036] This sequence, shown in Figure 8, shows the process by which layout operations at one location are deployed to the entire system via the central video central management device 140, and the three-stage synchronization process that takes place during this process: memory storage, database storage, and information redistribution. This clearly shows that the present invention achieves high-speed, highly reliable layout sharing functionality.

[0037] That is, display control unit 131 of video aggregation device 130a, which is the first video aggregation device, outputs a video window that can be moved by drag-and-drop operation to a display in charge, and updates its own layout state information in response to the moving operation of this video window. Then, communication unit 132, which is the first communication unit of video aggregation device 130a, synchronizes the difference in layout state information with synchronization unit 141 of video aggregation management device 140. Then, synchronization unit 141 of video aggregation management device 140 updates the layout state information stored in storage unit 149 based on the difference in layout state information input from video aggregation device 130a, and synchronizes with video aggregation devices 130b-130d, which are second video aggregation devices. Furthermore, display control unit 131 of video aggregation devices 130b-130d, which are second video aggregation devices, outputs a video window to a second display device managed by the second video aggregation device, based on the layout state information synchronized by synchronization unit 141 and communication unit 132, which is a second communication unit.

[0038] Drawing function Services such as online drawing chats are already commonplace, allowing multiple people to draw text and illustrations on the same screen, but at present there are no services that allow multiple people to draw on top of a video stream in the background.

[0039] When using an electronic whiteboard, drawing is limited to the presenter only, and users must gather in front of the board. Furthermore, there is no function that allows users to freely move the layout of the aggregated video while simultaneously drawing text or illustrations.

[0040] In addition, electronic whiteboards are generally large, and the effort required to pull them out and the need for people to gather in the same conference room often means that they end up not being used.

[0041] FIG. 9 is a diagram showing the configuration of the drawing information. FIG. 9 is a block diagram showing an example of the "drawing information 24," a data structure used to identify and reproduce drawn strokes in the drawing function. The drawing information 24 includes a stroke number 241 that uniquely identifies each stroke. This number is used to manage the drawing order and editing history of multiple strokes. Next, a color field 242 is provided, which indicates the color attributes used when displaying the stroke, and stores RGB values, palette numbers, and the like. A thickness field 243 stores a value that specifies the stroke width, allowing the same appearance to be reproduced regardless of the device or resolution.

[0042] The position information includes a coordinate field 244 indicating the start point of the stroke and a bottom-right coordinate field 245 indicating the end point or diagonal point. Similar to window information, these coordinates are stored as normalized values, ensuring compatibility between display devices. Finally, paint information 246 stores, as needed, the sequence of intermediate points in the case of freehand drawing, shape attributes such as rectangle or circle, and whether or not the stroke is filled. With this configuration, the drawing information 24 comprehensively stores the visual and geometric attributes of the stroke, and by exchanging this information between the video aggregation devices 130a-130d and the video aggregation management device 140, real-time synchronization of drawing information across locations is possible.

[0043] FIG. 10 is a sequence diagram of the synchronization operation of the drawing information. 10 shows a timeline of the drawing information synchronization process executed between the video aggregation devices 130a-130d at each location and the video aggregation / management device 140 when an operator performs a drawing operation. First, when a stroke is drawn via an input device in the video aggregation device 130d (step S20), the video aggregation device 130d generates drawing information 24 that identifies the stroke and stores it in its internal memory. Next, in step S21, the generated drawing information 24 is transmitted to the video aggregation / management device 140 via the network.

[0044] When the video aggregation and management device 140 acquires the received drawing information 24, it first temporarily stores the information in a volatile memory in step S22, and then writes it to a permanent database and registers it as history in step S23. This ensures redundancy so that the latest annotation state can be restored even if a system failure occurs. The video aggregation and management device 140 then proceeds to step S24, where it transmits the stored drawing information 24 to the video aggregation device 130a, The drawing information 24 is then broadcast to 130b, 130c. Each device receives the drawing information 24 and redraws the same strokes in real time on the multi-displays 160a to 160c that it controls, thereby realizing consistent annotation display across all locations.

[0045] That is, the video aggregation device 130d, which is the first video aggregation device, further includes a drawing execution unit 134 that executes a freehand drawing operation on the aggregated video. A communication unit 132 serving as a first communication unit of the video aggregation device 130d serving as the first video aggregation device outputs the drawing data executed by the drawing execution unit 134 to a synchronization unit 141 of the video aggregation management device 140. Furthermore, synchronization unit 141 of video aggregation management device 140 synchronizes the synchronized drawing data with video aggregation devices 130a to 130d, which are second video aggregation devices. Display control unit 131 of video aggregation devices 130a to 130d, which are second video aggregation devices, outputs drawing data synchronized by communication unit 132, which is a second communication unit, and synchronization unit 141 to a multi-display, which is a second display device in charge.

[0046] In this way, FIG. 10 shows a series of processing flows in which a drawing operation performed at one location is instantly distributed to all devices via the central video central management device 140.

[0047] <Call function> FIG. 11 is a diagram showing the configuration of the user call information 25. As shown in FIG. The figure shows an example of the data structure of "user call information 25," which is used to grasp and synchronize the voice communication status of each operator in the call management function of the present invention. The user call information 25 first stores user ID information 251, which uniquely identifies each operator. Next, a call status 252 is provided, indicating whether the operator is currently on a call. This allows the video aggregation device 130a-130d and the video aggregation management device 140 to visualize the speaker and control the right to speak. A microphone on / off status 253, indicating whether the operator's microphone is muted, is also stored and used as control information to prevent voice interference between remote locations. Furthermore, in preparation for the operation of multiple call channels, a participating channel 254 is provided, indicating the voice channel in which the operator is participating. As described above, the user call information 25 aggregates and stores attributes essential for integrated implementation of voice communication functions such as right-to-speak management, mute control, and channel switching. By exchanging this information in real time between the video aggregation management device 140 and the video aggregation devices 130a-130d, the consistency of calls and operational efficiency at all locations are improved.

[0048] FIG. 12 is a diagram showing the configuration of the task information 26. As shown in FIG. 12 is a conceptual diagram showing an example of the data structure of task information 26 used to smoothly facilitate business collaboration between bases in the present invention. Task information 26 integrally holds the request content and party information when an operator requests another base or another person in charge to perform a process such as switching video, searching for information, or presenting materials. Task information 26 first stores sender user ID information 261 that identifies the operator who sent the request. Next, requested user ID information 262 is provided that indicates the person who will receive the request. These two IDs uniquely identify the transmission and reception path of the task.

[0049] In this embodiment, task information 26 is mutually transmitted and updated between video aggregation devices 130a to 130d or via video aggregation management device 140, thereby ensuring that requests are shared reliably and in real time, and preventing delays and omissions in the transmission of instructions between remote locations.

[0050] FIG. 13 is a sequence diagram of the synchronization operation of the call information. 13 is a sequence diagram showing the flow of call status synchronization processing executed between video aggregation devices 130a-130d and video aggregation / management device 140 when an operator performs an operation to join a voice call. First, when an operator presses the call join button on video aggregation device 130d located at location d (step S30), video aggregation device 130d updates its own user call information 25 and generates call status information indicating the participation status. In the following step S31, this call status information is transmitted to central video aggregation / management device 140 via the network.

[0051] The video aggregation management device 140 acquires the received call status information and records the call participation flag of the video aggregation device 130d in volatile memory in step S32. At the same time, the information is also written to a database as needed to prepare for recovery in the event of a communication failure. Next, the video aggregation management device 140 broadcasts the updated call status information to the video aggregation devices 130a to 130c in step S33, and each device updates its own user call information 25 based on the received information.

[0052] After the update is reflected, the video aggregation device 130a updates its call screen and immediately redraws the participant list and speech status as shown in step S34, step S35, and step S36, respectively, for the video aggregation device 130b, 130c. This allows the status of call participants to be synchronized in real time at all locations, and controls such as speaking rights and mute operations can be performed in a consistent manner. Figure 13 visualizes the above processing sequence and demonstrates that the present invention achieves cross-site call management functions with high immediacy and reliability.

[0053] These first and second video aggregation devices 130a to 130d further include a microphone for recording audio and a speaker for reproducing audio. Communication unit 132 of video aggregation device 130d, which is a first video aggregation device, synchronizes call information for video aggregation devices 130a to 130c, which are second video aggregation devices, with synchronization unit 141 of video aggregation management device 140. The synchronization unit of video aggregation management device 140 synchronizes call information with video aggregation devices 130a to 130c, which are second video aggregation devices. Video aggregation devices 130a to 130c output a busy message in response to an incoming call from video aggregation device 130d, and then video aggregation devices 130a to 130c respond to a connection request from video aggregation device 130d.

[0054] FIG. 14 is a sequence diagram of the synchronization operation of the call information. 1 is a sequence diagram showing how state transitions from a call request through call reception and response to call establishment are synchronized between video aggregation devices 130a-130d and video aggregation management device 140 in the call management function. First, when the operator of video aggregation device 130d presses the call join button (step S40), video aggregation device 130d generates call status information indicating the call request and transmits this to video aggregation management device 140 in step S41. Upon acquiring the received call status information, video aggregation management device 140 records in volatile memory in step S42 that a call is being made from video aggregation device 130d to another client, and then broadcasts the call status information to the other video aggregation devices 130a-130c in step S43.

[0055] The video aggregation device 130b at the site that received the call displays a screen indicating an incoming call in step S44, and when the operator completes the answering operation, generates call status information indicating the answer in step S45 and transmits it to the video aggregation management device 140. The video aggregation management device 140 confirms in step S46 that both video aggregation devices 130a and 130b have transitioned to an in-call state, and saves this state in memory and, if necessary, in a database in step S47. Furthermore, in step S48, the call status information indicating the in-call state is redistributed to all video aggregation devices 130a to 130d, and the video aggregation device 130b and the video aggregation device 130d update their screens to an in-call display in step S49 and step S50, respectively, and the video aggregation devices 130a and 130c similarly redraw the call participant lists.

[0056] As such, Figure 14 shows the procedure by which call status information is instantly shared and updated via the video central management device 140 at each stage, including outgoing call request, incoming call display, response, and call establishment, and makes it clear that the present invention ensures high synchronization accuracy and responsiveness even in voice calls across locations.

[0057] Communication unit 132 of video aggregation device 130d, which is the first video aggregation device, synchronizes call information for video aggregation device 130c, which is the second video aggregation device, with synchronization unit 141 of video aggregation management device 140. The synchronization unit of video aggregation management device 140 synchronizes call information with video aggregation device 130c, which is the second video aggregation device. Video aggregation device 130c outputs an incoming call from video aggregation device 130d by displaying a busy call, and then video aggregation device 130c responds to a connection request from video aggregation device 130d.

[0058] According to the present invention, staff can monitor images similar to the aggregated images displayed on a large multi-display while in their respective positions on different floors, and can quickly share information through calls and drawing.

[0059] Previously, video aggregation functions could only be achieved with expensive hardware, but now they can be achieved with software, resulting in lower prices and space savings. In order to use a single multi-vision processor to display multiple images on a large multi-display, the multi-vision processor's housing had to be large, and there was also a limit to the number of images that could be displayed due to restrictions on the hardware specifications.

[0060] As a result of the effects of the present invention, by managing the layout status information of aggregated videos using a video aggregation management device, it is possible to connect multiple video aggregation devices in parallel and display a large number of videos on a large multi-display without relying on the constraints of hardware specifications. Furthermore, the effects of the present invention realize an electronic whiteboard function that allows multiple people to draw text and illustrations on top of the aggregated video, enabling information sharing between remote locations in real time.

[0061] The configuration and effects of the present invention will be described below.

[0062] [1] First and second video aggregation devices (video aggregation devices 130a to 130d) each including a display control unit (131) that receives a plurality of pieces of video information (150a to 150d) via a network line, aggregates the received video information (150a to 150d), and displays the information on a corresponding display device based on assigned display device information (22) indicating which of a plurality of display devices (multi-displays 160a to 160c) the information is to be displayed on; A video aggregation and management device (140), The video central management device (140) a storage unit (149) storing layout state information (21) indicating at which position on the plurality of display devices a video window relating to each of the video information is to be displayed; a synchronization unit that synchronizes the layout state information (21) between the first and second video aggregation devices (video aggregation devices 130a to 130d); and The first video aggregation device (video aggregation device 130a) has a first communication unit (communication unit 132) that transmits and receives data to and from the video aggregation management device (140), The second video aggregation device (video aggregation devices 130b to 130d) has a second communication unit (communication unit 132) that transmits and receives data to and from the video aggregation management device (140). A video aggregation display system (100) characterized by:

[0063] This replaces the previously expensive and large multi-vision processor with a general-purpose computer and software, making it possible to display aggregated images from multiple screens at multiple locations in real time at low cost.

[0064] [2] The display control unit (131) of the first video aggregation device (video aggregation device 130d) displays a first aggregated video on one or more first assigned display devices, The display control unit (131) of the second video aggregation device (video aggregation devices 130a to 130c) displays the second aggregated video on one or more second assigned display devices. 2. The video aggregation display system according to claim 1.

[0065] This allows the display load to be distributed to the video aggregation devices at each location, reducing frame drops, and the display surface can be easily expanded by simply adding more video aggregation devices.

[0066] [3] The display control unit (131) of the first video aggregation device (video aggregation device 130d) outputs a video window that can be moved by a drag-and-drop operation to the first assigned display device, and updates its own layout state information (21) in response to the moving operation of the video window; The first communication unit (communication unit 132) synchronizes the difference in the layout state information (21) with the synchronization unit (141) of the video centralized management device (140), The synchronization unit (141) of the video aggregation management device (140) updates the layout state information (21) stored in the storage unit (149) based on the difference in the layout state information (21) input from the first video aggregation device (video aggregation device 130d), and synchronizes with the second video aggregation device (video aggregation devices 130a to 130c), The display control unit (131) of the second video aggregation device (video aggregation devices 130a to 130c) outputs the video window to the second assigned display device based on the layout state information (21) synchronized with the second communication unit (communication unit 132) by the synchronization unit (141). 3. The video aggregation display system (100) according to claim 2.

[0067] When an operator freely positions the video window by dragging it, the changes are synchronized with all locations in sub-seconds, allowing the situation to be shared quickly without any layout inconsistencies.

[0068] [4] The first video aggregation device (video aggregation device 130d) further includes a drawing execution unit (134) that executes a freehand drawing operation on the aggregated video, The first communication unit (communication unit 132) of the first video aggregation device (video aggregation device 130d) outputs the drawing data executed by the drawing execution unit (134) to the synchronization unit (141) of the video aggregation management device (140); The synchronization unit (141) of the video aggregation management device (140) synchronizes the synchronized drawing data with the second video aggregation device (video aggregation devices 130a to 130c), The display control unit (131) of the second video aggregation device (video aggregation devices 130a to 130c) outputs the drawing data synchronized with the second communication unit (communication unit 132) by the synchronization unit (141) to the second assigned display device. 3. The video aggregation display system (100) according to claim 2.

[0069] This allows text and figures drawn on the video to be instantly shared between remote locations, enabling visual communication like that of an electronic whiteboard, regardless of location.

[0070] [5] The first and second video aggregation devices (video aggregation devices 130a to 130d) further include a microphone for recording audio and a speaker for reproducing audio, The first communication unit (communication unit 132) of the first video aggregation device (video aggregation device 130d) synchronizes call information for the second video aggregation device (video aggregation devices 130a to 130c) with the synchronization unit (141) of the video aggregation management device (140), The synchronization unit (141) of the video aggregation management device (140) synchronizes the call information with the second video aggregation device, The second video aggregation device (video aggregation devices 130a to 130c) outputs the incoming call from the first video aggregation device (video aggregation device 130d). 3. The video aggregation display system (100) according to claim 2.

[0071] Voice calls linked to video annotations can be managed centrally using the same user interface, and speaker information is consistent across all locations, preventing interference and enabling smooth calls.

[0072] [6] The second video aggregation device (video aggregation devices 130a to 130c) outputs a message indicating an incoming call from the first video aggregation device (video aggregation device 130d), and then responds to a connection request from the first video aggregation device (video aggregation device 130d). 7. A video aggregation display system (100) according to claim 6.

[0073] This means that the state transition from the notification of an incoming call to the response is automatically reflected in all devices, ensuring that calls are established reliably without any missed calls.

[0074] [7] receiving a plurality of pieces of video information (150a to 150d) via a network line, aggregating the received video information (150a to 150d), and, based on assigned display device information (22) indicating which of a plurality of display devices (multi-displays 160a to 160c) is to be displayed on which assigned display device, the first video aggregation device (video aggregation device 130d) among first and second video aggregation devices (video aggregation devices 130a to 130d) having a display control unit (131) for displaying on the assigned display device modifies layout state information (21) indicating at which position of the plurality of display devices the video window is to be displayed, by changing the video window; a step in which a first communication unit (communication unit 132) of the first video aggregation device (video aggregation device 130d) outputs the modified layout state information (21) to a video aggregation management device (140); a step in which a synchronization unit (141) of the video aggregation management device (140) having a storage unit (149) storing layout state information (21) indicating at which position of the plurality of display devices a video window related to each of the video information is to be displayed synchronizes the layout state information (21) with the second video aggregation device (video aggregation devices 130a to 130c); a step in which the display control unit (131) of the second video aggregation device (video aggregation devices 130a to 130c) displays a video window based on the synchronized layout state information (21) on a responsible display device; A video aggregation display method comprising:

[0075] This allows the layout change procedure to be defined as a software method, and by combining differential synchronization and history storage, the latest state can be instantly restored even in the event of a failure.

[0076] <<Variation>> The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. It is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0077] The above-described configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware such as an integrated circuit. The above-described configurations, functions, etc. may be realized by software by a processor interpreting and executing a program that realizes each function. Information such as the programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or on a storage medium such as a flash memory card or a DVD (Digital Versatile Disk).

[0078] In each embodiment, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are interconnected. [Explanation of symbols]

[0079] 100 Video Aggregation Display System 160a Multi-Display 160b Multi-Display 160c multi-display 160d display 130a Video consolidation device 130b Video Aggregation Device 130c Video Aggregator 130d Video Aggregator 161a~161d Display 131 Display control unit 132 Communications Department 133 Call Execution Unit 134 Drawing Executive Team 140 Video centralized management device 141 Synchronization Unit 170 Network 180 HUB 150a~150n Video information 142 Call status management department 160 Multi-Display 130 Video Aggregation Device 139 Storage section 21 Layout Status Information 22 Display device information in charge 149 Memory section 110 screens 116 "Video selection" button 117 "Square Frame" button 118 "Freehand" button 119 "Eraser" button 120 "Clear all" button 121 "Operation mode switch" button 122a "Personal" button 122b "Share" button 114a "Pattern switch" button 114b "Pattern switch" button 114c "Pattern switch" button 114d "Pattern switch" button 115a "Update" button 115b "Update" button 115c "Update" button 115d Update button 23 Window Information 231 window number 232 Video number 233 Audio presence / absence flag 234 top left coordinate 235 Lower right coordinate 3 Display screen 31 Video Window 311 Background Layer 312 Video Display Layer 313 User Interface Layer 24 Drawing Information 241 Stroke Number 242 Color Field 243 Thickness Field 244 coordinate fields 245 Bottom Right Coordinate Field 246 Paint Information 25 User call information 251 User ID information 252 Call Status 253 Microphone on / off status 254 participating channels 26 Task Information 261 Source user ID information 262 Requested user ID information 137 Display control unit 148 Synchronization Unit 138 Communications Department

Claims

1. Receive multiple video information via a network line, When displaying on a multi-display, displaying any of the multiple video windows on a display device connected to itself based on layout state information indicating at which positions on the multi-display the multiple video windows generated from the multiple pieces of video information should be arranged, and assigned display device information indicating on which of the multiple display devices constituting the multi-display the multiple video windows should be displayed, a display control unit that, when displaying on a single display, generates a plurality of video windows from the plurality of pieces of video information, aggregates the received video information in accordance with layout state information that indicates at which positions on the single display the plurality of video windows should be arranged, and displays an aggregated image in which the plurality of video windows are aggregated on the assigned display device that is the single display; a first video aggregation device and a second video aggregation device, a video centralized management device; The video centralized management device includes: a storage unit that stores the layout state information; a synchronization unit that synchronizes the layout state information between the first and second video aggregation devices; and the first video aggregation device has a first communication unit that transmits and receives data to and from the video aggregation management device, the second video aggregation device has a second communication unit that transmits and receives data to and from the video aggregation management device; A video aggregation display system characterized by:

2. the display control unit of the first video aggregation device displays a first aggregated video generated from the plurality of pieces of video information on a single first assigned display device that is the single display, or displays a plurality of video windows generated from the plurality of pieces of video information on a plurality of first assigned display devices that are the multi-display, thereby displaying the first aggregated video; The display control unit of the second video aggregation device displays the second aggregated video generated from the plurality of pieces of video information on the single second assigned display device that is the single display, or displays the second aggregated video by displaying a plurality of video windows generated from the plurality of pieces of video information on a plurality of second assigned display devices that are the multi-display.

2. The video aggregation display system according to claim 1.

3. the display control unit of the first video aggregation device outputs a video window that can be moved by a drag-and-drop operation to the first assigned display device, and updates its own layout state information in response to the moving operation of the video window; the first communication unit synchronizes the difference in layout state information with the synchronization unit of the video centralized management device; the synchronization unit of the video aggregation management device updates the layout state information stored in the storage unit based on the difference in the layout state information input from the first video aggregation device, and synchronizes with the second video aggregation device; the display control unit of the second video aggregation device outputs the video window to the second assigned display device based on the layout state information synchronized with the second communication unit by the synchronization unit; 3. The video aggregation display system according to claim 2.

4. The first image aggregation device further includes a drawing execution unit that executes freehand drawing on the aggregated image, the first communication unit of the first video aggregation device outputs drawing data executed by the drawing execution unit to the synchronization unit of the video aggregation management device; the synchronization unit of the video aggregation management device synchronizes the drawing data output to the synchronization unit with the second video aggregation device; The display control unit of the second video aggregation device outputs the drawing data synchronized with the second communication unit by the synchronization unit to the second assigned display device.

3. The video aggregation display system according to claim 2.

5. The first and second video aggregation devices further include a microphone for recording audio and a speaker for reproducing audio; the first communication unit of the first video aggregation device synchronizes call information for the second video aggregation device with the synchronization unit of the video aggregation management device; the synchronization unit of the video central management device synchronizes the call information with the second video central management device; the second video aggregation device outputs the incoming call from the first video aggregation device. The video aggregation display system according to claim 2 .

6. the second video aggregation device outputs a notification of an incoming call from the first video aggregation device, and then responds to a connection request from the first video aggregation device; The video aggregation display system according to claim 5 .

7. receiving a plurality of pieces of video information via a network line; When displaying on a multi-display, a display control unit of the first or second video aggregation device displays any of the multiple video windows on a display device connected to itself based on layout state information indicating at which positions on the multi-display multiple video windows generated from the multiple pieces of video information should be placed, and assigned display device information indicating on which of the multiple display devices constituting the multi-display the multiple video windows should be displayed; When displaying on a single display, a display control unit of the first or second video aggregation device generates a plurality of video windows from the plurality of pieces of video information, aggregates the received video information according to layout state information indicating at which positions on the single display the plurality of video windows should be arranged, and displays an aggregated image in which the plurality of video windows are aggregated on the assigned display device which is the single display; a step of modifying, by the first video aggregation device, layout state information indicating whether the video window is to be displayed on the multi-display or the single display, in accordance with a change in the position of the video window; a step in which a first communication unit of the first video aggregation device outputs the modified layout state information to a video aggregation management device; a synchronization unit of the video aggregation management device having a storage unit that stores the layout state information, synchronizing the modified layout state information with the second video aggregation device; a step in which the display control unit of the second video aggregation device displays a video window based on the modified layout state information synchronized with the video aggregation management device on a display device connected to itself among a plurality of display devices constituting a multi-display, or on a single display device that is a responsible display device; A video aggregation display method comprising:

Images