Real-time Editing System

Abstract

A real-time editing system (1) comprises: a display device (10) whereby operation commands representing edit data processing or video processing are input; a control device (20) that receives an operation command from the display device (10), performs edit data processing, and issues a processing command for commanding video processing; and a processing device (30) that receives the processing command from the control device (20) and performs video processing.

Description

[Technical Field]

[0001] The present invention relates to a real-time editing system. [Background technology]

[0002] Conventionally, editing software that runs on a general computer has been proposed for HD video (resolution 1920 x 1080 pixels). Also, a real-time editing system has been proposed for ultra-high-resolution video of 8K (7680 x 4320 pixels) at 60P (60 frames per second) or 120P (120 frames per second) (Patent Documents 1 and 2). This real-time editing system is composed of multiple decoding devices, compositing devices, and control devices.

[0003] The inventions described in Patent Documents 1 and 2 are complete real-time editing systems for a single video output. In this real-time editing system, editing data (a timeline) that stores raw video in chronological order is created, and editing work is performed by referencing portions of the raw video and cutting and pasting them onto the timeline. In this real-time editing system, when playback processing is performed on the timeline that is the editing result, decoding and compositing processing of the raw video is performed sequentially from the first frame, and the result is output as a video signal.

[0004] In this real-time editing system, the load of decoding and compositing processes increases depending on the resolution and frame rate of the target video, so parallel distributed processing is used. However, this real-time editing system does not perform calculations when playback is not being performed, resulting in excess processing capacity. To address this issue of excess processing capacity, it is thought that an effective solution would be to perform calculations as background processing that does not require user operation when the user is not performing playback operations.

[0005] Furthermore, in this real-time editing system, the user must operate a terminal (for example, a personal computer or a control device) directly connected to the editing system. To address the problem of operating a terminal directly connected to this real-time editing system, a means known as Virtual Desktop Infrastructure (VDI), which allows a remote terminal to be operated from a remote terminal via a network, is considered to be effective.

[0006] Furthermore, an editing system for producing content in a remote environment has also been proposed (Patent Document 3). In this editing system, an editing server that performs editing and an editing client that issues instructions to this editing server work together via a network. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] International Publication No. 2018 / 008076 [Patent Document 2] Japanese Patent Publication No. 2020-042125 [Patent Document 3] Japanese Patent Application Laid-Open No. 2003-309805 Summary of the Invention [Problem to be solved by the invention]

[0008] In the inventions described in Patent Documents 1 and 2, even when using the background processing and VDI, there is still a one-to-one correspondence between real-time editing systems and users. For multiple users to simultaneously perform editing tasks, it is necessary to operate the same number of editing systems as the users, which creates a problem of considerable surplus processing capacity. Furthermore, with VDI, the entire screen operated by the user is sent and received in real time, which generates a huge amount of network traffic and causes problems with poor response to operations.

[0009] Furthermore, the invention described in Patent Document 3 assumes editing processing on a relatively coarse-grained content unit basis, and has the problem of not being able to provide the response required for real-time editing. For example, this conventional technology is premised on editing processing that compiles multiple video works onto a single recording medium, which takes several minutes to several tens of minutes, making it difficult to apply to real-time editing.

[0010] Therefore, an object of the present invention is to provide a real-time editing system that reduces excess processing power and provides good response. [Means for solving the problem]

[0011] In order to solve the above problem, the real-time editing system of the present invention is configured to include a display device into which operation commands representing predetermined editing data processing or video processing are input, a control device that receives the operation commands from the display device, performs editing data processing, and generates processing commands representing video processing, and a processing device that receives the processing commands from the control device and performs video processing.

[0012] The display device comprises an operation command sending means for generating an operation command in response to a user's operation and sending the generated operation command to the control device, an operation response receiving means for receiving an operation response corresponding to the operation command from the control device and displaying it on the display screen, a first video data receiving means for receiving compressed video data that has been subjected to video processing from the control device, a compressed video data restoring means for restoring the compressed video data received by the first video data receiving means, and a video data display means for displaying the video data restored by the compressed video data restoring means on the display screen.

[0013] The control device comprises an operation command receiving means for receiving an operation command from the display device, an edited data processing means for performing edited data processing of the operation command, an operation response sending means for sending the result of the edited data processing as an operation response to the display device, a processing command sending means for generating a processing command representing video processing of the operation command and sending the generated processing command to the processing device, a second video data receiving means for receiving video data from the processing device, a compressed video data generating means for compressing the video data received by the second video data receiving means, and a first video data sending means for sending the compressed video data generated by the compressed video data generating means to the display device.

[0014] The processing device includes a processing command receiving means for receiving a processing command from the control device, a video processing means for performing video processing of the processing command, and a second video data transmitting means for transmitting the video data that has been subjected to the video processing to the control device.

[0015] In other words, in a real-time editing system, UI (User Interface) processing is assigned to the display device, edited data processing to the control device, and video processing to the processing device. This results in a client-server configuration in which the display device acts as the editing client, and the control device and processing device act as the editing server. In this case, the real-time editing system can be considered as a state in which the editing clients indirectly share the processing device, which makes it possible to standardize the processing power of the entire system and reduce excess computing power.

[0016] Furthermore, the real-time editing system sends operation commands as soon as the user operates them and receives the operation responses, which reduces network traffic compared to VDI, which sends and receives the entire screen in real time. [Effects of the Invention]

[0017] According to the present invention, it is possible to reduce excess computing power and obtain good response. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a schematic configuration diagram of a real-time editing system according to a first embodiment. [Figure 2] 1 is a block diagram showing the configuration of a real-time editing system according to a first embodiment. [Figure 3] 1 is a block diagram showing the configuration of a real-time editing system according to a first embodiment. [Figure 4] 4 is a flowchart showing edited data processing in the real-time editing system according to the first embodiment. [Figure 5] 4 is a flowchart showing a video data processing process of the real-time editing system according to the first embodiment. [Figure 6] FIG. 10 is a block diagram showing the configuration of a real-time editing system according to a second embodiment. [Figure 7] FIG. 10 is a block diagram showing the configuration of a real-time editing system according to a third embodiment. [Figure 8] FIG. 10 is a block diagram showing the configuration of a real-time editing system according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. However, each embodiment described below is intended to embody the technical idea of ​​the present invention, and unless otherwise specified, the present invention is not limited to the following. Furthermore, the same means will be given the same reference numerals, and their description may be omitted.

[0020] (First embodiment) [Outline of the real-time editing system] The real-time editing system 1 will be outlined with reference to FIG. 1, the real-time editing system 1 is a client-server type editing system made up of an editing client 2 that issues commands to an editing server 3 (described later) and the editing server 3 that performs editing processing and video processing according to the commands. Specifically, the real-time editing system 1 includes a display device 10 as the editing client 2, and a control device 20 and a processing device 30 as the editing server 3.

[0021] Operation commands representing predetermined editing data processing or video processing are input to the display device 10. Specifically, when a user operates a general UI, the display device 10 transmits the operation commands to the control device 20, receives an operation response, and reflects it in the UI.

[0022] The control device 20 receives operation commands from the display device 10, processes edited data, and generates processing commands that represent video processing. Specifically, the control device 20 operates the edited data in accordance with the operation commands received from the display device. Furthermore, as necessary, the control device 20 transmits processing commands to the processing device 30 and transmits the processing results of the processing device 30 to the display device 10 as operation responses.

[0023] The processing device 30 receives a processing command from the control device 20 and performs video processing. Specifically, the processing device 30 executes the video processing instructed by the processing command received from the control device 20 and transmits the processing result to the control device 20.

[0024] Here, the user operates the real-time editing system 1 via input means such as a mouse provided on the display device 10. The method of operating the display device 10 is similar to that of a general window system, so details will be omitted. In addition, in the real-time editing system 1, communication is established between the devices, and these devices are connected by information sharing means for sending and receiving various commands and responses. Examples of this information sharing means include a network such as Ethernet and inter-process shared memory. In this case, in the real-time editing system 1, the intermediate state of the editing results is held as an arbitrary data structure in the editing data of the control device 20.

[0025] Processing related to video editing can be broadly divided into UI processing, edited data processing, and video processing. Here, UI processing refers to processing related to the UI, such as screen display and mouse operation. Edited data processing refers to processing related to edited data, such as adding or deleting raw video to edited data and playing back edited data. Video processing refers to processing related to raw video, such as decoding and compositing raw video.

[0026] Of these processes, UI processing and edited data processing are executed as needed while the user is editing. In contrast, video editing processing is executed sporadically in response to specific operations such as playing back edited data. In other words, UI processing and edited data processing are executed more frequently than video editing processing. On the other hand, the amount of processing per unit time for video editing processing is overwhelmingly greater than that for UI processing and edited data processing.

[0027] Taking into consideration the characteristics of the above processes, the real-time editing system 1 allocates UI processing to the display device 10, edited data processing to the control device 20, and video processing to the processing device 30. In other words, the real-time editing system 1 is a client-server system in which the display device 10 serves as the editing client 2, and the control device 20 and processing device 30 serve as the editing server 3. With this system configuration, the real-time editing system 1 realizes a state in which multiple editing clients 2 indirectly share a large number of processing devices 30, thereby equalizing the processing power across the entire system and reducing excess computing power.

[0028] Except for operations that require video processing, most of the operation commands are completed by the control device 20. In this case, the network traffic generated between the editing client 2 and the editing server 3 is small, related to the sending and receiving of operation commands, and can be kept to one round trip. Furthermore, when sending video that has undergone video processing to the display device 10, only the video portion needs to be sent, which significantly reduces the amount of communication data compared to sending the entire display screen via VDI. This allows the real-time editing system 1 to significantly improve response during remote operations.

[0029] [Real-time editing system configuration] The configuration of the real-time editing system 1 will be described with reference to FIGS. FIG. 2 shows one display device 10, one control device 20, and one processing device 30 as an example of the minimum configuration of the real-time editing system 1, but the number of each device is arbitrary.

[0030] The control device 20 and the processing device 30 may also be implemented in any manner. As shown in Fig. 2, the control device 20 and the processing device 30 may be implemented on a single piece of hardware, or as shown in Fig. 3, the control device 20 and the processing device 30 may be implemented on separate pieces of hardware. In this embodiment, the control device 20 and the processing device 30 are implemented as a single editing server 3.

[0031] Communication between devices can be broadly divided into sending and receiving commands and responses, and sending and receiving video data. In the former, processing content is used as a command and the processing result is used as a response, and data of about several hundred bytes is sent and received between devices at a time. In this case, inter-process communication using a general remote procedure call (RPC) can be established over shared memory or a network.

[0032] The latter involves transmitting and receiving video data in units of frames of moving images between devices. In this case, one frame of video data can be as large as several tens of megabytes. As a method for transmitting and receiving such large amounts of data, RDMA (Remote Direct Memory Access) using optical communication has been proposed (Patent Document 1). In recent years, RDMA can also be used in common network protocols such as Ethernet (registered trademark) and InfiniBand, so it is sufficient to establish an inter-process communication path using a known protocol.

[0033] <Display device> As shown in FIG. 2, the display device 10 includes an input means 11, an operation command transmitting means 12, an operation response receiving means 13, a video data receiving means (first video data receiving means) 14, a compressed video data restoring means 15, a video buffer 16, a video data display means 17, and a display screen 18.

[0034] The input means 11 is used by the user to input various operations. For example, the input means 11 may be a mouse, a trackball, or a keyboard. The input means 11 also outputs the results of the user's operations to the operation command transmission means 12.

[0035] The operation command transmitting means 12 generates an operation command in response to a user's operation and transmits the generated operation command to the control device 20. Here, the operation command transmitting means 12 transmits the operation command using an RPC. That is, the operation command transmitting means 12 transmits RPC parameters corresponding to each operation command as a packet to the operation command receiving means 21.

[0036] For example, consider the case where a user uses input means 11 to select a raw video and drag and drop it onto the timeline. In this case, operation command transmission means 12 generates an operation command for editing data processing, "register raw video in editing data," and transmits it to control device 20. Also, consider the case where a user uses input means 11 to press a play button. In this case, operation command transmission means 12 generates an operation command representing editing data processing, "start playback of edited data," and transmits it to control device 20. Also, if a user uses input means 11 to designate a specific frame of a raw video and presses a decode button, operation command transmission means 12 generates an operation command representing video processing, "decode a specified frame in the raw video," and transmits it to control device 20.

[0037] The operation response receiving means 13 receives an operation response (e.g., operation completion) corresponding to the operation command from the control device 20 and displays it on the display screen 18. Here, the operation response receiving means 13 receives the operation response using an RPC. That is, the operation response receiving means 13 receives RPC parameters corresponding to each operation response as a packet.

[0038] The video data receiving means 14 receives the compressed video data that has been subjected to video processing from the control device 20. Here, the video data receiving means 14 receives the compressed video data by inter-process communication such as RDMA. Then, the video data receiving means 14 outputs the received compressed video data to the compressed video data restoring means 15.

[0039] The compressed video data restoration means 15 restores the compressed video data received by the video data receiving means 14. In other words, the compressed video data restoration means 15 restores (decodes) the compressed video data input from the video data receiving means 14. Here, known video codecs such as H.264, H.265, and JPEG-XG can be used to compress and restore the video data. This makes it possible to compress, for example, video data of several hundred megabits per second to approximately a few megabits per second, which is 1 / 100th of the original speed. The compressed video data restoration means 15 then stores the restored video data in a video buffer 16.

[0040] The video buffer 16 is a buffer memory that temporarily stores the video data input from the compressed video data restoration means 15. For example, the video buffer 16 may be a general frame memory.

[0041] The video data display means 17 displays the video data restored by the compressed video data restoration means 15 on the display screen 18. Here, the video data display means 17 reads out the video data from the video buffer 16 at a constant frame rate and displays it on the display screen 18.

[0042] The display screen 18 displays the user's operation results (the operation response from the operation response receiving means 13) and the video data that has been subjected to video processing (the video data from the video data display means 17). That is, the display screen 18 displays the results of the video processing (i.e., the reproduced video) as a moving image. For example, the display screen 18 may be a general flat panel display such as a liquid crystal display.

[0043] <Control device> As shown in FIG. 2, the control device 20 includes an operation command receiving means 21, an edited data processing means 22, an operation response transmitting means 23, an edited data storage means 24, a processing command transmitting means 25, a video data receiving means (second video data receiving means) 26, a video buffer 27, a compressed video data generating means 28, and a video data transmitting means (first video data transmitting means) 29.

[0044] The operation command receiving means 21 receives operation commands from the display device 10. Here, the operation command receiving means 21 receives operation commands using RPC. That is, the operation command receiving means 21 receives RPC parameters corresponding to each operation command as a packet from the operation command transmitting means 12. Then, the operation command receiving means 21 outputs the received operation commands to the edited data processing means 22.

[0045] The edited data processing means 22 processes edited data in response to an operation command. For example, in response to an operation command to "register raw footage in edited data," the edited data processing means 22 generates a data structure that references the raw footage and adds it to the edited data in the edited data storage means 24. The edited data processing means 22 then generates an operation response that indicates the result of the edited data processing (processing completion), and outputs the generated operation response to the operation response transmission means 23.

[0046] In addition, in the edited data processing, the specific data structure and processing contents are common, so a detailed description will be omitted. Furthermore, when an operation command representing a video processing operation is input, the edited data processing means 22 outputs the operation command to the processing command transmitting means 25 .

[0047] The operation response sending means 23 sends the result of the edited data processing as an operation response to the display device 10. Here, the operation response sending means 23 sends the operation response using an RPC. That is, the operation response sending means 23 sends RPC parameters corresponding to each operation response as a packet.

[0048] The edited data storage means 24 is a memory that stores edited data (timeline). For example, the edited data storage means 24 may be a storage medium such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0049] The processing command sending means 25 generates a processing command that indicates the video processing process of the operation command, and sends the generated processing command to the processing device 30. In other words, the processing command sending means 25 interprets the operation command input from the edited data processing means 22 and generates a processing command. Here, the processing command sending means 25 can send the processing command using an RPC, just like the operation command. This processing command is a command that indicates the content of the video processing, and corresponds to the rendering job in Patent Document 1.

[0050] The video data receiving means 26 receives video data from the processing device 30. The video data receiving means 26 then stores the received video data in a video buffer 27.

[0051] The video buffer 27 is a buffer memory that temporarily stores the video data input from the video data receiving means 26. For example, the video buffer 27 may be a general frame memory.

[0052] The compressed video data generation means 28 compresses the video data received by the video data receiving means 26. Here, the compressed video data generation means 28 reads the video data from the video buffer 27 and compresses it sequentially. At this time, the compressed video data generation means 28 can use the same video codec as the compressed video data restoration means 15. Then, the compressed video data generation means 28 outputs the generated compressed video data to the video data transmission means 29.

[0053] The video data transmission means 29 transmits the compressed video data generated by the compressed video data generation means 28 to the display device 10. Here, the video data transmission means 29 transmits the compressed video data by inter-process communication such as RDMA.

[0054] <Processing equipment> As shown in FIG. 2, the processing device 30 includes a processing command receiving means 31, a video processing means 32, and a video data transmitting means (second video data transmitting means) 33.

[0055] The processing command receiving means 31 receives processing commands from the control device 20. Here, the processing command receiving means 31 can receive processing commands using RPCs, similar to operation commands. Then, the processing command receiving means 31 outputs the received processing commands to the video processing means 32.

[0056] The video processing means 32 performs video processing in accordance with the processing command input from the processing command receiving means 31. In other words, the video processing means 32 executes the video processing specified in the processing command and generates video data (e.g., 1920 x 1080 pixel YUV data) that is the processing result. For example, in response to a processing command to "decode a specified frame in the raw video," the video processing means 32 generates video data obtained by decoding the frame specified in the raw video. The video processing means 32 then outputs the generated video data to the video data transmitting means 33. Since the image processing is a common technique, further detailed explanation will be omitted.

[0057] The video data transmission means 33 transmits the video data that has been subjected to video processing to the control device 20. In other words, the video data transmission means 33 transmits the video data input from the video processing means 32 to the control device 20.

[0058] As described above, the real-time editing system 1 can realize a real-time video editing function in which the results of the user's operations are reflected in the edited data, and the video resulting from the video processing can be viewed on the display screen 18.

[0059] [Real-time editing system operation] <Editing data processing> The editing data processing by the real-time editing system 1 will be described with reference to FIG. Here, it is assumed that the user has performed an operation for editing data processing. In this case, in step S1, the display device 10 generates an operation command for editing data processing in response to the operation by the user.

[0060] In step S2, the display device 10 transmits the operation command generated in step S1 to the control device 20. In step S3, the control device 20 performs editing data processing of the operation command received from the display device 10. In step S4, the control device 20 transmits the result of the edited data processing to the display device 10 as an operation response. In step S5, the display device 10 displays on the display screen 18 an operation response corresponding to the operation command received from the control device 20.

[0061] As described above, most of the operation commands are for editing data processing, which is completed between the display device 10 and the control device 20. Therefore, the network traffic required between the editing client 2 and the editing server 3 can be reduced to a small amount related to sending and receiving the operation commands, and can be reduced to one round trip.

[0062] <Video data processing> The video data processing performed by the real-time editing system 1 will be described with reference to FIG. Here, it is assumed that the user has performed an operation for the video data processing. In this case, in step S10, the display device 10 generates an operation command for the video data processing in response to the user's operation.

[0063] In step S11, the display device 10 transmits the operation command generated in step S10 to the control device 20. In step S12, the control device 20 interprets the operation command received from the display device 10 and generates a processing command. In step S13, the control device 20 transmits the processing command generated in step S12 to the processing device 30.

[0064] In step S14, the processing device 30 performs the image processing process in accordance with the processing command received from the control device 20. In step S15, the processing device 30 transmits the video data that has been subjected to the video processing to the control device 20.

[0065] In step S16 , the control device 20 stores the video data received from the processing device 30 in the video buffer 27 . In step S17, the control device 20 reads the video data from the video buffer 27 and compresses it. In step S18, the control device 20 transmits the compressed video data compressed in step S17 to the display device 10.

[0066] In step S19, the display device 10 decompresses the compressed video data received from the control device 20. In step S20, the display device 10 stores in the video buffer 16 the video data restored in step S19. In step S21, the display device 10 reads the video data from the video buffer 16 and displays it on the display screen 18.

[0067] [Actions and Effects] As described above, in the real-time editing system 1, UI (User Interface) processing is assigned to the display device 10, edited data processing to the control device 20, and video processing to the processing device 30. This results in a client-server configuration in which the display device 10 serves as the editing client 2, and the control device 20 and processing device 30 serve as the editing server 3. In this case, the real-time editing system 1 can be considered to be in a state in which the editing client 2 indirectly shares the processing device 30, making it possible to equalize the processing power of the entire system and reduce excess computing power.

[0068] For example, in the real-time editing system 1, the display device 10 may be implemented in a personal computer with low processing power, and the control device 20 and processing device 30 may be implemented in a high-performance workstation. In this case, the real-time editing system 1 can be used in such a way that raw footage and editing data are collectively managed on the workstation, and the user connects to the workstation only during editing work.

[0069] Furthermore, because the real-time editing system 1 sends operation commands and receives operation responses when the user performs an operation, it can reduce network traffic compared to sending and receiving the entire screen in real time, as in VDI.In other words, the real-time editing system 1 does not send each mouse operation and receive screen data as in VDI, but only sends an operation command when a change to the editing data is required as a result of mouse operation, etc., which is expected to significantly improve operation response.

[0070] Furthermore, in the real-time editing system 1, the compressed video data sent from the control device 20 to the display device 10 is smaller than the original video data, further reducing network traffic, further improving operational response, and enabling use over slower networks. In particular, the real-time editing system 1 has good operational response even when the display device 10 and control device 20 in remote locations are connected via a wide area network (WAN), making it possible to realize, for example, editing work from home or collaborative editing work between multiple locations.

[0071] Furthermore, because the real-time editing system 1 transmits and receives video data compressed using a video codec, there is no need to transmit and receive UI components (e.g., buttons and list boxes) on the display screen 18, allowing users to perform editing tasks with image quality nearly equivalent to that of the original video data. In contrast, with VDI, UI components on the screen displayed on the virtual desktop are also transmitted and received, generating a huge amount of network traffic and often reducing the gradation of the screen. This makes it difficult for users to perform editing tasks with image quality nearly equivalent to that of the original video data.

[0072] (Second embodiment) [Real-time editing system configuration] With reference to FIG. 6, the configuration of a real-time editing system 1B according to the second embodiment will be described with respect to differences from the first embodiment. As shown in FIG. 6, the real-time editing system 1B includes a plurality of display devices 10, control devices 20, and processing devices 30.

[0073] In this embodiment, the editing server 3B is made up of multiple pairs of control devices 20 and processing devices 30. The editing server 3B receives operation commands from each display device 10 and processes edited data. In the example of Fig. 6, the editing server 3B is made up of three pairs of control devices 201 to 203 and processing devices 301 to 303. In this case, there is a one-to-one correspondence between the control device 201 and the processing device 301, a one-to-one correspondence between the control device 202 and the processing device 302, and a one-to-one correspondence between the control device 203 and the processing device 303.

[0074] The editing client 2B includes a plurality of display devices 10. The number of display devices 10 may be greater than the number of control devices 20. In the example of Fig. 6, the editing client 2B includes four display devices 101-104.

[0075] A user operates the display device 10 to connect to the control device 20 and performs video editing work using the same procedure as in the first embodiment. Here, the correspondence between the display device 10 and the control device 20 is not fixed, and the display device 10 remains connected to the control device 20 only while the editing work continues. Specifically, the user logs in to one of the control devices 20 from the display device 10. In the example of FIG. 6, the user of the display device 101 logs in to the control device 201 if the control device 201 is not in use. The processing itself of the display device 10, the control device 20, and the processing device 30 is the same as in the first embodiment, and therefore further explanation will be omitted.

[0076] [Actions and Effects] As described above, the real-time editing system 1B can reduce excess processing power and achieve good response, similar to the first embodiment. Furthermore, the real-time editing system 1B can significantly reduce excess processing power because the entire processing power of the editing server 3B can be shared among multiple users. In other words, the real-time editing system 1B can achieve the same processing power at a lower cost than if separate editing devices were prepared for each user.

[0077] (Third embodiment) [Real-time editing system configuration] With reference to FIG. 7, the configuration of a real-time editing system 1C according to the second embodiment will be described with respect to differences from the second embodiment. 7, the editing server 3C further includes a processing device selection means 40 that assigns the video processing of the processing command transmitted from each control device 20 to one of the processing devices 30. This allows the control device 20 to use any of the processing devices 30 in the real-time editing system 1C.

[0078] The editing server 3C includes processing devices 30 equal to or greater than the number of control devices 20. In the example of FIG.

[0079] The processing device selection means 40 is disposed between the control devices 20 and the processing devices 30. The processing device selection means 40 distributes the video processing in the processing devices 30 by allocating processing commands from each control device 20 to one of the processing devices 30. Here, the processing device selection means 40 may perform fine-grained load balancing in units of a single processing command, or may perform coarse-grained load balancing over periods ranging from a few seconds to a few minutes. For example, the processing device selection means 40 performs load balancing using a typical round-robin method. This allows the real-time editing system 1C to evenly execute processing commands sent sporadically from multiple control devices across multiple processing devices, thereby evenly distributing the computational load throughout the system.

[0080] [Actions and Effects] As described above, the real-time editing system 1C can reduce excess processing power and achieve good response, similar to the second embodiment. Furthermore, the real-time editing system 1C can average out the load of video processing, which changes from moment to moment, in response to the user's editing operations, thereby reducing excess calculation processing in the entire system.

[0081] (Fourth embodiment) [Real-time editing system configuration] With reference to FIG. 8, the configuration of a real-time editing system 1D according to the fourth embodiment will be described, focusing on differences from the first embodiment. As shown in FIG. 8, the method of compressing and restoring video data in a display device 10D and a control device 20D differs from that in the first embodiment. In FIG. 8, for ease of understanding, components not directly related to the compression and decompression of video data are omitted from the illustration.

[0082] Well-known video codecs such as H.264, H.265, and JPEG-XG can be broadly divided into two types: Intra methods, which perform compression by utilizing the data correlation between pixels within a frame, and Long GOP (Group of Picture) methods, which perform compression by utilizing the data correlation between multiple frames.

[0083] With the Intra method, the codec's input and output correspond one-to-one, so the latency (delay time) associated with compression and decompression is limited to the computational processing time for compression and decompression. In contrast, the Long GOP method stores multiple frames of data inside the codec before performing computations, which means that latency increases in proportion to the amount of data stored. On the other hand, the Long GOP method has an advantage over the Intra method in terms of compression efficiency.

[0084] In real-time video editing, operations such as starting and stopping playback and jumping to specific frames are frequently performed, so the Intra method is preferable as it prioritizes operational response and can be expected to have low latency.On the other hand, in cases where playback will continue for a long period of time, the Long GOP method is preferable for the sake of reducing data volume.

[0085] Therefore, the compressed video data generation means 28D compresses the video data using the Intra method for several frames after the playback start operation. Then, after video data sufficient to absorb the latency of the Long GOP method has been accumulated in the video buffer 16, the compressed video data generation means 28D compresses the video data using the Long GOP method.

[0086] As shown in FIG. 8, the compressed video data generating means 28D includes an encoder selecting means 280, an Intra encoder 281, and a Long encoder 282. The encoder selection means 280 selects whether to compress video data using the Intra method or the Long GOP method. Specifically, the encoder selection means 280 selects the Intra method from the playback start frame up to a preset number of frames. Then, the encoder selection means 280 selects the Long GOP method after the preset number of frames has been exceeded from the playback start frame.

[0087] The Intra encoder 281 compresses video data in the Intra format based on the selection result of the encoder selection means 280 . The Long encoder 282 compresses the video data in the Long GOP format based on the selection result of the encoder selection means 280 . As in the first embodiment, the video data transmitting means 29 transmits the compressed video data generated by the compressed video data generating means 28D to the video data receiving means 14.

[0088] The compressed video data restoration means 15D includes a decoder selection means 150, an Intra decoder 151, and a Long decoder 152. The decoder selection means 150 selects whether to decompress compressed video data using the Intra method or the Long GOP method. Here, like the encoder selection means 280, the decoder selection means 150 selects either the Intra method or the Long GOP method.

[0089] The Intra decoder 151 restores the compressed video data in the Intra format based on the selection result of the decoder selection means 150 . The Long decoder 152 restores the compressed video data in the Long GOP format based on the selection result of the decoder selection means 150 . The video data restored by the compressed video data restoration means 15D is stored in the video buffer 16, as in the first embodiment.

[0090] [Actions and Effects] As described above, the real-time editing system 1D can achieve both good operational response and a reduction in data volume.

[0091] (Variation) Although the embodiments have been described in detail above, the present invention is not limited to the above-described embodiments, and includes design modifications and the like within the scope of the present invention.

[0092] In the third embodiment, all control devices are connected to a single processing device selection means, but this is not limiting. For example, each control device may be provided with a processing device selection means, and processing commands may be sent from each processing device selection means to one of the processing devices. Also, for example, multiple editing servers may be provided, and processing commands may be sent to any processing device provided in any editing server. In this way, the real-time editing system can adopt a scalable configuration depending on the balance between the scale of the entire system and the processing power of the editing server. [Explanation of symbols]

[0093] 1,1B,1C,1D real-time editing system 2, 2B, 2C editing client 3, 3B, 3C editing server 10,101~104,10D Display device 11 Input Methods 12 Operation command transmission means 13 Operation response receiving means 14 Video data receiving means (first video data receiving means) 15,15D Compressed video data restoration means 16 video buffers 17 Video data display means 18 Display screen 20,201~203,20D Control device 21 Operation command receiving means 22 Editing data processing means 23 Operation response sending method 24 Editing data storage means 25 Processing command transmission means 26 Video data receiving means (second video data receiving means) 27 Video Buffer 28,28D Compressed video data generation means 29 Video data transmission means (first video data transmission means) 30, 301~303 Processing equipment 31 Processing command receiving means 32 Image processing means 33 Video data transmission means (second video data transmission means) 40 Processing device selection means 150 decoder selection means 151 Intra decoder 152 Long Decoder 280 Encoder selection means 281 Intra Encoder 282 Long Encoder

Claims

1. A real-time editing system comprising: a display device to which an operation command representing a predetermined editing data process or video processing process is input; a control device that receives the operation command from the display device, performs the editing data process, and generates a processing command representing the video processing process; and a processing device that receives the processing command from the control device and performs the video processing, The display device includes: an operation command transmitting means for generating the operation command in response to an operation by a user and transmitting the generated operation command to the control device; an operation response receiving means for receiving an operation response corresponding to the operation command from the control device and displaying the operation response on a display screen; a first video data receiving means for receiving the compressed video data that has been subjected to the video processing from the control device; a compressed video data restoring means for restoring the compressed video data received by the first video data receiving means; a video data display means for displaying the video data restored by the compressed video data restoration means on the display screen, The control device an operation command receiving means for receiving the operation command from the display device; edited data processing means for processing edited data of the operation command; an operation response transmitting means for transmitting a result of the edited data processing to the display device as the operation response; a processing command transmitting means for generating the processing command representing the image processing processing of the operation command and transmitting the generated processing command to the processing device; a second video data receiving means for receiving the video data from the processing device; a compressed video data generating means for compressing the video data received by the second video data receiving means; a first video data transmission means for transmitting the compressed video data generated by the compressed video data generation means to the display device; The processing device includes: a processing command receiving means for receiving the processing command from the control device; a video processing means for performing video processing according to the processing command; a second video data transmission means for transmitting the video data that has been subjected to the video processing to the control device.

2. the operation command transmitting means and the operation command receiving means transmit and receive the operation command using a remote procedure call; the operation response sending means and the operation response receiving means send and receive the operation response using a remote procedure call; 2. The real-time editing system according to claim 1, wherein said processing command transmitting means and said processing command receiving means transmit and receive said processing commands using a remote procedure call.

3. 2. The real-time editing system according to claim 1, wherein the first video data transmitting means and the first video data receiving means transmit and receive the compressed video data by inter-process communication.

4. The display device is a plurality of devices, an editing server configured with a plurality of sets of the control device and the processing device, 2. The real-time editing system according to claim 1, wherein the editing server receives the operation command from each display device and processes the edited data.

5. 5. The real-time editing system according to claim 4, wherein said editing server further comprises a processing device selection means for allocating the video processing of the processing command transmitted by said control device to one of said processing devices.

6. 5. The real-time editing system according to claim 4, comprising a plurality of said editing servers.

7. the compressed video data generating means compresses the video data in an Intra format from a playback start frame to a preset number of frames, and compresses the video data in a Long GOP format after the number of frames from the playback start frame has been exceeded; 2. The real-time editing system according to claim 1, wherein the compressed video data restoration means restores the compressed video data from the playback start frame to the number of frames in an Intra format, and restores the compressed video data from the playback start frame to the number of frames in a Long GOP format after the number of frames has been exceeded.

Images