Information processing device, information processing program and recording medium

Abstract

To simplify and accelerate video content production and to reduce its cost by using a network camera.SOLUTION: An information processing device of the present invention is an information processing device that outputs video content using video from a plurality of video sources including a network camera as material, and includes: switching means for performing switching on the basis of a video signal that has been transferred without a transmission node for a video source confirming reception at a reception node; playlist generation means for generating a playlist on the basis of a video source that has been stitched and information on timing; subject tracking means for detecting a moving object in an image on the basis of the video signal that was transferred without the transmission node confirming the reception at the reception node with a PTZ-enabled network camera, and performing control of and tracking the network camera so that the moving object is continuously captured near the center of a screen; and information processing means for performing distribution in a transfer scheme that continues to re-transmit the video content until an acknowledgement at the reception node is confirmed.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing device that performs information processing related to the production or distribution of video content made from video images from a plurality of video sources. [Background technology]

[0002] Generally, video content such as TV programs is created by displaying video signals captured by multiple imaging devices from different directions (angles of view) of the subject on a monitor screen, and then using a switcher (video switching means) to switch (switch) between the video images to be used in the video content for viewers and the video images that will not be used by operating a switch button, etc.

[0003] Network cameras that transmit video signals via computer networks such as the Internet or LANs have become widespread. In recent years, video cameras and digital cameras for business, broadcasting, and general photography have also appeared that can transmit the captured video via computer networks.

[0004] Therefore, there is a need to use these network cameras (cameras that can transmit video via a computer network) to create video content by switching video sources, thereby reducing the effort required for line setup and equipment costs, and making video content production easier, faster, and less costly.

[0005] In recent years, HTTP streaming, which transmits video in accordance with the HyperText Transfer Protocol (HTTP), has become the mainstream method of video distribution over the Internet. HTTP streaming segments (divides) video data into Transport Stream (TS) files of very short duration (for example, 10 seconds), transfers the segmented data together with a playlist (in formats such as M3U or MPD files) that instructs playback, and then plays the received TS files continuously on the viewer's device according to the playlist. HTTP streaming includes formats such as HLS (HTTP Live Streaming) and MPEG-DASH (Dynamic Adaptive Streaming over HTTP).

[0006] There are two methods for transmitting data on a computer network: TCP (Transmission Control Protocol), which continues to retransmit the same data (packet) until the receiving node acknowledges receipt, and UDP (User Datagram Protocol), which allows the sending node to transfer data without acknowledging receipt by the receiving node. HTTP streaming uses TCP (Transmission Control Protocol) for data transfer, which reduces degradation of signal quality due to data transfer. It is also the same transfer method as general file transfers on web servers, which can reduce distribution costs.

[0007] Patent Document 1 discloses a camera system technology that has a plurality of camera devices that packetize and output captured video signals, and a relay device that is connected to the plurality of camera devices and relays the video signals transmitted from the plurality of camera devices, where the relay device has a receiving unit that receives each video signal synchronized between the plurality of camera devices based on a synchronization signal, and a switch unit that selects and outputs the video signals output from the plurality of camera devices based on a control signal for selecting video, thereby making it possible to select a desired video from videos from the plurality of camera devices and simplifying the system configuration.

[0008] Patent Document 2 also discloses technology for a video transmission device and monitor system that can be applied to an IP-based broadcasting system, by comprising a video transmission device that receives an uncompressed video signal and generates an IP packet stream of uncompressed video data while simultaneously compressing the uncompressed signal and generating an IP packet stream of compressed video data; a video distribution system that distributes an IP packet stream of uncompressed video data from among the IP packet streams received from multiple video transmission devices as high-quality video while simultaneously distributing an IP packet stream of compressed video data to a monitor system for video monitoring; and a monitor system that selects an IP packet stream of the compressed video data to display on a monitor and displays it on the monitor.

[0009] Patent document 3 also discloses a technology for an information processing device that can allocate video sources, including imaging devices, and camerawork to portions of input text, thereby determining the composition and how to switch between video sources, including multiple cameras, by looking at text (characters and sentences) that indicate the content structure, such as a script or interview notes.

[0010] Patent document 4 also discloses technology for a video streaming distribution device that generates a clip information file that identifies a video clip to be played back extracted from streaming data and a playback instruction file that instructs the playback of multiple segment files that make up the clip, edits the clip information file that includes the playback instruction file, and transfers the clip information file to a public site, thereby making it possible to publish on the Web a clip information file that includes a playlist that defines clips corresponding to highlight scenes while live streaming the video, for example, in a live sports broadcast.

[0011] Patent document 5 also discloses a content distribution technology that includes a playback means for playing a video file containing multiple TS packets, a UDP packet receiving means for decomposing the video file containing the multiple TS packets and receiving UDP packets transmitted via IP multicast, and a video file reconstruction means for reconstructing the video file from the UDP packets received by the UDP packet receiving means and supplying it to the playback means in accordance with HTTP, wherein the video file reconstruction means, if it does not receive at least one UDP packet out of the multiple UDP packets corresponding to the video file, reconstructs the video file in a form including only complete TS packets, thereby using IP multicast communication when distributing video in accordance with HTTP and further reducing playback loss as much as possible. [Prior art documents] [Patent documents]

[0012] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-62311 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-9881 [Patent Document 3] Japanese Patent Application Laid-Open No. 2012-14420 [Patent Document 4] Patent No. 5909546 [Patent Document 5] Patent No. 6490284 Summary of the Invention [Problem to be solved by the invention]

[0013] To transfer video signals from a network camera in real time, the Real-time Transport Protocol (RTP), based on the User Datagram Protocol (UDP), is used. This protocol transfers data without the sending node having to confirm receipt by the receiving node. However, because RTP communication does not retransmit data if it does not reach the other party, if a packet loss occurs on the network, where part of the transferred data is lost, the video and audio quality will be degraded. Therefore, this type of data cannot be used as is as the material for video content to be viewed by viewers. On the other hand, TCP / IP (Transmission Control Protocol / Internet Protocol), which is the standard protocol used in computer networks, prioritizes data quality and is therefore inferior in real-time performance (timing is off), meaning that video data transferred via TCP / IP is not suitable as live view video for switching or camerawork control. In addition, when a user selects PGM and PST using live view video and instructs them to be switched using a transition (a gradual switch accompanied by an effect), there was a problem in that it was not possible to output (produce or distribute) video content that reflected the transition in high-quality video data.

[0014] However, none of the prior art, including the techniques disclosed in the above documents, has solved these problems. The present invention has been made to solve the above problems, and aims to provide an information processing device that is suitable for outputting (producing or distributing) video content by performing "transitions," which are gradual switching with effects between images from multiple video sources, including network cameras. [Means for solving the problem]

[0015] The information processing device of the present invention is an information processing device that performs information processing to output (produce or distribute) video content using video from multiple video sources including a network camera as material, and has: a switching means that selects PGM and PST based on video signals transferred by a transfer method in which the node transmitting from the video source transfers data without confirming reception by the receiving node; a processing / combining means that processes and combines the PGM and PST selected by the switching means with a user-specified effect (switching effect); and an information processing means that outputs (produces or distributes) the video processed and combined by the processing / combining means by a transfer method that continues to retransmit the same data (packets) until it confirms reception notification from the node receiving the video. [Effects of the Invention]

[0016] According to the present invention, an information processing device can be realized that is suitable for outputting (producing or distributing) video content by performing "transitions," which are gradual switching with effects between videos from multiple video sources, including network cameras. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a block diagram showing a configuration of a main part of an information processing device according to an embodiment of the present invention; [Figure 2] FIG. 1 is a diagram illustrating an example of a mode in which an information processing device according to an embodiment is used. [Figure 3] 1 is a block diagram showing a functional module configuration of an information processing apparatus according to an embodiment of the present invention; [Figure 4] FIG. 2 is a block diagram showing a functional module configuration of the switcher terminal. [Figure 5] 10A and 10B are diagrams illustrating an example of a web application screen that the information processing device according to the embodiment displays on a touch panel display of a switcher terminal. [Figure 6]10 is a diagram showing an example of a camera operation panel that the information processing device according to the embodiment displays on a touch panel display of a switcher terminal. FIG. [Figure 7] 10 is a flowchart showing a flow of a process in which the information processing apparatus according to the present embodiment generates and outputs sequence data. [Figure 8] 10 is a flowchart illustrating an example of information processing in which the information processing device according to the present embodiment generates and distributes a playlist based on information regarding which video source was switched at what timing. [Figure 9] FIG. 10 is a diagram showing an example of a programmed camerawork panel that is displayed in response to pressing of programmed camerawork button 523. [Figure 10] 10 is a flowchart showing a processing flow relating to a programmed camerawork function. [Figure 11] FIG. 10 is a diagram illustrating the "size" of a shot. [Figure 12] 10A and 10B are diagrams illustrating an example of sequence data output by the information processing device according to the present embodiment. [Figure 13] FIG. 10 is a diagram showing an example of a playlist. [Figure 14] FIG. 10 is a diagram illustrating an example of EDL data. [Figure 15] 1 is a block diagram showing the configuration of a main part of a network camera that can be connected to an information processing device according to the present embodiment. [Figure 16] 2 is a block diagram showing a configuration of functional modules related to content reception and playback of a viewer terminal connectable to the information processing device according to the present embodiment. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0018] Throughout this specification, "switching" refers to switching a video source (or a video signal from a video source), and includes "cut switching," which is an instantaneous switch, and "transition," which is a switch that takes a certain amount of time.

[0019] "PTZ" is a general term for "pan," which moves the camera's angle of view horizontally, "tilt," which moves the camera's angle of view vertically, and "zoom," which expands and contracts the angle of view. Additionally, switching (cutting and transitions) and PTZ control (control of pan, tilt, and zoom) are collectively called "camera work."

[0020] Also, a unit of footage shot continuously from start to finish with one camera without any starts or stops in between is called a "shot." Furthermore, a single sequence of footage made up of shots or similar units switched and combined for a single scene is called a "scene." Also, a single recording of a scene with the same structure is called a "take."

[0021] Furthermore, video productions that are made available for viewing by viewers are referred to as "programs," "program content," "video content," etc. Furthermore, the number indicating the order in which scenes appear in the "video content" is called a "scene number." Also, the video of the video source being switched is called "PGM." Also, the video of the video source that is switched (before switching) is called "PST."

[0022] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

[0023] FIG. 1 is a block diagram showing the main configuration of an information processing apparatus 100 according to an embodiment of the present invention. The information processing device 100 includes a CPU 110 , a RAM 120 , a ROM 130 , a display 160 , an operation unit 170 , and a network I / F 190 , which are interconnected via an internal bus 180 .

[0024] Reference numeral 110 denotes a CPU, which controls the operation of the information processing device 100 itself (the operation of each unit for performing its functions), generates signals for controlling the operation of the video source, etc., and processes various data.

[0025] The RAM 120 is a writable, high-speed storage device into which the OS, various programs, and various data are loaded, and is also used as a working area for the OS and various programs.

[0026] The ROM 130 is a non-volatile read-only storage device, and is used as a permanent storage area for the BIOS, various basic programs, and various basic data.

[0027] The display 160 is a display device such as an LCD (liquid crystal display panel).

[0028] The operation unit 170 is a keyboard, a mouse, a joystick, a remote controller, a touch panel (including a touch display), or the like.

[0029] With a touch display, a user can operate the device and perform various functions by using touch gestures such as tapping (touching down and then touching up) or flicking (moving quickly while touching and then touching up) on GUI (Graphical User Interface) components such as icons with their fingers or a stylus pen.

[0030] The network I / F 190 is an I / F (interface) for connecting to a network, and enables connection to video sources such as a network camera 1500, a viewer terminal 1600, and a switcher terminal 400 via the network. The network includes, for example, the Internet, a wired LAN, a wireless LAN, a mobile phone network, and the like.

[0031] Fig. 2 is a diagram showing the overall configuration of a content output system including an information processing device according to an embodiment of the present invention. In Fig. 2, reference numeral 100 denotes an information processing device that performs content output (production or distribution), distributes video content to a viewer terminal 1600 via a network, and provides a switcher terminal 400 with a content production function using switching. The viewer terminal 1600 and the switcher terminal 400 are information terminals such as personal computers (hereinafter referred to as "PCs"), tablets, and smartphones.

[0032] In the example shown in the figure, an "event" (210) such as a ceremony, sport, game, musical performance, scientific experiment, news report, social information, meeting, drama, etc. is photographed from different directions (angles of view) by multiple cameras 1500-1, 1500-2, etc.

[0033] The switcher terminal 400 has, for example, a touch panel display in which a display and a touch panel are integrated, and accepts GUI operations by a user (content creator). The operation screen (GUI) displayed on this touch panel display is displayed as a web application on a web browser started on the switcher terminal 400 by the web server function of the information processing device 100. Video sources such as network cameras 1500 are connected to the switcher terminal 400 via a network, and device control such as data access and pan / tilt is possible after authentication using the device's IP address, user name, and password. The content creator produces video content by switching video sources based on video signals (real-time live view) transferred via UDP from the video sources to the switcher terminal 400 while progressing the event with cues (signals that trigger the event), etc. At this time, sequence data is generated that records which video sources were switched at what timing, and is transferred and saved to the RAM 120 of the information processing device 100.

[0034] Meanwhile, video data (video clips) from video sources such as cameras 1500-1, 1500-2, etc. are first stored (including cache or buffering) in storage means (storage media such as RAM or SD card) that the video sources have. When the video source is switched, the stored video data (video clip) is segmented into segment data (TS files), which are then transferred and saved in a specific storage area (for example, a folder on online storage).Function modules for data segmentation, transfer, saving, etc., are installed in the storage means (or the control means that manages it) as needed. Alternatively, if the video source itself has HTTP streaming functionality, segment data (TS files) are generated and stored within each video source. The time code and other time information attached as metadata to the segment data (TS files) is then referenced and compared with the timing at which the video source was switched, and the segment data (TS files) related to the switching timing is transferred and saved to a specific storage area (for example, a folder on online storage).

[0035] Next, the information processing device 100 according to an embodiment of the present invention generates a playlist (an index file such as an M3U file or an MPD file) that contains the file path and duration of the segment data relating to the timing at which the video source was switched, and distributes it to the viewer terminal 1600 (registers the playlist in a distribution directory). The viewer terminal 1600 references the playlist (M3U file, MPD file, etc.) and requests segment data, thereby performing streaming distribution of video content obtained by switching multiple video sources.

[0036] It is also possible to record and store segment data generated from video of each video source in the video source itself or in a recording device (recording media) connected to it without going through a computer network, and then distribute (acquire) the data from there to the viewer terminal 1600. In this case, the latency (delay) until the segment data can be acquired by the viewer terminal 1600 or the like is improved (delay is reduced).

[0037] As mentioned above, UDP transmission prioritizes real-time performance, resulting in reduced data quality (noise). Therefore, it is not appropriate to distribute UDP-transferred video signals directly to viewers or use them as material for creating video content. Therefore, the switcher terminal 400 performs switching while allowing users to monitor the progress of an event in real time using the video signals (live view) transmitted via UDP from each video source. It then generates and outputs "sequence data" that indicates which video data (video sources) were switched at what timing. This sequence data includes the IDs (identifiers) of the video data (video clips) transmitted from the video sources and is associated with the video data. It also contains information about which video data (video sources) were switched at what timing, based on the time codes of the in-point (the start point of switching insertion) and out-point (the end point of insertion). In other words, it allows users to reference the video data portions used as shots in scenes of video content. The user (content creator) can edit the video data without any loss of quality based on the above sequence data on a video editing terminal 240 connected to the recording medium 230, and add musical effects, subtitles, visual effects, etc. as appropriate to complete the video content that can be viewed by viewers.

[0038] Furthermore, the information processing device 100 according to this embodiment generates a playlist that lists file paths to video segment data at the timing when the video source is switched. Because TCP is used to transfer and distribute the segment data, data quality is ensured, and high-quality video and audio without noise is distributed to the viewer terminal 1600. In other words, the information processing device 100 according to this embodiment switches video sources based on live view video transferred via UDP from multiple video sources, including network cameras, generates a playlist for content distribution based on information about which video source was switched and when, and then, based on the generated playlist, the video data of the switched video source is acquired by the viewer terminal 1600 via TCP transfer.

[0039] By adopting the above-described configuration, the information processing device 100 according to this embodiment can output (produce or distribute) video content made from video data with little degradation in quality due to network transmission, while switching video sources in approximately real time in line with the progress of an "event" such as a ceremony such as a wedding or funeral, or a drama (play), and make it available for viewing by viewers. In recent years, WebRTC (Web Real-Time Communication) has emerged as a technology for real-time video communication using UDP in web browsers, and it is also possible to use WebRTC as a method for transferring live view video from the video source for switching. Also, if the web browser cannot display a video stream via UDP, it can send the ONVIF-specified snapshot acquisition command to the network camera to continuously acquire still images (JPEG images) and switch based on these. Even if still images (JPEG images) are continuous, they cannot be viewed by viewers as they are because their frame rate (number of frames displayed per unit time) is insufficient. However, based on information regarding which video source was switched at what timing when switching was performed and the video data output from the video source, video content can be output (produced or distributed) without noise or a drop in frame rate and made available for viewers to view. In other words, even when a web browser does not support video streams via UDP, it is possible to realize an information processing means that displays a switching means on the web browser, allows the user to switch video sources, and outputs (produces or distributes) video content.

[0040] 3 is a block diagram showing the functional module configuration of the information processing device 100 according to this embodiment. The information processing device 100 includes a storage unit 301, a communication unit 302, a segment data generation unit 303, a playlist generation unit 304, a playlist distribution unit 305, a segment data distribution unit 306, a switcher unit 307, and a system clock 308. 3 are realized by information processing by software stored in the device using various hardware resources. For example, the switcher means 307 is realized by the CPU 110 processing information using software stored in the ROM 130 of the information processing device 100, and displaying a touch GUI of a web application on the touch panel display of the switcher terminal 400.

[0041] The storage means 301 stores video content in the form of a combination of video clip files output from a video source and sequence data recording the timing of switching, or segment data obtained by dividing the video clip files. In addition, the IDs (identifiers), passwords, etc. of users of the switcher terminal (users of the web application), content viewers, and advertisers are stored as a producer database, viewer database, advertiser database, etc., respectively.

[0042] The communication means 302 transmits and receives various data and commands to and from the switcher terminal 400, the viewer terminal 1600, etc. For example, it communicates with the web browser of the switcher terminal 400 to display a web application. It also streams content in response to a request from the web browser of the viewer terminal 1600.

[0043] The segment data generation means 303 generates segment data (TS files) by dividing the video data output from the video source into small time intervals (for example, 10 seconds) in accordance with a standard format such as "MPEG-2 TS." At this time, the storage means 301 stores the ID of the video source, time data indicating the time the video data was captured, file paths to the generated segments, etc. as segment data information.

[0044] The playlist generating means 304 generates a playlist that includes file paths and durations of the segment data generated by the segment data generating means 303, the segment data relating to the switching timing of the switched video source.

[0045] The playlist distribution means 305 distributes a playlist to the viewer terminal 1600 in response to a request from the viewer terminal 1600. Specifically, the playlist is registered and saved in a distribution directory (folder).

[0046] The segment data distribution means 306 distributes the segment data (TS files) to the viewer terminal 1600. Specifically, the segment data (TS files) are registered and saved in a specific directory (folder), and the file path is recorded in a playlist.

[0047] As the event progresses, the switcher means 307 selects and switches one of the video sources connected to the switcher terminal 400 as a constituent material for the video content of the event at that timing, in response to instructions from the content creator user. The system clock 308 keeps time in synchronization with a time server.

[0048] FIG. 15 is a block diagram showing the main configuration of a network camera 1500 that can be connected to the information processing device 100 according to this embodiment. The network camera 1500 includes a CPU 1510, a RAM 1520, a ROM 1530, an imaging unit 1540, a memory I / F 1550, a network I / F 1560, a PTZ control unit 1570, and an external I / F 1580, which are interconnected via an internal bus 1590.

[0049] A CPU 1510 controls the operation of the network camera 1500 . The RAM 1520 is a writable, high-speed storage device into which the OS, various programs, and various data are loaded, and is also used as a working area for the OS and various programs. The ROM 1530 is a non-volatile read-only storage device that is used as a permanent storage area for the BIOS, various basic programs, and various basic data. An identification ID (such as an IP address) for identifying the network camera 1500 is held in the RAM 1520 or the ROM 1530 . The imaging unit 1540 photoelectrically converts an optical image using an imaging sensor such as a CCD or CMOS, and transfers the image to the CPU. The memory I / F 1550 is connected to recording media such as SD cards and HDDs, and is used to write and read video signals and various data. The network I / F 1560 serves as an interface for communicating with the information processing device 100 and various clients via a network in accordance with protocols such as the Internet Protocol and ONVIF. ONVIF is a protocol established by the standardization organization Open Network Video Interface Forum that implements commands for instructing device control and the start of distribution of images and metadata. The PTZ control unit 1570 obtains the pan-tilt-zoom state of a PTZ camera having a pan-tilt-zoom mechanism, and performs PTZ control according to operation instructions. The external I / F 1580 is used to connect accessories such as a microphone and viewfinder. These do not necessarily have to be directly attached to the camera body; they can be connected via a cable to a terminal or wirelessly. When audio is picked up by the microphone, it is recorded as an audio track integrated with the video signal.

[0050] FIG. 16 is a block diagram showing the configuration of functional modules related to content reception and playback of a viewer terminal 1600 connectable to the information processing device 100 according to this embodiment. The viewer terminal 1600 comprises a communication means 1601 , a playlist acquisition means 1602 , a segment data acquisition means 1603 , a buffer means 1604 , and a playback means 1605 .

[0051] The communication unit 1601 communicates with the information processing device 100 using a protocol such as HTTP. The playlist acquisition means 1602 requests the information processing device 100 to acquire a playlist, and acquires the playlist from the information processing device 100. Then, before playback of all segment data described in the acquired playlist is completed, the playlist acquisition means 1602 resends the playlist acquisition request and reacquires (reloads) the playlist. If the acquired playlist indicates that it is the last playlist, the playlist acquisition ends. The segment data acquisition means 1603 acquires the segment data in the order in which they are listed in the acquired playlist, that is, it sends an acquisition request to a URL (Uniform Resource Locator) indicating the file path of the TS file listed in the playlist. The buffer means 1604 buffers (temporarily holds) the acquired segment data for a certain length of time. The playback means 1605 reads and plays back the segment data buffered in the buffer means 1604 in the order listed in the playlist.

[0052] FIG. 4 is a block diagram showing the functional module configuration of the switcher terminal 400. As shown in FIG. In FIG. 4, reference numeral 401 denotes a video source thumbnail display means that generates thumbnail images from video signals (live view) of video sources connected to the information processing device 100 and displays them on a touch panel display of the switcher terminal 400.

[0053] A main menu display means 402 displays a main menu on the switcher terminal 400 . 403 is a volume level display means that displays the volume level of the audio signal of a video source (hereinafter referred to as "PGM") selected (switched) by the user from among the video sources connected to the information processing device 100 on the touch panel display of the switcher terminal 400. 404 is an "ON AIR" indicator display means, which displays on the touch panel display of the switcher terminal 400 that the user has instructed to start and that sequence data, etc., consisting of information such as the video source being switched and the time of switching, is being output. Reference numeral 405 denotes an output start / end control means, which controls the start and end of output of sequence data and the like.

[0054] Reference numeral 406 denotes a video monitor means, which displays the video signal (PGM) of the video source selected (switched) by the user on a monitor section 506 on the touch panel display of the switcher terminal 400 . Reference numeral 407 denotes a video source search, addition, and deletion means for searching, adding, and deleting video sources that can be connected to the information processing device 100 . Reference numeral 408 denotes a switching means that selects (switches) one of the video sources connected to the information processing device 100 as PGM in response to a user instruction, and displays the video signal on a monitor unit 506 on the touch panel display of the switcher terminal 400 . Reference numeral 409 denotes a camera control means, which generates a camera control command based on a user operation and transmits it to the target camera. Reference numeral 410 denotes a transition means, which controls switching from a video source selected as PGM to another video source over a certain period of time. 411 is a property means for writing / reading properties to / from the sequence (timeline) of the video source. 412 is a device setting means for setting and adjusting the display text size, screen brightness, volume, Wi-Fi connection, etc. on the switcher terminal 400 touch panel display.

[0055] 413 is the system clock, which keeps time in sync with the time server. Reference numeral 414 denotes a data output means that outputs the generated sequence data in a data format such as SMIL (Synchronized Multimedia Integration Language), CSV (Comma Separated Value), EDL (Edit Decision List), or AAF (Advanced Authoring Format). 415 is a programmed camerawork means that programs a series of camerawork (switching and PTZ control) for each scene. Reference numeral 416 denotes a camerawork execution means, which sequentially executes the camerawork programmed by the programmed camerawork means 415 based on user instructions.

[0056] Network cameras include those capable of panning, tilting, and zooming. The information processing device 100 according to this embodiment has a PTZ control means for controlling the panning, tilting, and zooming of such network cameras. By adopting this configuration, it is possible to control the pan, tilt, and zoom of the network camera in near real time in line with the progress of an "event" such as a wedding, funeral, drama, or other such event, and create video content using video data with minimal degradation in quality due to network transmission, and make it available for viewing by viewers. In addition, in this embodiment, the switching means and the PTZ control means are provided integrally. This configuration makes it possible to "stage" video content by linking switching and PTZ control in line with the progression of "events" such as ceremonies such as weddings and funerals, dramas, etc. For example, in video content created by filming a wedding, the camera may zoom in on a close-up of the bride as she reads a "letter of thanks to her parents," and then switch to a camera showing her parents the moment the reading of the letter finishes.

[0057] FIG. 5 is a diagram showing an example of a web application screen that the information processing apparatus 100 according to the embodiment of the present invention displays on the touch panel display of the switcher terminal 400. 501 is the GUI of the switcher application. 502 is a menu button for calling up the main menu. 503 is an audio meter that indicates the volume level. Reference numeral 504 denotes an "ON AIR" indicator that displays whether the device is in recording (sequence data recording) or broadcasting (streaming) mode. Reference numeral 505 denotes a start button that instructs the start of "ON AIR" (start of recording or broadcasting). When start button 505 is pressed and recording or broadcasting begins, start button 505 is replaced with stop button 505-2. Reference numeral 506 denotes a monitor section, which is an area for displaying the video (PGM) of the video source being selected (switched).

[0058] Reference numeral 507 denotes a video source section, which is an area for displaying thumbnails of switchable video sources. Video source thumbnails 5071, 5072, ... are displayed in the video source section 507. The video source thumbnails are thumbnail icons that indicate multiple video sources connected to the information processing device 100 via a network, and display thumbnails of the video output from each video source. In the example of FIG. 2, cameras 1500-1, 1500-2, ... are the video sources, and the video being captured by these cameras is displayed in real time as video source thumbnails. When no video is being captured, the last frame up to that point is displayed in a still state, or a so-called "black screen" is displayed. 5, images captured by three PTZ (pan-tilt-zoom) cameras and one video image are displayed. The video image here refers to video data that has been captured in advance and stored in a recording medium such as the RAM 120 of the information processing device 100 or online storage. Such video images can also be used as material for video content. A plus button 508 is used to search for and add a video source to be displayed in the video source section 507 . Reference numeral 509 denotes a camera control button for controlling each video source. When the user presses the camera control button 509, a camera operation panel 600 is displayed. A camera operation panel 600 is displayed for each camera. In addition, the panel can be moved and displayed as a so-called floating window. Reference numeral 510 denotes a switching type button for changing the switching type, and is composed of a cut selection button 510-1 for selecting a "cut," which is an instantaneous switch, and a transition selection button 510-2 for selecting a "transition," which is a gradual switch accompanied by an effect. The cut selection button 510-1 and the transition selection button 510-2 are so-called toggle buttons that can be selected mutually exclusively (alternatively).

[0059] 511 is a rating button that allows a user (content creator) to rate a recorded take. 512 is a timeline area that displays a so-called "filmstrip" (a thumbnail sequence of shots that make up a scene). Reference numeral 513 denotes a comment area in the timeline area, 514 a time code scale, and 515 a video source thumbnail display area that displays (thumbnails of) the video source that was switched at the timing of that time code in the timeline area. The length of each video source thumbnail along the time code scale 514 corresponds to the duration of that "shot." Comment text entered by the user is displayed in the comment area 513. Reference numeral 516 denotes a play button, and when the play button 516 is pressed, the recorded video sequence (recorded in RAM 120 by connecting live views from each video source) is played back from the timing indicated by a sequence marker 517 on the time code scale 514 and displayed on the monitor unit 506. When the play button 516 is pressed, it is replaced by a play stop button 516-2. Furthermore, if there is no video sequence that can be played (if the video sequence has not been recorded), the play button 516 is disabled (grayed out). Reference numeral 518 denotes a property button, which displays a property panel, allowing scene titles, take numbers, etc. to be assigned or changed for the video sequence displayed as a timeline in the timeline area 512. Also, from the property panel, scenes and takes that have been created up to that point can be called up and played back. 519 is a magnification slider that adjusts the magnification at which the timeline is displayed. 520 is a fader, which is a switching slider for performing switching operations during transitions. Reference numeral 521 denotes a scene title combo box that displays the scene title. 522 is a take number text box that displays the take number. Numeral 523 denotes a programmed camerawork button that displays a programmed camerawork panel for performing the programmed camerawork function. The programmed camerawork function will be described later.

[0060] Since the display of the switcher terminal 400 is configured as a touch panel display integrated with the touch panel, the user can execute various functions by using a finger or a stylus pen to perform touch gestures such as tapping (touching down and touching up) or flicking (moving quickly while touching and touching up) on GUI components such as icons. For example, by tapping on thumbnails 5071, 5072, ... in the video source section 507, the video source corresponding to that thumbnail can be selected (switched) as the PGM. At this time, a red frame is added to that thumbnail to indicate that the corresponding video source has been selected. The video of that video source is also displayed on the monitor section 506. The ID of that video source (or the video clip generated from that video source) and the time of tapping (time code) are also recorded as sequence data. When the user taps the start button 505, recording of the sequence data and the video sequence (a sequence in which live views of the switched video sources are connected together) starts. When the user touches one of the thumbnails in the video source section 507 and then touches down on another thumbnail, the video source corresponding to the later-touched thumbnail is set as the so-called preset video (hereinafter referred to as "PST") to be switched to next. Then, when the previously touched thumbnail is touched up while the later-touched thumbnail is still being touched, the transition means 410 gradually changes the PGM video (video from the video source of the previously touched thumbnail) displayed on the monitor section 506 to the video of the video source set as the PST (video from the video source of the later-touched thumbnail) with an effect (transition effect). In other words, in this embodiment, it is possible to instruct a transition by a touch gesture on the touch display means. With this configuration, the user can distinguish between a "cut (instantaneous change)" and a "transition" by using the difference between a "one-point touch" and a "two-point touch," making the operation method easy to understand and convenient.

[0061] The user can also perform a transition by operating the fader 520. That is, when the user operates the fader 520, the transition means 410 gradually switches the PGM video displayed on the monitor unit 506 to the video of the video source set as PST, depending on the amount of sliding of the fader 520. Such transition control by operating the fader is in line with the operability of conventional switcher devices, and is convenient because it allows the user to perform a transition at the timing desired. When switching (cutting and transitioning) of a video source is performed, the ID of the video source (or video clip) and the times (time codes) at which the switching starts and ends are recorded as sequence data. Reference numeral 524 denotes a transition effect pull-down for selecting a transition effect. Transition effects include "dissolve," "wipe," "slide," "squeeze," "iris," and "zoom." The transition effect pull-down 521 is enabled when the transition selection button 510-2 is selected (it is grayed out when the transition selection button 510-2 is not selected). A duration time up / down button 525 is used to set the duration time, which is the transition switching time. The duration time up / down button 522 is enabled when the transition selection button 510-2 is selected (it is grayed out when the transition selection button 510-2 is not selected).

[0062] FIG. 6 is a diagram showing an example of a camera operation panel that the information processing device according to this embodiment displays on the touch panel display of the switcher terminal 400. The camera operation panel 600 has a video display section 601, a "Subject tracking" check box 602, a "Tracking even during PGM" radio button 603a, a "Do not track during PGM" radio button 603b, a "Size setting tracking" check box 604, a "Size setting" pull-down 605, a "Adjust continuity" check box 606, a close button 607, a subject designation frame 608, a pan / tilt controller 609, a zoom controller 610, a focus controller 611, and a setting menu button 612. The video display section 601 is an area that displays the video captured by the corresponding camera. The pan-tilt controller 609 is a GUI that resembles a joystick for controlling the pan-tilt of the corresponding camera. The zoom controller 610 is a slider for controlling the zoom of the corresponding camera.

[0063] The user can also perform PTZ control of the corresponding camera by pinching in or out on the video display unit 601. That is, by touching down at two points on the video display unit 601 simultaneously and bringing the touched positions closer together (pinch in), the user can perform PTZ control of the corresponding camera so that the angle of view (angle) corresponding to the range displayed across the entire video display unit 601 at the time of touch down is zoomed out and panned / tilted toward the midpoint between the two touched positions (see FIG. 6(b)). Also, by touching down at two points on the video display unit 601 simultaneously and moving the touched positions away from each other (pinch out), the user can perform PTZ control of zooming in and panned / tilted so that the range near the midpoint between the two touched down positions expands to the range displayed across the entire video display unit 601 (see FIG. 6(c)). By using such an operation method, the user can conveniently perform pan / tilt and zoom operations in an integrated manner using touch gestures. When such an operation is performed, an ONVIF command such as the following is sent to the network camera. AreaZoom=x1-x0,y1-y0,w1-w0,h1-h0 Here, x1, y1 are the coordinates of the midpoint of the two-point touch after the pinch operation, x0, y0 are the coordinates of the midpoint of the two-point touch before the pinch operation, w1, h1 are the width and height between the touch points after the pinch operation, and w0, h0 are the width and height between the touch points before the pinch operation. However, the width and height of the entire video display unit 601 are assumed to be 200 and 200, respectively.

[0064] Generally, the so-called EASE IN / EASE OUT operation of PTZ is considered smooth and suitable, where the operation starts slowly, gradually accelerates, and then slows down further in the middle before stopping. Therefore, when PTZ control is performed by the above pinch operation, a speed control command may be automatically sent to perform EASE IN / EASE OUT PTZ operation within the duration of the PTZ operation. Furthermore, when pan / tilt and zoom are performed in an integrated manner, a subject that once entered the angle of view due to pan / tilt may be cut off from the angle of view again (no longer included in the angle of view) by zooming in, or a subject that once entered the angle of view due to zooming out may be cut off from the angle of view again by pan / tilt, which is generally considered undesirable and is described as "flickering." Therefore, the change in the optical axis angle of pan / tilt and the change in focal length due to zooming may be calculated and controlled so that such subject movement does not occur.

[0065] When a user controls the PTZ, it is convenient to know how much room there is for PTZ adjustment relative to the angle of view being shot at that time. Therefore, when the zoom or pan / tilt approaches a limit at which further operation will stop, the display of each side of the display frame of the video display unit 601 can be made different from when the limit is not approached, thereby visually notifying the user that there is little room for PTZ adjustment. For example, the frame surrounding the video display unit 601 may be displayed with an unclear border, with the edges of the image inside gradually blurred. As the image approaches the wide-angle end, at which zooming out is no longer possible, or the telephoto end, at which zooming in is no longer possible, the degree of blurring decreases, and the border indicating the frame becomes more clearly displayed. Similarly, as the pan or tilt limit is approached, the border indicating the edge of the pan or tilt direction becomes more clearly displayed. This display allows the user to optimally control the PTZ while taking into account the available PTZ space. Note that the notification method using the display frame described above is one way of intuitively notifying the user of the PTZ limit; other notification methods, such as a text message such as "You are approaching the limit of zooming out," may also be used. The focus controller 604 is a slider for controlling the focus of the corresponding camera. Depending on the capabilities of the corresponding camera, it can also be set to autofocus mode. The setting menu button 612 is a button for displaying a menu for setting the white balance, iris, frame rate, etc. of the corresponding camera. Note that auto white balance, auto iris, etc. can also be set depending on the capabilities of the corresponding camera.

[0066] The "Subject Tracking" checkbox 602 is a checkbox for setting automatic PTZ tracking of a specific subject. The technology of an automatic tracking function that controls pan and tilt so that a moving object detected on the screen is kept near the center of the screen has already been put to practical use in fields such as surveillance cameras. By selecting (checking) the "Subject Tracking" checkbox 602, the user can set the corresponding camera to automatically track a specific subject. When the "Subject Tracking" checkbox 602 is selected (checked), a subject designation frame 608 is superimposed on the video display section 601, and the user can move the frame by touch operation or the like to set the subject at the frame's position as the subject to be tracked. The "Track during PGM" radio button 603a and "Do not track during PGM" radio button 603b are radio buttons for setting whether or not to perform the above-mentioned automatic tracking (automatic PTZ) when the corresponding camera is switched, and are enabled when the "Subject Tracking" checkbox 602 is selected (checked) (when the "Subject Tracking" checkbox 602 is not selected (checked), the buttons are grayed out). The above-mentioned subject tracking is primarily aimed at keeping the subject within the screen, which can result in awkward and unnatural camerawork. Therefore, by not tracking during PGM, which is the video that the viewer sees, it is possible to eliminate the risk of unnatural camerawork.

[0067] The "Size setting tracking" checkbox 604 is a checkbox for setting the tracking of a specific subject while maintaining the "size," and is enabled when the "Subject tracking" checkbox 602 is selected (checked) (it is grayed out when the "Subject tracking" checkbox 602 is not selected (checked)). Here, "size" refers to the size of the subject within the frame, based on a human subject, and includes full shot (FS), full figure (FF), knee shot (KS), waist shot (WS), bust shot (BS), up shot (US), and close up (CU) (see Figure 11). The "Size setting" pull-down 605 is a pull-down for selecting and setting the above-mentioned "Size" and is enabled when the "Size setting tracking" checkbox 604 is selected (checked) (it is grayed out when the "Size setting tracking" checkbox 604 is not selected (checked)).

[0068] The "Adjust Transition" checkbox 606 is a checkbox for setting whether or not to perform automatic PTZ adjustments to ensure a smooth transition (cutback) between the video from the video source that was previously PGM when the video from the corresponding camera is switched (changed to PGM). This checkbox is enabled when the "Tracking Size Settings" checkbox 604 is selected (checked) (it is grayed out when the "Tracking Size Settings" checkbox 604 is not selected (checked)). When an image is switched from PGM, if the "size" of the shots before and after the switch is the same (for example, if both shots before and after the switch are "bust shots"), the transition (transition) will appear unnatural if the length from the top of the subject's head to the top of the screen is different. Similarly, if the "size" of the shot after the switch is tighter than the "size" of the shot before the switch (for example, if a "waist shot" is switched to a "bust shot"), the transition (transition) will also appear unnatural unless the length from the top of the subject's head to the top of the screen is shorter before and after the switch. Also, if the "size" of the shot after the switch is looser than the "size" of the shot before the switch (for example, if the shot is switched from a "bust shot" to a "waist shot"), it will look unnatural if the length from the top of the subject's head to the top of the screen is not longer before and after the switch. Therefore, when the "Adjust continuity" checkbox 606 is selected, the image of the video source set as the PGM is analyzed to determine or estimate the "size" and the length from the top of the person's head to the top of the screen, and when the "size" of the PGM and the size set in the "Size setting" pull-down 605 are the same, the corresponding camera is tilted so that the length from the top of the person's head to the top of the screen is also the same. Also, when the size set in the "Size setting" pull-down 605 is tighter than the "size" of the PGM, the corresponding camera is tilted so that the length from the top of the person's head to the top of the screen is shorter than that of the PGM. Also, when the size set in the "Size setting" pull-down 605 is looser than the "size" of the PGM, the corresponding camera is tilted so that the length from the top of the person's head to the top of the screen is longer than that of the PGM. By doing so, it is possible to eliminate the possibility that when an image is switched from PGM, the connection between the shots before and after the switch will look unnatural.

[0069] The close button 607 is a button for closing (hiding) the camera operation panel 600. The subject designation frame 608 is a marker (GUI) that allows the user to designate and set the subject at the position of the frame as the subject to be tracked. The camera operation panel 600 can be displayed for each camera, and can also be moved around as a so-called floating window. As described above, the information processing device 100 according to this embodiment allows a PTZ command to be given to a PTZ-capable network camera by touch gestures on the touch display means. This configuration allows the user to perform pan-tilt and zoom operations in an integrated manner. Furthermore, it is extremely convenient because it is easy to intuitively understand how the angle of view is changed by PTZ control. In addition, the information processing device 100 according to this embodiment detects a moving object in an image based on a video signal transferred by a transfer method in which the node transmitting from a PTZ-capable network camera transfers data without confirming reception by the node receiving it, and performs PTZ control of the network camera so as to keep the detected moving object captured near the center of the screen. This is because such PTZ control needs to be performed based on near-real-time live view video, but as already explained, live view video transferred via UDP is not suitable for use as video content for viewers to view. Therefore, by performing information processing such as TCP / IP transfer or HTTP streaming on video data other than the live view video, it is possible to output (produce or distribute) high-quality video content.

[0070] FIG. 11 is a diagram for explaining "size." Full Shot (FS) is the widest angle of view of the camera. A full figure (FF) is the size that includes the figure from the toes to the top of the head. Knee Shot (KS) is the size that covers from the knee to the top of the head. Waist shot (WS) is the size that covers from the waist to the top of the head. A bust shot (BS) is the size that covers the chest to the top of the head. An up-shot (US) is a size where the face fills the entire screen. A close-up (CU) is a shot in which body parts such as eyes, mouth, hands, and feet fill the entire screen.

[0071] 7 is a flowchart showing the flow of processing in which the information processing device 100 according to this embodiment displays video source thumbnails corresponding to video sources, changes the video displayed on the monitor unit 506 in response to a switching operation (including a fader operation) by the user, and generates and records sequence data when in a recording state. The processing of this flowchart is realized by the CPU 110 of the information processing device 100 controlling each unit based on an input signal and a program. 5 on the touch panel display of the switcher terminal 400 (step S701). At this point, the information processing device 100 is ready to accept GUI operations by the user. If the user operation is to press the plus button 508 (YES in S702), the CPU 110 searches for video sources that can be connected to the network and displays them as a list (S703). When the user selects one or more video sources and closes the list (YES in S704), video source thumbnails of the selected video sources are additionally displayed in the video source section 507 (S705). If the user's operation is the pressing of a camera control button on the PTZ camera (YES in S706), the CPU 110 displays the camera operation panel 600 (S707). As described above, the camera operation panel 600 has the video display unit 601, pan / tilt controller 602, zoom controller 603, focus controller 604, setting menu button 605, and close button 606. Note that specifications such as the lens focal length at the wide-angle and telephoto zoom ends and the pan / tilt movement range are acquired using the ONVIF protocol described above when the PTZ camera is added as a video source. In addition, ONVIF commands and the like are generated in response to the user's operation on the camera operation panel 600, and the PTZ camera is controlled (S708). When the user presses the close button 606 (YES in S709), the camera operation panel 600 is closed (S710).

[0072] If the user operation is the pressing of the start button 505 (YES in S711), the CPU 110 sends a command to the video source connected to the information processing device 100 through the network (the video source whose thumbnail is displayed in the video source section 507) to start recording to online storage, recording media, etc. (S712). Also, if the scene title combo box 515 is blank (YES in S713), a scene title input panel is displayed to prompt the user to input a scene title (S714). Here, the scene title is a combination of the title (program title) of the video content as the "finished product" and the scene number (scene number), for example, "Watanabe Family Wedding Scene 1." The take number in the take number text box 516 is set to the most recent take number for that scene title plus one (S715). The start button 505 is replaced with the stop button 505-2 (S716). The system also starts generating and recording sequence data consisting of the ID of the currently switched video source (or video clip) and the time of switching (time code) (S717). A URL (Uniform Resource Locator) can be used for the video source or video clip ID. The timeline area 512 displays a so-called filmstrip, in which the names or thumbnails of the switched video sources are arranged along the time code (S718).

[0073] When the user taps one of the video source thumbnails (YES in S719), CPU 110 displays the video of the video source corresponding to that video source thumbnail on monitor unit 506 (S720). It also displays a red frame around that thumbnail (S721) to indicate that the corresponding video source has been selected (a so-called "tally display"). It also obtains the time (time code) from system clock 413 (S722). If the user touches one thumbnail in the video source section 507 and then touches down on another thumbnail (YES in S723), the video source is switched to the video source corresponding to the thumbnail touched first (S724), and the video source corresponding to the thumbnail touched later is set as PST (preset video) (S725). The thumbnail set as PST may be displayed with, for example, a flashing red frame. When the two touched thumbnails are touched up at approximately the same time and the user operates the fader 520 (YES in S726), the PGM video displayed on the monitor unit 506 is gradually switched to the PST video in accordance with the amount of sliding of the fader 520 (S727). The effect used for the switch (transition effect) is the effect selected by the user when they press the effect change button 510. Note that the default value for the transition effect may be set to "dissolve," as this is a commonly used transition effect. If the user does not operate the fader 520 (NO in S726) and the subsequently touched thumbnail remains touched while the previously touched thumbnail is touched up (YES in S728), the PGM video is gradually switched to the PST video over the user-specified Duration Time (the time required for switching) (S729). Then, the switching of the video source is recorded as sequence data (S730). Note that the Duration Time is specified by selecting a value in the "Auto Transition Duration Time" combo box on the switching effect selection panel that is displayed when the effect change button 510 is pressed.

[0074] When the user presses the stop button 505-2 (YES in S731), a command to stop recording to the recording medium is sent to the video source (the video source whose thumbnail is displayed on the video source section 507) connected to the information processing device 100 via the network (S732). The stop button 505-2 is also replaced with the start button 505 (S733). The recording of the sequence data is also stopped (S734). When the user presses the menu button 502 (YES in S735), the main menu is displayed (S736). When the user selects the "Save Data" command on the main menu (YES in S737), the sequence data is saved or output to a location designated by the user (S738). Furthermore, when the user selects the "End" command (YES in S739), the processing of this flow ends.

[0075] Fig. 12 shows an example of sequence data recorded or output by the information processing device 100 according to this embodiment. In Fig. 12, the data has a structure described in SMIL (Synchronized Multimedia Integration Language) format. SMIL is a language recommended for standardization by the World Wide Web Consortium (W3C) that synchronizes the playback of various data formats, such as video, audio, and text. In the example of sequence data in FIG. 12, first, information unrelated to temporal behavior is described in elements. The meta attributes of an element are used to define properties of the data (such as program title, scene number, take number, rating, remarks, and notes) and to assign values ​​to those properties. The layout attribute of an element is used to determine the display position of the object. <layout type="″text / smil-basic-layout″">< / layout> By writing this, you can apply the default layout values ​​to all objects.

[0076] Information related to temporal behavior is described in the element. <seq>Below the element, objects that are played back in time sequence are described. In other words, the sequence of switched video sources (video clips) is <seq>of the element's child hierarchy <video>It is written as an element. each <video>The element uses the src attribute to specify a unique ID that identifies the output file from the video source. Such an ID can be, for example, a UMID (Unique Material Identifier) ​​defined by the Society of Motion Picture and Television Engineers (SMPTE). Alternatively, the src attribute can be specified as a file path to the location where the output file is saved (e.g., src="D:\cam1\Clip1.avi"). The clipBegin attribute specifies the start point (in point) of a video clip (each output file from a video source) where switching will be performed, and the clipEnd attribute specifies the end point (out point) of the insertion. The clipBegin and clipEnd attributes can be specified using time codes, for example, clipBegin="hh:mm:ss:ff". When switching between video clips involves a transition effect, for example, <transition>It is described using elements. <transition>The type attribute of the element describes the type of effect, and the dur attribute describes the duration of the transition.

[0077] The data format in Fig. 12 is merely an example, and various other formats are possible, such as CSV (Comma Separated Value) format, EDL (Edit Decision List), AAF (Advanced Authoring Format), or a newly defined data format. When sequence data is output in EDL (Edit Decision List) format, it can be read and edited using existing video editing applications, such as Adobe Premiere Pro and Apple's Final Cut Pro.

[0078] FIG. 14 shows an example of EDL data. FIG. 14 shows an example in which program recording (recording of sequence data) starts 20 seconds after each video source starts outputting video data. In the figure, 1401 indicates the title of the sequence. Also, 1402 indicates whether the time code of the edited sequence is in drop frame format or non-drop frame format. Also, 1403 indicates whether the time code of the video clip sequence of each video source is in drop frame format or non-drop frame format. Also, 1404 represents the shot number. Also, 1405 represents the ID of the video source. Also, 1406 represents a "2-channel audio / video" configuration. Also, 1407 indicates the switching type. Also, 1408 represents the time code of the IN point of the video clip of the video source. Also, 1409 represents the time code of the out point of the video clip of the video source. Also, 1410 represents the time code of the IN point of the edited video clip (video content). Also, 1411 represents the time code of the out point of the edited video clip (video content). When sequence data is output in a compatible data format such as EDL, it becomes possible to create video content by switching video sources, including network cameras, using existing video editing applications.

[0079] FIG. 8 is a flowchart showing an example of information processing in which the information processing device 100 according to the embodiment of the present invention generates and distributes a playlist based on information regarding which video source was switched at what timing. When a browser is launched on a user terminal (such as a PC, tablet, or smartphone) and the URL (Uniform Resource Locator) of a "video content production service" provided by the information processing device 100 according to an embodiment of the present invention is input or specified, the information processing device 100 displays a homepage on (the display unit of) the user terminal (S801) and also displays a "Producer User Login" button (S802). When the user selects the "Producer User Login" button (YES in S803), authentication is performed using the user ID, password, etc. (S804), and if the authentication result is positive (YES in S805), a GUI of the switcher application such as 501 in Figure 5 is displayed on the display of the user terminal (S806). If the user operation is to press the plus button 508 (YES in S807), network-connectable video sources are searched for and displayed as a list (S808). When the user selects one or more video sources and the list is closed (YES in S809), video source thumbnails of the selected video sources are additionally displayed in the video source section 507 (S810).

[0080] If the user's operation is the pressing of a camera control button on the PTZ camera (YES in S811), the camera operation panel 600 is displayed (S812). As described above, the camera operation panel 600 has a video display area 601, a "Subject tracking" check box 602, a "Tracking even during PGM" radio button 603a, a "Do not track during PGM" radio button 603b, a "Size setting tracking" check box 604, a "Size setting" pull-down 605, a "Contact" radio button 606a, a "Do not contact" radio button 606b, a close button 607, and a subject designation frame 608. When the user performs a touch gesture operation on the video display unit 601 (YES in S813), a control command corresponding to the operation is transmitted to perform PTZ camera control of the corresponding camera (S814). Furthermore, when subject tracking is set (YES in S815), the corresponding PTZ camera is set to tracking mode (S816). Note that the thumbnail of the corresponding PTZ camera in the video source section 507 indicates that it is in tracking mode. When the user presses the close button 607 (YES in S817), the camera operation panel 600 is closed (S818).

[0081] If the user operation is the pressing of start button 505 (YES in S819), a command is sent to the video source connected to the user terminal via the network (the video source whose thumbnail is displayed in video source section 507) to instruct it to output video data to a specific storage area (S820). In addition, segment data is generated by dividing the video data output to the storage area (S821). Also, a playlist is generated that describes file paths that refer to segment data generated from the video data of the video source being switched at that time (S822), and is registered and saved in the distribution directory (folder) (S823).Also, start button 505 is replaced with stop button 505-2 (S824). Also, the generation and recording of sequence data consisting of the ID of the video source (or video clip) being switched at that time and the time of switching (time code) is started (S825). When the user taps one of the video source thumbnails (YES in S826), first, the switching time T1 is obtained from the system clock 308 (S827). The switching type set by the switching type button 510 is then determined, and if "Cut" has been selected (YES in S828), first the video of the video source corresponding to that video source thumbnail is displayed on the monitor unit 506 (S829). A red frame is added to that thumbnail (S830) to indicate that the corresponding video source has been selected (a so-called "tally display"). The imaging times added to the segment data generated from the switched video source are then compared with the switching time T1, and a playlist is generated by entering file paths to segment data relating to video captured after time T1 (S831).

[0082] When "Transition" is selected as the switching type of the switching type button 510 (NO in S828), the PGM video displayed on the monitor unit 506 is gradually switched to the video (PST) of the video source corresponding to the tapped video source thumbnail using a user-specified effect (switching effect) for a user-specified Duration Time (switching time) (S832). A red frame is also added to the tapped video source thumbnail (S833). Segment data relating to the video (PGM) of the previously switched video source and the video (PST) of the newly switched video source captured between time T1 and Duration Time (switching time) is processed and combined with a user-specified effect (switching effect) to generate segment data for the switching video (S834), which is then registered and saved in a specific directory (folder) (S835). A playlist is then generated by entering the file paths of the segment data for the switching video (S836). Then, the playlist in the distribution directory (folder) is overwritten and updated with the newly created playlist (S837). When the user presses the stop button 505-2 (YES in S838), a command is sent to the connected video source (the video source whose thumbnail is displayed in the video source section 507) to stop outputting video data to a specific storage area (S839). The playlist is then marked as the last playlist, and the playlist in the distribution directory (folder) is overwritten or updated (S840). The stop button 505-2 is then replaced with the start button 505 (S841). Recording of the sequence data is then stopped (S842). When the user presses the menu button 502 (YES in S843), the main menu is displayed (S844). When the user selects the "Save Data" command in the main menu (YES in S845), the sequence data is saved or output to a location designated by the user (S846). Furthermore, when the user selects the "End" command (YES in S847), the processing of this flow ends.

[0083] FIG. 13 shows an example of a playlist in m3u format. FIG. 13 shows an example in which switching occurs from Camera_1 to Camera_2 at 999 seconds (16 minutes and 39 seconds) from the start of the program.

[0084] The segment data for the images from Camera_1 and Camera_2 are http: / / www.contentsvision.com / sample / ts / camera_1 / http: / / www.contentsvision.com / sample / ts / camera_2 / In addition, the segment data will be given a file name that indicates the number of seconds in milliseconds (1 / 1000) from the time when the video related to that data started to be captured.

[0085] In the figure, 1301 indicates the file format. Also, 1302 represents the version number of the file format. Also, 1303 indicates the overall number of the segment data (TS file) at the top of the list. Also, 1304 indicates how many seconds each segment data (TS file) is divided into. Also, 1305 indicates the time length of the segment data (TS file) indicated by the file path on the next line. Also, 1306 represents the file path (URL) to the segment data (TS file).

[0086] In this way, video content can be distributed via HTTP streaming by switching between multiple video sources, including network cameras. TCP is used to transfer video data in HTTP streaming, allowing viewers to watch high-quality video content with reduced noise.

[0087] (Programmed camerawork function) The programmed camerawork function of the information processing device according to this embodiment will be described below. Generally, ceremonies such as weddings and plays (dramas) have a prepared schedule and scenario, and the camerawork (switching and PTZ) for the video content related to the ceremony or play (drama) is carried out according to that schedule and scenario. Therefore, if a series of camerawork can be programmed in advance and execution instructions can be given in accordance with the progress of the event, it is convenient as it reduces camerawork operation errors and the burden.

[0088] The programmed camerawork function of the information processing device of this embodiment is a function that allows the user to program (input) a series of camerawork in advance, and executes the programmed camerawork sequentially in response to user instructions. Camera work input and execution instructions are made by displaying the programmed camera work panel.

[0089] An example of the programmed camerawork panel is shown in Figure 9. The programmed camerawork panel is displayed as a so-called floating window on the display of the switcher terminal 400 when the programmed camerawork button 523 is pressed by the user. The programmed camerawork panel 901 has a monitor section 902 , a REC button 903 , an EDIT button 904 , a NEXT button 905 , a camerawork field 906 , an event number field 907 , a text field 908 , a BACK button 909 , a SKIP button 910 , and a pop-up 911 .

[0090] The camera work column 906 is a column showing the camera work in the order of progression. Here, camera work refers to cut switching or transition of the video source or PTZ control of the PTZ camera. The event number column 907 is a column showing the numbers in the order in which the camera work is performed. The text field 908 is a field for displaying text (such as a scenario, progress chart, or script) that serves as a guide for the timing of camerawork execution. The displayed text may be, for example, song lyrics, musical progression, beat count, or bar count. It may also be entered by the user from an operation unit such as a keyboard, or it may be read from a file, scanned from a paper manuscript, or photographed. If the file or manuscript to be read has a table or layout, only the text may be extracted and read, or the table or layout may be preserved and pasted. A so-called continuity (storyboard) may also be pasted. The monitor section 902 is an area that displays the video signal (PGM) of the video source that has been selected (switched). The REC button 903 is a button for issuing an instruction to start recording (recording of sequence data). The EDIT button 904 is a button for switching to a mode in which the camerawork field and text field can be edited (changed, inserted, deleted, and rearranged). The NEXT button 905 is a button for executing the focused camera work and moving the focus to the next camera work. The BACK button 909 is a button for moving the focus to the previous camera work. The SKIP button 910 is a button for moving the focus to the next camera work. Pop-up 911 is a "speech bubble" that is displayed near the cursor position in the text and has a "BS button," a "new block button," and a "keyboard button," and is displayed for editing the text.

[0091] The user can set the programmed camera work panel 901 to EDIT mode (editable mode) and set "video source switching" and "PTZ control" in the camera work field. Furthermore, camerawork executed in the main window (GUI 501 in Fig. 5) is saved as data and can be loaded into the programmed camerawork panel 901 and reused as programmed camerawork. In other words, for example, after switching and PTZ control are performed in the first take of a certain scene, that camerawork can be used as programmed camerawork in subsequent takes (reshoots), reducing the effort and mistakes involved in camerawork operations.

[0092] FIG. 10 is a flowchart showing the flow of processing related to the programmed camerawork function of the information processing device 100 according to this embodiment. When the user presses the programmed camerawork button 523, the programmed camerawork panel 901 is displayed as a so-called floating window (S1001). The information processing device 100 first determines whether there is data on camerawork executed in the main window, and if there is such data (YES in S1002), it compares it with the programmed camerawork data (camerawork data displayed on the programmed camerawork panel) (S1003). If the camerawork data executed in the main window differs from the programmed camerawork data (YES in S1004), the system displays the message "Do you want to convert the executed camerawork to programmed camerawork?" and waits for the user to select YES or NO (S1005). When the user selects YES (YES in S1006), the executed camerawork is converted into programmed camerawork and displayed in the camerawork column (S1007). When the user presses the EDIT button (YES in S1008), the programmed camera work panel is put into EDIT mode (S1009). Specifically, the REC button and NEXT button are disabled (grayed out).

[0093] In EDIT mode, when the user taps a blank spot in the camera work field (YES in S1010), a dialog box with options (radio buttons) for "Video Source Switching" and "PTZ" is displayed at that spot (S1011). However, if a PTZ-enabled video source is not assigned to that spot in the camera work field, the "PTZ" option is unavailable (grayed out). When "Video Source Switching" is selected (YES in S1012), a dialog box is displayed (S1013) having the options (radio buttons) of "Cut" and "Transition", a pull-down list for selecting the video source, and a GUI similar to that of the camera control panel 601. When the user selects "cut" and one of the video sources and the angle of view of the video source after switching is determined (YES in S1014), the cut line, the name of the video source, and a thumbnail showing the angle of view of the video source after switching are added and displayed in the camera work column (S1015).

[0094] When the user selects "Transition" and one of the video sources and determines the angle of view of the video source after switching (YES in S1016), a cut line, the name of the selected video source, and a thumbnail showing the angle of view of the video source after switching are added and displayed in the camera work field (S1017). In addition, a band indicating the transition, a combo box indicating the switching time, and an icon indicating the type of transition effect are displayed (S1018). The combo box indicating the switching time and the icon indicating the type of transition effect have default values. The user can change the values ​​by tapping the combo box or icon. Each dialog has a cancel or close button, and when the user instructs to cancel or close, the dialog returns to the previous state.

[0095] When the user selects "PTZ" (YES in S1019), a PTZ dialog box having a GUI similar to that of the camera control panel 601 is displayed (S1020). When the user operates the GUI on the PTZ dialog and decides the angle of view after PTZ is performed (YES in S1021), a band indicating the cut line and PTZ, a combo box indicating the switching time, and a thumbnail indicating the angle of view after PTZ are displayed in the camera work column (S1022). Note that the combo box indicating the switching time has a default value. The user can change the value by tapping the combo box.

[0096] When the user presses and holds the text field (YES in S1023), a cursor is displayed in the text (S1024). Also, a pop-up (speech bubble) is displayed (S1025). The pop-up has a BS button, a new block button, and a keyboard button. When the user presses the BS button (YES in S1026), a backspace is performed to delete the character in front of the cursor (S1027). If the object to be deleted is a "cut line" rather than a character, the text in the text block where the cursor is located is combined with the previous block, and the text blocks corresponding to subsequent event numbers are shifted forward one by one. When the user selects the new block button (YES in S1028), the text after the cursor position in the text block is moved to a new block created immediately after the text block, and the text blocks corresponding to subsequent event numbers are shifted forward one by one (S1029). When the keyboard button is pressed by the user (YES in S1030), a floating keyboard is displayed, allowing text to be inserted (S1031).

[0097] When the user presses the MENU button (YES in S1032), a menu is displayed (S1033). The menu has the commands "Read text," "Save programmed camerawork data," and "Read programmed camerawork data." When the user instructs "Text Import" (YES in S1034), a file import window is displayed (S1035). The file import window has options (radio buttons) for "Overwrite" and "Insert" and "OK" and "Cancel" buttons. Note that the "OK" button is disabled (grayed out) when no file is selected in the window. When the user selects "overwrite" and loads a file (YES in S1036), all text in the text field is overwritten with the text from the loaded file (S1037). When the user selects "insert" and loads a file (YES in S1038), the text of the loaded file is inserted at the cursor position (S1039). When the user instructs "save programmed camerawork data" (YES in S1040), a data save window is displayed (S1041).

[0098] When the user instructs "Load programmed camerawork data" (YES in S1042), a data file load window is displayed (S1043). The data file load window has a check box labeled "Load executed camerawork" and "OK" and "Cancel" buttons. Note that if the check box is not checked or if no file is selected in the window, the "OK" button is disabled (grayed out). Also, if the check box is checked, file selection in the window is disabled (grayed out). When the camera work data is read (YES in S1044), the camera work column is overwritten (S1045).

[0099] When the user ends the EDIT mode (when the EDIT button that was displayed in the depressed state is tapped again) (YES in S1046), the programmed camerawork panel is put into execution mode (S1047). Specifically, the REC button and NEXT button are enabled. When the user taps the NEXT button (YES in S1048), the camera movement of the focused row is performed and the focus is moved to the next row (S1049). When the camera work for the last line is executed (YES in S1050), no line is focused, and the NEXT button is disabled (grayed out) (S1051). When the user taps the REC button (YES in S1052), REC starts (S1053). The operation during REC is the same as when REC is instructed in step S711 of FIG. When the user taps on any of the rows in the "camera work column," "event number column," or "text column" (YES in S1054), the focus is moved to that row (S1055). When the user presses the close button or taps anywhere on the main panel (YES in S1056), the programmed camera work panel closes (S1057). Note that REC / Stop REC and the display / hide of the programmed camera work panel are independent of each other. For example, even if recording is in progress when the programmed camera work panel is closed, REC will continue.

[0100] The data used to save and load programmed camerawork is written by assigning numbers (event numbers) to video source switching and PTZ control, and relating the timing to the progress chart text and storyboard. Since it is common to sequence data in that it describes "switching of video sources," the format of sequence data may be extended and used. An example of extending sequence data written in SMIL (Synchronized Multimedia Integration Language) format to programmed camerawork data will be described below.

[0101] First, each <video>An event_Num attribute is added to the element, describing the order (event number) in which the camerawork is performed. <video src="″D:\cam1\Clip1.mxf″" clipBegin="″hh:mm:ss:ff" clipEnd="″hh:mm:ss:ff″" event_Num="″1″">< / video> Also, <seq>In the child hierarchy of an element, for example <onvif>Describe PTZ control as an element. <onvif CamID="″cam1″" AbsolutePanTilt="″X,Y″" AreaZoom="″x,y,w,h″" dur="″hh:mm:ss:ff″" event_Num="″2″" / > Also, corresponding to the order (event number) in which camera work is executed, describe text (such as a progress chart or script) that serves as a timing guideline for executing that camera work as a <text_block> element. <text_block=″Dear guests, thank you very much for coming to today's wedding reception of ○○○○. Now, the bride and groom will enter. Please give them a big round of applause. event_Num=″1″ / >

[0102] As described above, the user can program a series of camera works in advance and sequentially execute them in the preset order by instructing the NEXT button. This can reduce the labor and operation errors of camera work operations. Also, the user can plan the timing of camera work execution in contrast to the progress chart or script, further improving the operability. Also, the camera work executed in the main window can be used as programmed camera work. This reduces the labor of programming (inputting / setting) camera work. Also, an embodiment in which the above text (such as scenarios, progress charts, scripts, song lyrics of songs, picture contents, scores, etc.) is reused to disclose and explain the content of video content, or plans and projects, and to recruit or match advertisers, collaborators, staff, etc. is also suitable. When done in this way, it becomes possible to build and operate platforms for planning, fundraising, production, distribution, circulation, etc. centered on video content creation. Another preferred embodiment is one in which, when programmed camerawork is executed in comparison with the above-mentioned text (such as a scenario, progress chart, script, song lyrics, storyboard, or sheet music) and video content is produced, the text is used to notify matched advertisers, collaborators, and staff, or to allow them to access all or part of the produced video content. In this case, the notification or access to all or part of the video content (playback, viewing, confirmation, etc.) can prompt actions (actions / executions) from advertisers, collaborators, staff, etc., thereby accelerating the progress of projects related to video content production and distribution.

[0103] (Video content creation) As explained above, when an event is recorded by switching between multiple video sources using the information processing device 100 of this embodiment, video clips that record the video signals from the video sources and sequence data that references the switched parts of the video clips in the order in which they were switched are created. This sequence data also has the properties of a "playlist" that indicates the playback order of multiple video clips (segments), so by playing the video clips (segments) according to the sequence data on a video player, the video content of the recorded event can be viewed and played. For example, if the sequence data is written in SMIL format, the video content can be played using a web browser as a video player. Furthermore, if the sequence data is in a data format that can be read by a video editing device (video editing application), the video content of the recorded event can be written (output) as a single video file using video clips from the video source as raw data.

[0104] Since sequence data is generated for each recording (one take of one scene), when video content consists of multiple scenes, the sequence data for each scene is made to correspond to the multiple scenes (sequence data combination). For example, in the sequence data example in Figure 12 (an example of sequence data written in SMIL format), if elements at a lower level than the element are combined in the order of the "scene number" written in the meta attribute of the element, they will be combined into a playlist that plays multiple scenes sequentially in the order of their scene numbers.

[0105] Furthermore, when a scene is "reshot" and multiple takes exist for each scene, the "rating" written in the meta attribute of the element can be referenced, and the takes with the highest rating (if there are multiple, for example, the one recorded later) can be linked in order of scene number to create a "playlist," which allows the user to view and play video content made up of scenes that use the takes they want. In cases where "reshoots" are common, such as with video content for plays (dramas), this is convenient as it allows the efficiency of the video editing process by deciding which takes to use for each scene. Because text editing of sequence data (including combined data) produces results similar to those of video editing, the configuration of a video editing device (video editing application) and the resources required can be simplified and made lighter. For example, a video editing web application can be configured using a web page that has a text editor area for editing SMIL and a monitor area for playing and displaying that SMIL. Furthermore, if the video server side references the file into which the information processing device 100 of this embodiment writes the sequence data, and based on that data, sequentially transmits the video of the portion of the video clip into which the video data of the video source has been written that has been switched by the information processing device 100, live streaming (with a slight delay) can be performed even during event recording (while the sequence data is being written).

[0106] The information processing device 100 according to this embodiment switches between video sources based on live view video transferred in near real time from multiple video sources, including network cameras, using a transfer method in which the transmitting node transfers data without confirming receipt by the receiving node, and creates or distributes video content based on information regarding which video source was switched at what timing and the video data output from the video source. This configuration makes it possible to create or distribute video content based on video data with minimal degradation in quality due to network transmission, while switching between video sources in near real time in line with the progress of an "event" such as a ceremony, drama, or other ceremonial event. This reduces the time and effort required for line setup and equipment costs, and simplifies and speeds up video content production, reducing costs. In other words, an information processing device suitable for switching video sources, including network cameras, to create or distribute video content is realized.

[0107] (Other embodiments) The present invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention.

[0108] The present invention can also be realized by supplying software (programs) that realize the functions of the above-described embodiments to a system or device via a network or various storage media, and having the computer (or CPU, MPU, etc.) of the system or device read and execute the programs. Therefore, the program code itself installed on a computer to implement the functional processing of the present invention also embodies the present invention. In other words, the present invention also includes the computer program itself for implementing the functional processing of the present invention. In this case, as long as it has the program functionality, it may be in the form of object code, a program executed by an interpreter, a script or macro executed by an application program such as a web browser, or an API (Application Programming Interface). It may also be implemented by being incorporated as part of another web service such as a social networking service (SNS) using web programming techniques such as "mashup." The present invention can also be realized as a so-called web application that uses a web browser. Therefore, the web server pointed to by the URL (Uniform Resource Locator) constitutes an embodiment of the present invention, regardless of whether it is a single piece of hardware. [Explanation of symbols]

[0109] 100...information processing device, 110...CPU, 120...RAM, 130...ROM, 160...display, 170...operation unit, 190...network I / F, 180...internal bus, 210...event, 400...switcher terminal, 1500...video source such as network camera, 1600...viewer terminal, 301...storage means, 302...communication means, 303...segment data generation means, 304...playlist generation means, 305...playlist distribution means, 306...segment data distribution means, 307...switcher means, 308...system clock, 401...video source thumbnail display means, 402...memory menu display means, 403...volume level display means, 404...REC indicator display means, 405...sequence recording start / end control means, 406...PGM video display means, 407...video source search / add / delete means, 408...switching means, 409...camera control means, 410...transition means, 411...property means, 412...device setting means, 413...system clock, 414...sequence data output means, 415...programmed camerawork means, 416...camerawork execution means, 501...GUI, 502...menu button, 503...audio meter, 504...ONAIR indicator, 505...Start button, 506...Monitor section, 507...Video source section, 508...Plus button, 509...Camera control button, 510...Switching type button, 511...Rating button, 512...Timeline area, 513...Comment area, 514...Timecode scale, 515...Video source thumbnail display area, 516...Play button, 517...Sequence marker, 518...Properties button, 519...Magnification slider, 520...Fader, 521...Scene type Torque combo box, 522...Take number text box, 523...Programmed camera work button, 524...Transition effect pull-down, 525...Duration time up / down, 600...Camera operation panel, 601...Video display section, 602..."Subject tracking" check box, 603a..."Tracking even during PGM" radio button, 603b..."No tracking during PGM" radio button, 604..."Size setting tracking" check box, 605..."Size setting" pull-down, 606... "Adjust continuity" checkbox, 607...close button, 608...subject selection frame, 609...pan-tilt controller, 610...zoom controller, 611...focus controller, 612...setting menu button, 901...programmed camera work panel, 902...monitor section, 903...REC button, 904...EDIT button, 905...NEXT button, 906...camera work field, 907...event number field, 908...text field, 909...BACK button, 910...SKIP button , 911... Pop-up (speech bubble), 1510... CPU (camera), 1520... RAM (camera), 1530... ROM (camera), 1540... Imaging unit (camera), 1550... Memory I / F (camera), 1560... Network I / F (camera), 1570... PTZ control unit (camera), 1580... External I / F (camera), 1590... Internal bus (camera), 1601... Viewer terminal communication means, 1602... Playlist acquisition means, 1603... Segment data acquisition means, 1604... Buffer means, 1605... Playback means< / onvif> < / seq> < / video> < / transition> < / transition> < / video> < / video> < / seq> < / seq>

Claims

1. An information processing device that performs information processing for distributing video content using video from multiple video sources, including network cameras, as source material, A switching means that switches based on a video signal transmitted by a transmission method in which a node transmitting from the video source transfers data without confirming receipt by a receiving node, The system includes a playlist generation means that generates a playlist based on information regarding which video source was switched by the switching means and at what timing, The information processing apparatus is characterized in that the switching means is capable of performing a transition accompanied by a switching effect.

2. A program for causing a computer to function as each of the means of the information processing apparatus described in claim 1.

3. A computer-readable recording medium storing a program for causing a computer to function as each of the means of the information processing apparatus described in claim 1.

Images