Information processing device, information processing program, and recording medium

Abstract

To simplify and accelerate video content production and to reduce its cost by using a network camera.SOLUTION: An information processing device of the present invention is an information processing device that outputs video content using video from a plurality of video sources including a network camera as material, and includes: switching means for performing switching on the basis of a video signal that has been transferred without a transmission node for a video source confirming reception at a reception node; playlist generation means for generating a playlist on the basis of a video source that has been stitched and information on timing; subject tracking means for detecting a moving object in an image on the basis of the video signal that was transferred without the transmission node confirming the reception at the reception node with a PTZ-enabled network camera, and performing control of and tracking the network camera so that the moving object is continuously captured near the center of a screen; and information processing means for performing distribution in a transfer scheme that continues to re-transmit the video content until an acknowledgement at the reception node is confirmed.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an information processing apparatus that performs information processing related to the production or distribution of video content using videos of a plurality of video sources as materials.

Background Art

[0002] Generally, for the creation of video content such as TV programs, video signals captured by a plurality of imaging devices from different directions (viewing angles) with respect to the subject are projected onto a monitor screen, and a switcher (video switching means) switches between the video to be used and the video not to be used for the video content to be provided for viewing by the viewer by operating a switching button or the like (switching).

[0003] By the way, network cameras that transmit video signals through computer networks such as the Internet and LANs have been widespread. In recent years, among video cameras and digital cameras for business / broadcast use and general photography, those that can transmit the captured video through a computer network have also emerged.

[0004] Therefore, there is a problem / need to use such network cameras (cameras that can transmit video through a computer network) in the creation of video content by switching video sources, reduce the labor and equipment costs such as line setting, and simplify, speed up, and reduce the cost of video content production.

[0005] Furthermore, in recent years, HTTP streaming, which transmits video in accordance with HTTP (HyperText Transfer Protocol), has become the mainstream method for video distribution over the internet. HTTP streaming segments (decomposes) video data into short-time (for example, 10-second) TS (Transport Stream) files, and transmits them via HTTP data along with a playlist (available in formats such as M3U and MPD files) that instructs playback. The viewer's terminal then plays the received TS files continuously according to the playlist. HTTP streaming includes formats such as HLS (HTTP Live Streaming) and MPEG-DASH (Dynamic Adaptive Streaming over HTTP).

[0006] In computer networks, there are two methods for transmitting data: one using TCP (Transmission Control Protocol), which continuously retransmits the same data (packet) until the receiving node confirms receipt, and the other using UDP (User Datagram Protocol), which transfers data without the sending node confirming receipt by the receiving node. HTTP streaming uses TCP (Transmission Control Protocol) for data transfer, which suppresses signal quality degradation during data transfer. Furthermore, it uses the same transfer method as typical file transfers on web servers, making it possible to reduce delivery costs.

[0007] Patent Document 1 discloses a camera system technology that includes multiple camera devices that packetize and output video signals obtained by imaging, and a relay device connected to the multiple camera devices that relays the video signals transmitted from the multiple camera devices. The relay device includes a receiving unit that receives synchronized video signals between the multiple camera devices based on a synchronization signal, and a switching unit that selects and outputs video signals output from the multiple camera devices based on a control signal for selecting video. This allows for the selection of a desired video from among the videos from multiple camera devices and simplifies the system configuration.

[0008] Furthermore, Patent Document 2 discloses a video transmission device and a monitor system applicable to an IP-based broadcasting system, comprising: a video transmission device that receives an uncompressed video signal and generates an IP packet stream of uncompressed video data, and at the same time compresses the uncompressed signal and generates an IP packet stream of compressed video data; a video distribution system that distributes an IP packet stream of uncompressed video data as high-quality video from among the IP packet streams received from a plurality of the video transmission devices, and at the same time distributes an IP packet stream of compressed video data to a monitor system for video monitoring; and a monitor system that selects an IP packet stream to be displayed on the monitor from among the IP packet streams of compressed video data and displays it on the monitor.

[0009] Furthermore, Patent Document 3 discloses a technology for an information processing device that, by assigning video sources including an imaging device and camera work to portions of input text, can determine what kind of composition to use and how to switch between video sources including multiple cameras by looking at text (characters and sentences) that indicate the content structure, such as a script or interview notes.

[0010] Furthermore, Patent Document 4 discloses a video streaming distribution device that generates a clip information file that identifies video clips for playback extracted from streaming data, and a playback instruction file for instructing the playback of multiple segment files that constitute a clip, edits the clip information file including the playback instruction file, and transfers the clip information file to a public site, thereby enabling the publication of a clip information file including a playlist defining clips corresponding to highlight scenes on the Web while live streaming the video, for example, in a sports broadcast.

[0011] Furthermore, Patent Document 5 discloses a content distribution technology that uses IP multicast communication when distributing video in accordance with HTTP, and further reduces playback loss as much as possible, by having a playback means for playing a video file containing multiple TS packets, a UDP packet receiving means for receiving UDP packets transmitted via IP multicast by decomposing the video file containing multiple TS packets, and a video file reconstruction means for reconstructing the video file from the UDP packets received by the UDP packet receiving means and supplying it to the playback means in accordance with HTTP, wherein the video file reconstruction means reconstructs the video file in a manner that includes only complete TS packets when at least one of the multiple UDP packets corresponding to the video file is not received. [Prior art documents] [Patent Documents]

[0012] [Patent Document 1] Japanese Patent Publication No. 2015-62311 [Patent Document 2] Japanese Patent Publication No. 2016-9881 [Patent Document 3] Japanese Patent Publication No. 2012-14420 [Patent Document 4] Patent No. 5909546 [Patent Document 5] Patent No. 6490284 [Overview of the project] [Problems that the invention aims to solve]

[0013] To transmit video signals from network cameras in real time, RTP (Real-time Transport Protocol), which is based on UDP (User Datagram Protocol) and does not require the transmitting node to confirm receipt by the receiving node, is used. However, because RTP communication does not retransmit data if it does not reach the recipient, if packet loss occurs on the network, resulting in the loss of some of the transmitted data, the video and audio quality will deteriorate. Therefore, such data cannot be used directly as material for video content intended for viewers. On the other hand, TCP / IP (Transmission Control Protocol / Internet Protocol), which is the standard protocol used in computer networks, prioritizes data quality over real-time performance (timing is off), which means that video data transmitted via TCP / IP is unsuitable as live view footage for switching and camera work control. Furthermore, when a user selected PGM or PST via live view video and instructed the system to switch between them using a transition (a gradual switch with an effect), there was a problem in that it was not possible to output (produce or distribute) video content that reflected the transition in high-quality video data.

[0014] However, none of the prior art, including the technologies disclosed in the above-mentioned documents, solved these problems. The present invention was made to solve the above-mentioned problems, and aims to provide an information processing device suitable for outputting (producing or distributing) video content by performing a "transition," which is a gradual switching of video from multiple video sources, including network cameras, accompanied by effects. [Means for solving the problem]

[0015] The information processing apparatus of the present invention is an information processing apparatus that performs information processing for outputting (producing or distributing) video content using videos of a plurality of video sources including a network camera as materials, and is based on a video signal transferred by a transfer method in which a node that transmits from the video source transfers data without confirming the reception of a node that receives. Switching means for selecting PGM and PST, processing and synthesizing means for processing and synthesizing the selected PGM and PST with an effect (switching effect) designated by the user, and the same data (packet) until confirming a reception notification of a node that receives the video processed and synthesized by the processing and synthesizing means. Information processing means for outputting (producing or distributing) by a transfer method that continues to retransmit.

Effect of the Invention

[0016] According to the present invention, an information processing apparatus suitable for outputting (producing or distributing) video content by performing a "transition" which is a gradual switching with an effect on videos of a plurality of video sources including a network camera can be realized.

Brief Description of the Drawings

[0017] [Figure 1] It is a block diagram showing a main part configuration of the information processing apparatus according to the present embodiment. [Figure 2] It is a diagram showing an example of a state in which the information processing apparatus according to the present embodiment is used. [Figure 3] It is a block diagram showing a functional module configuration of the information processing apparatus according to the present embodiment. [Figure 4] It is a block diagram showing a functional module configuration of a switch terminal. [Figure 5] It is a diagram showing an example of a WEB application screen displayed on a touch panel display of a switch terminal by the information processing apparatus according to the present embodiment. [Figure 6]It is a diagram showing an example of a camera operation panel that an information processing apparatus according to this embodiment displays on a touch panel display of a switcher terminal. [Figure 7] It is a flowchart showing the flow of a process in which an information processing apparatus according to this embodiment generates and outputs sequence data. [Figure 8] It is a flowchart showing an example of information processing in which an information processing apparatus according to this embodiment generates and distributes a playlist based on information regarding which video source is switched at which timing. [Figure 9] It is a diagram showing an example of a programmed camera work panel that is displayed in response to pressing of a programmed camera work button 523. [Figure 10] It is a flowchart showing the flow of a process related to a programmed camera work function. [Figure 11] It is a diagram explaining the "size" of a shot. [Figure 12] It is a diagram showing an example of sequence data output by an information processing apparatus according to this embodiment. [Figure 13] It is a diagram showing an example of a playlist. [Figure 14] It is a diagram showing an example of EDL data. [Figure 15] It is a block diagram showing a main configuration of a network camera that can be connected to an information processing apparatus according to this embodiment. [Figure 16] It is a block diagram showing a configuration of a functional module related to content reception and playback of a viewer terminal that can be connected to an information processing apparatus according to this embodiment.

Embodiments for Carrying Out the Invention

[0018] Throughout this specification, "switching" means switching the video source (video signal from), and includes "cut switching" which is an instantaneous switch and "transition" which is a switch taking a certain amount of time.

[0019] Furthermore, the functions of "panning" (moving the camera's field of view horizontally), "tilting" (moving the camera's field of view vertically), and "zooming" (enlarging or reducing the field of view) are collectively referred to as "PTZ." Furthermore, switching (cut switching and transitions) and PTZ control (pan, tilt, and zoom control) are collectively referred to as "camera work."

[0020] Furthermore, a unit of video footage shot continuously from beginning to end with a single camera, without any start / stop in between, is called a "shot." Furthermore, a continuous sequence of images composed of shots or similar units of footage, arranged in a sequence, is called a "scene." Furthermore, a single recording session for a scene with the same structure is called a "take."

[0021] Furthermore, video productions made available for viewers to watch are referred to as "programs," "program content," "video content," etc. Furthermore, the numbers indicating the order in which scenes appear in the "video content" mentioned above are called "scene numbers." Furthermore, the video from the switching video source is called "PGM". Furthermore, the video source that is switched over (the one before the switch) is called "PST".

[0022] Preferred embodiments of the present invention will be described below with reference to the attached drawings.

[0023] Figure 1 is a block diagram showing the main components of an information processing device 100 according to an embodiment of the present invention. The information processing device 100 consists of a CPU 110, RAM 120, ROM 130, display 160, operation unit 170, and network I / F 190, all interconnected via an internal bus 180.

[0024] 110 is the CPU, which controls the operation of the information processing unit 100 itself (the operation of each part necessary for it to function), generates signals to control the operation of the video source, and processes various types of data.

[0025] RAM120 is a writable, high-speed storage device where the OS, various programs, and various data are loaded. It is also used as a workspace for the OS and various programs.

[0026] ROM130 is a non-volatile, read-only storage device used as a persistent storage area for the BIOS, various basic programs, and various basic data.

[0027] Display 160 is a display device such as an LCD (liquid crystal display panel).

[0028] The control unit 170 includes a keyboard, mouse, joystick, remote controller, touch panel (including touch display), etc.

[0029] On a touch display, users can operate the device and perform various functions by using touch gestures such as tapping (touching down and then touching up) or flicking (quickly moving while touching and then touching up) on GUI (Graphical User Interface) elements such as icons with their fingers or a stylus pen.

[0030] The network interface 190 is an interface for connecting to a network, allowing video sources such as network cameras 1500, viewer terminals 1600, and switcher terminals 400 to be connected via the network. Networks include, for example, the internet, wired LANs, wireless LANs, and mobile phone networks.

[0031] Figure 2 shows the overall configuration of a content output system including an information processing device according to an embodiment of the present invention. In Figure 2, 100 is an information processing device that performs content output (production or distribution), which distributes video content to a viewer terminal 1600 via a network and also provides a content production function by switching to a switcher terminal 400. The viewer terminal 1600 and the switcher terminal 400 are, for example, information terminals such as personal computers (hereinafter referred to as "PCs"), tablets, and smartphones.

[0032] In the example shown in the figure, "events" (210) such as ceremonies, sports, games, musical performances, scientific experiments, news reports, social information, meetings, and dramas (plays) are filmed from different directions (angles of view) by multiple cameras 1500-1, 1500-2, ...

[0033] The switcher terminal 400 has, for example, a touch panel display that integrates a display and a touch panel, and accepts GUI operations from the user (content creator). The operation screen (GUI) displayed on this touch panel display is displayed as a web application on a web browser launched on the switcher terminal 400 by the web server function of the information processing device 100. The switcher terminal 400 is connected to video sources such as network cameras 1500 via the network. Authentication is performed using the device's IP address, username, and password, enabling data access and control of the devices such as pan and tilt. The content creator manages the event by issuing cues (starting signals), and creates video content by switching video sources based on video signals (real-time live view) transmitted via UDP from the video source to the switcher terminal 400. At this time, sequence data is generated that records which video source was switched and at what timing, and this data is transferred and stored in the RAM 120 of the information processing device 100.

[0034] On the other hand, video data (video clips) from video sources such as cameras 1500-1, 1500-2, etc. are first stored (including caching or buffering) in the storage means (storage media such as RAM or SD cards) of the video source. When the video source is switched, the stored video data (video clip) is subdivided into segment data (TS files) and transferred and saved to a specific storage area (for example, a folder on online storage). If necessary, functional modules for data subdivision, transfer, and storage are introduced into the storage means (or the control means that manages it). Alternatively, if the video source itself has HTTP streaming capabilities, segment data (TS files) are generated and stored within each video source. Then, time information such as time codes attached as metadata to the segment data (TS files) is referenced and compared with the timing when the video source was switched, and the segment data (TS files) related to the switching timing are transferred and saved to a specific storage area (for example, a folder on online storage).

[0035] Next, the information processing device 100 according to the embodiment of the present invention generates a playlist (an index file such as an M3U file or MPD file) that contains the file path and time length to the segment data related to the timing when the video source was switched, and distributes it to the viewer terminal 1600 (registers the playlist in the distribution directory). The viewer terminal 1600 requests segment data by referring to a playlist (M3U file, MPD file, etc.), thereby enabling the streaming delivery of video content with multiple video sources switched.

[0036] Alternatively, the segment data generated from the video of each video source may be recorded and stored on the video source itself or on a recording device (recording medium) connected to it without going through a computer network, and then distributed (data acquired) to the viewer terminal 1600. In this case, the latency (delay) until the segment data becomes available for acquisition by the viewer terminal 1600, etc., is improved (delay is reduced).

[0037] As mentioned earlier, UDP transmission prioritizes real-time performance, resulting in reduced data quality (noise). Therefore, it is not appropriate to directly distribute UDP-transmitted video signals for viewer viewing or to use them as material for creating video content. The switcher terminal 400 uses UDP-transmitted video signals (live view) from each video source to monitor the event's progress in real-time while performing switching. It generates and outputs "sequence data" indicating which video data (video source) was switched and at what timing. This sequence data includes the ID (identifier) ​​of the video data (video clip) sent from the video source and is associated with the video data. Furthermore, the time codes at the in-point (start of switching insertion) and out-point (end of insertion) provide information on which video data (video source) was switched at what timing. In other words, the video data portions used as shots within a scene of video content are accessible. The user (content creator) can edit video data without loss of quality based on the above sequence data on a video editing terminal 240 connected to the recording media 230, and then add appropriate music effects, subtitles, video effects, etc., to complete video content that can be made available for viewing by viewers.

[0038] Furthermore, the information processing device 100 according to this embodiment generates a playlist containing file paths to video segment data at the timing when the video source is switched. Since TCP is used for the transfer and distribution of segment data, data quality is ensured, and high-quality video and audio without noise are delivered to the viewer terminal 1600. In other words, the information processing device 100 according to this embodiment switches video sources based on live view video transferred via UDP from multiple video sources, including network cameras, generates a content distribution playlist based on information about which video source was switched and at what timing, and based on the generated playlist, the video data of the switched video source is acquired by TCP transfer to the viewer terminal 1600.

[0039] By adopting the above configuration, the information processing device 100 according to this embodiment can output (produce or distribute) video content using video data with minimal quality degradation due to network transmission, while switching video sources in near real-time in accordance with the progress of "events" such as weddings, funerals, and other ceremonies or dramas (plays), and make it available for viewing by viewers. In recent years, WebRTC (Web Real-Time Communication) has emerged as a technology for real-time video communication using UDP in web browsers, and it is also possible to use WebRTC as a method for transferring live view video for switching from the video source. Furthermore, if the web browser cannot display the video stream via UDP, the network camera may be sent a snapshot acquisition command as defined by ONVIF, as described above, to continuously acquire still images (JPEG images), and the camera may be switched based on these images. Still images (JPEG images), even if continuous, cannot be directly presented to viewers because their frame rate (number of display frames per unit time) is insufficient. However, based on information regarding which video source was switched and at what timing, along with the video data output from the said video source, it is possible to output (produce or distribute) video content without noise or frame rate degradation for viewers to watch. In other words, even when a web browser does not support video streams via UDP, it becomes possible to implement an information processing system that displays a switching mechanism on the web browser, allowing the user to switch video sources and output (produce or distribute) video content.

[0040] Figure 3 is a block diagram showing the functional module configuration of the information processing device 100 according to this embodiment. The information processing device 100 includes a storage means 301, a communication means 302, a segment data generation means 303, a playlist generation means 304, a playlist distribution means 305, a segment data distribution means 306, a switcher means 307, and a system clock 308. The functions shown in Figure 3 are realized by information processing performed by software stored within the device, utilizing various hardware resources. For example, the switcher means 307 is realized by having the CPU 110 process information using software stored in the ROM 130 of the information processing device 100, and displaying a touch GUI of a web application on the touch panel display of the switcher terminal 400.

[0041] The storage means 301 stores video content in the form of a combination of video clip files output from a video source and sequence data recording the switching timing, or segment data obtained by subdividing the video clip files. Furthermore, the IDs (identifiers) and passwords of users of the switcher terminal (users of the web application), content viewers, and advertisers are stored in the creator database, viewer database, advertiser database, etc., respectively.

[0042] The communication means 302 transmits and receives various data and commands between the switcher terminal 400, the viewer terminal 1600, and other devices. For example, it communicates with the web browser of the switcher terminal 400 to display a web application. It also streams content in response to requests from the web browser of the viewer terminal 1600.

[0043] The segment data generation means 303 subdivides the video data output from the video source into segments (TS files) in minute time intervals (for example, 10 seconds), and generates these segments in accordance with a standard format such as "MPEG-2 TS". At the same time, it stores the ID of the video source, time data indicating when the video data was captured, and the file path to the generated segments as segment data information in the storage means 301.

[0044] The playlist generation means 304 generates a playlist that includes the file path and duration of the segment data related to the switching timing of the switched video source, from the segment data generated by the segment data generation means 303.

[0045] The playlist distribution means 305 distributes the playlist to the viewer terminal 1600 in response to a request from the viewer terminal 1600. Specifically, it registers and saves the playlist in a distribution directory (folder).

[0046] The segment data distribution means 306 distributes segment data (TS files) to the viewer terminal 1600. Specifically, it registers and saves the segment data (TS files) in a specific directory (folder) and lists the file path in the playlist.

[0047] The switcher means 307, in accordance with the progress of the event and at the instruction of the content creator user, selects and switches one of the video sources connected to the switcher terminal 400 as the constituent material for the event's video content at that time. The system clock 308 synchronizes with the time server to keep time.

[0048] Figure 15 is a block diagram showing the main components of a network camera 1500 that can be connected to the information processing device 100 according to this embodiment. The network camera 1500 consists of a CPU 1510, RAM 1520, ROM 1530, imaging unit 1540, memory interface 1550, network interface 1560, PTZ control unit 1570, and external interface 1580, all interconnected via an internal bus 1590.

[0049] The 1510 is the CPU, which controls the operation of the network camera 1500. RAM1520 is a writable, high-speed storage device where the OS, various programs, and various data are loaded. It is also used as a workspace for the OS and various programs. ROM1530 is a non-volatile, read-only storage device used as a persistent storage area for the BIOS, various basic programs, and various basic data. The identification ID (such as an IP address) used to identify the network camera 1500 is stored in RAM 1520 or ROM 1530. The imaging unit 1540 converts an optical image using an imaging sensor such as a CCD or CMOS into an electrical image and transfers it to the CPU. The Memory I / F1550 connects to recording media such as SD cards and HDDs to write and read video signals and various types of data. The network interface 1560 is responsible for communication with the information processing device 100 and various clients over the network, following protocols such as the Internet Protocol and ONVIF. ONVIF is a protocol developed by the Open Network Video Interface Forum that implements commands for controlling devices and initiating the distribution of images and metadata. The PTZ control unit 1570, in a PTZ camera having a pan-tilt-zoom mechanism, obtains pan-tilt-zoom states and performs PTZ control according to operation instructions. External I / F1580 is used to connect accessories such as microphones and viewfinders. These do not necessarily have to be directly attached to the camera body; they can be connected via a cable through a terminal, or they can be connected wirelessly. When audio is picked up by the microphone, it is recorded as an integrated audio track with the video signal.

[0050] Figure 16 is a block diagram showing the configuration of a functional module related to content reception and playback of a viewer terminal 1600 that can be connected to the information processing device 100 according to this embodiment. The viewer terminal 1600 includes a communication means 1601, a playlist acquisition means 1602, a segment data acquisition means 1603, a buffer means 1604, and a playback means 1605.

[0051] The communication means 1601 communicates with the information processing device 100 using a protocol such as HTTP. The playlist acquisition means 1602 requests the information processing device 100 to acquire a playlist and acquires the playlist from the information processing device 100. Before playback of all segment data listed in the acquired playlist is completed, the playlist acquisition request is resent to acquire (reload) the playlist again. If the acquired playlist indicates that it is the last playlist, the playlist acquisition is terminated. The segment data acquisition means 1603 acquires segment data in the order listed in the acquired playlist. That is, it sends an acquisition request to a URL (Uniform Resource Locator) that indicates the file path of the TS files listed in the playlist. The buffering means 1604 buffers (temporarily holds) the acquired segment data for a certain period of time. The playback means 1605 reads the segment data buffered in the buffer means 1604 in the order listed in the playlist and plays it back.

[0052] Figure 4 is a block diagram showing the functional module configuration of the switcher terminal 400. In Figure 4, 401 is a video source thumbnail display means, which generates a thumbnail image from the video signal (live view) of the video source connected to the information processing device 100 and displays it on the touch panel display of the switcher terminal 400.

[0053] 402 is a main menu display means, which displays the main menu on the switcher terminal 400. 403 is a volume level display means, which displays the volume level of the audio signal of the user-selected (switched) video source (hereinafter referred to as "PGM") among the video sources connected to the information processing device 100 on the touch panel display of the switcher terminal 400. 404 is an "ON AIR" indicator display means that the user has instructed it to start, and it displays on the touch panel display of the switcher terminal 400 that it is outputting sequence data consisting of information such as the video source being switched and the time of switching. 405 is an output start / end control means, which controls the start and end of output such as sequence data.

[0054] 406 is a video monitoring device, which displays the video signal (PGM) of the user-selected (switched) video source on the monitor unit 506 on the touch panel display of the switcher terminal 400. 407 is a means for searching, adding, and deleting video sources, and performs searching, adding, and deleting video sources that can be connected to the information processing device 100. 408 is a switching means that, upon user instruction, selects (switches) one of the video sources connected to the information processing device 100 as a PGM and displays the video signal on the monitor unit 506 on the switcher terminal 400's touch panel display. 409 is a camera control device that generates camera control commands based on user input and sends them to the target camera. 410 is a transition mechanism that controls the switching of a video source selected as a PGM to another video source over a certain period of time. 411 is a property means, which writes / reads properties to / from the video source sequence (timeline). 412 is a device configuration means, used to configure and adjust settings such as the display text size, screen brightness, volume, and WiFi connection on the switcher terminal 400's touch panel display.

[0055] 413 is the system clock and synchronizes with the time server to keep time. 414 is a data output method that outputs the generated sequence data in data formats such as SMIL (Synchronized Multimedia Integration Language), CSV (Comma Separated Value), EDL (Edit Decision List), and AAF (Advanced Authoring Format). 415 is a programmed camera work mechanism that programs a series of camera operations (switching and PTZ control) for each scene. 416 is a camera work execution means, which sequentially executes camera work programmed by the programmed camera work means 415 based on user instructions.

[0056] Network cameras include those capable of panning, tilting, or zooming. The information processing device 100 according to this embodiment has PTZ control means for controlling the panning, tilting, and zooming of such network cameras. By adopting this configuration, it becomes possible to create video content using video data with minimal quality degradation due to network transmission, while controlling the pan, tilt, and zoom of network cameras in near real-time in accordance with the progress of "events" such as weddings, funerals, and dramas (plays), and to make it available for viewers to watch. Furthermore, in this embodiment, the switching means and the PTZ control means are provided as an integrated unit. By adopting this configuration, it becomes possible to create video content that incorporates "direction" by coordinating switching and PTZ control in line with the progress of "events" such as weddings, funerals, and dramas (plays). For example, in video content created by filming a wedding, the camera could zoom in on the bride as she reads her "letter of gratitude to her parents," and then switch to a camera showing her parents the moment she finishes reading the letter.

[0057] Figure 5 shows an example of a web application screen displayed on the touch panel display of a switcher terminal 400 by an information processing device 100 according to an embodiment of the present invention. 501 is the GUI for the switcher application. 502 is the menu button that brings up the main menu. 503 is an audio meter that indicates the volume level. 504 is the "ON AIR" indicator, which shows whether the device is recording (sequence data recording) or broadcasting (streaming). Button 505 is the start button that signals the start of "ON AIR" (start of recording or broadcasting). When start button 505 is pressed and recording or broadcasting begins, start button 505 is replaced by stop button 505-2. 506 is the monitor section, which displays the video (PGM) of the selected (switched) video source.

[0058] 507 is the video source section, an area that displays thumbnails of switchable video sources. The video source section 507 displays video source thumbnails 5071, 5072, ... The video source thumbnails are thumbnail icons that represent multiple video sources connected to the information processing device 100 via a network, and each video source displays a thumbnail of the video it is outputting. In the example in Figure 2, cameras 1500-1, 1500-2, ... are video sources, and the video they are capturing is displayed in real time as video source thumbnails. If no capture is taking place, the last frame up to that point is displayed in a static state, or a so-called "black screen" is displayed. In the example shown in Figure 5, images captured by three PTZ (pan-tilt-zoom) cameras and a single video feed are displayed. This video feed refers to, for example, pre-recorded video data stored in a recording medium such as the RAM 120 of the information processing device 100 or online storage. Such video feeds can also be used as material for video content. 508 is a plus button, used to search for and add video sources to be displayed on the video source unit 507. Button 509 is a camera control button for controlling each video source. When the user presses camera control button 509, the camera operation panel 600 is displayed. The camera operation panel 600 is displayed for each camera. It can also be displayed as a so-called floating window, and its position can be moved. The 510 is a switching type button that changes the switching type. It consists of a cut selection button 510-1 for selecting "cut," which is an instantaneous switch, and a transition selection button 510-2 for selecting "transition," which is a gradual switch accompanied by an effect. The cut selection button 510-1 and the transition selection button 510-2 are mutually exclusive (alternative) selectable, so-called toggle buttons.

[0059] 511 is a rating button that allows users (content creators) to rate (evaluate) the recorded takes. 512 is the timeline area that displays the so-called "filmstrip" (a sequence of thumbnail shots that make up a scene). 513 is the comment area in the timeline region, 514 is the timecode scale, and 515 is the video source thumbnail display area in the timeline region that displays the video source (thumbnail) that was switched at the timing of that timecode. The length of each video source thumbnail along the timecode scale 514 corresponds to the duration of that "shot". The comment area 513 displays comment text entered by the user. Button 516 is the play button. When the play button 516 is pressed, the recorded video sequence (recorded in RAM 120 by connecting live views from each video source) is played back from the timing indicated by the sequence marker 517 on the timecode scale 514 and displayed on the monitor unit 506. When the play button 516 is pressed, it is replaced by the stop play button 516-2. Also, if there is no playable video sequence (no video sequence has been recorded), the play button 516 is disabled (grayed out). Button 518 is the properties button, which displays the properties panel, allowing you to add or change scene titles, take numbers, etc., to the video sequence displayed on the timeline in the timeline area 512. You can also recall and play previously created scenes and takes from the properties panel. 519 is a zoom slider that adjusts the zoom level at which the timeline is displayed. 520 is a fader, a slider used to switch between effects during transitions. 521 is a scene title combo box that displays the scene title. 522 is a take number text box that displays the take number. Button 523 is the Programmed Camera Work button, which displays the Programmed Camera Work panel for performing the Programmed Camera Work function. The Programmed Camera Work function will be explained later.

[0060] Since the display of the switcher terminal 400 is configured as a touch panel display integrated with the touch panel, users can execute various functions by performing touch gestures such as tapping (touching down and then touching up) or flicking (quickly moving while touching and then touching up) on GUI elements such as icons using their fingers or a stylus pen. For example, by tapping on the thumbnails 5071, 5072, ... of the video source unit 507, the corresponding video source can be selected (switched) as a PGM. At this time, a red frame is placed around the thumbnail to indicate that the corresponding video source has been selected. The video from that video source is also displayed on the monitor unit 506. In addition, the ID of the video source (or the video clip generated from that video source) and the time (timecode) when it was tapped are recorded as sequence data. When the user taps the start button 505, recording of sequence data and video sequence (a sequence of live views of switched video sources connected in one continuous sequence) begins. When a user touches one of the thumbnails in the video source unit 507 and then touches down on another thumbnail, the video source corresponding to the second thumbnail touched is set as the next preset video (hereinafter referred to as "PST"). Then, when the first thumbnail touched is touched up while the second thumbnail is still being touched, the transition means 410 gradually changes the PGM video displayed on the monitor unit 506 from the video source of the first thumbnail touched to the video of the PST (the video source of the second thumbnail touched), accompanied by an effect (transition effect). In other words, in this embodiment, transitions can be instructed by touch gestures on the touch display means. With this configuration, users can distinguish between "cuts (instant switching)" and "transitions" by the difference between "one-point touch" and "two-point touch," making the operation method easy to understand and convenient.

[0061] The user can also perform transitions by operating the fader 520. That is, when the user operates the fader 520, the transition means 410 gradually switches the PGM video displayed on the monitor unit 506 to the video source set as the PST, according to the amount the fader 520 is slid. This type of transition control by fader operation is consistent with the operability of conventional switcher devices and is convenient because it allows the user to perform transitions at their desired timing. When video source switching (cuts and transitions) is performed, the ID of the video source (or video clip) and the start and end times (timecodes) of the switch are recorded as sequence data. 524 is a transition effect dropdown menu that allows you to select a transition effect. Transition effects include "Dissolve," "Wipe," "Slide," "Squeeze," "Iris," and "Zoom." The transition effect dropdown menu 521 is activated when the transition selection button 510-2 is selected (it is grayed out when the transition selection button 510-2 is not selected). 525 is a Duration Time Up / Down control that sets the Duration Time, which is the time it takes for a transition to switch. Duration Time Up / Down 522 is activated when the Transition Selection button 510-2 is selected (it is grayed out when the Transition Selection button 510-2 is not selected).

[0062] Figure 6 shows an example of a camera operation panel displayed on the touch panel display of the switcher terminal 400 by the information processing device according to this embodiment. The camera operation panel 600 includes a video display unit 601, a "Subject Tracking" checkbox 602, a "Tracking during PGM" radio button 603a, a "Do not track during PGM" radio button 603b, a "Size Setting Tracking" checkbox 604, a "Size Setting" pull-down menu 605, a "Adjust Continuity" checkbox 606, a close button 607, a subject selection frame 608, a pan-tilt controller 609, a zoom controller 610, a focus controller 611, and a setting menu button 612. The video display unit 601 is an area that displays the video being captured by the corresponding camera. The Pan-Tilt Controller 609 is a joystick-like GUI used to control the pan and tilt of compatible cameras. The Zoom Controller 610 is a slider used to control the zoom of a compatible camera.

[0063] The user can also perform PTZ control of the corresponding camera by pinching in or out on the video display unit 601. Specifically, by simultaneously touching down two points on the video display unit 601 and bringing the touch positions closer together (pinch-in), the user can perform PTZ control of the corresponding camera so that the field of view (angle) corresponding to the area displayed across the entire video display unit 601 at the time of touch-down zooms out and pans and tilts toward the midpoint between the two touch positions (see Figure 6(b)). Conversely, by simultaneously touching down two points on the video display unit 601 and moving the touch positions further apart (pinch-out), the user can perform PTZ control of zoom-in and pans and tilts so that the area near the midpoint between the two touch positions expands to cover the entire video display unit 601 (see Figure 6(c)). This operating method allows users to perform pan-tilt and zoom operations in an integrated manner using touch gestures, which is convenient. When this operation is performed, the following ONVIF command is sent to the network camera: AreaZoom=x1-x0,y1-y0,w1-w0,h1-h0 Here, x1 and y1 are the coordinates of the midpoint of the two touch points after the pinch operation, x0 and y0 are the coordinates of the midpoint of the two touch points before the pinch operation, w1 and h1 are the width and height between the touch points after the pinch operation, and w0 and h0 are the width and height between the touch points before the pinch operation. However, the width and height of the entire video display unit 601 are set to 200 and 200, respectively.

[0064] In general, PTZ (Push-Through) movement is considered most suitable for smooth motion if it uses the so-called EASE IN / EASE OUT method, which involves starting slowly, gradually accelerating, and then slowing down further in the middle before stopping. Therefore, when PTZ control is performed using the pinch operation described above, a speed control command may be automatically sent to ensure that the PTZ operation becomes an EASE IN / EASE OUT operation within the duration of the PTZ movement. Furthermore, when pan-tilt and zoom are performed in an integrated manner, it is generally considered undesirable, often described as "flickering," when a subject that has entered the field of view due to pan-tilt is cut off again (no longer in the field of view) by zooming in, or a subject that has entered the field of view due to zooming is cut off again by pan-tilt. Therefore, the change in the optical axis angle of pan-tilt and the change in focal length due to zooming may be calculated and controlled to prevent such entry and exit of the subject.

[0065] Incidentally, when a user is performing PTZ control, it is convenient to know how much PTZ range is available relative to the field of view being captured at that time. Therefore, when the user approaches a limit where further operation would stop the zoom or pan-tilt, the display on each side of the display frame of the video display unit 601 is changed to something different from when the user is not approaching such a limit, thereby visually notifying the user that there is little PTZ range left. For example, the frame surrounding the image display unit 601 is displayed with an indistinct boundary, blurring the edges of the inner image. When approaching the wide-angle end where further zooming out is impossible, or the telephoto end where further zooming in is impossible, the degree of blurring is reduced, and the boundary line of the frame becomes clearly visible. Similarly, when approaching the limit of panning or tilting, the boundary line indicating the panning or tilting direction becomes clearly visible. This display allows the user to perform PTZ control appropriately while considering the remaining PTZ range. Note that the above notification method using the display unit frame is one way to intuitively notify the user of the PTZ limit, and other notification methods, such as displaying text like "Approaching the limit of zooming out," may also be used. The focus controller 604 is a slider used to control the focus of the compatible camera. Depending on the capabilities of the compatible camera, it can also be set to autofocus mode. The settings menu button 612 displays a menu for setting the white balance, iris, frame rate, etc., of the compatible camera. Depending on the capabilities of the compatible camera, it may also be possible to set it to auto white balance, auto iris, etc.

[0066] The "Subject Tracking" checkbox 602 is used to set the system to automatically track a specific subject using PTZ (Plan-Til-Through) motion. Automatic tracking technology, which controls pan and tilt to keep a detected moving object near the center of the screen, is already in practical use in fields such as surveillance cameras. By selecting (checking) the "Subject Tracking" checkbox 602, the user can set the corresponding camera to automatically track a specific subject. When the "Subject Tracking" checkbox 602 is selected (checked), a subject designation frame 608 is superimposed on the video display unit 601, and the user can move the frame using touch operations, etc., to set the subject at the frame's position as the subject to be tracked. The "Tracking during PGM" radio button 603a and the "Do not track during PGM" radio button 603b are radio buttons that set whether or not to perform the above-mentioned automatic tracking (automatic PTZ) when the corresponding camera is being switched. They are activated when the "Subject Tracking" checkbox 602 is selected (checked) (they are grayed out when the "Subject Tracking" checkbox 602 is not selected (checked)). The main purpose of the above-mentioned subject tracking is to keep the subject in the frame, which may result in awkward and unnatural camera work. Therefore, by not performing tracking during PGM, which is the video that viewers see, it is possible to eliminate the risk of unnatural camera work.

[0067] The "Size Setting Tracking" checkbox 604 is used to set the system to track a specific subject while maintaining its "size," and is enabled when the "Subject Tracking" checkbox 602 is selected (checked). (When the "Subject Tracking" checkbox 602 is not selected (checked), it is grayed out.) Here, "size" refers to how large a subject appears within the frame, relative to a human subject, and includes full shot (FS), full figure (FF), knee shot (KS), waist shot (WS), bust shot (BS), up shot (US), and close-up (CU) (see Figure 11). The "Size Settings" dropdown 605 is used to select and set the "Size" mentioned above, and is enabled when the "Size Setting Tracking" checkbox 604 is selected (checked). (When the "Size Setting Tracking" checkbox 604 is not selected (checked), it is grayed out.)

[0068] The "Adjust transition" checkbox 606 is a checkbox that sets whether or not to perform automatic PTZ adjustment to ensure a smooth transition (cutback) when the corresponding camera's image is switched (to PGM) with the image from the previously PGM video source. This is enabled when the "Size setting tracking" checkbox 604 is selected (checked). (When the "Size setting tracking" checkbox 604 is not selected (checked), it is grayed out.) When an image is switched from PGM, if the "size" of the shots before and after the switch is the same (for example, if both were "bust shots"), the transition will appear unnatural if the length from the top of the subject's head to the top of the screen is different. Similarly, if the "size" of the shot after the switch is tighter than the "size" of the shot before the switch (for example, if switching from a "waist shot" to a "bust shot"), the transition will appear unnatural if the length from the top of the subject's head to the top of the screen is not shorter before and after the switch. Furthermore, if the "size" of the shot after the switch is looser than the "size" of the shot before the switch (for example, when switching from a "bust shot" to a "waist shot"), it will look unnatural if the length from the top of the subject's head to the top of the screen does not increase before and after the switch. Therefore, when the "Adjust connection" checkbox 606 is selected, the image of the video source designated as PGM is analyzed to determine or estimate the "size" and the length from the top of the subject's head to the top of the screen. If the "size" of the PGM and the setting size in the "Size setting" dropdown 605 are the same, the corresponding camera is tilted so that the length from the top of the subject's head to the top of the screen is also the same. Also, if the setting size in the "Size setting" dropdown 605 is tighter than the "size" of the PGM, the corresponding camera is tilted so that the length from the top of the subject's head to the top of the screen is shorter than that of the PGM. Also, if the setting size in the "Size setting" dropdown 605 is looser than the "size" of the PGM, the corresponding camera is tilted so that the length from the top of the subject's head to the top of the screen is longer than that of the PGM. By doing this, it becomes possible to eliminate the risk of unnatural transitions between shots when switching from a program (PGM) to an image.

[0069] The close button 607 is used to close (hide) the camera operation panel 600. The subject designation frame 608 is a marker (GUI) that allows the user to indicate and set the subject at the frame's location as the subject to be tracked. The camera operation panel 600 can be displayed for each camera. Furthermore, it can be displayed as a so-called floating window, allowing its position to be moved. As described above, the information processing device 100 according to this embodiment allows for PTZ control of a PTZ-capable network camera by touch gestures on a touch display. With this configuration, the user can perform pan-tilt and zoom operations in an integrated manner. Furthermore, it is extremely convenient as it is intuitively easy to understand how the field of view is changed by PTZ control. Furthermore, the information processing device 100 according to this embodiment detects moving objects in an image based on a video signal transmitted by a transmission method in which a node transmitting from a PTZ-capable network camera transmits data without confirming receipt by a receiving node, and performs PTZ control of the network camera to keep the detected moving objects near the center of the screen. This PTZ control needs to be performed based on near real-time live view video. However, as already explained, live view video transmitted via UDP is not suitable for use in video content provided to viewers. Therefore, by performing information processing such as TCP / IP transmission or HTTP streaming on video data separate from the live view video, it becomes possible to output (produce or distribute) high-quality video content.

[0070] Figure 11 is a diagram illustrating "size". Full Shot (FS) is the widest angle of view the camera can achieve. A full figure (FF) is a size that includes everything from the toes to the top of the head. The Knee Shot (KS) is a size that covers from the knee to the top of the head. A waist shot (WS) is a size that covers from the waist to the top of the head. A bust shot (BS) is a size that includes the area from the chest to the top of the head. A close-up shot (US) is a size where the face fills the entire screen. A close-up (CU) is a size where a body part such as the eyes, mouth, hands, or feet fills the entire screen.

[0071] Figure 7 is a flowchart showing the processing flow of the information processing device 100 according to this embodiment, which displays a video source thumbnail corresponding to the video source, changes the video displayed on the monitor unit 506 in response to a switching operation (including fader operation) by the user, and generates and records sequence data when in REC mode. The processing in this flowchart is realized by the CPU 110 of the information processing device 100 controlling each part based on input signals and programs. The CPU 110 displays the GUI 501 shown in Figure 5 on the touch panel display of the switcher terminal 400 (step S701). At this point, the information processing device 100 is ready to accept GUI operations from the user. If the user presses the plus button 508 (YES in S702), the CPU 110 searches for network-connectable video sources and displays them in a list (S703). When the user selects one or more video sources and closes the list (YES in S704), the video source thumbnails of the selected video sources are added and displayed in the video source section 507 (S705). If the user's operation is the pressing of a camera control button on the PTZ camera (YES in S706), the CPU 110 displays the camera operation panel 600 (S707). As mentioned above, the camera operation panel 600 has a video display unit 601, a pan-tilt controller 602, a zoom controller 603, a focus controller 604, a setting menu button 605, and a close button 606. The lens focal length at the wide-angle and telephoto ends of the zoom, the pan-tilt movable range, and other specifications are acquired using the ONVIF protocol, etc., as mentioned above when the PTZ camera is added as a video source. In addition, ONVIF commands, etc., are generated in response to the user's operation on the camera operation panel 600 to control the PTZ camera (S708). When the user presses the close button 606 (YES in S709), the camera operation panel 600 is closed (S710).

[0072] If the user's operation is pressing the start button 505 (YES in S711), the CPU 110 sends a command to the information processing device 100 instructing the network-connected video source (the video source displayed as a thumbnail in the video source unit 507) to start recording to online storage or recording media (S712). Also, if the scene title combo box 515 is blank (YES in S713), the scene title input panel is displayed to prompt the user to enter a scene title (S714). Here, the scene title is a combination of the title of the video content as a "finished product" (program title) and the scene number, for example, "Watanabe Family Wedding Scene 1". Furthermore, the take number in the take number text box 516 is set to the latest take number of the scene title plus 1 (S715). Also, the start button 505 is replaced with the stop button 505-2 (S716). In addition, the generation and recording of sequence data consisting of the ID of the currently switched video source (or video clip) and the time of switching (timecode) is started (S717). Note that a URL (Uniform Resource Locator) can be used for the ID of the video source or video clip. Also, a so-called filmstrip is displayed in the timeline area 512, in which the names or thumbnails of the switched video sources are arranged according to the timecode (S718).

[0073] When a user taps one of the video source thumbnails (YES in S719), the CPU 110 displays the video of the corresponding video source on the monitor unit 506 (S720). It also places a red frame around the thumbnail (S721) to indicate that the corresponding video source is selected (a so-called "tally display"). It also obtains the time (time code) from the system clock 413 (S722). If the user touches one of the thumbnails on the video source unit 507 and then touches down on another thumbnail (YES in S723), the system switches to the video source corresponding to the previously touched thumbnail (S724), and sets the video source corresponding to the later touched thumbnail as a PST (preset video) (S725). Thumbnails set as PST may be displayed with, for example, a blinking red frame. When the two touched thumbnails are touched up almost simultaneously, and the user operates the fader 520 (YES in S726), the PGM video displayed on the monitor unit 506 is gradually switched to the PST video according to the amount the fader 520 is slid (S727). For the effect during the switch (transition effect), the effect selected by the user by pressing the effect change button 510 is used. Note that the initial value of the transition effect may be set to "dissolve" because it is a commonly used transition effect. If the user does not operate fader 520 (NO in S726), and the thumbnail that was touched later remains touched while the user touches up from the thumbnail that was touched earlier (YES in S728), the PGM video will be gradually switched to the PST video at the user-specified Duration Time (time required for switching) (S729). Then, the video source switching is recorded as sequence data (S730). The Duration Time is specified by selecting a value in the "Auto Transition Duration Time" combo box on the switching effect selection panel that appears when the effect change button 510 is pressed.

[0074] When the user presses the stop button 505-2 (YES in S731), the information processing device 100 sends a command to the network-connected video source (the video source displayed as a thumbnail on the video source unit 507) instructing it to stop recording to the recording medium (S732). It also swaps the stop button 505-2 with the start button 505 (S733). Furthermore, it stops recording the sequence data (S734). When the user presses the menu button 502 (YES in S735), the main menu is displayed (S736). When the user selects the "Save Data" command from the main menu (YES in S737), the sequence data is saved or output to a user-specified location (S738). Additionally, if the user selects the "End" command (YES in S739), the processing of this flow will be terminated.

[0075] Figure 12 shows an example of sequence data recorded or output by the information processing device 100 according to this embodiment. In Figure 12, the data has a structure described in SMIL (Synchronized Multimedia Integration Language) format. Here, SMIL is a language recommended for standardization by the W3C (World Wide Web Consortium) for synchronizing the playback of various data formats such as video, audio, and text. In the example sequence data in Figure 12, information unrelated to temporal behavior is first described in the elements. The meta attribute of an element is used to define the properties of the data (such as program title, scene number, take number, rating, notes, etc.) and to assign values ​​to those properties. The `layout` attribute of an element is used to determine the display position of an object. <layout type="″text / smil-basic-layout″">< / layout> By writing it this way, you can apply the default layout values ​​to all objects.

[0076] Information related to temporal behavior is described in the element. Also, in its child hierarchy <seq>Below the element, objects that are played consecutively in time are described. In other words, a sequence of switched video sources (video clips) is: <seq>The child hierarchy of an element <video>It is described as an element. each <video>The element uses the `src` attribute to specify a unique ID that identifies the output file from the video source. Such IDs can include, for example, the UMID (Unique Material Identifier) ​​defined by the SMPTE (Society of Motion Picture and Television Engineers). Alternatively, the `src` attribute can be specified as the file path to where the output file is stored (for example, `src=″D:\cam1\Clip1.avi″`). The clipBegin attribute specifies the start position (in point) of the transition insertion in a video clip (each output file from the video source), and the clipEnd attribute specifies the end position (out point) of the insertion. The clipBegin and clipEnd attributes can be specified using timecodes, for example, clipBegin=″hh:mm:ss:ff″. When switching between video clips involves transition effects, for example, <transition>It is described using elements. <transition>The element's `type` attribute specifies the type of effect, and the `dur` attribute specifies the duration of the transition.

[0077] The data format shown in Figure 12 is merely an example; various other formats are possible. For example, data formats such as CSV (Comma Separated Value), EDL (Edit Decision List), AAF (Advanced Authoring Format) are acceptable, as are newly defined data formats. When sequence data is exported in EDL (Edit Decision List) format, it becomes possible to import the data into existing video editing applications and edit the video. Representative video editing applications include Adobe Premiere Pro and Apple Final Cut Pro.

[0078] Figure 14 shows an example of EDL data. Figure 14 shows an example where program recording (recording of sequence data) begins 20 seconds after the start of video data output from each video source. In the diagram, 1401 represents the sequence title. Additionally, 1402 indicates whether the timecode of the edited sequence is in drop-frame or non-drop-frame format. Additionally, 1403 indicates whether the timecode of the video clip sequence for each video source is in drop-frame or non-drop-frame format. Also, 1404 represents the shot number. Additionally, 1405 represents the ID of the video source. Furthermore, 1406 represents a "2-channel audio / video" configuration. Furthermore, 1407 represents the switching type. Additionally, 1408 represents the timecode of the in-point of the video clip in the video source. Additionally, 1409 represents the timecode of the out point of the video clip in the video source. Furthermore, 1410 represents the timecode of the in-point of the edited video clip (video content). Furthermore, 1411 represents the timecode of the out point of the edited video clip (video content). When sequence data is output in a compatible data format such as EDL, it becomes possible to create video content with switched video sources, including network cameras, using existing video editing applications.

[0079] Figure 8 is a flowchart showing an example of information processing in which an information processing device 100 according to an embodiment of the present invention generates and distributes a playlist based on information regarding which video source was switched and at what timing. When a browser is launched on a user terminal (PC, tablet, smartphone, etc.) and the URL (Uniform Resource Locator) of the "video content production service" provided by the information processing device 100 according to an embodiment of the present invention is entered or instructed, the information processing device 100 displays the homepage on the user terminal (display unit) (S801). It also displays a "Creator User Login" button (S802). When the user clicks the "Creator User Login" button (YES in S803), authentication is performed using the user ID, password, etc. (S804). If the authentication result is positive (YES in S805), the GUI of the switcher application, as shown in 501 in Figure 5, is displayed on the user terminal's display (S806). If the user presses the plus button 508 (YES in S807), network-connectable video sources are searched for and displayed in a list (S808). When the user selects one or more video sources and closes the list (YES in S809), the video source thumbnails of the selected video sources are added and displayed in the video source section 507 (S810).

[0080] If the user's operation is the pressing of a camera control button on the PTZ camera (YES in S811), the camera operation panel 600 is displayed (S812). As mentioned above, the camera operation panel 600 has a video display section 601, a "subject tracking" checkbox 602, a "track even during PGM" radio button 603a, a "do not track during PGM" radio button 603b, a "size setting tracking" checkbox 604, a "size setting" pull-down 605, a "adjust connection" radio button 606a, a "do not adjust connection" radio button 606b, a close button 607, and a subject selection frame 608. When the user performs a touch gesture operation on the video display unit 601 (YES in S813), a control command corresponding to the operation is sent to control the PTZ camera of the corresponding camera (S814). Furthermore, when subject tracking is enabled (YES in S815), the corresponding PTZ camera is set to tracking mode (S816). The thumbnail of the corresponding PTZ camera in the video source unit 507 also displays that it is in tracking mode. When the user presses the close button 607 (YES in S817), the camera operation panel 600 is closed (S818).

[0081] If the user's operation is pressing the start button 505 (YES in S819), a command is sent to the user terminal to the network-connected video source (the video source displayed as a thumbnail in the video source unit 507) instructing it to output video data to a specific storage area (S820). In addition, segment data is generated by subdividing the video data output to the storage area (S821). Furthermore, a playlist is generated (S822) containing file paths that reference segment data generated from the video data of the video source being switched at that time, and this playlist is registered and saved in the distribution directory (folder) (S823). Also, the start button 505 is swapped with the stop button 505-2 (S824). Furthermore, the generation and recording of sequence data consisting of the ID of the video source (or video clip) being switched at that time and the time of switching (timecode) is initiated (S825). When the user taps one of the video source thumbnails (YES in S826), first, the time T1 switched from the system clock 308 is obtained (S827). Then, the switching type set by the switching type button 510 is determined, and if "Cut" is selected (YES in S828), the monitor unit 506 first displays the video source corresponding to the video source thumbnail (S829). A red frame is also placed around the thumbnail (S830) to indicate that the corresponding video source is selected (a so-called "tally display"). The imaging time attached to the segment data generated from the switched video source is then compared with the switching time T1, and a playlist is generated by listing the file paths to the segment data related to the video captured after time T1 (S831).

[0082] When "Transition" is selected as the switching type for the switching type button 510 (NO in S828), the PGM video displayed on the monitor unit 506 is gradually switched to the video (PST) of the video source corresponding to the tapped video source thumbnail, with a user-specified Duration Time (switching time) and a user-specified effect (switching effect) (S832). A red frame is also placed around the tapped video source thumbnail (S833). Furthermore, segment data for the switched video is generated by processing and combining the segment data related to the footage captured between time T1 and the Duration Time (switching time) of the previously switched video source (PGM) and the newly switched video source (PST) with a user-specified effect (switching effect) (S834), and this data is registered and saved in a specific directory (folder) (S835). Finally, a playlist is generated by listing the file path of this switched video segment data (S836). Then, the newly generated playlist overwrites and updates the playlist in the distribution directory (folder) (S837). When the user presses the stop button 505-2 (YES in S838), a command is sent to the connected video source (the video source displayed as a thumbnail on the video source unit 507) instructing it to stop outputting video data to a specific memory area (S839). The playlist in the distribution directory (folder) is also overwritten and updated with a note indicating that it is the last playlist (S840). The stop button 505-2 is then replaced with the start button 505 (S841). The recording of sequence data is also stopped (S842). When the user presses the menu button 502 (YES in S843), the main menu is displayed (S844). When the user selects the "Save Data" command from the main menu (YES in S845), the sequence data is saved or output to a user-specified location (S846). Additionally, if the user selects the "End" command (YES in S847), the processing of this flow will be terminated.

[0083] Figure 13 shows an example of a playlist in m3u format. Figure 13 shows an example where the camera switches from Camera_1 to Camera_2 at 999 seconds (16 minutes and 39 seconds) from the start of the program.

[0084] The segment data related to the video from Camera_1 and Camera_2 are as follows: http: / / www.contentsvision.com / sample / ts / camera_1 / http: / / www.contentsvision.com / sample / ts / camera_2 / The data shall be registered and stored. In addition, the segment data shall have the number of milliseconds (1 / 1000) of the time when the video imaging related to that data began included as the file name.

[0085] In the diagram, 1301 represents the file format. Additionally, 1302 represents the version number of the file format. Additionally, 1303 indicates the overall position of the first segment data (TS file) in the list. Furthermore, 1304 indicates how many seconds or less each segment of data (TS file) is divided into. Furthermore, 1305 represents the time length of the segment data (TS file) indicated by the file path on the next line. Furthermore, 1306 represents the file path (URL) to the segment data (TS file).

[0086] As described above, video content from multiple video sources, including network cameras, can be streamed via HTTP. TCP is used for transferring video data in HTTP streaming, making it possible to provide viewers with high-quality video content with reduced noise.

[0087] (Programmed camera work function) The programmed camera work function of the information processing device according to this embodiment will be described below. Generally, ceremonies such as weddings and theatrical productions (dramas) have pre-prepared schedules and scripts, and the camera work (switching and PTZ) for video content related to these ceremonies and theatrical productions is carried out according to these schedules and scripts. Therefore, it is convenient to be able to pre-program a series of camera movements and instruct them to be executed according to the progress of the event, thereby reducing errors and the burden on camera operators.

[0088] The programmed camera work function of the information processing device in this embodiment is a function that allows the user to pre-program (input) a series of camera movements, and then executes those programmed camera movements sequentially according to the user's instructions. Camera work input and execution instructions are performed by displaying the Programmed Camera Work Panel.

[0089] Figure 9 shows an example of a programmed camera work panel. The programmed camera work panel is displayed as a so-called floating window on the display of the switcher terminal 400 when the programmed camera work button 523 is pressed by the user. The programmed camera work panel 901 includes a monitor section 902, a REC button 903, an EDIT button 904, a NEXT button 905, a camera work section 906, an event number section 907, a text section 908, a BACK button 909, a SKIP button 910, and a pop-up 911.

[0090] The camera work section 906 is a section that shows the camera work in chronological order. Here, camera work refers to cut changes or transitions of the video source or PTZ control for a PTZ camera. The event number field 907 represents the number indicating the order in which the camera work is performed. Text field 908 is a field for displaying text (such as a scenario, progress chart, or script) that serves as a timing guide for camera work. The displayed text may include things like song lyrics, scale progression, beat count, or measure count. It can also be entered by the user via a keyboard or other control panel, imported from a file, scanned from a paper document, or photographed. If the imported file or document contains tables or layouts, only the text may be extracted and imported, or the tables and layouts may be preserved and pasted. In addition, so-called storyboards (continuity, storyboard) may be pasted. The monitor section 902 is an area that displays the video signal (PGM) of the selected (switched) video source. The REC button 903 is used to signal the start of recording (sequence data recording). The EDIT button 904 is used to enter a mode where you can edit (change, insert, delete, and reorder) the camera work and text fields. The NEXT button 905 is used to execute the currently focused camera action and move the focus to the next camera action. The BACK button 909 is used to move the focus to the previous camera movement. The SKIP button 910 is used to shift focus to the next camera movement. Pop-up 911 is a "speech bubble" that appears near the cursor position within the text and contains a "Backspace button," a "Edit block button," and a "Keyboard button," and is displayed for editing the text.

[0091] The user can set the Programmed Camera Work Panel 901 to EDIT mode (editable mode) and configure settings such as "Video Source Switching" and "PTZ Control" in the Camera Work section. Furthermore, camera work performed in the main window (GUI 501 in Figure 5) is saved as data and can be loaded into the Programmed Camera Work Panel 901 and reused as programmed camera work. In other words, for example, after performing switching or PTZ control in the first take of a scene, that camera work can be used as programmed camera work in subsequent takes (retakes), reducing the effort and errors involved in camera work operation.

[0092] Figure 10 is a flowchart showing the processing flow related to the programmed camera work function of the information processing device 100 according to this embodiment. When the user presses the programmed camera work button 523, the programmed camera work panel 901 is displayed as a so-called floating window (S1001). The information processing device 100 first determines whether there is data for camera work performed in the main window, and if such data exists (YES in S1002), it compares it with the programmed camera work data (camera work data displayed on the programmed camera work panel) (S1003). Then, if the data for the camera work executed in the main window differs from the programmed camera work data (YES in S1004), the message "Do you want to convert the executed camera work to programmed camera work?" is displayed, and the system waits for the user to select YES / NO (S1005). If the user selects YES (YES in S1006), the executed camera work is made into a programmed camera work and displayed in the camera work field (S1007). When the user presses the EDIT button (YES in S1008), the programmed camera work panel is set to EDIT mode (S1009). Specifically, the REC button and NEXT button are disabled (grayed out).

[0093] In EDIT mode, when the user taps a blank area in the Camera Work section (YES in S1010), a dialog box with the options "Switch Video Source" and "PTZ" (radio buttons) will be displayed at that location (S1011). However, if a PTZ-capable video source is not assigned to that location in the Camera Work section, the "PTZ" option will be grayed out and unavailable. When "Switch Video Source" is selected (YES in S1012), a dialog box is displayed with the options "Cut" and "Transition" (radio buttons), a pull-down list for selecting a video source, and a GUI similar to that of the camera control panel 601 (S1013). When the user selects "Cut" and one of the video sources, and the aspect ratio of the switched video source is determined (YES in S1014), the cut line, the name of the video source, and a thumbnail representing the aspect ratio of the switched video source are added and displayed in the camera work section (S1015).

[0094] When the user selects a "transition" and one of the video sources, and the field of view of the transitioned video source is determined (YES in S1016), the camera work section is updated with a cut line, the name of the selected video source, and a thumbnail representing the field of view of the transitioned video source (S1017). A transition bar, a combo box indicating the transition time, and an icon indicating the type of transition effect are also displayed (S1018). The combo box indicating the transition time and the icon indicating the type of transition effect have default values. The user can change the values ​​by tapping the combo box or icon. Each dialog box also has a cancel or close button, and when the user selects cancel or close, it returns to the previous state.

[0095] When the user selects "PTZ" (YES in S1019), a PTZ dialog box with a GUI similar to that of the camera control panel 601 is displayed (S1020). When the user performs GUI operations on the PTZ dialog and determines the field of view after PTZ (YES in S1021), the camera work section displays a cut line and a band indicating PTZ, a combo box indicating the switching time, and a thumbnail indicating the field of view after PTZ (S1022). The combo box indicating the switching time has a default value. The user can change the value by tapping the combo box.

[0096] When the user long-presses the text field (YES in S1023), a cursor is displayed within the text (S1024). A pop-up (speech bubble) is also displayed (S1025). The pop-up has a Backspace button, a Edit / Block button, and a keyboard button. When the user presses the BS button (YES in S1026), a backspace operation is performed to delete the character immediately preceding the cursor (S1027). If the target of deletion is a "cut line" rather than a character, the text in the text block where the cursor was located is merged with the previous block, and the text blocks corresponding to subsequent event numbers are shifted forward one position. When the user presses the edit block button (YES in S1028), the text from the cursor position in that text block moves to a newly created block immediately after that text block, and the text blocks corresponding to subsequent event numbers shift one position backward (S1029). When a keyboard button is pressed by the user (YES in S1030), a floating keyboard is displayed, allowing text to be inserted (S1031).

[0097] When the user presses the MENU button (YES in S1032), the menu is displayed (S1033). The menu contains the commands "Load Text", "Save Programmed Camera Work Data", and "Load Programmed Camera Work Data". When the user selects "Load Text" (YES in S1034), the file loading window is displayed (S1035). The file loading window has "Overwrite" and "Insert" options (radio buttons) and "OK" and "Cancel" buttons. Note that the "OK" button is disabled (grayed out) when no file is selected in the window. When the user selects "Overwrite" and a file is loaded (YES in S1036), all the text in the text field is overwritten with the text from the loaded file (S1037). When the user selects "Insert" and loads a file (YES in S1038), the text from the loaded file is inserted at the cursor's location (S1039). When the user instructs to "Save Programmed Camera Work Data" (YES in S1040), the data save window is displayed (S1041).

[0098] When the user instructs "Load Programmed Camera Work Data" (YES in S1042), the data file loading window is displayed (S1043). The data file loading window has a checkbox labeled "Load executed camera work" and "OK" and "Cancel" buttons. Note that the "OK" button is disabled (grayed out) when the checkbox is not checked or when no file is selected in the window. Also, when the checkbox is checked, file selection in the window is disabled (grayed out). When camera work data is loaded (YES in S1044), the camera work field is overwritten (S1045).

[0099] When the user exits EDIT mode (when the EDIT button, which was displayed in a pressed-down state, is tapped again) (YES in S1046), the programmed camera work panel is put into execution mode (S1047). Specifically, the REC button and NEXT button are enabled. When the user taps the NEXT button (YES in S1048), the camera work on the focused row is executed and the focus moves to the next row (S1049). When the camera work for the last row is executed (YES in S1050), all rows will not be in focus, and the NEXT button will be disabled (grayed out) (S1051). When the user taps the REC button (YES in S1052), recording begins (S1053). The operation during recording is the same as when the REC command is given in step S711 in Figure 7. When the user taps any of the rows in the "Camera Work" column, "Event Number" column, or "Text" column (YES in S1054), the focus is moved to that row (S1055). When the user presses the close button or taps any part of the main panel (YES in S1056), the programmed camera work panel is closed (S1057). Note that REC / REC stop and the display / hide of the programmed camera work panel are independent of each other. For example, even if REC is in progress when the programmed camera work panel is closed, REC will continue.

[0100] The data used for saving and loading programmed camera work consists of video source switching and PTZ control, each assigned a number (event number), and described in a way that correlates them with the timing of the schedule text and storyboard. Since it shares similarities with sequence data in that it describes "switching video sources," it may be used by extending the format of the sequence data. The following describes an example of extending sequence data written in SMIL (Synchronized Multimedia Integration Language) format into programmed camera work data.

[0101] First, each <video>The element will have an `event_Num` attribute added to it, which will specify the order in which the camera movements will be executed (event number). <video src="″D:\cam1\Clip1.mxf″" clipBegin="″hh:mm:ss:ff" clipEnd="″hh:mm:ss:ff″" event_Num="″1″">< / video> Also, <seq>In the child hierarchy of an element, for example <onvif>Describe PTZ control as an element. <onvif CamID="″cam1″" AbsolutePanTilt="″X,Y″" AreaZoom="″x,y,w,h″" dur="″hh:mm:ss:ff″" event_Num="″2″" / > Also, corresponding to the order (event number) in which camera work is executed, describe text (such as a progress chart or script) that serves as a timing guideline for executing that camera work as a <text_block> element. <text_block=″Dear guests, thank you very much for coming to today's wedding reception of ○○○○. From now on, the bride and groom will enter. Please give them a big round of applause. event_Num=″1″ / >

[0102] As described above, the user can program a series of camera work in advance and sequentially execute it in the preset order by instructing the NEXT button. This can reduce the labor and operation errors of camera work operations. Also, the user can plan the timing of camera work execution in contrast to the progress chart or script, further improving the operability. Also, the camera work executed in the main window can be used as programmed camera work. This reduces the labor of programming (inputting / setting) the camera work. Also, an embodiment in which the above text (such as scenarios, progress charts, scripts, song lyrics of songs, picture contents, scores, etc.) is reused to disclose and explain the content of video content, or plans and projects, and recruit or match advertisers, collaborators, staff, etc. is also suitable. When done in this way, it becomes possible to construct and operate platforms for planning, fundraising, production, distribution, circulation, etc. centered on video content creation. Furthermore, when camera work programmed in conjunction with the above-mentioned text (such as scenarios, schedules, scripts, song lyrics, storyboards, and musical scores) is executed and video content is produced, an embodiment is also preferable in which the text is reused to notify advertisers, collaborators, and staff who have been matched, or to make all or part of the produced video content accessible. In this case, the notification or access to all or part of the video content (playback, viewing, confirmation, etc.) can prompt action (behavior / execution) by advertisers, collaborators, and staff, thereby accelerating the progress of projects related to video content production and distribution.

[0103] (Creation of video content) As described above, when the information processing device 100 according to this embodiment switches between multiple video sources and records events, a video clip recording the video signals from the video sources and sequence data referencing the switched portions within the video clip in the order they were switched are created. This sequence data also functions as a "playlist," indicating the playback order of multiple video clips (or sections). Therefore, by playing the video clips (or sections) according to the sequence data using a video player, the video content of the recorded event can be viewed and played. For example, if the sequence data is written in SMIL format, a web browser can be used as a video player to play the video content. Furthermore, if the sequence data is in a format that can be read by a video editing device (video editing application), the video clips from the video source can be used as source data, and the video content of the recorded event can be written out (output) as a single video file.

[0104] Since sequence data is generated for each recording session (one take per scene), when video content consists of multiple scenes, the sequence data for each scene must be combined to correspond to all scenes (sequence data merging). For example, in the example sequence data in Figure 12 (an example of sequence data written in SMIL format), if elements at a lower level than the element are combined in the order of the "scene number" specified in the element's meta attribute, it will result in a playlist that combines multiple scenes and plays them sequentially in scene number order.

[0105] Furthermore, if a scene is reshot and multiple takes exist for each scene, the "rating" indicated in the element's meta attribute can be referenced. The take with the highest rating (or, if there are multiple, the one recorded later) can be combined into a "playlist" in scene number order, allowing the user to view and play video content consisting of scenes using their desired take. This is particularly convenient for video content such as plays (dramas), where reshots are common, as it streamlines the process of deciding which take to use for each scene and editing the video. Since editing sequence data (including combined data) as text itself produces results similar to video editing, the configuration and resource requirements of video editing equipment (video editing applications) can be simplified and made lighter. For example, a web application for video editing can be configured using a web page that has a text editor area for editing SMIL and a monitor area for playing back and displaying the SMIL. Furthermore, if the video server side references the file on which the information processing device 100 according to this embodiment writes sequence data, and based on that data sequentially transmits the video of the switched portion of the video clip on which the video data of the video source is written by the information processing device 100, then live streaming (with some delay) can be performed even during event recording (while sequence data is being written).

[0106] The information processing device 100 according to this embodiment switches between video sources based on live view video transmitted in near real-time using a transmission method in which transmitting nodes, including network cameras, transfer data without confirming receipt by receiving nodes. It then creates or distributes video content based on information regarding which video source was switched and at what timing, as well as the video data output from the video sources. By adopting this configuration, it becomes possible to create or distribute video content using video data with minimal quality degradation due to network transmission, while switching video sources in near real-time in accordance with the progress of "events" such as weddings, funerals, and other ceremonies or dramas (plays). In this way, the effort involved in setting up lines and the equipment costs are reduced, making video content production simpler, faster, and more cost-effective. In other words, an information processing device suitable for switching video sources, including network cameras, to create or distribute video content is realized.

[0107] (Other embodiments) The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention.

[0108] Furthermore, the present invention can also be realized by supplying software (programs) that realize the functions of the embodiments described above to a system or device via a network or various storage media, and by a process in which the computer (or CPU, MPU, etc.) of that system or device reads and executes the program. Therefore, in order to implement the functional processing of the present invention on a computer, the program code installed on the computer itself also implements the present invention. In other words, the present invention includes the computer program itself for implementing the functional processing of the present invention. In this case, as long as it has the functionality of a program, it may take the form of object code, a program executed by an interpreter, a script or macro executed by an application program such as a web browser, or an API (Application Programming Interface). It may also be implemented by being incorporated as part of another web service such as a Social Networking Service (SNS) using web programming techniques such as "Mashup". Furthermore, the present invention can also be implemented as a so-called web application that utilizes a web browser. Therefore, the web server pointed to by its URL (Uniform Resource Locator) constitutes an implementation of the present invention, regardless of whether it is a single piece of hardware or not. [Explanation of Symbols]

[0109] 100...Information processing device, 110...CPU, 120...RAM, 130...ROM, 160...Display, 170...Operation unit, 190...Network I / F, 180...Internal bus, 210...Event, 400...Switcher terminal, 1500...Video source such as network camera, 1600...Viewer terminal, 301...Storage means, 302...Communication means, 303...Segment data generation means, 304...Playlist generation means, 305...Playlist distribution means, 306...Segment data distribution means, 307...Switcher means, 308...System clock, 401...Video source thumbnail display means, 402...Me 403... New display means, 404... Volume level display means, 405... REC indicator display means, 406... Sequence recording start / end control means, 407... PGM video display means, 408... Video source search / add / delete means, 409... Switching means, 410... Camera control means, 411... Transition means, 412... Property means, 413... Device setting means, 414... System clock, 415... Sequence data output means, 416... Programmed camera work means, 501... GUI, 502... Menu button, 503... Audio meter, 504... ONAIR indicator, 505... Start button, 506... Monitor section, 507... Video source section, 508... Plus button, 509... Camera control button, 510... Switching type button, 511... Rating button, 512... Timeline area, 513... Comment area, 514... Timecode scale, 515... Video source thumbnail display area, 516... Play button, 517... Sequence marker, 518... Properties button, 519... Magnification slider, 520... Fader, 521... Scene time Torconbo box, 522…Take number text box, 523…Programmed camera work button, 524…Transition effect pulldown, 525…Duration time up / down, 600…Camera operation panel, 601…Video display, 602…"Subject tracking" checkbox, 603a…"Tracking during PGM" radio button, 603b…"Do not track during PGM" radio button, 604…"Size setting tracking" checkbox, 605…"Size setting" pulldown, 606… "Adjust connection" checkbox, 607... Close button, 608... Subject selection frame, 609... Pan / tilt controller, 610... Zoom controller, 611... Focus controller, 612... Settings menu button, 901... Programmed camera work panel, 902... Monitor section, 903... REC button, 904... EDIT button, 905... NEXT button, 906... Camera work section, 907... Event number section, 908... Text section, 909... BACK button, 910... SKIP button 911...Pop-up (speech bubble), 1510...CPU (camera), 1520...RAM (camera), 1530...ROM (camera), 1540...Imaging unit (camera), 1550...Memory I / F (camera), 1560...Network I / F (camera), 1570...PTZ control unit (camera), 1580...External I / F (camera), 1590...Internal bus (camera), 1601...Viewer terminal communication means, 1602...Playlist acquisition means, 1603...Segment data acquisition means, 1604...Buffer means, 1605...Playback means< / onvif> < / seq> < / video> < / transition> < / transition> < / video> < / video> < / seq> < / seq>

Claims

1. An information processing device that performs information processing for distributing video content using video from multiple video sources, including network cameras, as source material, A switching means that switches based on a video signal transmitted by a transmission method in which a node transmitting from the video source transfers data without confirming receipt by a receiving node, A sequence data generation means that generates sequence data that records which video source was switched and at what timing, A playlist generation means generates a playlist based on information regarding which video source was switched by the switching means and at what timing, The system includes an information processing means that distributes video content using a transfer method that continuously retransmits the same data (packets) until the receiving node confirms receipt, The switching means is capable of performing transitions that involve switching effects. An information processing device characterized in that, when the switching of video is accompanied by a transition effect, the sequence data records the type of transition effect and the duration of the transition.

2. A program for causing a computer to function as each of the means of the information processing apparatus described in claim 1.

3. A computer-readable recording medium storing a program for causing a computer to function as each of the means of the information processing apparatus described in claim 1.

Images