Camera management device and camera management method

Abstract

The present invention provides a camera management device that groups multiple camera devices together based on their ability to photograph the same subject. [Solution] The video distribution device 20, as a camera management device, is communicatively connected to multiple camera devices 10a etc. that photograph any of the subjects located in multiple locations, and multiple sound pickup devices (sound pickup units 13a etc.) installed in accordance with each of the multiple camera devices, and comprises a state information acquisition unit 241 and a grouping processing unit 242. The state information acquisition unit 241 acquires video information captured by the multiple camera devices or sound information picked up by the multiple sound pickup devices, and acquires state information that shows the change in state quantities related to the subject over time for each acquired piece of information. The grouping processing unit 242 groups the camera devices among the multiple camera devices that correspond to video information or sound information where the peak of the state quantity is within a predetermined time range as a group of camera devices belonging to a group that photographs subjects located in the same location.

Description

Technical Field

[0007]

[0001] The present invention relates to a camera management device and a camera management method.

Background Art

[0002] There is a multi-angle distribution system that arranges a plurality of video information captured by a plurality of camera devices on the same screen and distributes it to a viewer's terminal. By distributing the video information arranged in this way, the viewer can simultaneously visually recognize subjects photographed from different angles.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the multi-angle distribution system as described above, there may be a case where a plurality of video information arranged on the same screen is captured from subjects located at different places.

[0005] For example, when distributing a relay video of a golf tournament, there may be a case where video information of a plurality of camera devices that have photographed the 1st hole from different angles and video information of a plurality of camera devices that have photographed the 10th hole, which is a different location from the 1st hole, from different angles are arranged on the same screen.

[0006] In such a case, the multi-angle distribution system performs processing such as synchronizing or superimposing the same telop on a plurality of video information that captures subjects located at the same place among the video information arranged on the same screen. In order to perform such processing, it is necessary to group in advance the camera devices that capture subjects located at the same place.

[0007] When this grouping is done manually, it is necessary to regroup each time a camera device moves and the subject changes during the broadcast, which presents a problem as monitoring the target of each camera device and processing the grouping becomes time-consuming. Furthermore, as the number of camera devices used for broadcasting increases, the grouping process becomes more complex and time-consuming, and the risk of setting errors also increases.

[0008] The present invention aims to provide a camera management device and a camera management method that group multiple camera devices together according to the group of camera devices that photograph subjects located at the same location. [Means for solving the problem]

[0009] A camera management device according to one or more embodiments for achieving the above objective comprises: a plurality of camera devices for photographing any of subjects located in a plurality of locations, and a plurality of sound-collecting devices installed in conjunction with each of the plurality of camera devices for collecting ambient sounds, which are communicated to a plurality of camera devices and a state information acquisition unit that acquires video information captured by the plurality of camera devices or sound information collected by the plurality of sound-collecting devices, and acquires state information indicating the change over time of a state quantity related to the subject for each acquired piece of information; and a grouping processing unit that groups camera devices among the plurality of camera devices, where the peak of the state quantity indicated by the state information is within a predetermined time range, as a group of camera devices belonging to a group that photographs subjects located in the same location.

[0010] Furthermore, a camera management method according to one or more embodiments includes a camera management device that is communicatively connected to a plurality of camera devices that photograph any of a subject located in a plurality of locations, and a plurality of sound-collecting devices installed in conjunction with each of the plurality of camera devices to collect ambient sounds. The camera management device acquires video information captured by the plurality of camera devices or sound information collected by the plurality of sound-collecting devices. For each piece of acquired information, it acquires state information indicating the temporal change of a state quantity related to the subject. The camera devices corresponding to the video information or sound information in which the peak of the state quantity indicated by the state information is within a predetermined time range are grouped together as a group of camera devices that photograph subjects located in the same location. [Effects of the Invention]

[0011] According to a camera management device and camera management method according to one or more embodiments, multiple camera devices can be grouped together according to the group of camera devices that photograph subjects located at the same location. [Brief explanation of the drawing]

[0012] [Figure 1] This is an overall diagram showing the configuration of a video distribution system using a video distribution device as a camera management device according to one embodiment of the present invention. [Figure 2] This is a block diagram showing the configuration of a camera device used in a video distribution system according to one embodiment of the present invention. [Figure 3] This flowchart shows the grouping procedure performed by a video distribution device according to one embodiment of the present invention. [Figure 4] (a) to (d) are graphs showing the change over time in the amount of motion in video information acquired by a video distribution device according to one embodiment of the present invention. [Figure 5] Figure 4 shows a graph with detection windows set to detect the amount of motion in each video information within a predetermined time range ta. [Figure 6] This flowchart shows the pre-distribution processing procedure performed by a video distribution device according to one embodiment of the present invention. [Figure 7] It is an enlarged view of the change in the amount of movement of the video information within the detection window of FIG. 5. [Figure 8] It is an example of the composite video information distributed by the video distribution device according to an embodiment of the present invention and displayed on the viewer terminal. [Figure 9] It is another example of the composite video information distributed by the video distribution device according to an embodiment of the present invention and displayed on the viewer terminal. [Figure 10] It is an example of the updated composite video information distributed by the video distribution device according to an embodiment of the present invention and displayed on the viewer terminal.

Mode for Carrying Out the Invention

[0013] Hereinafter, a video distribution system using a video distribution device as a camera management device according to an embodiment of the present invention will be described with reference to the drawings. In this embodiment, a case where the video distribution system live-distributes the situation of a golf tournament will be described as an example.

[0014] 〈Configuration of a Video Distribution System Using a Video Distribution Device According to an Embodiment〉 FIG. 1 is an overall view showing the configuration of a video distribution system 1 using a video distribution device 20 according to this embodiment. The video distribution system 1 shoots and distributes the situation of the play being carried out at a plurality of holes located at a plurality of locations in a golf tournament.

[0015] The video distribution system 1 includes a first camera device 10a and a second camera device 10b that are fixedly installed near the first area AR1 near the 1st hole of the golf course and target the 1st hole for shooting. The video distribution system 1 also includes a third camera device 10c and a fourth camera device 10d that are fixedly installed near the second area AR2 near the 10th hole located at a location different from the irst hole and target the 10th hole for shooting, a video distribution device 20, and a viewer terminal 30. In the following description, the camera devices 10a, 10b, 10c, and 10d may sometimes be simply referred to as the camera device 10.

[0016] The areas to be photographed within the video distribution system 1 are not limited to two, and may be three or more, or only one area may be the subject of photography. Also, the number of camera devices installed corresponding to one area is not limited to two, and may be three or more, or only one device may be used. In addition, in FIG. 1, only one viewer terminal 30 is shown for simplicity of explanation, but in reality, a plurality of viewer terminals 30 are communicably connected to the video distribution device 20.

[0017] FIG. 2 is a block diagram showing the configurations of the first camera device 10a, the second camera device 10b, the third camera device 10c, and the fourth camera device 10d.

[0018] The first camera device 10a includes an imaging unit 11a, a video encoder 12a, a sound collection unit 13a, an audio encoder 14a, and a transmission unit 15a. The imaging unit 11a photographs a subject. The imaging unit 11a images the surroundings to generate video information. The video encoder 12a encodes the video information captured by the imaging unit 11a into video information Pa. The sound collection unit 13a collects the surrounding sounds to generate sound information. The audio encoder 14a encodes the sound information collected by the sound collection unit 13a into sound information Sa. The transmission unit 15a transmits the video information Pa encoded by the video encoder 12a and the sound information Sa encoded by the audio encoder 14a to the video distribution device 20.

[0019] The second camera device 10b has the same configuration as the first camera device 10a, generates video information Pb and sound information Sb, and transmits them to the video distribution device 20. Also, the third camera device 10c has the same configuration as the first camera device 10a, generates video information Pc and sound information Sc, and transmits them to the video distribution device 20. Also, the fourth camera device 10d has the same configuration as the first camera device 10a, generates video information Pd and sound information Sd, and transmits them to the video distribution device 20.

[0020] Each sound collection unit and each audio encoder may be provided outside each camera device and communicably connected to the corresponding camera device.

[0021] The first camera device 10a and the second camera device 10b photograph a predetermined area, including the first hole of the golf course, from different angles. The third camera device 10c and the fourth camera device 10d photograph a predetermined area, including the tenth hole, from different angles.

[0022] The video distribution device 20 is built on a communication network using, for example, cloud computing technology, and includes a first receiving unit 21a, a second receiving unit 21b, a third receiving unit 21c, and a fourth receiving unit 21d, a first buffer 22a, a second buffer 22b, a third buffer 22c, and a fourth buffer 22d, an additional information storage unit 23, a CPU 24, a composite video distribution unit 25, and an instruction receiving unit 26. In the following description, the first receiving unit 21a, the second receiving unit 21b, the third receiving unit 21c, and the fourth receiving unit 21d may be simply referred to as the receiving unit 21.

[0023] The first receiving unit 21a receives video information and sound information captured by the first camera device 10a. More specifically, the first receiving unit 21a receives video information Pa and sound information Sa from the first camera device 10a. The second receiving unit 21b receives video information and sound information captured by the second camera device 10b. The third receiving unit 21c receives video information and sound information captured by the third camera device 10c. The fourth receiving unit 21d receives video information and sound information captured by the fourth camera device 10d.

[0024] The first buffer 22a temporarily stores video and audio information received by the first receiver 21a. More specifically, the first buffer 22a temporarily stores video information Pa and audio information Sa. The second buffer 22b temporarily stores video and audio information received by the second receiver 21b. The third buffer 22c temporarily stores video and audio information received by the third receiver 21c. The fourth buffer 22d temporarily stores video and audio information received by the fourth receiver 21d. In the following description, the first buffer 22a, the second buffer 22b, the third buffer 22c, and the fourth buffer 22d may be simply referred to as buffer 22.

[0025] The additional information storage unit 23 stores additional information to be added as text overlays to the video information being distributed.

[0026] The CPU 24 is, for example, the CPU (Central Processing Unit) of a general-purpose microcomputer, and by installing and executing a predetermined video distribution program, it configures one or more information processing units as shown below. The CPU 24 includes a state information acquisition unit 241, a grouping processing unit 242, a delay time calculation unit 243, and a video synthesis processing unit 244.

[0027] The state information acquisition unit 241 acquires video information captured by camera devices 10a, 10b, 10c, and 10d, or sound information captured, and for each acquired piece of information, it acquires state information indicating the temporal change of state quantities related to the subject. Here, if the information acquired from camera device 10a is encoded, the state information acquisition unit 241 may decode each piece of information before acquiring the state information indicating the temporal change of state quantities related to the subject. In the video distribution device 20, since it receives encoded video information Pa, Pb, Pc, and Pd from camera device 10, the state information acquisition unit 241 may decode the video information Pa, Pb, Pc, and Pd to obtain video information. Here, if the acquired information is encoded in an encoding scheme that includes information about motion vectors between frames, such as H.264, decoding is not necessarily required.

[0028] The grouping processing unit 242 groups camera devices that correspond to imaging information or sound information in which the peak of the state quantity acquired by the state information acquisition unit 241 is within a predetermined time range, as a group of camera devices belonging to the same group, meaning that they are a group that photographs subjects located at the same location.

[0029] The delay time calculation unit 243 calculates the delay time for the output timing of each video information in order to output the video information captured by multiple camera devices that the grouping processing unit 242 has determined to belong to the same group, in a synchronized timing manner. The delay time calculation unit 243 sends an instruction to buffers 22a, 22b, 22c, and 22d to delay the output timing by holding the video information by the calculated delay time.

[0030] The video synthesis processing unit 244 acquires multiple video information appropriately held and output in buffers 22a, 22b, 22c, and 22d to generate synthesized video information. The video synthesis processing unit 244 also adds the same additional information acquired from the additional information storage unit 23 to the video information captured by multiple camera devices determined by the grouping processing unit 242 to belong to the same group within the generated synthesized video information. The video synthesis processing unit 244 also acquires multiple audio information appropriately held and output in buffers 22a, 22b, 22c, and 22d, processes it, and adds it to the synthesized video information. Regarding the processing of audio information to be added to the synthesized video information, the video synthesis processing unit 244 may synthesize all audio information, extract and synthesize multiple audio pieces, or extract only one audio piece. The video synthesis processing unit 244 determines the manner of processing the audio information according to instructions from, for example, the instruction receiving unit 26, which will be described later. Here, the synthesized video information generated by the video synthesis processing unit 244 consists of encoded video information and encoded audio information.

[0031] The composite video distribution unit 25 distributes the composite video information generated by the video synthesis processing unit 244 to the viewer terminal 30.

[0032] The instruction receiving unit 26 receives a switching instruction from the viewer terminal 30 regarding the display of video information within the composite video information and sends it to the video synthesis processing unit 244. The instruction receiving unit 26 also receives a switching instruction from the viewer terminal 30 regarding the mode of sound information within the composite video information and sends it to the video synthesis processing unit 244.

[0033] The viewer terminal 30 is a personal computer or smart device operated by a user viewing the streamed video, and includes a terminal communication unit 31, a decoder 32, a display unit 33, and an input unit 34. The viewer terminal 30 may also include an audio decoder and an audio output unit (not shown).

[0034] The terminal communication unit 31 communicates with the video distribution device 20. The decoder 32 receives the composite video information distributed from the video distribution device 20 via the terminal communication unit 31 and decodes the encoded video information. The display unit 33 plays the video by displaying the information decoded by the decoder 32. The input unit 34 receives information about user operations. The viewer terminal 30 may decode the encoded sound information from the composite video information received from the video distribution device 20 using an audio decoder (not shown) and play the combined sound information using an audio output unit (not shown).

[0035] <Operation of video distribution system 1 according to one embodiment> The operation of the video distribution system 1 according to this embodiment will be described as follows: [1] Grouping process, which groups the multiple camera devices 10a, 10b, 10c, and 10d within the video distribution system 1 according to the subject to be photographed; and [2] Pre-distribution processing, which is performed for each grouped group of camera devices for distribution.

[0036] [1] Grouping process As examples of grouping processes, we will explain (1a) grouping based on the amount of movement of the subject and (1b) grouping based on sound information.

[0037] (1a) Grouping process based on the amount of motion of the subject While the video distribution system 1 is in operation, the imaging unit 11a of the first camera device 10a films golfers playing on the first hole at a predetermined angle. The video information captured by the imaging unit 11a is encoded into video information Pa by the video encoder 12a. In addition, while the imaging unit 11a is filming, the sound collection unit 13a captures ambient sounds. The sound information captured by the sound collection unit 13a is encoded into sound information Sa by the audio encoder 14a. The encoded video information Pa and sound information Sa are sequentially transmitted to the video distribution device 20 by the transmission unit 15a.

[0038] Furthermore, the imaging unit 11b of the second camera device 10b captures golfers playing on the first hole from a different angle than the first camera device 10a. The second camera device 10b generates encoded video information Pb and audio information Sb by processing them in the same way as the first camera device 10a, and transmits them sequentially to the video distribution device 20.

[0039] Furthermore, the imaging unit 11c of the third camera device 10c photographs golfers playing on the 10th hole, which is located in a different location from the 1st hole, at a predetermined angle. The third camera device 10c generates encoded video information Pc and audio information Sc by processing them in the same manner as the first camera device 10a, and transmits them sequentially to the video distribution device 20.

[0040] Furthermore, the imaging unit 11d of the fourth camera device 10d captures golfers playing on the 10th hole from a different angle than the third camera device 10c. The fourth camera device 10d generates encoded video information Pd and audio information Sd by processing them in the same way as the first camera device 10a, and transmits them sequentially to the video distribution device 20.

[0041] Figure 3 is a flowchart showing the grouping process performed by the video distribution device 20. In the video distribution device 20, the first receiving unit 21a receives video information Pa and sound information Sa transmitted from the first camera device 10a, the second receiving unit 21b receives video information Pb and sound information Sb transmitted from the second camera device 10b, the third receiving unit 21c receives video information Pc and sound information Sc transmitted from the third camera device 10c, and the fourth receiving unit 21d receives video information Pd and sound information Sd transmitted from the fourth camera device 10d.

[0042] The status information acquisition unit 241 determines whether or not the timing for executing the grouping process has arrived (step S1). The status information acquisition unit 241 determines that the timing for executing the grouping process has arrived, for example, when it determines that the movement of the camera device body has stopped after movement has been detected in any of the camera devices 10a, 10b, 10c, or 10d, or when a predetermined time has elapsed since the timing for executing the previous grouping process, or when it receives a signal from the video distribution device 20 via a communication unit (not shown) of any of the camera devices 10 that means that the grouping process will be executed, or when the video distribution device 20 is equipped with an input unit (not shown) and information meaning that the user will execute the grouping process is input via the input unit.

[0043] The status information acquisition unit 241 determines the movement and stopping status of the camera devices 10a, 10b, 10c, and 10d based on sensing information from inertial sensors (not shown) built into the camera devices 10a, 10b, 10c, and 10d. Specifically, the determination is made based on whether the output value of the inertial sensor of any of the camera devices 10 indicates that the camera device 10 has moved and then stopped moving. Here, the movement of the camera device 10 means both that it is carried by a person and its installation position is changed, and that the field of view and imaging direction are changed by the pan-tilt movement of the camera device 10 itself.

[0044] Incidentally, while the video distribution system 1 was envisioned for live streaming of a golf tournament, the content to be streamed is not limited to this. The video distribution system 1 can be applied to scenes where a subject operating within the field of view of the camera device 10 is filmed at a fixed point and streamed. Specifically, the video distribution system 1 can be applied to streaming gymnastics competitions where competitions are held simultaneously in multiple areas within a gymnasium. In gymnastics competitions, if a competition in a particular area being filmed by the camera device 10 ends during streaming, the camera device 10 may not move its position, but the imaging direction and field of view of the camera device 10 may be changed by pan-tilt operation, etc., to stream competitions in areas other than the original area. In such cases, the state information acquisition unit 241 may determine that the timing for executing grouping processing has arrived when it detects the pan-tilt operation of the camera device 10 and then determines that the pan-tilt has stopped.

[0045] When the state information acquisition unit 241 determines that the timing for executing the grouping process has arrived ("YES" in step S1), it starts acquiring video information Pa received by the first receiver 21a, video information Pb received by the second receiver 21b, video information Pc received by the third receiver 21c, and video information Pd received by the fourth receiver 21d (step S2), and acquires information on the change over time of the amount of motion in the video information as a state quantity related to the subject for each piece of video information (step S3). In step S3, the state information acquisition unit 241 acquires information on the change over time of the amount of motion based on video information from the time the timing for executing the grouping process has arrived to an arbitrary period. Here, the arbitrary period can be, for example, 35 seconds, taking into account the time it takes for a golfer to hit a tee shot, but is not limited to this. If the state information acquisition unit 241 does not determine that the timing for executing the grouping process has arrived ("NO" in step S1), it repeats step S1.

[0046] Figure 4(a) is a graph showing the change over time of the amount of motion in the video information Pa acquired by the state information acquisition unit 241, Figure 4(b) is a graph showing the change over time of the amount of motion in the video information Pb, Figure 4(c) is a graph showing the change over time of the amount of motion in the video information Pc, and Figure 4(d) is a graph showing the change over time of the amount of motion in the video information Pd.

[0047] Existing techniques can be used to detect the amount of motion of a subject in video information. These existing techniques include, for example, the interframe difference method, which calculates the difference in pixel values ​​between consecutive frames; the optical flow method, which determines the motion of an object between frames as a vector field; and the background difference method, which pre-models the background of the video and detects the difference between this model and the video information as a moving object.

[0048] When a fixed camera is used to film a golfer hitting a tee shot, the background is mostly still, so the amount of movement of the golfer is directly reflected in the video information, with the amount of movement peaking at the moment of the shot.

[0049] In scenes where the subject to be streamed does not move from the same position, such as when a fixed camera is used to film a golfer hitting a tee shot, the state information acquisition unit 241 does not need to acquire information on the temporal change in the amount of motion from all areas of the video information in step S3. In this case, the state information acquisition unit 241 may acquire information on the temporal change in the amount of motion from the video footage cropped to include the area in which the golfer is shown and its surroundings. In other words, the state information acquisition unit 241 may acquire information on the temporal change in the amount of motion from only a portion of the video information.

[0050] The grouping processing unit 242 identifies the timing at which the amount of motion peaks for each piece of video information (step S4). Specifically, the grouping processing unit 242 identifies time t1 as the timing at which the amount of motion of video information Pa peaks, time t2 as the timing at which the amount of motion of video information Pb peaks, time t3 as the timing at which the amount of motion of video information Pc peaks, and time t4 as the timing at which the amount of motion of video information Pd peaks.

[0051] The grouping processing unit 242 generates grouping information for multiple camera devices corresponding to video information whose identified time falls within a predetermined time range, grouping them as a group of camera devices belonging to the same group that capture the same subject (step S5). Then the process returns to step S1.

[0052] For example, as shown in Figure 5, when the grouping processing unit 242 detects that a peak in motion has occurred in the video information Pa at time t1, it sets a detection window F for detecting the amount of motion in each video information within a predetermined time range ta that includes time t1. Here, the predetermined time range ta can be a time that takes into account whether or not it is possible to determine the peak in motion, and is, for example, 1 second, but is not limited to this. The grouping processing unit 242 monitors whether or not there is video information within the detection window F in which a peak in motion occurs.

[0053] Here, the grouping processing unit 242 detects the peak of the motion amount of the video information Pb within the detection window F. As a result, the grouping processing unit 242 recognizes the first camera device 10a, which captured the video information Pa, and the second camera device 10b, which captured the video information Pb, as a group of camera devices belonging to the first group that captures the same subject.

[0054] Here, even if video information Pa and video information Pb are filming the same subject, a golfer, the reasons why the peak motion amount may be staggered include, for example, variations in processing delay in equipment such as the imaging units 11a and 11b of camera devices 10a and 10b or video encoders 12a and 12b, differences in the network line delay between camera devices 10a and 10b and the video distribution device 20, and differences in the communication path in the network line.

[0055] Similarly, the grouping processing unit 242 recognizes the third camera device 10c, which captured the video information Pc, and the fourth camera device 10d, which captured the video information Pd, as a group of camera devices belonging to the second group that capture the same subject. The subjects captured by the first group of camera devices 10a and 10b are different subjects and are located in different places than the subjects captured by the second group of camera devices 10c and 10d.

[0056] As a result, the grouping processing unit 242 generates grouping information in which the first camera device 10a and the second camera device 10b form the first group, and the third camera device 10c and the fourth camera device 10d form the second group.

[0057] The grouping processing unit 242 may, for example, leave the grouping information of all camera devices 10 in an undetermined state during the initial setup of the video distribution device 20. Furthermore, if the time-dependent change in the amount of motion of the video information of any camera device 10 does not show a waveform that clearly indicates a peak in the amount of motion as shown in Figure 5, the grouping processing unit 242 may leave the grouping information of that camera device 10 in an undetermined state. By doing so, it is possible to suppress the generation of incorrect grouping information.

[0058] The grouping processing unit 242 may perform the grouping process in a manner other than the processing procedure shown in the flowchart in Figure 3. For example, when the grouping processing unit 242 detects that a peak in motion has occurred in the video information Pa, it may detect the video information in which the peak occurred within a predetermined time range from the detected time t1, identify the camera device that captured the detected video information, and group the identified camera device with the first camera device 10a that captured the video information Pa as the same group.

[0059] (1b) Grouping process based on sound information The first camera device 10a, the second camera device 10b, the third camera device 10c, and the fourth camera device 10d each generate encoded video and audio information in the same manner as in (1a) and transmit it sequentially to the video distribution device 20.

[0060] In the video distribution device 20, when it determines that the timing for executing the grouping process has arrived ("YES" in step S1), the state information acquisition unit 241 starts acquiring sound information Sa received by the first receiving unit 21a, sound information Sb received by the second receiving unit 21b, sound information Sc received by the third receiving unit 21c, and sound information Sd received by the fourth receiving unit 21d (step S2). For each sound information, it acquires information on the change in volume over time within the sound information as a state quantity related to the subject of the associated video information (step S3). Existing techniques can be used for detecting the change in volume over time within the sound information. In this case, the state information acquisition unit 241 may also acquire information on the change in volume over time from sound information in a predetermined frequency band, for example, a frequency band specific to the sound of a golf shot. If the state information acquisition unit 241 does not determine that the timing for executing the grouping process has arrived ("NO" in step S1), it repeats step S1.

[0061] The grouping processing unit 242 identifies the timing at which the volume peaked for each sound information (step S4). The grouping processing unit 242 generates grouping information for camera devices that correspond to sound information whose identified time falls within a predetermined time range, as a group of camera devices belonging to the same group (step S5). Then the process returns to step S1.

[0062] The video distribution device 20 may perform grouping by combining the processes of (1a) and (1b). This allows the video distribution device 20 to perform grouping with even higher accuracy.

[0063] Furthermore, the video distribution device 20 may perform grouping using at least one of the processes in (1a) and (1b) and the brightness information of the video information. When using brightness information, for example, multiple video information items whose average brightness is within a predetermined range are recognized as belonging to the same group and grouping is performed. This allows the video distribution device 20 to perform grouping with even higher accuracy, taking into account differences in brightness due to weather conditions, etc., at the location being filmed.

[0064] Alternatively, the video distribution device 20 may perform grouping by calculating the degree of similarity of subjects using a trained model based on the video information. For example, the video distribution device 20 takes multiple video pieces of information as input and uses a trained model that has been trained to output the probability that the people contained in each video piece of information are the same person. It then recognizes that videos that have been determined to contain the same person belong to the same group and performs grouping.

[0065] Figure 6 is a flowchart showing the pre-distribution processing procedure performed by the video distribution device 20. When grouping information is generated by the grouping process in the video distribution device 20 ("YES" in step S11), the delay time calculation unit 243 calculates the delay time of the output timing for each piece of video information within each group (step S12).

[0066] As an example, the case in which the delay time calculation unit 243 calculates the delay time of the output timing for each piece of video information within the first group will be explained with reference to Figure 7. Figure 7 is an enlarged view of the change in the amount of motion of the video information Pa and Pb within the detection window F of Figure 5.

[0067] The delay time calculation unit 243 sets the delay time of the video information Pb, which has the latest peak in motion within the first group, to 0. The delay time calculation unit 243 also calculates the difference Δt1 between the time t2 when the motion of the video information Pb with a delay time of 0 peaks and the time t1 when the motion of other video information Pa within the first group peaks, as the delay time of the video information Pa relative to the video information Pb.

[0068] Similarly, the delay time calculation unit 243 sets the delay time of the video information Pd, in which the peak of motion occurred latest within the second group, to 0, and calculates the difference Δt2 between the time t4 when the motion of video information Pd peaks and the time t3 when the motion of video information Pc peaks as the delay time of video information Pc relative to video information Pd.

[0069] The delay time calculation unit 243 sends instructions to buffers 22a, 22b, 22c, and 22d to delay the output timing by holding each video information for the calculated delay time (step S13). Specifically, the delay time calculation unit 243 sends an instruction to the first buffer 22a to hold the video information Pa for a delay time Δt1 minute, and an instruction to the third buffer 22c to hold the video information Pc for a delay time Δt2 minutes.

[0070] Each buffer 22a and 22c holds the video information Pa and Pc received by the receiving units 21a and 21c for a specified time, delaying it before sending it to the video synthesis processing unit 244. Since the delay time for buffers 22b and 22d is 0, these buffers 22b and 22d send the video information Pb and Pd received by the receiving units 21b and 21d to the video synthesis processing unit 244 without delay.

[0071] The video synthesis processing unit 244 acquires video information Pa, Pb, Pc, and Pd output from each buffer 22a, 22b, 22c, and 22d, and generates synthesized video information by appropriately arranging these multiple video pieces within the same screen frame. Since the video information Pa, Pb, Pc, and Pd output from each buffer 22a, 22b, 22c, and 22d are appropriately delayed, the video synthesis processing unit 244 can generate synthesized video information by synchronizing multiple video pieces captured by a grouped camera device based on the delay time calculated by the delay time calculation unit 243.

[0072] Here, the video synthesis processing unit 244, when viewing the synthesized video information from the direction of viewing, arranges the video information Pa, Pb, Pc, and Pd vertically in small screens on the left side of the screen frame, and on the right side of the screen frame, it arranges one of these video information pieces, in this case video information Pa, in a larger screen than the small screen, thereby generating the synthesized video information V1.

[0073] Furthermore, the video synthesis processing unit 244 adds the same additional information obtained from the additional information storage unit 23 to the video information captured by multiple camera devices that the grouping processing unit 242 has determined to belong to the same group, within the generated synthesized video information V1.

[0074] The video synthesis processing unit 244, for example, obtains the names of the second group of golfers playing on the first hole from the additional information storage unit 23 and adds the obtained information as captions to the video information Pa and Pb, respectively. The video synthesis processing unit 244 also obtains the names of the third group of golfers playing on the tenth hole from the additional information storage unit 23 and adds the obtained information as captions to the video information Pc and Pd, respectively.

[0075] Furthermore, the video synthesis processing unit 244 may add the additional information obtained from the additional information storage unit 23 to the video information of only one of the same group. For example, the video synthesis processing unit 244 may obtain the name information of the second group of golfers playing on the first hole from the additional information storage unit 23 and add the obtained information as a caption to only one of the video information Pa or Pb. Alternatively, the video synthesis processing unit 244 may obtain the name information of the third group of golfers playing on the tenth hole from the additional information storage unit 23 and add the obtained information as a caption to only one of the video information Pc or Pd.

[0076] The composite video distribution unit 25 sequentially distributes the composite video information V1 generated by the video synthesis processing unit 244 to the viewer terminal 30 (step S14). Then the process returns to step S11.

[0077] On the viewer terminal 30, the decoder 32 receives the composite video information V1 transmitted from the video distribution device 20 via the terminal communication unit 31 and decodes the encoded video information within the composite video information V1. The display unit 33 plays the composite video by displaying the information decoded by the decoder 32. The decoding by the decoder 32 and the display by the display unit 33 can be performed using a web browser on the viewer terminal 30.

[0078] Figure 8 is a screen configuration diagram showing an example of the composite video information V1 displayed on the display unit 33. In this screen configuration diagram, video information Pa, Pb, Pc, and Pd are displayed vertically in small screens on the left side of the display unit 33, while video information Pa is displayed in a large screen on the right side of the display unit 33. This allows viewing from different angles for multiple halls, and also allows viewing of predetermined video information in an enlarged state. Each piece of video information has a corresponding caption attached.

[0079] The video synthesis processing unit 244 may determine the arrangement of each video information in the synthesized video information VI based on the grouping information so that video information from the same group is adjacent to each other. For example, as shown in Figure 8, the synthesized video information VI is generated such that video information Pa and Pb from the same group are adjacent to each other, and video information Pc and Pd from another group are adjacent to each other. In yet another example, the video synthesis processing unit 244 may generate the synthesized video information VI by arranging only one video from each of the same groups based on the grouping information. For example, the video synthesis processing unit 244 may generate a synthesized video information VI in which only two video pieces are arranged, which are created by arranging either video information Pa or Pb with either video information Pc or Pd. In other words, the video synthesis processing unit 244 may determine the arrangement of the video information in the synthesized video information VI based on the grouping information generated by the grouping processing unit 242.

[0080] While video distribution by the video distribution system 1 continues, the processes in steps S1 to S5 and steps S11 to S14 are executed at predetermined timings.

[0081] During the distribution of the composite video information V1, if the user performs an operation in the input unit 34 to select a small screen other than the video information Pa among the small screens displayed on the left side of the display unit 33, for example, the small screen of video information Pb, a switching instruction is sent to the video distribution device 20 via the terminal communication unit 31 to switch the video information to be displayed on the large screen on the right side of the display unit 33 to the selected video information Pb.

[0082] In the video distribution device 20, the instruction receiving unit 26 receives the switching instruction transmitted from the viewer terminal 30 and sends it to the video synthesis processing unit 244. Based on the instruction received from the instruction receiving unit 26, the video synthesis processing unit 244 switches to position the video information Pb on the right side of the screen frame and updates the synthesized video information V1.

[0083] The composite video distribution unit 25 sequentially distributes the composite video information V1 updated by the video synthesis processing unit 244 to the viewer terminal 30, and the distributed composite video information V1 is displayed on the display unit 33 of the viewer terminal 30. As a result, the video information displayed on the large screen to the right of the display unit 33 is switched to the video information of the selected screen.

[0084] In step S14 described above, the placement positions of the video information Pa, Pb, Pc, and Pd in ​​the composite video information generated by the video synthesis processing unit 244 are not limited to the placement positions shown in composite video information V1. For example, when the user performs a screen switching operation, the video synthesis processing unit 244 may divide the screen frame into four equal areas and place the video information Pa, Pb, Pc, and Pd in ​​each area to generate composite video information V2. Once composite video information V2 is generated in this way, the display unit 33 of the viewer terminal 30 displays screen information as shown in Figure 9.

[0085] Next, we will explain the case where the third camera device 10c moves from near the 10th hole to near the 1st hole while the composite video information V2 is being distributed. When the third camera device 10c starts moving, the state information acquisition unit 241 detects the movement of the third camera device 10c based on the sensing information from the inertial sensor inside the third camera device 10c. Subsequently, when the third camera device 10c arrives near the 10th hole and is fixed in place and stops moving, the state information acquisition unit 241 detects this and starts grouping.

[0086] In this grouping process, the grouping processing unit 242 generates grouping information in which the first camera device 10a, the second camera device 10b, and the third camera device 10c form the first group, and only the fourth camera device 10d forms the second group.

[0087] Based on this grouping information, the delay time calculation unit 243 performs processing to appropriately delay the output timing of the video information Pa, pb, Pc, and Pd, and the video synthesis processing unit 244 generates synthesized video information V3 in which this video information is arranged.

[0088] Furthermore, the video synthesis processing unit 244 acquires additional information for each group from the additional information storage unit 23 and adds it as text overlays to each video information. Specifically, the video synthesis processing unit 244 acquires the names of the second group of golfers playing on the first hole and adds them as text overlays to the video information Pa, Pb, and Pc respectively, and acquires the names of the third group of golfers playing on the tenth hole and adds them as text overlays to the video information Pd.

[0089] As the synthesized video information V3 is generated and distributed in this manner, the display unit 33 of the viewer terminal 30 displays screen information as shown in Figure 10.

[0090] The above-mentioned [1] grouping process and [2] pre-distribution processing may be started after golf play has begun in the golf tournament being broadcast, or each camera device may perform test imaging before broadcasting and perform the processing in advance based on the results. By performing the [1] grouping process and [2] pre-distribution processing before broadcasting, it is possible to deliver composite video information with appropriately adjusted output timing and captions for each subject from the start of broadcasting.

[0091] In the embodiment described above, the case where the video distribution system 1 live streams the status of a golf tournament, and the grouping processing unit 242 groups the camera devices based on the timing when the amount of motion in the video information or the volume of the sound information reaches its peak, was explained. However, the invention is not limited to this, and in cases such as when the video distribution system 1 live streams the status of a fireworks display, the grouping processing unit 242 may group the camera devices based on the timing when the brightness of the video information reaches its peak. Alternatively, the grouping processing unit 242 may group the camera devices based on the timing when the frequency of the sound information reaches its peak.

[0092] Furthermore, the video distribution device 20 may be configured to allow switching of elements (motion amount, volume, sound frequency, brightness, etc.) or combinations thereof within the video or sound information used for grouping the camera devices, depending on the content being distributed.

[0093] Furthermore, the video distribution device 20 does not necessarily need to generate composite video information that synchronizes multiple video information captured by a group of camera devices and displays it on the same screen. If multiple video information captured by a group of camera devices is not synchronized, the CPU 24 does not need to have a delay time calculation unit 243, and steps S12 and S13 may be skipped in the pre-distribution processing procedure executed by the video distribution device 20 shown in Figure 6.

[0094] [Effects of the Embodiment] According to the embodiment described above, the video distribution device as a camera management device is communicatively connected to a plurality of camera devices that photograph any of a plurality of subjects, and a plurality of sound-gathering devices installed in conjunction with each of the plurality of camera devices to collect ambient sound, and comprises a state information acquisition unit and a grouping processing unit. The state information acquisition unit acquires video information captured by the plurality of camera devices or sound information collected by the plurality of sound-gathering devices, and acquires state information indicating the change over time of a state quantity related to the subject for each acquired piece of information. The grouping processing unit groups the camera devices among the plurality of camera devices that correspond to video information or sound information in which the peak of the state quantity indicated by the state information is within a predetermined time range, as a group of camera devices belonging to the same group.

[0095] According to this system, multiple camera devices can be automatically grouped together based on whether they are capturing the same subject. Therefore, administrators do not need to constantly monitor or manually input changes in the subject of each camera device during video streaming, and grouping can be performed appropriately even when the number of camera devices increases.

[0096] Furthermore, the grouping processing unit may perform grouping processing when it detects movement of one of the multiple camera devices and determines that this movement has stopped, or at predetermined time intervals. This allows the grouping information to be updated appropriately and automatically even if the camera device moves and the subject changes.

[0097] Furthermore, the system may also include a video synthesis processing unit that generates composite video information by synchronizing multiple video information captured by a group of camera devices grouped by the grouping processing unit and displaying it on the same screen, or a video synthesis processing unit that generates composite video information by adding the same additional information to multiple video information captured by a group of camera devices and displaying it on the same screen.

[0098] This allows multiple video recordings captured by a group of camera devices to be displayed appropriately on the same screen, making it easier to recognize them as a collection of video recordings of the same subject.

[0099] Furthermore, the status information acquisition unit may acquire, as status information, information indicating the temporal change in motion or brightness for each video information captured by multiple camera devices, or information indicating the temporal change in volume or frequency for each sound information captured by multiple sound pickup devices. This allows multiple camera devices to be automatically and accurately grouped according to which camera devices are capturing the same subject. [Explanation of symbols]

[0100] 1. Video distribution system 10a, 10b, 10c, 10d Camera equipment 11a, 11b, 11c, 11d Imaging unit 12a, 12b, 12c, 12d video encoders 13a, 13b, 13c, 13d sound collection section 14a, 14b, 14c, 14d audio encoders 15a, 15b, 15c, 15d Transmitter 20 Video distribution device 21a, 21b, 21c, 21d Receiver 22a, 22b, 22c, 22d buffer 23 Additional Information Storage Unit 24 CPU 25. Composite Video Distribution Department 26 Instruction receiving unit 30 Viewer terminals 31 Terminal Communication Unit 32 Decoders 33 Display section 34 Input section 241 Status Information Acquisition Unit 242 Grouping Processing Unit 243 Delay Time Calculation Unit 244 Video Synthesis Processing Unit

Claims

1. Multiple camera devices for photographing subjects located in multiple locations, and multiple sound-gathering devices installed in conjunction with each of the multiple camera devices for capturing ambient sounds, are communicated to each of these devices. A state information acquisition unit acquires video information captured by the aforementioned multiple camera devices or sound information captured by the aforementioned multiple sound-collecting devices, and for each acquired piece of information, acquires state information indicating the temporal changes in state quantities related to the subject. A grouping processing unit groups together the camera devices among the plurality of camera devices whose peak state quantity indicated by the state information is within a predetermined time range and corresponds to video information or sound information, as a group of camera devices belonging to a group that photographs subjects located at the same location. A camera management device equipped with the following features.

2. The camera management device according to claim 1, wherein the grouping processing unit executes the grouping process when it determines that the movement of one of the multiple camera devices has stopped after detecting the movement of one of the main bodies.

3. A delay time calculation unit calculates the delay time for the output timing of each piece of video information, The camera management device according to claim 1, further comprising: an image synthesis processing unit that generates composite image information for displaying multiple video information captured by a group of camera devices grouped by the grouping processing unit on the same screen in synchronization based on the delay time calculated by the delay time calculation unit.

4. The state information acquisition unit uses the following as the state information: Information showing the change over time in motion or brightness for each image information captured by the aforementioned multiple camera devices, or The camera management device according to claim 1, which acquires information showing the change in volume or frequency over time for each sound information picked up by the plurality of sound picking devices.

5. Multiple camera devices for photographing subjects located in multiple locations, and a camera management device that is communicatively connected to multiple sound-gathering devices installed in conjunction with each of the multiple camera devices for capturing ambient sounds, The system acquires video information captured by the multiple camera devices or sound information captured by the multiple sound-gathering devices, and for each acquired piece of information, it acquires state information showing the temporal changes in state quantities related to the subject. A camera management method that groups camera devices corresponding to video information or audio information in which the peak of the state quantity indicated by the aforementioned state information is within a predetermined time range, as a group of camera devices belonging to a group that photographs subjects located at the same location.

Images