Control unit, control method, and program

The control device automates switching between imaging devices for efficient media data distribution, addressing inefficiencies in existing systems by using functional information to manage device usage and event capture.

JP2024162792APending Publication Date: 2024-11-21CANON KK
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2023078698
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-11
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Existing systems require manual setup and specialized knowledge for switching between multiple imaging devices, making them inefficient and prone to missed events like accidents, and lack flexibility in device usage time settings.

Method used

A control device that automatically switches between multiple imaging devices based on functional information, using a communication system to manage media data distribution, and adjusts device usage time based on performance and event detection.

Benefits of technology

Enables seamless and automatic switching among imaging devices for media data distribution, enhancing event capture and reducing the need for manual setup and specialized knowledge.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024162792000001_ABST
    Figure 2024162792000001_ABST
Patent Text Reader

Abstract

To automatically select an imaging apparatus that receives media data for distribution from a plurality of imaging apparatuses.SOLUTION: A control unit has communication means that communicates with a plurality of imaging apparatuses, and control means that controls the communication means to receive media data from the plurality of imaging apparatuses. The control means performs control of selecting a predetermined imaging apparatus used for distribution of the media data from the plurality of imaging apparatuses according to predetermined time, and controls the predetermined time on the basis of function information received from the plurality of imaging apparatuses.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a system for transmitting media data received from a plurality of imaging devices to a distribution device. [Background technology]

[0002] In a system that distributes images from multiple different viewpoints captured by multiple imaging devices, multi-angle distribution is known in which images from multiple viewpoints are distributed by switching between the imaging devices that capture the images to be distributed from among the multiple imaging devices.

[0003] Patent Document 1 describes a method of switching between multiple imaging devices installed on a course such as a track in a stadium, which capture images for distribution according to preset conditions. Patent Document 2 describes a method of monitoring road conditions while switching between images transmitted from multiple imaging devices installed on a road, and switching to a specific imaging device to obtain precise images when an accident, congestion, or other event is detected. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 119034 [Patent Document 2] Patent No. 4133691 Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, in order to perform multi-angle distribution, it is necessary to set conditions for switching between multiple imaging devices in advance. In Patent Document 2, since multiple imaging devices can only be switched in sequence, it is not possible to set the usage time for each imaging device, and if an event such as an accident is detected, it becomes impossible to switch to another imaging device.

[0006] Furthermore, multi-angle distribution requires specialized knowledge and preparation time because it is necessary to connect multiple image capture devices, set switching methods, and set switching conditions.

[0007] The present invention has been made in view of the above-mentioned problems, and has an object to realize a technique that can automatically switch between imaging devices to be used for distributing media data from among a plurality of imaging devices. [Means for solving the problem]

[0008] In order to solve the above problems and achieve the objective, the control device of the present invention has a communication means for communicating with a plurality of imaging devices, and a control means for receiving media data from the plurality of imaging devices via the communication means, and the control means controls switching of a predetermined imaging device among the plurality of imaging devices to be used for distributing the media data in accordance with a predetermined time, and controls the predetermined time based on functional information received from the plurality of imaging devices. Effect of the Invention

[0009] According to the present invention, it is possible to automatically switch between a plurality of imaging devices to be used for distributing media data. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a system configuration diagram of the present embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating the configuration of a control device according to the embodiment. [Diagram 3] FIG. 1 is a block diagram illustrating the configuration of an imaging apparatus according to an embodiment of the present invention. [Figure 4] 5 is a diagram illustrating a processing sequence of media data by the control device of the embodiment; [Diagram 5] 5A and 5B are diagrams showing examples of how media data is displayed on the control device of the embodiment. [Figure 6] 4 is a flowchart showing a control process of the control device of the embodiment. [Figure 7] 4 is a flowchart showing a control process of the control device of the embodiment. [Figure 8] 5A to 5C are diagrams for explaining a method of scoring the performance of the imaging device according to the present embodiment. [Figure 9] 5 is a timing chart illustrating a switching process of the imaging apparatus according to the present embodiment. [Figure 10] 4 is a flowchart showing a control process of the control device of the embodiment. [Figure 11] 5A to 5C are diagrams for explaining a usage time correction method in the control process of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] <System configuration> The system of this embodiment will be described with reference to FIG.

[0013] The system of this embodiment includes a distribution device 10, a control device 20, and multiple imaging devices 30. The multiple imaging devices 30 include a first imaging device 30A and a second imaging device 30B, but in this embodiment, the multiple imaging devices 30A and 30B may be collectively referred to as imaging devices 30. The multiple imaging devices 30A and 30B capture images from different viewpoints for each imaging device, collect sounds around the imaging devices, generate media data in a predetermined format (for example, RTMP) including at least one of images and sounds, and transmit the media data to the control device 20. The control device 20 controls the shooting operation and the sound recording operation for each imaging device for the multiple imaging devices 30A and 30B. In addition, the control device 20 receives media data in a predetermined format (for example, RTMP) including images captured by each imaging device and / or sounds collected by each imaging device from the imaging devices. Then, the control device 20 applies a predetermined data process to the media data received from the imaging devices to generate data for playback in the control device 20 or data for distribution by the distribution device 10. The distribution device 10 receives media data to be distributed from the control device 20, and distributes the data to an external device such as a user device via an IP network such as the Internet or a LAN (Local Area Network).

[0014] The multiple imaging devices 30A, 30B and the control device 20 are connected to be able to communicate with each other through a network by a wireless connection method such as wireless LAN or a wired connection method such as wired LAN. In the case of a wireless connection method, the multiple imaging devices 30A, 30B and the control device 20 are directly connected by P2P or connected via an IP network. The media data transmitted from the imaging devices 30A, 30B to the control device 20 or the media data transmitted from the control device 20 to the distribution device 10 includes at least one of an image and an audio. The images include moving images or still images, and the audio may include audio analysis data, subtitle data, etc.

[0015] The imaging device 30 is a digital camera capable of taking images and collecting audio, but is not limited to a digital camera and may be a personal computer (notebook PC or tablet PC), a smartphone, a web camera such as a surveillance camera, a medical camera, or the like.

[0016] The control device 20 is a personal computer (notebook PC or tablet PC), a smartphone, etc. The control device 20 generates media data for playback without applying a data conversion process described later in FIG. 4 to the media data received from the imaging device 30, or generates media data in a data format suitable for distribution by the distribution device 10 by applying a data conversion process, and transmits the media data to the distribution device 10. The control device 20 also has a function of determining from which of the multiple imaging devices 30A and 30B the media data received is to be transmitted to the distribution device 10.

[0017] The distribution device 10 is a server computer such as a service site that simultaneously distributes (for example, streaming) media data to multiple user terminals. The service site provides services for various purposes and uses, such as entertainment, promotion, and training.

[0018] <Configuration of the control device 20> Next, the configuration and functions of the control device 20 of this embodiment will be described with reference to FIG.

[0019] FIG. 2 is a block diagram illustrating the configuration of the control device 20 of the present embodiment.

[0020] The control unit 201 includes an arithmetic processor such as a CPU or MPU that controls the entire control device 20, and realizes the control processing of the flowchart described below by executing programs stored in the nonvolatile memory 202. The work memory 203 is a RAM or the like, and is also used as a work memory for expanding constants and variables for the operation of the control unit 201, programs read from the nonvolatile memory 202, and the like. The control unit 201 also controls the display of the display unit 206 described below.

[0021] The non-volatile memory 202 stores an OS (operating system), which is basic software executed by the control unit 201, and applications that cooperate with the OS to realize applied functions. In this embodiment, the non-volatile memory 202 also stores a camera application for communicating with the imaging device 30 and a distribution application for communicating with the distribution device 10.

[0022] The processing of the control device 20 in this embodiment is realized by reading software provided by an application. It is assumed that the application has software for utilizing basic functions of the OS installed in the control device 20. It is also possible for the OS of the control device 20 to have software for realizing the control processing in this embodiment. The application has a function of remotely controlling the imaging device 30 to receive media data, and a function of transmitting the media data received from the imaging device 30 to the distribution device 10.

[0023] The control unit 201 switches between the imaging devices 30A and 30B, based on the function information received from the imaging device 30 by the communication unit 207, the imaging device (hereinafter, distribution source imaging device) used by the distribution device 10 to distribute media data. The control unit 201 can also switch the distribution imaging device based on the media data received from the imaging device 30. For example, the control unit 201 detects a subject from the media data received from the imaging device, and switches the distribution imaging device based on the detected subject. The control unit 201 includes a data processing unit 201a. The data processing unit 201a applies a data conversion process, which will be described later in FIG. 4, to the media data received from the imaging device 30 to generate media data in a data format suitable for distribution by the distribution device 10.

[0024] A timer 204 is a timing unit that measures the time used for various controls and the time of a built-in clock.

[0025] The operation unit 205 is made up of operation members such as various switches and buttons that receive various operations from a user and notify the control unit 201. The operation unit 205 also includes a touch input function as a touch screen that is integrally configured with the display unit 206 described below. The operation unit 205 includes a power switch that turns the power of the control device 20 on or off and starts or shuts down the system.

[0026] The display unit 206 displays images, characters for interactive operations, etc. The display unit 206 is, for example, a display device such as a liquid crystal display or an organic EL display. The display unit 206 may be integrated with the control device 20, or may be an external device connected to the control device 20.

[0027] The communication unit 207 is communicably connected to external devices such as the distribution device 10 and the imaging device 30 via a wireless antenna or a wired cable, and transmits and receives various information and data. The various information and data are, for example, function information and media data received from the imaging device 30, and media data transmitted to the distribution device 10. The communication unit 207 includes an interface for communicating with the external device via wireless LAN. The control device 20 may be directly connected to the imaging device 30 via the communication unit 207, or may be connected via wireless LAN. The communication unit 207 is not limited to wireless LAN, and may use a wireless communication module such as infrared communication, Bluetooth (registered trademark), or Wireless USB, or a wired connection interface such as a USB cable, HDMI (registered trademark), or IEEE1394. The communication unit 207 may also be an interface for wireless communication via a wide area network (WAN) such as 4G / LTE or 5G.

[0028] The power supply control unit 208 includes a power supply unit, a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to be energized, and controls the DC-DC converter based on instructions from the control unit 201 to supply the required voltage for the required period to each unit of the control device 20. The power supply unit includes primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, and Li-ion batteries, and an AC adapter.

[0029] <Configuration of imaging device 30> Next, the configuration and functions of the imaging device 30 of this embodiment will be described with reference to FIG.

[0030] The control unit 301 includes an arithmetic processor such as a CPU or MPU that controls the entire imaging device 30, and executes a program stored in the non-volatile memory 302 to realize communication processing and control processing with the control device 20. Note that instead of the control unit 301 controlling the entire device, the entire device may be controlled by multiple hardware devices sharing the processing.

[0031] The control unit 301 controls the imaging process by the imaging unit 304 and the communication process with the control device 20 by the communication unit 307 .

[0032] The imaging unit 304 includes a group of lenses including a zoom lens and a focus lens, and a shutter with an aperture function. The imaging unit 304 also includes an image sensor configured with a CCD or CMOS that converts a subject image into an electric signal, and an A / D converter that converts an analog image signal output from the image sensor into a digital signal. Under the control of the control unit 301, the imaging unit 304 converts the subject image light formed by the lens included in the imaging unit 304 into an electric signal using the image sensor, performs noise reduction processing, and outputs image data consisting of a digital signal.

[0033] The control unit 301 performs resizing such as pixel interpolation and reduction and color conversion processing on image data captured by the imaging unit 304. The control unit 301 also compresses and encodes the processed still image data using JPEG or the like, or encodes only moving image data or both moving image data and audio data using a moving image compression format such as H.264 / MPEG-4 / AVC to generate an image file, and records the image file on the recording medium 309. The control unit 301 also performs a predetermined calculation process using the captured image data, and controls the focus lens, aperture, and shutter of the imaging unit 304 based on the calculation result to perform AF (autofocus) processing and AE (autoexposure) processing. The control unit 301 also generates predetermined streaming data (for example, RTMP (Real-Time Messaging Protocol)) from media data including image data and / or audio data, and transmits the data to the control device 20.

[0034] Non-volatile memory 302 is an electrically erasable and recordable memory, and for example, an EEPROM is used. Constants, programs, etc. for the operation of control unit 301 are recorded in non-volatile memory 302. The program here refers to a program for executing a control process described later in FIG. 6.

[0035] The work memory 303 is used as a working area for expanding constants and variables for the operation of the control unit 301, programs read from the non-volatile memory 302, etc. The work memory 303 is also used as a buffer memory for temporarily storing image data captured by the imaging unit 304, and an image display memory for the display unit 306.

[0036] The operation unit 305 is composed of operation members such as various switches, buttons, and a touch screen that accept various operations from the user. The operation unit 305 includes a power switch that turns on or off the power of the imaging device 30 and starts or shuts down the system, a shooting switch that shoots an image, and a playback button that plays back the shot image. The operation unit 305 also includes a touch screen that is integrally formed with the display unit 306 described later. The shooting switches include a still image shooting switch that instructs the control unit 301 to shoot a still image, and an action shooting switch that instructs the control unit 301 to start or stop shooting (recording) a video. In response to an instruction from the still image shooting switch or the video shooting switch, the control unit 301 controls the imaging unit 304 to execute shooting preparation operations such as AF processing and AE processing, and also executes a series of shooting operations from reading a signal from the imaging unit 304 to writing image data to the recording medium 309.

[0037] The display unit 306 displays a live view image, a captured image, and text for interactive operations. The display unit 306 is, for example, a display device such as a liquid crystal display or an organic EL display. The display unit 306 may be integrated with the imaging device 30 or may be an external device connected to the imaging device 30. The imaging device 30 may be connected to the display unit 306 and may control the display of the display unit 306.

[0038] The communication unit 307 is communicably connected to an external device such as the control device 20 via a wireless antenna or a wired cable, and transmits and receives various information and data. The various information and data are, for example, function information and media data to be transmitted to the control device 20. The communication unit 307 includes an interface for communicating with the external device via wireless LAN. The imaging device 30 may be directly connected to the control device 20 via the communication unit 307, or may be connected via wireless LAN. The communication unit 307 is not limited to wireless LAN, and may use a wireless communication module such as infrared communication, Bluetooth (registered trademark), or Wireless USB, or a wired connection interface such as a USB cable, HDMI (registered trademark), or IEEE1394. The communication unit 207 may also be an interface for wireless communication via a wide area network (WAN) such as 4G / LTE or 5G.

[0039] The communication unit 307 includes an AP mode in which the imaging device 30 operates as an access point (AP) in infrastructure mode, and a CL mode in which the imaging device 30 operates as a client (CL) in infrastructure mode. By operating the communication unit 307 in the CL mode, the imaging device 30 can operate as a CL in infrastructure mode. When the imaging device 30 operates as a CL, it can connect to a surrounding AP and participate in a network formed by the AP. By operating the communication unit 307 in the AP mode, the imaging device 30 can operate as a simple AP (hereinafter, a simple AP) which is a type of AP but has more limited functions. When the imaging device 30 operates as a simple AP, the imaging device 30 forms a network by itself. Peripheral devices of the imaging device 30 recognize the imaging device 30 as an AP and can participate in the network formed by the imaging device 30. A program for operating the imaging device 30 in the AP mode or the CL mode is stored in the non-volatile memory 302.

[0040] The audio input unit 308 is one or more microphones built into the imaging device 30 or connected via an audio terminal, and converts an analog audio signal generated by collecting audio around the imaging device 30 into a digital signal and outputs it to the control unit 301. The control unit 301 performs various types of audio processing on the digital audio signal generated by the audio input unit 308 to generate audio data.

[0041] Image files generated by the control unit 301 are recorded on the recording medium 309, and image files already recorded on the recording medium 309 are read out by the control unit 301. The recording medium 309 may be a memory card or a hard disk drive that is attached to the imaging device 30, or may be a flash memory or a hard disk drive built into the imaging device 30. It is sufficient that the imaging device 30 is at least capable of accessing the recording medium 309.

[0042] <Media data processing sequence> Next, a media data processing sequence performed by the control device 20 of this embodiment will be described with reference to FIG.

[0043] 4 is realized by the control unit 201 of the control device 20 functioning as the data processing unit 201a, reading data before and after processing from the work memory 203, and writing data to the work memory 203. Note that in this embodiment, an example will be described in which streaming data received from the multiple imaging devices 30A, 30B includes video and audio, and media data for playback is generated without applying data conversion processing to the streaming data, or media data in a data format suitable for distribution by the distribution device 10 is generated by applying data conversion processing.

[0044] In the reception process 402, the control device 20 receives streaming data from the multiple imaging devices 30A, 30B via the communication unit 207, and stores the data in the work memory 203. The format of the streaming data received in the reception process 402 is, for example, RTMP. Furthermore, in the reception process 402, media data including video and audio is generated from the streaming data stored in the work memory 203. The format of the media data generated from the streaming data is, for example, FLV.

[0045] In the DEMUX (demultiplexer) process 403, the media data input from the reception process 402 is separated into video data and audio data. The video data separated from the media data is in a format of, for example, H.264. The audio data separated from the media data is in a format of, for example, AAC.

[0046] In the decoding processes 404 and 405, the video data and audio data separated in the DEMUX process 403 are converted into a general-purpose format. Since the video data and audio data have been subjected to advanced encoding processing, the decoding processes 404 and 405 decode the data into a general-purpose data format, whereby the video data and audio data are generated as media data for playback that can be played back in the playback process 406.

[0047] In the encoding processes 407 and 408, the video data and audio data generated in the decoding processes 404 and 405 are subjected to a high-level encoding process again to generate encoded video data and audio data. The format of the encoded video data is, for example, H.264. The format of the encoded audio data is, for example, AAC. In the encoding processes 407 and 408, encoding processes in a data format that can be received by the distribution device 10 are applied.

[0048] A MUX (multiplexer) process 409 generates media data by synthesizing the video data and audio data encoded in the encoding processes 407 and 408. The format of the media data by synthesizing the video data and audio data is, for example, FLV.

[0049] In the transmission process 410, the media data generated in the MUX process 409 is converted into streaming data and transmitted to the distribution device 10. Note that the data format applied in the MUX process 409 and the transmission process 410 is a data format that can be received by the distribution device 10.

[0050] The data format applied in each process of the data processing flow in Fig. 4 is not limited to the above-mentioned example. For example, the formats of the video data and audio data may be determined according to the processing speed of decoding and encoding, and the quality characteristics of the video and audio, in addition to conforming to the data format output by the imaging device 30 and the data format that the distribution device 10 can receive.

[0051] FIG. 5 shows an example of a display of an image reproduced in the reproduction process 406 in FIG.

[0052] A display area 501 corresponds to the display screen of the display unit 206 of the control device 20. An image 502 is an image input from the first imaging device 30A. An image 503 is an image input from the second imaging device 30B.

[0053] Image 502 is an example of an image with an angle of view that includes the entire subject. Image 503 is an example of an image from a different viewpoint than image 502, and has an angle of view that enlarges part of the subject compared to image 502.

[0054] A thick dashed attention frame is displayed around image 502. The attention frame indicates which of the multiple imaging devices the image was received from. In addition to the attention frame, information such as characters, figures, and coloring that can identify the imaging device that outputs the image with the attention frame may be added and displayed. Which of the media data received from the multiple imaging devices is to be transmitted to distribution device 10 is determined by a control process described later with reference to Figs. 6 to 11. In addition, the user can select the media data that distribution device 10 uses for distribution by touching image 502 or image 503.

[0055] <Control processing> Next, the control process performed by the control device 20 of this embodiment will be described with reference to FIGS.

[0056] Fig. 6 is a flowchart showing the control process of the control device 20 of this embodiment. The process of Fig. 6 is realized by the control unit 201 expanding a program stored in the non-volatile memory 202 into the work memory 203, executing the program, and controlling each component of the control device 20. The process of Fig. 6 is started when a camera application is started in the control device 20, the control device 20 and the multiple image capturing devices 30A, 30B are connected to each other so as to be able to communicate with each other, and the multiple image capturing devices 30A, 30B are in a shooting standby state.

[0057] The control device 20 of this embodiment switches between the multiple imaging devices 30A, 30B as a distribution source imaging device used by the distribution device 10 to distribute media data, based on function information for each imaging device received from the multiple imaging devices 30A, 30B.

[0058] The control device 20 compares the functional information received from the multiple imaging devices 30A, 30B with a threshold value, receives media data from the delivery image imaging device that is automatically switched based on the comparison result, and transmits it to the delivery device 10.

[0059] In step S601, the control unit 201 performs initial settings for the multiple imaging devices 30A, 30B. In the initial settings for the imaging devices, settings necessary for distribution of media data to the distribution device 10 are performed, such as the shooting mode of the imaging devices, the image resolution and format, exposure, and focus. If there are model-specific items such as whether or not to display a tally lamp during video shooting, depending on the model of the imaging device, these may be set. In addition, the control unit 201 receives function information from the multiple imaging devices 30A, 30B, acquires information on the size and number of pixels of the image sensor equipped in the imaging devices, the maximum aperture F value of the lens, and whether or not image noise removal is performed, and stores the information in the work memory 203.

[0060] In step S602, the control unit 201 determines whether or not the initial settings for the multiple imaging devices 30A and 30B are complete. The control unit 201 simultaneously transmits an initial setting instruction to the multiple imaging devices 30A and 30B, waits until it receives a response indicating that the settings are complete from each of the imaging devices 30A and 30B, and when it receives the response indicating that the settings are complete from each of the imaging devices 30A and 30B, it determines that the initial settings are complete and proceeds to step S603.

[0061] In step S603, the control unit 201 performs initial settings for the distribution device 10. In this embodiment, since the control device 20 is connected to the distribution device 10 via an IP network, the control device 20 performs initial settings for the distribution device 10 using a WebAPI or REST API interface. In the initial settings for the distribution device 10, user authentication and distribution session designation are performed. Depending on the distribution device to be connected, an API key may be obtained by performing the initial settings.

[0062] In step S604, the control unit 201 determines the role allocation of the multiple imaging devices 30A and 30B. The control unit 201 determines the functions of the multiple imaging devices 30A and 30B, and determines a main imaging device that plays a central role in distributing media data to the distribution device 10, and a sub imaging device that plays a role in improving the distribution quality by adding variations to the media data. A method for determining the role allocation of the imaging devices will be described later with reference to FIG. 7. The control unit 201 determines the role allocation for the first imaging device 30A and the second imaging device 30B, and stores it in the work memory 203. The control unit 201 selects the imaging device determined as the main imaging device as a distribution source imaging device that will first transmit media data to the distribution device 10 when the distribution device 10 starts distributing the media data. If the control unit 201 has determined the role allocation for all of the imaging devices 30A and 30B, the control unit 201 advances the process to step S605.

[0063] In step S605, the control unit 201 requests the plurality of image capture devices 30A and 30B to transmit media data. In this case, the control unit 201 may request the transmission of media data only from the main image capture device determined in step S604.

[0064] In step S606, the control unit 201 performs distribution start processing. The control unit 201 transmits, among the media data received from the multiple imaging devices 30A, 30B, the media data received from the main imaging device determined in step S604 to the distribution device 10. The distribution device 10 simultaneously distributes the media data received from the source imaging device to multiple user terminals. Upon starting distribution of the media data, the control unit 201 starts measuring the elapsed time since the main imaging device started to be used as the source imaging device using the timer 204. The data processing flow from receiving the media data from the imaging device to transmitting it to the distribution device 10 is as described in FIG. 4.

[0065] In step S607, the control unit 201 determines whether or not an instruction to switch the source imaging device by a user operation has been received. If it is determined that the instruction to switch has not been received, the control unit 201 advances the process to step S608. If it is determined that the instruction to switch has been received, the control unit 201 advances the process to step S611. In this embodiment, not only is the media data to be transmitted to the distribution device 10 switched between the media data of the main imaging device and the media data of the sub-imaging device based on the time (usage time) that the imaging device is used as the source imaging device, but the user can also arbitrarily switch the source imaging device. The selection method includes a method of touching the image areas 502 and 503 in FIG. 5, a method of selecting by gaze detection, and a method of selecting by hand gesture detection. The selection method by gaze detection is a method of selecting the imaging device that captured the image by performing gaze detection on the image area and determining whether or not the image is gazed on for a predetermined time. The selection method by hand gesture detection is a method of performing hand gesture detection on the image area, and selecting the imaging device that captured the image when a hand gesture of a predetermined direction and size is detected on the image. The gaze detection process and the hand gesture detection process can be performed using known techniques.

[0066] In step S608, the control unit 201 acquires the elapsed time since the start of use as the source imaging device, which is held by the timer 204 in step S606.

[0067] In step S609, the control unit 201 determines whether it is time to switch the source imaging device. If the control unit 201 determines that it is not time to switch the source imaging device, the process proceeds to step S613, and if it is time to switch the source imaging device, the process proceeds to step S610. The control unit 201 compares the usage time (reference value or correction value) of the source imaging device stored in the work memory 203 with the elapsed time since the source imaging device started to be used as the source imaging device. If the elapsed time since the source imaging device started to be used as the source imaging device exceeds the usage time of the source imaging device, the control unit 201 determines that it is time to switch the source imaging device, and the process proceeds to step S610.

[0068] In step S610, the control unit 201 acquires the usage time of the imaging device that will be the next source imaging device. The control unit 201 determines the imaging device that will be the next source imaging device from among the multiple imaging devices 30A and 30B, determines the usage time of the source imaging device, and stores it in the work memory 203. A method for determining the usage time of the source imaging device will be described later with reference to FIG.

[0069] In step S611, the control unit 201 sets the usage time of the source imaging device selected by the user to a predetermined time, and stores it in the work memory 203. The usage time of the source imaging device selected by the user is a fixed value, but may be changeable by a touch operation, gaze detection, or hand gesture detection.

[0070] In step S612, the control unit 201 switches the current source imaging device based on the usage time of the source imaging device set in step S610 or S611. If an imaging device other than the source imaging device has stopped transmitting media data, the control unit 201 requests the imaging device that will be the next source imaging device to transmit media data, and executes the switching process after confirming that the media data before and after the switching of the source imaging device can be received. In the switching process of the source imaging device, in addition to immediately switching the source imaging device, it is also possible to notify the user by a screen or sound that the source imaging device is being switched. When the source imaging device is switched, the elapsed time from the start of use of the source imaging device before the switch, which is stored in the work memory 203, is reset and measurement is resumed, so that the elapsed time from the start of use of the next source imaging device can be measured.

[0071] In step S613, the control unit 201 determines whether or not the user has issued an instruction to end distribution by operating the operation unit 205 on the UI screen of the camera application displayed on the display unit 206. If the control unit 201 determines that an instruction to end distribution has been received, the process proceeds to step S614. If the control unit 201 determines that an instruction to end distribution has not been received, the process returns to step S607, and distribution of the media data by the distribution device 10 and transmission of the media data by the distribution source imaging device continue.

[0072] In step S614, the control unit 201 performs distribution end processing. The control unit 201 stops the transmission of the media data to the distribution device 10, instructs the distribution device 10 to stop the distribution of the media data, and further requests the multiple imaging devices 30A and 30B to stop transmitting the media data.

[0073] FIG. 7 is a flowchart showing the function determination process of the imaging device in step S604 of FIG.

[0074] In step S701, the control unit 201 acquires functional information from the multiple imaging devices 30A, 30B. The functional information includes information related to the imaging angle of view and information related to image quality performance. The information related to the imaging angle of view includes the focal length of the lens and electronic zoom magnification. The information related to the image quality performance includes information related to the size and number of pixels of the image sensor, the maximum aperture of the lens, and whether or not noise reduction processing is performed on the video. In the case of a zoom lens, the maximum aperture of the lens is the maximum aperture of the lens at the stopped zoom position after the angle of view adjustment is performed. The noise reduction processing on the video includes whether or not three types of noise reduction processing are performed: color noise reduction, luminance noise reduction, and cyclic noise reduction.

[0075] In step S702, the control unit 201 performs a calculation process to score the image quality performance based on the information on the image quality performance acquired in step S701. Fig. 8 is a diagram for explaining a method of scoring the image quality performance, and Fig. 8(a) to Fig. 8(d) show examples of scores according to the information on the image quality performance.

[0076] Figure 8(a) shows the number of points according to the size of the image sensor. For a 35mm full-size image sensor, the number of points is 5, for an APS-C size, the number of points is 4, and for a small sensor of 1 inch or less, the number of points is 1. APS-H and Four Thirds sizes are considered to be equivalent to APS-C size.

[0077] Figure 8(b) shows the scores according to the number of pixels of the image sensor. In this example, an image sensor with 24 million pixels or more is assigned 5 points, an image sensor with 12 million pixels is assigned 3 points, and an image sensor with fewer than 12 million pixels is assigned 1 point. If the image sensor has more than 12 million pixels but less than 24 million pixels, it is considered based on the 12 million pixel classification, and for example, if it has 20 million pixels, it is assigned 3 points. When outputting 4K resolution, 12 million pixels is one standard, but if the pixel count is 24 million pixels or more, it is more than double that of 4K, so it is possible to improve the S / N ratio while maintaining the resolution by pixel interpolation, so a higher score is assigned to 24 million pixels or more.

[0078] Figure 8(c) shows the scores according to the maximum aperture of the lens. In this example, a lens with a maximum aperture of F2.0 or less is given 5 points, a lens with a medium maximum aperture of F5.6 is given 3 points, and a lens with a maximum aperture of F5.6 is given 1 point. For example, a lens with an F4.0 aperture, which is between F2.0 and F5.6, is given the same classification as F5.6 and is given 3 points.

[0079] Fig. 8(d) shows the scores for video noise reduction processing. An example is shown in which a video chrominance noise reduction process is given a score of 1, a video luminance noise reduction process is given a score of 1, and a video cyclic noise reduction process is given a score of 1.

[0080] The scores obtained as shown in Figures 8(a) to 8(d) are added together to give a score for image quality performance. For example, if a camera is equipped with an APS-C size 24-megapixel image sensor, a lens with an open F-number of F2.8, and capable of performing video noise reduction processing, the score is 4+5+3+(1+1+1)=15 points. As another example, if a camera is equipped with a 1-inch size 20-megapixel image sensor, a built-in zoom lens with an open F-number of F1.8 at the wide-angle end, and capable of performing video noise reduction processing, the score is 1+3+5+(1+1+1)=12 points.

[0081] In step S703, the control unit 201 determines whether the focal lengths of the lenses of all the imaging devices 30A, 30B are the same. If the control unit 201 determines that the focal lengths of the lenses of all the imaging devices are the same based on the function information of the multiple imaging devices 30A, 30B, the control unit 201 proceeds to step S707, and if the control unit 201 determines that the focal lengths are different, the control unit 201 proceeds to step S704.

[0082] In step S704, if the control unit 201 determines that the focal length of the lens of the imaging device being evaluated is the widest angle for the multiple imaging devices 30A, 30B, the control unit 201 proceeds to S705, and if the control unit 201 determines that the focal length is not the widest angle, the control unit 201 proceeds to S706.

[0083] In step S705, the control unit 201 determines the role allocation of the imaging device to be determined as the main imaging device because the focal length of the lens of the imaging device to be determined is the widest angle and the media data includes an image with an angle of view that includes the entire subject.

[0084] In step S706, the control unit 201 determines the role allocation as a sub-imaging device because the focal length of the lens of the imaging device to be determined is on the magnification side and the media data includes an image with an angle of view that enlarges a portion of the subject, and produces a variation of media data including an image with an angle of view that includes the entire subject.

[0085] In step S707, since the focal length of the lenses of all the imaging devices is the same and it is not possible to determine the division of roles based on the difference in the angle of view, the division of roles of the imaging devices is determined based on the image quality performance. If the control unit 201 determines that the image quality performance value of the imaging device to be judged is a higher score (highest score) than the image quality performance values ​​of other imaging devices that are not the subject of judgment based on the image quality performance value calculated in step S702, the control unit 201 proceeds to process S708, and if it determines that the image quality performance value is not the highest score, the control unit 201 proceeds to process S709.

[0086] In step S708, since the image quality performance value of the imaging device to be determined is the highest, the control unit 201 determines the division of roles with the imaging device to be determined as the main imaging device so that media data with higher image quality will be the focus of distribution.

[0087] In step S709, since the image quality performance value of the imaging device being determined is not the highest point, the control unit 201 determines the role allocation of the imaging device being determined as a secondary imaging device so that media data with relatively low image quality performance becomes variation in the distribution.

[0088] FIG. 9 is a timing chart for explaining the process of switching the source imaging device in steps S607 to S612 in FIG.

[0089] 9(a) to 9(c), the horizontal axis represents time, and an example is shown in which the source imaging device is switched in order along the time axis.

[0090] Fig. 9(a) illustrates a state in which the switching time between imaging devices 1 and 2 is constant. In Fig. 9(a), imaging devices 1 and 2 are switched at the same time interval, and based on the elapsed time since the source imaging device started to be used, acquired in step S608, it is determined in step S609 whether it is time to switch the source imaging device. The usage time of the source imaging device can be set to any value, but a time of, for example, 60 seconds is set as the initial value.

[0091] Fig. 9(b) shows an example in which imaging devices 1 and 2 are switched when one of imaging devices 1 and 2 is assigned as the main imaging device and the other imaging device is assigned as the sub-imaging device. Fig. 9(b) shows an example in which imaging device 1 is assigned as the main imaging device and imaging device 2 is assigned as the sub-imaging device. Transmission of media data is started from imaging device 1, which is the main imaging device, and short periods of media data from imaging device 2, which is the sub-imaging device, are periodically transmitted between the media data from imaging device 1.

[0092] 9(c) shows an example in which imaging devices 1 to 3 are switched when the main imaging device of imaging devices 1 to 3 is assigned to imaging device 1 and the sub-imaging devices are assigned to imaging devices 2 and 3. The usage time of imaging device 1, which is the main imaging device, is set to be longer than the usage times of imaging devices 2 and 3, which are the sub-imaging devices. Regarding the media data of imaging devices 2 and 3, which are the sub-imaging devices, switching is also performed for the multiple sub-imaging devices according to the usage time so that the media data of imaging device 2, which has a longer usage time, is delivered before the media data of imaging device 3.

[0093] FIG. 10 is a flow chart showing the process of acquiring the usage time of the source imaging device in step S610 of FIG.

[0094] The processes in steps S1001 and S1002 in FIG. 10 are similar to the processes in steps S701 and S702 in FIG.

[0095] In step S1003, the control unit 201 determines the role sharing between the multiple imaging devices 30A and 30B. The control unit 201 refers to the results determined in step S604 in Fig. 6 and the process in Fig. 7, and if it is determined that the imaging device to be determined is the main imaging device, the process proceeds to S1004, and if it is determined that the imaging device is the sub imaging device, the process proceeds to S1005.

[0096] In step S1004, the control unit 201 acquires a reference value (reference use time) of the use time of the main imaging device.

[0097] In step S1005, the control unit 201 acquires a reference value (reference use time) of the use time of the sub imaging device.

[0098] The reference value of the usage time of the main imaging device is longer than the reference value of the usage time of the sub imaging device.

[0099] In step S704 in FIG. 7, the imaging device with the relatively wide-angle focal length is set as the main imaging device, and the imaging device with the relatively enlarged focal length is set as the sub-imaging device, so that the usage time of the imaging device with the relatively wide-angle focal length is longer and the usage time of the imaging device with the relatively enlarged focal length is shorter.

[0100] Furthermore, in step S707 in FIG. 7, an imaging device with a relatively high image quality performance value is set as the main imaging device, and an imaging device with a relatively low image quality performance value is set as the sub-imaging device, so that the usage time of the imaging device with the relatively high image quality performance value is longer and the usage time of the imaging device with the relatively low image quality performance value is shorter.

[0101] In step S1006, the control unit 201 determines whether the image quality performance value of the image capture device to be determined is a higher score (highest score) than the image quality performance values ​​of other image capture devices that are not the subject of determination, based on the image quality performance value calculated in step S1002. If the control unit 201 determines that the image quality performance value of the image capture device to be determined is the highest score, the control unit 201 proceeds to process S1007, and if the control unit 201 determines that the image quality performance value of the image capture device to be determined is not the highest score, the control unit 201 proceeds to process S1008.

[0102] In step S1007, the control unit 201 corrects the reference usage time acquired in step S1004 or S1005 based on the image quality performance. Fig. 11 shows an example of correcting the reference usage time based on the image quality performance. If the image quality performance of either the main imaging device or the sub imaging device is high, 10% of the time is added to the reference time.

[0103] In step S1008, the control unit 201 analyzes the media data received from the imaging device to be determined, and identifies whether the subject to be captured is speaking. The identification method can be a known method such as a method of performing face organ detection on the media data to identify the movement of the feature points of the lips and corners of the mouth, or a method of using deep learning to obtain the speaker's facial feature amount by machine learning.

[0104] In step S1009, the control unit 201 determines whether or not a speaker is present within the imaging angle of view range of the media data received from the imaging device to be determined, based on the speaker identification result in step S1008. If the control unit 201 determines that a speaker is present, the process proceeds to step S1010, and if the control unit 201 determines that a speaker is not present, the process ends.

[0105] In step S1010, the control unit 201 determines whether the imaging device being determined is a sub-imaging device or not, and if it is determined that it is a sub-imaging device, proceeds to step S1011, and if it is determined that it is not a sub-imaging device, terminates the processing.

[0106] In step S1011, the control unit 201 corrects the usage time of the imaging device to be determined as a sub-imaging device. Fig. 11 shows an example of correcting the usage time of the sub-imaging device when a speaker is present within the imaging angle of view of the media data of the sub-imaging device. When a speaker is present within the imaging angle of view of the media data of the sub-imaging device, 10% of time is added to the reference time. In other words, the usage time of the sub-imaging device is corrected twice, when the image quality performance is high and when a speaker is present. Then, the usage time of the sub-imaging device capturing media data including a speaker becomes relatively longer, and the usage time of the sub-imaging device not capturing media data including a speaker becomes relatively shorter.

[0107] According to the embodiment described above, the usage time of the distribution source imaging device used by the distribution device 10 for distributing media data is corrected based on the function information of the multiple imaging devices 30A and 30B. In this manner, the distribution source imaging device can be automatically switched among the multiple imaging devices 30A and 30B.

[0108] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0109] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0110] The disclosure of this specification includes the following control device, control method, and program. [Configuration 1] A communication means for communicating with a plurality of imaging devices; a control unit for receiving media data from the plurality of imaging devices via the communication unit; the control means performs control to switch a predetermined imaging device to be used for distributing the media data among the plurality of imaging devices in accordance with a predetermined time; a control device that controls the predetermined time based on function information received from the plurality of imaging devices. [Configuration 2] the control means performs control to switch between a first imaging device and a second imaging device as the predetermined imaging device in accordance with the predetermined time; The control device according to configuration 1, characterized in that first media data received from the first imaging device or second media data different from the first media data received from the second imaging device is used for distribution. [Configuration 3] 3. The control device according to configuration 2, wherein the control means controls the predetermined time so that a usage time of the first imaging device and a usage time of the second imaging device are the same. [Configuration 4] The control device according to configuration 2, wherein the control means controls the predetermined time based on the functional information so that the usage time of the first imaging device is longer than the usage time of the second imaging device. [Configuration 5] 5. The control device according to any one of configurations 2 to 4, wherein the second media data is data that adds variation to the first media data. [Configuration 6] the control means performs control to switch the predetermined imaging device among a first imaging device, a second imaging device, and a third imaging device in accordance with the predetermined time; The control device according to configuration 1, characterized in that first media data received from the first imaging device, second media data different from the first media data received from the second imaging device, or third media data different from the first media data and the second media data received from the third imaging device is used for distribution. [Configuration 7] the control means determines, based on the function information, that a usage time of the first imaging device is longer than a usage time of the second imaging device, 7. The control device according to configuration 6, wherein the predetermined time is controlled so that a usage time of the second imaging device is longer than a usage time of the third imaging device. [Configuration 8] 8. The control device according to configuration 6 or 7, wherein the second media data and the third media data are data that add variation to the first media data. [Configuration 9] The functional information includes information regarding a photographing angle of view and information regarding image quality performance, 9. The control device according to any one of configurations 1 to 8, wherein the control means converts information about the shooting angle of view and information about image quality performance into a score for each imaging device. [Configuration 10] The information regarding the photographing angle of view includes a focal length of a lens and a magnification of an electronic zoom, 10. The control device according to configuration 9, wherein the information regarding image quality performance includes the size and number of pixels of the image sensor, the maximum aperture of the lens, and whether or not noise reduction processing is performed. [Configuration 11] The control device according to configuration 10, characterized in that the control means extends the usage time of the imaging device whose focal length is relatively on the wide-angle side and shortens the usage time of the imaging device whose focal length is relatively on the enlargement side based on the information regarding the shooting angle of view. [Configuration 12] The control device according to configuration 10, characterized in that the control means extends the usage time of an imaging device having relatively high image quality performance and shortens the usage time of an imaging device having relatively low image quality performance based on the information related to the image quality performance. [Configuration 13] image processing means for identifying a speaker from the media data; The control device according to any one of configurations 1 to 12, characterized in that the control means relatively extends the usage time of an imaging device that is generating media data including the speaker, and relatively shortens the usage time of an imaging device that is not generating media data including the speaker. [Configuration 14] 14. The control device according to any one of configurations 1 to 13, wherein the control means switches the predetermined imaging device in response to a user operation. [Configuration 15] 15. The control device according to configuration 14, wherein the control means switches the predetermined imaging device by detecting a user's line of sight or hand gesture with respect to the media data. [Configuration 16] The control device according to any one of configurations 1 to 15, characterized in that the control means converts the media data received from the specified imaging device into a data format suitable for distribution by a distribution device and transmits the converted data to the distribution device. [Configuration 17] 17. The control device according to any one of configurations 1 to 16, wherein the media data includes video and audio generated by the imaging device. [Configuration 18] A control method for a control device, comprising: receiving media data from a plurality of imaging devices by a communication means; The step of performing control to switch a predetermined imaging device to be used for distributing the media data among the plurality of imaging devices in accordance with a predetermined time, A control method comprising: controlling the predetermined time based on functional information received from the plurality of image capture devices. [Configuration 19] A program for causing a computer to function as a control device according to any one of configurations 1 to 17. [Explanation of symbols]

[0111] 10... distribution device, 20... control device, 30, 30A, 30B... imaging device, 201... control unit, 207... communication unit

Claims

1. A communication means for communicating with a plurality of imaging devices; a control unit for receiving media data from the plurality of imaging devices via the communication unit; the control means performs control to switch a predetermined imaging device to be used for distributing the media data among the plurality of imaging devices in accordance with a predetermined time; a control device that controls the predetermined time based on function information received from the plurality of imaging devices.

2. the control means performs control to switch between a first imaging device and a second imaging device as the predetermined imaging device in accordance with the predetermined time; The control device according to claim 1 , wherein the control device uses for distribution first media data received from the first imaging device or second media data different from the first media data received from the second imaging device.

3. 3. The control device according to claim 2, wherein the control means controls the predetermined time so that a usage time of the first image pickup device and a usage time of the second image pickup device are the same.

4. 3. The control device according to claim 2, wherein the control means controls the predetermined time based on the function information so that a usage time of the first image capture device is longer than a usage time of the second image capture device.

5. 3. The control device according to claim 2, wherein the second media data is data that adds a variation to the first media data.

6. the control means performs control to switch the predetermined imaging device among a first imaging device, a second imaging device, and a third imaging device in accordance with the predetermined time; The control device according to claim 1, characterized in that first media data received from the first imaging device, second media data different from the first media data received from the second imaging device, or third media data different from the first media data and the second media data received from the third imaging device is used for distribution.

7. the control means determines, based on the function information, that a usage time of the first imaging device is longer than a usage time of the second imaging device, 7. The control device according to claim 6, wherein the predetermined time is controlled so that a usage time of the second image pickup device is longer than a usage time of the third image pickup device.

8. The control device according to claim 6 , wherein the second media data and the third media data are data that add variations to the first media data.

9. The functional information includes information regarding a photographing angle of view and information regarding image quality performance, The control device according to claim 1 , wherein the control means converts information about the photographing angle of view and information about image quality performance into a score for each image pickup device.

10. The information regarding the photographing angle of view includes a focal length of a lens and a magnification of an electronic zoom, 10. The control device according to claim 9, wherein the information regarding image quality performance includes a size of an image sensor, a number of pixels, a maximum aperture of a lens, and whether or not noise reduction processing is performed.

11. The control device according to claim 10, characterized in that the control means extends the usage time of the imaging device whose focal length is relatively on the wide-angle side and shortens the usage time of the imaging device whose focal length is relatively on the enlargement side based on the information regarding the shooting angle of view.

12. The control device according to claim 10 , wherein the control means extends the usage time of an imaging device having a relatively high image quality performance and shortens the usage time of an imaging device having a relatively low image quality performance based on the information regarding the image quality performance.

13. image processing means for identifying a speaker from the media data; The control device according to claim 1, characterized in that the control means relatively increases the usage time of an imaging device that is generating media data including the speaker, and relatively decreases the usage time of an imaging device that is not generating media data including the speaker.

14. The control device according to claim 1 , wherein the control means switches the predetermined imaging device in response to a user operation.

15. The control device according to claim 14 , wherein the control means switches the predetermined imaging device by detecting a user's line of sight or a hand gesture with respect to the media data.

16. 2. The control device according to claim 1, wherein the control means converts the media data received from the predetermined imaging device into a data format suitable for distribution by a distribution device, and transmits the converted data to the distribution device.

17. The control device according to claim 1 , wherein the media data includes video and audio generated by the imaging device.

18. A control method for a control device, comprising: transmitting and receiving data between the plurality of imaging devices by a communication means; The step of performing control to switch a predetermined imaging device to be used for distributing media data among the plurality of imaging devices in accordance with a predetermined time, A control method comprising: controlling the predetermined time based on functional information received from the plurality of image capture devices.

19. A program for causing a computer to function as the control device according to any one of claims 1 to 17.

Citation Information

Patent Citations

  • road monitoring system

    JP4133691B2

  • Image capture system, image capture method, and program

    WO2017119034A1