Image processing apparatus and image processing method

Abstract

To provide an image processing apparatus that can appropriately switch pieces of moving image data obtained from a plurality of imaging apparatuses placed at geographically different positions.SOLUTION: An image processing apparatus selects, in predetermined order, one of a plurality of pieces of moving image data obtained from a plurality of imaging apparatuses placed at geographically different positions, and outputs the selected moving image data to the outside. The image processing apparatus dynamically changes the length of a period for outputting the selected data to the outside on the basis of one or more of a change over time of the plurality of pieces of moving image data and a change over time of information on viewing of the moving image data being selected.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method, and more particularly to a technique for switching between multiple images. [Background technology]

[0002] Patent Document 1 proposes that a mobile terminal having an in-camera and an out-camera automatically switches which camera's video to transmit to the outside depending on whether the mobile terminal is oriented vertically or horizontally. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-56930 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 targets multiple cameras installed in the same device, and therefore cannot be applied to switching between images captured by multiple imaging devices located in different geographical locations.

[0005] In one aspect, the present invention provides an image processing device and an image processing method that are capable of appropriately switching between video data obtained by a plurality of image capture devices that are placed in different geographical locations. [Means for solving the problem]

[0006] In one aspect, the present invention provides an image processing device that selects, in a predetermined order, one of a plurality of video data obtained by a plurality of imaging devices each located in a different geographical location, and outputs the selected video data to an external device, characterized in that the image processing device has a control means that dynamically changes the length of the period for which the selected video data is output to an external device based on one or more of changes over time in the plurality of video data and changes over time in information related to viewing of the selected video data. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an image processing device and an image processing method that are capable of appropriately switching between video data obtained by a plurality of image capture devices that are placed in different geographical locations. [Brief explanation of the drawings]

[0008] [Figure 1] Schematic diagram of a distribution system according to an embodiment. [Figure 2] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus according to an embodiment; [Figure 3] FIG. 1 is a block diagram showing an example of the functional configuration of an imaging device that can be used in an embodiment. [Figure 4] Schematic diagram of data processing applied by an image processing apparatus according to an embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of source rotation according to an embodiment. [Figure 6] FIG. 1 is a diagram for explaining the operation of an image processing device according to an embodiment. [Figure 7] FIG. 10 is a diagram for explaining the operation of a timer calculation unit in the embodiment. [Figure 8] FIG. 10 is a diagram for explaining the control operation of source rotation in the embodiment. [Figure 9] 1 is a flowchart illustrating the operation of an image processing apparatus according to an embodiment of the present invention; [Figure 10] 1 is a flowchart illustrating the operation of an image processing apparatus according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0009] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] In the following, the present invention will be described as being implemented in a personal computer as an example of an image processing device. However, the present invention can be implemented in any electronic device having one or more arithmetic circuits or processors configured to be able to execute a program. Typical examples of such electronic devices include, but are not limited to, tablet devices and smartphones.

[0011] 1 is a schematic diagram showing an example of a usage environment of an image processing device according to an embodiment. Multiple cameras 300, 311, and 312 are connected to an image processing device 201 via wired or wireless connections. The wireless connections may be direct connections such as P2P connections, or indirect connections via a network. Hereinafter, the multiple cameras 300, 311, and 312 may be collectively referred to as multiple cameras 300.

[0012] Cameras 300, 311, and 312 are installed in different geographical locations. The camera ranges of cameras 300, 311, and 312 may be the same or different. For example, some or all of cameras 300, 311, and 312 may have angles of view set so as to capture the same range from different viewpoints. Furthermore, at least one of the camera direction and angle of view of cameras 300, 311, and 312 may be remotely controllable.

[0013] Furthermore, cameras 300, 311, and 312 are set to continuously capture moving images and immediately output the captured moving image data to image processing device 201. The moving image data output by cameras 300, 311, and 312 may include audio analysis data, subtitle data, and the like in addition to image data and audio data.

[0014] The image processing device 201 selects video data to be output to the distribution system 210 from the video data received from the cameras 300, 311, and 312. The image processing device 201 may apply image processing or convert the data format before outputting the selected video data to the distribution system 210. The method by which the image processing device 201 selects video data to be output will be described later.

[0015] The distribution system 210 distributes to a plurality of viewers the video data output from the image processing device 201. The configuration and distribution method of the distribution system 210 are not directly related to the present invention, and therefore detailed description thereof will be omitted.

[0016] 2 is a block diagram showing an example of the functional configuration of the image processing device 201. The nonvolatile memory 205 may be electrically erasable and recordable. The nonvolatile memory 205 stores programs executed by the control unit 202, various setting values, and the like. The operation of the image processing device 201, which will be described later using flowcharts, is realized by the control unit 202 loading the programs stored in the nonvolatile memory 205 into the system memory 206 and executing them. The nonvolatile memory 205 can also store data such as video data received from the cameras 300, 311, and 312.

[0017] The control unit 202 has at least one arithmetic circuit or processor. The control unit 202 realizes the operation of the image processing device 201 by loading a program stored in a nonvolatile memory 205 into a system memory 206 and executing the program.

[0018] In the figure, functional blocks 202A to 202E shown inside control unit 202 are schematic representations of representative functions realized by execution of programs by control unit 202. Therefore, in the following description, the operations performed by functional blocks 202A to 202E inside control unit 202 are actually performed by control unit 202.

[0019] Note that one or more of the functions realized by the control unit 202 executing a program may be implemented by a hardware circuit (such as an ASIC or FPGA) separate from the control unit 202, or may be implemented using a hardware circuit.

[0020] The system memory 206 is a volatile memory such as a RAM. The system memory 206 is used to temporarily store programs executed by the control unit 202, variables and constants required for program execution, intermediate data, etc. The system memory 206 is also used as a buffer memory for video data, etc., and as a video memory for the display unit 209.

[0021] A system timer 208, under the control of the control unit 202, acquires the time of the built-in clock of the image processing device 201 and measures a set time.

[0022] The operation unit 203 is a general term for input devices that are provided in the image processing device 201 and can be operated by a user. The operation unit 203 may include a power button for switching the image processing device 201 on and off, and input devices (such as a keyboard, mouse, or touch panel) for operating a user interface provided by a program.

[0023] A power supply control unit 204 supplies power of an appropriate format and voltage to each unit of the image processing device 201 from a power supply such as a commercial power supply, an AC adapter, or a battery.

[0024] The display unit 209 is, for example, a liquid crystal display. The display unit 209 displays a user interface, video data, text data, etc. provided by a program executed by the control unit 202. The display unit 209 may be a touch display, in which case the display unit 209 also functions as the operation unit 203. The display unit 209 may be an external device of the image processing device 201.

[0025] The communication I / F unit 207 enables the image processing device 201 to communicate with external devices. The communication I / F unit 207 supports one or more well-known wired and wireless communication protocols and has a circuit, an antenna, and / or a connector corresponding to the supported communication protocol. In this embodiment, the image processing device 201 communicates with the cameras 300, 311, and 312 and the distribution system 210 via the communication I / F unit 207.

[0026] 3 is a block diagram showing an example of the functional configuration of camera 300. Note that while the functional configuration of camera 300 will be described here as a representative example, cameras 311 and 312 also have the same functional configuration. Note that one or more of cameras 300, 311, and 312 are not limited to digital cameras, and may be any electronic device capable of simultaneously capturing video and externally outputting the captured video data. Typical examples of such electronic devices include, but are not limited to, tablet devices and smartphones.

[0027] Control unit 301 has at least one arithmetic circuit or processor. Control unit 301 loads a program stored in nonvolatile memory 303 into system memory 304 and executes it to realize the operation of camera 300. Furthermore, image processing unit 301A described inside control unit 301 is a schematic representation of the function realized by control unit 301 executing the program. Therefore, in the following description, the operation performed by image processing unit 301A is actually performed by control unit 301.

[0028] Note that one or more of the functions realized by the control unit 301 executing a program may be implemented by a hardware circuit (such as an ASIC or FPGA) separate from the control unit 301, or may be implemented using a hardware circuit.

[0029] The imaging unit 302 has an imaging element and a lens (optical system) that generates an optical image of a subject on the imaging element. The imaging element may be, for example, a known CCD or CMOS color image sensor with a primary color Bayer array color filter. The imaging element has a pixel array in which multiple pixels are arranged two-dimensionally, and peripheral circuits for reading out signals from each pixel. Each pixel accumulates charge according to the amount of incident light through photoelectric conversion. By reading out from each pixel a signal having a voltage according to the amount of charge accumulated during the exposure period, a group of pixel signals (analog image signals) representing the subject image generated by the lens can be obtained.

[0030] The analog image signal output by the imaging unit 302 is supplied to the image processing unit 301A. The image processing unit 301A applies predetermined image processing to the analog image signal to generate evaluation values ​​and image data according to the intended use.

[0031] The image processing applied by the image processing unit 301A can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include A / D conversion, signal amplification, reference level adjustment, defective pixel correction, and the like. Color interpolation, also known as demosaicing, is performed when the image sensor is equipped with a color filter, and is a process of interpolating the values ​​of color components that are not included in the individual pixel data that make up the image data. The correction processing can include white balance adjustment, tone correction, correction of image degradation caused by optical aberration of the lens (image restoration), correction of the effects of lens vignetting, color correction, and the like. The detection process may include detection of characteristic regions (for example, face regions or human body regions) and their movements, person recognition processing, and the like. Data processing can include processes such as area extraction (trimming), compositing, scaling, encoding and decoding, header information generation (data file generation), etc. Data processing also includes the generation of image data for display or image data for recording. The evaluation value calculation process can include processes such as generating signals and evaluation values ​​used in autofocus (AF) detection, and generating evaluation values ​​used in automatic exposure control (AE). Special effect processing can include adding a blur effect, changing color tones, relighting, and the like. Note that these are examples of processes that can be applied by the image processing unit 301A, and do not limit the processes that can be applied by the image processing unit 301A.

[0032] During video shooting, the imaging unit 302 continues shooting at a predetermined frame rate. The image processing unit 301A generates video data using an image signal obtained from the imaging unit 302 and an audio signal obtained from a microphone (not shown). The image processing unit 301A records the generated video data on a recording medium 307, or outputs the generated video data to an external device such as the image processing device 201 via the communication I / F unit 308.

[0033] The nonvolatile memory 303 is an electrically erasable and recordable nonvolatile memory, and stores programs executed by the control unit 301, various setting values ​​of the camera 300, and the like.

[0034] The system memory 304 is a volatile memory such as a RAM. The system memory 304 is used to temporarily store programs executed by the control unit 301, variables and constants required for program execution, intermediate data, etc. The system memory 304 is also used as a buffer memory for video data, etc., and as a video memory for the display unit 306.

[0035] The operation unit 305 is a collective term for input devices (buttons, switches, dials, etc.) provided for the user to input various instructions to the camera 300. The input devices constituting the operation unit 305 are named according to their assigned functions. For example, the operation unit 305 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, directional keys, and a confirm key. The release switch is a switch for recording still images, and the control unit 301 recognizes a half-pressed state of the release switch as an instruction to prepare for shooting and a full-pressed state as an instruction to start shooting. The control unit 301 also recognizes a press of the video recording switch in shooting standby mode as an instruction to start video recording, and a press of the video recording switch during video recording as an instruction to stop recording. Note that the functions assigned to the same input device may be variable. The input device may also be software buttons or keys using a touch display. The operation unit 305 may also include an input device compatible with non-contact input methods such as voice input and eye-gaze input.

[0036] When the control unit 301 recognizes the shooting preparation instruction, it starts still image shooting preparation processing such as AF (autofocus) processing, AE (auto exposure) processing, etc. Furthermore, when the control unit 301 recognizes the shooting start instruction, it executes a series of processes from shooting to recording the still image.

[0037] The display unit 306 is, for example, a display provided on the surface of the housing of the camera 300. By continuously capturing video and displaying the generated video data for display on the display unit 306, the display unit 306 can function as an electronic viewfinder (EVF). Video data used to make the display unit 306 function as an EVF is called a live view image. The display unit 306 is also used to display image data recorded on the recording medium 307, information about the camera 300, a menu screen, and the like. The display unit 306 may be an external device.

[0038] The communication I / F unit 308 enables the camera 300 to communicate with external devices. The communication I / F unit 308 supports one or more well-known wired and wireless communication protocols and has a circuit, an antenna, and / or a connector according to the supported communication protocol. In this embodiment, the camera 300 communicates with the image processing device 201 through the communication I / F unit 308.

[0039] In the following description, it is assumed that the communication I / F unit 207 of the image processing device 201 and the communication I / F unit 308 of the camera 300 support the same wireless LAN protocol and communicate in accordance with the wireless LAN protocol. However, other well-known communication protocols such as USB and Bluetooth (registered trademark) may also be used.

[0040] The recording medium 307 records image data generated based on image capture using the imaging unit 302. The recording medium 307 may be, for example, a removable memory card, but may also be a non-removable non-volatile memory, a hard disk drive, or the like.

[0041] Fig. 4 is a diagram showing a schematic flow of processing that the image processing device 201 performs on data received from the camera 300. The processing shown by the blocks in Fig. 4 is performed by the control unit 202 reading a program stored in the nonvolatile memory 205 into the system memory 206 and executing it.

[0042] It is assumed here that the video data output from the cameras 300, 311, and 312 can be live-streamed through the distribution system 210. In other words, it is assumed that the cameras 300, 311, and 312 are currently capturing video, and the generated video data is being supplied to the image processing device 201.

[0043] Of the three systems of video data received from cameras 300, 311, and 312, image processing device 201 selects one system of video data and outputs it to distribution system 210. However, for the sake of simplicity, the selection (switching) process of video data to be output to distribution system 210 will not be described here. Only the process applied to one system of video data to be output to distribution system 210 from reception to output will be described.

[0044] In reception processing 401, control unit 202 receives video data from cameras 300, 311, and 312 via communication I / F unit 207. Control unit 202 stores the received video data in system memory 206. The video data received in reception processing 401 may have a format according to the specifications of cameras 300, 311, and 312, for example, a format conforming to a known streaming protocol such as RTMP, RTSP, HLS, or MPEG-DASH. Then, control unit 202 extracts video and audio data from the received video data. For example, control unit 202 extracts FLV format data from RTMP format video data.

[0045] Then, the control unit 202 separates the extracted video and audio data into video data and audio data by a DEMUX process 402. For example, the control unit 202 separates data in FLV format into video data in H.264 format and audio data in AAC format.

[0046] The control unit 202 applies decoding processing to each of the separated video data and audio data. Specifically, the control unit 202 applies decoding processing 403 to the video data and decoding processing 404 to the audio data. The decoding processing 403 and 404 are intended to convert the data into a data format compatible with the playback processing 405. Therefore, the data format after decoding may differ depending on the specifications of the playback processing 405.

[0047] The control unit 202 applies a playback process 405 to the video data and audio data after the decoding processes 403 and 404 have been applied. The playback process 405 may be, for example, a display process on the display unit 209 for the video data, and an output process from a speaker for the audio data.

[0048] The control unit 202 also applies encoding processes 407 and 406 to the video data and audio data after applying decoding processes 403 and 404, respectively. The encoding processes 407 and 406 are intended to convert the data into a data format compatible with the distribution system 210. Therefore, the data format after encoding may differ depending on the specifications of the distribution system 210.

[0049] The control unit 202 applies multiplexing (MUX) processing 408 to the video data and audio data after applying encoding processing 407 and 406. Whether multiplexing processing 408 is necessary and the format in which the data is multiplexed may differ depending on the specifications of the distribution system 210.

[0050] The control unit 202 applies a transmission process 409 to the video data to which the multiplexing process 408 has been applied as necessary, to the distribution system 210 using the communication I / F unit 207 .

[0051] In at least one of the playback process 405 and the encoding processes 406 and 407, a process of converting one or more of the resolution, frame rate, data rate, etc. may be applied depending on the necessity and settings.

[0052] FIG. 5 is a diagram schematically illustrating source rotation as an example of a method for selecting video data to be output to the distribution system 210, performed by the image processing device 201. Source rotation is a method for repeatedly selecting input image data from multiple systems in order. The selection order is assumed to be stored in advance in the non-volatile memory 205 as setting information. The selection order may also be changeable by the user via the operation unit 203. FIG. 5 shows an example in which source rotation is performed in the order of video data from camera 300 → video data from camera 311 → video data from camera 312 → video data from camera 300 → ...

[0053] The order of selection may be dynamically determined by any method. The control unit 202 may determine the next video data to be selected, for example, randomly, or based on an evaluation value obtained from the image. Furthermore, consecutive selection of video data from the same camera may be permitted or prohibited. Furthermore, video data from the same camera may not be selected until all video data have been selected once.

[0054] The image processing device 201 dynamically changes the length of the period for selecting video data from each camera. The basic operation of the image processing device 201 (control unit 202) for selecting video data will be described with reference to Fig. 6, and then the operation for dynamically changing the length of the period will be described.

[0055] As described above, function blocks 202A to 202D in control unit 202 are functional blocks that represent the operations executed by control unit 202. Timer calculation unit 202C has a plurality of timers that correspond one-to-one to a plurality of cameras that are image data supply sources. In the example of FIG. 6, timer calculation unit 202C has timer 600 corresponding to camera 300, timer 611 corresponding to camera 311, and timer 612 corresponding to camera 312. In the initial state, timers 600, 611, and 612 measure the same time. Note that the timers 600, 611, and 612 may be of any format, but here, as an example, they are countdown timers, and initial values ​​corresponding to the measurement times are set.

[0056] Assume that the camera from which video data is first selected is camera 300. The camera from which video data is next selected is determined by rotation arrangement unit 202B using one of the methods described above. By referring to rotation arrangement unit 202B, it is possible to identify the camera from which video data is next selected.

[0057] The control unit 202 references the rotation arrangement unit 202B and starts selecting video data from the camera 300 in the reception process 401. As a result, the video data from the camera 300 starts to be output from the communication I / F unit 207 to the distribution system 210. The control unit 202 also starts measuring time using the timer 600 corresponding to the camera 300. The rotation arrangement unit 202B decides which camera's video data to select next.

[0058] Thereafter, when it is detected that the count value of the timer 600 has reached 0, the control unit 202 refers to the rotation array unit 202B. The control unit 202 starts selecting video data from the camera 312 in the reception process 401. As a result, the video data from the camera 312 starts to be output from the communication I / F unit 207 to the distribution system 210. The control unit 202 also starts measuring time using the timer 612 corresponding to the camera 312. At this point, the control unit 202 may reset the value of the timer 600 to its initial value.

[0059] When it is detected that the count value of the timer 612 has reached 0, the control unit 202 refers to the rotation array unit 202B. The control unit 202 starts selecting video data from the camera 311 in the reception process 401. As a result, the video data from the camera 311 starts to be output from the communication I / F unit 207 to the distribution system 210. The control unit 202 also starts measuring time using the timer 611 corresponding to the camera 311. At this point, the control unit 202 may reset the value of the timer 600 to its initial value.

[0060] When it is detected that the count value of timer 611 has reached 0, control unit 202 refers to rotation array unit 202B. Control unit 202 starts selecting video data from camera 300 in reception processing 401. As a result, video data from camera 300 begins to be output from communication I / F unit 207 to distribution system 210. Control unit 202 also starts measuring time using timer 600 corresponding to camera 300. At this point, control unit 202 may reset the value of timer 612 to an initial value.

[0061] Thereafter, when control unit 202 repeats the same process, video data is output to distribution system 210 at regular intervals from the three cameras, camera 300, camera 312, and camera 311, in that order. This completes the basic operation of selecting video data.

[0062] Next, we will explain the operation of dynamically changing the selection period of each video data by the control unit 202 (timer calculation unit 202C). Here, as an example, we will explain the operation of dynamically changing the selection period of video data based on one or more of an evaluation value related to changes over time in the video data and changes over time in information related to viewing of the video being output to the distribution system.

[0063] Examples of evaluation values ​​for changes over time in video data are: -Full frame movement, - The speed or acceleration of the object in the frame, - Subject movement (changes in behavior or facial expression), Brightness changes in the whole frame or in the subject area, - Frame rate changes, Changes in the volume or dominant frequencies of the voice These evaluation values ​​can be detected by any known method, and therefore detailed explanations of the detection or calculation methods of each evaluation value will be omitted.

[0064] The video analysis unit 202D calculates an evaluation value relating to the change over time in the video data for the image data received from one or more of the cameras 300, 311, 312 via the communication I / F unit 207, and stores the calculated evaluation value in the system memory 206.

[0065] In addition, examples of changes over time in information about video data being output to a distribution system include: Changes in audience numbers, -Changes in the number of comments on social media regarding video data currently being distributed, the number of likes and / or dislikes; Changes in the cumulative amount of so-called tips The viewing-related information is acquired by the viewing response detection unit 202E from the distribution system 210 and stored in the system memory 206.

[0066] Then, timer calculation unit 202C increases or decreases the time measured by timers 600, 611, and 612 based on one or more of the evaluation value relating to the change over time in the video data and the change over time in the information relating to the viewing of the video being distributed, thereby changing the length of the selection period (distribution period) for the video data from cameras 300, 311, and 312.

[0067] The timer calculation unit 202C increases the selection period (delays the timing of switching) if the change over time in either the evaluation value related to the change over time in the video data or the information related to the viewing of the video being distributed is equal to or greater than a first threshold. Furthermore, the timer calculation unit 202C decreases the selection period (delays the timing of switching) if the change over time in either the evaluation value related to the change over time in the video data or the information related to the viewing of the video being distributed is less than a second threshold. The first threshold is equal to or greater than the second threshold. This allows automatic adjustment so that the selection period for video data with little variation is short and the selection period for video data with a lot of variation is long. Therefore, for example, when one person is live streaming using multiple imaging devices, it is possible to automatically switch video data so as not to bore viewers. The amount of increase or decrease per time may be fixed or, as described below, may vary depending on the type of evaluation value.

[0068] If the timer is a countdown timer, the timer calculation unit 202C can increase the selection period (delay the timing of switching) by increasing the count value of the timer. Similarly, the timer calculation unit 202C can decrease the selection period (advance the timing of switching) by decreasing the count value of the timer. If the timer is a count-up timer, the relationship between the increase / decrease in the count value and the increase / decrease in the selection period is reversed. Note that the increase / decrease in the count value of the timer based on the evaluation value and the count operation of the timer to measure the selection period are executed independently.

[0069] The control unit 202 changes the length of the selection period of the selected (distributed) video data each time the rating value is updated. The rating value can be calculated every n frames (n is an integer equal to or greater than 1) of video data, or at regular intervals. The frequency of calculating the rating value can be determined in advance and stored in the non-volatile memory 205.

[0070] In addition, the control unit 202 also changes the length of the selection period for unselected video data based on the above-mentioned evaluation value. The frequency of calculating the evaluation value for unselected video data may be lower than the frequency of calculating the evaluation value for selected video data. Alternatively, the calculation of the evaluation value and the adjustment of the selection period may not be performed until the remaining time of the selection period for the selected video data becomes less than a threshold value.

[0071] It should be noted that an upper limit to the length of the selection period can be set regardless of whether the data is currently selected or not. For selected video data, the selection period ends when the elapsed time from the start of selection reaches the upper limit of the length of the selection period, regardless of the count value of the timer. This causes the video data to be distributed to be switched at least every time the upper limit of the length of the selection period is reached. For non-selected video data, the count value (initial value) of the timer is set so as not to exceed the count value corresponding to the upper limit of the length of the selection period. The upper limit of the length of the selection period (time or count value) can be stored in non-volatile memory 205.

[0072] The lower limit of the length of the selection period is 0. Furthermore, control unit 202 may skip the selection of video data (camera) when the count value (initial value) of the timer corresponding to the video data (camera) to be selected next is equal to or less than a predetermined lower limit value greater than 0. This makes it possible to prevent viewers from feeling annoyed by switching in a short period of time.

[0073] Furthermore, the control unit 202 may put some of the cameras 300, 311, and 312 into a sleep state. Video data is not supplied from cameras that are in a sleep state. Therefore, the control unit 202 sequentially selects video data from cameras that are not in a sleep state. For example, when the camera 300 is in a sleep state, the control unit 202 alternately selects video data from the cameras 311 and 312. The control unit 202 can put the cameras connected to the communication I / F unit 207 into sleep mode or wake them up by sending commands to the cameras via communication I / F unit 207.

[0074] A specific example of the operation of the timer calculation unit 202C will be described using FIG. 7. Here, a combination of four of the multiple evaluation values ​​described above is used. The control unit 202 is currently selecting (distributing) video data from the camera 300. Video data from the camera 311 is being supplied to the image processing device 201 but is not selected (this state is called standby). The camera 312 is in a sleep state. Therefore, no video data is being supplied from the camera 312 to the image processing device 201. In this case, the image processing device 201 alternately selects the video data supplied from the camera 300 and the video data supplied from the camera 311 and outputs the selected data to the distribution system 210.

[0075] In this specification, selecting video data supplied from camera 300 for output to the distribution system may be expressed as selecting camera 300. The same applies to cameras 311 and 312.

[0076] 7, a weight (sensitivity) is set for each camera and for each of the four types of evaluation values. Here, the weight can be set as an integer between 0 and 10, with 10 being the largest weight. The timer calculation unit 202C does not take into account evaluation values ​​with a weight of 0. The magnitude of the weight can be reflected in at least one of the magnitude of the threshold applied to the amount of change and the magnitude of the increment / decrement of the count value.

[0077] For example, in the case of the amount of change in volume, the threshold is 5 db when a reference weight (e.g., 5) is set, and the count value is increased or decreased when the absolute value of the amount of change in volume is equal to or greater than the threshold. The increment or decrement may be constant. In this case, the timer calculation unit 202C reduces the threshold to less than 5 db (e.g., 2 db) when a weight greater than the reference weight (e.g., 10) is set, and increases the threshold to more than 5 db (e.g., 10 db) when a weight smaller than the reference weight (e.g., 1) is set. This makes it easier for the count value to increase or decrease in response to changes in the evaluation value for evaluation values ​​with a large weight, and makes it harder for the count value to increase or decrease in response to changes in the evaluation value for evaluation values ​​with a small weight. In other words, the sensitivity of the increase or decrease in the count value to changes in the evaluation value can be controlled by the magnitude of the evaluation value weight.

[0078] Furthermore, when reflecting the weight in the increment / decrement unit of the count value, the timer calculation unit 202C can set the increment / decrement unit to 1 / 2 of the increment / decrement unit when the standard weight of 5 is set, if the weight is 1, and can set the increment / decrement unit to double when the weight is 10. Note that the standard weight and the specific method of reflecting the weight are merely examples.

[0079] The timer calculation unit 202C determines the final adjustment amount based on the adjustment amount of the count value or the initial value calculated for each evaluation value. For example, the timer calculation unit 202C can determine the total of the adjustment amounts as the final adjustment amount when a weight is reflected in the threshold value of the evaluation value, or the average or median of the adjustment amounts when a weight is reflected in the unit of increase or decrease of the count value.

[0080] FIG. 8 is a diagram showing example setting values ​​for timers 600, 611, and 612 managed by the timer calculation unit 202C. The timer calculation unit 202C manages timers for each camera. For each timer, a range (upper and lower limits), the shortest time required for selection, and a reference initial value can be set for the count value or measurement time. While FIG. 8 shows a case where the setting value is time, it may also be a count value. Furthermore, the lower limit of the measurement time is set to the minimum delivery time, and the upper limit is set to the maximum delivery time. Furthermore, the shortest time required for selection is set to "skip." It is not necessary to set anything other than the reference initial value.

[0081] The reference initial value is a reference value that is set when the timer is initialized. For example, in the example shown in Fig. 8, the reference initial value of the timer 600 of the camera 300 is 50 seconds (or a count value for measuring 50 seconds). Also, the reference initial value of the timer 611 of the camera 311 is 2 minutes (or a count value for measuring 2 minutes).

[0082] The reference initial value shown in Fig. 8 may be changed by adjustment by the timer calculation unit 202C before being set as the timer's initial value. For example, the reference initial value of the timer corresponding to the standby camera (timer 611 corresponding to camera 311 in the example of Fig. 8) is increased or decreased according to the amount of change in the evaluation value while the video data of camera 300 is selected. Then, when the video data selected by the image processing device 201 is switched from the video data of camera 300 to the video data of camera 311, the value at that time is set as the initial value of timer 611, and timer 611 starts operating. In the following description, when the term "initial value" is simply used, it means the value held as the timer's initial value at that time, and may or may not be equal to the reference initial value.

[0083] The minimum delivery time corresponds to the lower limit of the length of the selection period. The control unit 202 controls the selection period for video data from a camera corresponding to a timer for which a minimum delivery time is set so that it does not fall below the minimum delivery time. Specifically, during standby, the control unit 202 can control the timer calculation unit 202C to adjust the initial value so that the initial value does not fall below the minimum delivery time. Furthermore, after selection, the control unit 202 can control the timer calculation unit 202C to adjust the count value so that the selection period does not fall below the minimum delivery time.

[0084] The maximum delivery time corresponds to the upper limit of the length of the selected period. The control unit 202 controls the selected period for video data from a camera corresponding to a timer for which the maximum delivery time is set so that it does not exceed the maximum delivery time. Specifically, during standby, the control unit 202 can control the timer calculation unit 202C to adjust the initial value so that the initial value does not exceed the maximum delivery time. Furthermore, after selection, the control unit 202 can control the timer calculation unit 202C to adjust the count value so that the selected period does not exceed the maximum delivery time.

[0085] "Skip" is the lower limit of the initial value required for selection. When a camera is selected, if the initial value set in the corresponding timer is equal to or less than the "skip" setting (skip value), the control unit 202 skips the selection of video data from that camera. In the example of FIG. 8, assume that the video data to be distributed is to be switched from video data from camera 311 to video data from camera 300. In this case, if the initial value set in timer 600 corresponding to the next selected camera 300 is 5 seconds or less, the control unit 202 skips the distribution of the video data from camera 300. Furthermore, the control unit 202 initializes the timer 600 to the standard initial value (50 seconds). In the example of FIG. 8, because camera 312 is in a sleep state, the control unit 202 continues to select video data from camera 311. In this case, the control unit 202 may reset the distribution time of the video data from camera 311 to 0, or may not reset the distribution time to 0 if the maximum distribution time has not been reached. Note that if the camera to be selected is switched because the maximum delivery time has been reached, and if skipping the selection of the camera to be switched to would not change the selected camera, the skip may be disabled. If a minimum delivery time is set, the skip value is set to be smaller than the minimum delivery time.

[0086] Next, the operation of the image processing device 201 will be described using the flowcharts shown in Figures 9 and 10. The operation described below is performed by the control unit 202 reading a program stored in the nonvolatile memory 205 into the system memory 206 and executing it. In the following description, it is assumed that the image processing device 201 operates in the environment shown in Figures 1 and 2.

[0087] In S101, the control unit 202 (rotation arrangement unit 202B) refers to the camera selection order pre-stored in the non-volatile memory 205. Here, it is assumed that the order follows the source rotation described above. The order may also be changeable by the user.

[0088] In S102, the control unit 202 identifies the camera that supplies the video data to be distributed in accordance with the reference order.

[0089] In S103, the control unit 202, for example, refers to the system memory 206, and determines whether the initial value to be set in the timer corresponding to the camera identified in S102 is equal to or less than the skip value. If the control unit 202 determines that the initial value is equal to or less than the skip value, it executes S104; if not, it executes S105. Note that if a skip value is not set in the timer, the control unit 202 executes S105.

[0090] In S104, the control unit 202 initializes the initial value of the timer corresponding to the camera identified in S102 to the reference initial value (FIG. 8). After that, the control unit 202 executes S102.

[0091] In S105, the control unit 202 selects the video data supplied from the camera identified in S102 as video data to be output to the distribution system 210. The control unit 202 also starts measuring a selection period using the system timer 208. From the second time onwards, in S105, the video data (the camera supplying the data) to be output to the distribution system 210 is switched.

[0092] When S105 is executed for the first time, the control unit 202 starts applying processing to the video data to be distributed for output to the distribution system 210, as described with reference to Fig. 4. Also, the video analysis unit 202D starts calculating evaluation values ​​for the video data to be distributed and video data supplied from cameras that are not selected and not in a sleep state.

[0093] In S106, the control unit 202 adjusts the initial value of the timer corresponding to the standby camera based on the evaluation value calculated by the video analysis unit 202D for the video data supplied from the standby camera. Details of the operation in S106 will be described later.

[0094] In S107, the control unit 202 (timer calculation unit 202C) counts down the count value of the timer corresponding to the selected camera based on the system timer 208. S107 is an operation for measuring a unit time, and the specific operation such as counting down or counting up may differ depending on the type of timer.

[0095] In S108, the control unit 202 (timer calculation unit 202C) determines the amount of adjustment for the count value of the timer corresponding to the selected camera based on the evaluation value calculated by the video analysis unit 202D for the video data supplied from the selected camera (i.e., being distributed). If the determined adjustment amount is an increase, the control unit 202 executes S109, and if it is a decrease, the control unit 202 executes S110.

[0096] In S109, the control unit 202 (timer calculation unit 202C) increases the count value of the timer corresponding to the selected camera by the adjustment amount, and then the control unit 202 executes S106.

[0097] In S110, the control unit 202 (timer calculation unit 202C) decreases the count value of the timer corresponding to the selected camera by the adjustment amount. Thereafter, the control unit 202 executes S111.

[0098] In S111, the control unit 202 determines whether the count value of the timer corresponding to the selected camera has reached 0, and if it is determined that it has reached 0, executes S104, and if not, executes S113.

[0099] If a minimum delivery time is set for the timer corresponding to the selected camera, the control unit 202 determines whether the selected period is equal to or longer than the minimum delivery time before executing S104. If the control unit 202 determines that the selected period is equal to or longer than the minimum delivery time, it executes S104, and if not, it executes S106. This makes it possible to prevent the selected period from being shorter than the minimum delivery time.

[0100] Furthermore, if a maximum distribution time is set for the timer corresponding to the selected camera, the control unit 202 determines whether or not the selected period has reached the maximum distribution time after executing S109. If it is determined that the selected period has reached the maximum distribution time, the control unit 202 executes S104, and if not, it executes S106. This makes it possible to prevent the selected period from exceeding the maximum distribution time.

[0101] In S113, the control unit 202 determines whether or not to continue the distribution of the video data, and if it is determined to continue, executes S106, and if it is not determined to continue, ends the output of the video data to the distribution system 210. For example, the control unit 202 can determine to continue the distribution of the video data if the user has not instructed via the operation unit 203 to end the distribution.

[0102] FIG. 10 is a flowchart relating to the operation of adjusting the initial value of the timer corresponding to the standby camera in S106.

[0103] S202 is the same operation as S108. That is, the control unit 202 (timer calculation unit 202C) determines the amount of adjustment to the initial value of the timer corresponding to the standby camera based on the evaluation value calculated by the video analysis unit 202D for the video data supplied from the standby camera. The control unit 202 executes S203 if the determined adjustment amount is an increase amount, and executes S204 if it is a decrease amount.

[0104] In S203, the control unit 202 (timer calculation unit 202C) increases the initial value of the timer corresponding to the camera on standby by the adjustment amount. After that, the control unit 202 executes S107.

[0105] In S204, the control unit 202 (timer calculation unit 202C) decreases the initial value of the timer corresponding to the standby camera by the adjustment amount. After that, the control unit 202 executes S107.

[0106] As described above, according to this embodiment, an image processing device can be provided that can appropriately switch between images captured by multiple imaging devices located at different geographical locations. Specifically, the timing of switching between images is delayed when there is a certain amount of change in the image, and is accelerated when there is no change. Therefore, the image processing device according to this embodiment can prevent a decrease in viewer interest in the distribution, for example, due to the continued distribution of video data with little change.

[0107] (Other embodiments) In the above-described embodiment, the count value of the timer corresponding to the selected camera is adjusted based on the evaluation value of the video data supplied from the selected camera. However, the count value of the timer corresponding to the selected camera may be adjusted taking into account the evaluation value of the video data supplied from the standby camera.

[0108] For example, if the amount of change in the video data supplied from the standby camera is greater than the amount of change in the video data supplied from the selected camera, a predetermined decrease may be applied to the adjustment amount of the count value of the timer corresponding to the selected camera. This makes it possible to control so that switching to the standby camera occurs more easily when the amount of change in the video data supplied from the standby camera is greater than the amount of change in the video data supplied from the selected camera.

[0109] The length of the selected period may also be taken into consideration when adjusting the count value. For example, if the selected period exceeds a predetermined time, a predetermined decrease may be applied to the adjustment. This makes it possible to control the switch to a standby camera more likely when the broadcast time exceeds the predetermined time.

[0110] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0111] The disclosure of the present embodiment includes the following image processing device, image processing method, and program. (Item 1) An image processing device that selects one of a plurality of pieces of video data obtained by a plurality of image capture devices placed in different geographical locations in a predetermined order, and outputs the selected video data to an external device, An image processing device characterized by having a control means for dynamically changing the length of the period for which the selected video data is output to the outside based on one or more of the changes over time of the multiple video data and the changes over time of information regarding the viewing of the selected video data. (Item 2) Item 1. An image processing device according to item 1, wherein the changes over time in the plurality of video data include one or more of the movement of the entire frame of the video data, the speed or acceleration of the subject's movement within the frame, changes in the subject's movement or facial expression, changes in brightness of the entire frame or the subject area, changes in frame rate, and changes in the volume or main frequency of the audio. (Item 3) The image processing device described in item 1 or 2, wherein the change over time in the viewing information includes one or more of a change in the number of viewers, a change in the number of comments on SNS regarding the selected video data, the number of likes and / or dislikes, and a change in the cumulative amount of tips. (Item 4) The image processing device described in any one of items 1 to 3, characterized in that the control means calculates an evaluation value for one or more of the changes over time of the multiple video data and the changes over time of information regarding viewing of the selected video data, and increases the length of the period when the evaluation value exceeds a threshold. (Item 5) The image processing device described in any one of items 1 to 4, characterized in that the control means calculates an evaluation value for one or more of the changes over time of the multiple video data and the changes over time of information regarding viewing of the selected video data, and reduces the length of the period if the evaluation value does not exceed a threshold. (Item 6) 6. The image processing device according to item 4 or 5, wherein the control means changes the unit of increase or decrease of the threshold value or the length of the period based on the weight set for the evaluation value. (Item 7) The image processing device described in any one of items 1 to 6, characterized in that the control means dynamically changes the length of the period for non-selected video data among the plurality of video data based on changes over time of the non-selected video data. (Item 8) The image processing device described in item 7 is characterized in that, when switching the video data to be selected according to the predetermined order, the control means skips the selection of the video data to be selected next if the length of the period for the video data to be selected next is less than a predetermined length. (Item 9) The image processing device described in item 7 or 8, characterized in that the control means dynamically changes the length of the period for the unselected video data so that the length of the period does not fall below a predetermined length. (Item 10) The image processing device described in any one of items 1 to 9, characterized in that the control means switches the selected video when the length of the period during which the selected video data is selected reaches a predetermined length, even if the dynamically changed period length has not been reached. (Item 11) 11. The image processing device according to any one of items 1 to 10, wherein the control means reduces the period when the length of the period during which the selected video data is selected exceeds a predetermined length. (Item 12) An image processing method executed by an image processing device that selects, in a predetermined order, one of a plurality of video data obtained by a plurality of image capture devices placed in different geographical locations, and outputs the selected video data to an external device, comprising: An image processing method characterized by dynamically changing the length of the period for which the selected video data is output to the outside based on one or more of the changes over time of the multiple video data and the changes over time of information regarding viewing of the selected video data. (Item 13) 12. A program for causing a computer to function as a control unit of the image processing device according to any one of items 1 to 11.

[0112] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention. [Explanation of symbols]

[0113] 201...image processing device, 202...control unit, 203...operation unit, 205...nonvolatile memory, 206...system memory, 207...communication I / F unit, 208...system timer, 300, 311, 312...camera

Claims

1. An image processing device that selects one of a plurality of pieces of video data obtained by a plurality of image capture devices placed in different geographical locations in a predetermined order, and outputs the selected video data to an external device, An image processing device characterized by having a control means for dynamically changing the length of the period for which the selected video data is output to the outside based on one or more of the changes over time of the multiple video data and the changes over time of information regarding the viewing of the selected video data.

2. The image processing device of claim 1, wherein the changes over time in the multiple video data include one or more of the movement of the entire frame of the video data, the movement speed or acceleration of the subject within the frame, changes in the subject's movement or facial expression, changes in brightness of the entire frame or the subject area, changes in frame rate, and changes in audio volume or main frequency.

3. The image processing device of claim 1, wherein the changes over time in the viewing information include one or more of the following: a change in the number of viewers, a change in the number of comments on social media regarding the selected video data, the number of likes and / or dislikes, and a change in the cumulative amount of tips.

4. The image processing device described in claim 1, characterized in that the control means calculates an evaluation value for one or more of the changes over time of the multiple video data and the changes over time of information regarding viewing of the selected video data, and increases the length of the period when the evaluation value exceeds a threshold value.

5. The image processing device described in claim 1, characterized in that the control means calculates an evaluation value for one or more of the changes over time of the multiple video data and the changes over time of information regarding viewing of the selected video data, and reduces the length of the period if the evaluation value does not exceed a threshold.

6. 5. The image processing apparatus according to claim 4, wherein the control means changes the unit of increase or decrease of the threshold value or the length of the period based on a weight set for the evaluation value.

7. The image processing device according to claim 1, characterized in that the control means dynamically changes the length of the period for unselected video data among the plurality of video data based on changes over time in the unselected video data.

8. The image processing device described in claim 7, characterized in that when switching the video data to be selected according to the specified order, the control means skips the selection of the next video data to be selected if the length of the period for the next video data to be selected is less than a predetermined length.

9. 8. The image processing device according to claim 7, wherein the control means dynamically changes the length of the period for the non-selected video data so that the length of the period does not fall below a predetermined length.

10. The image processing device according to claim 1, characterized in that the control means switches the selected video when the length of the period during which the selected video data is selected reaches a predetermined length, even if the length of the dynamically changed period has not been reached.

11. 2. The image processing device according to claim 1, wherein the control means reduces the period when the length of the period during which the selected video data is selected exceeds a predetermined length.

12. An image processing method executed by an image processing device that selects, in a predetermined order, one of a plurality of video data obtained by a plurality of image capture devices placed in different geographical locations, and outputs the selected video data to an external device, comprising: An image processing method characterized by dynamically changing the length of the period for which the selected video data is output to the outside based on one or more of the changes over time of the multiple video data and the changes over time of information regarding viewing of the selected video data.

13. A program for causing a computer to function as the control means of the image processing device according to any one of claims 1 to 11.

Images