Image processing device, image processing method, and program

Abstract

This image processing device is provided with an image processing unit which, on the basis of a plurality of input images, which are images obtained by imaging an event using a plurality of imaging devices, generates a distribution image in which a first area, which is a main image region, and a second area, which is an image region different from the first area, are formed within a screen. Further, the image processing device is provided with a control unit which is capable of executing control in accordance with a plurality of switching control modes relating to switching of the input images included in the distribution image, and which controls the image processing unit such that a captured image displayed in the second area is switched on the basis of any one of the plurality of switching control modes.

Description

[Technical Field]

[0001] The present technology relates to an image processing device, an image processing method, and a program, and to a technical field of processing images to be distributed when an event is captured by multiple image capturing devices. [Background technology]

[0002] Patent Document 1 below discloses a technique for controlling the resolution in accordance with a priority based on an operation history when images input from a plurality of cameras are compressed and combined. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] WO98 / 06222 publication Summary of the Invention [Problem to be solved by the invention]

[0004] For example, at an event such as a live music concert, images are sometimes taken using a plurality of imaging devices (cameras), and main line images are generated and distributed while switching between the images taken by each imaging device. However, each user has different tastes and viewing purposes, and the images that are distributed are not necessarily the images that the user wants to see. For example, when watching a live music concert, there are fans of individual band members, people who want to use the images as reference for practicing their instruments, people who are interested in the equipment, people who want to enjoy the overall view of the music performance, and so on, and each person has different tastes and purposes, and the content they want to see will also be different.

[0005] Therefore, an object of the present technology is to enable generation of distribution images that allow more suitable image switching in response to users with diverse preferences. [Means for solving the problem]

[0006] The image processing device according to the present technology includes an image processing unit that generates a distribution image based on a plurality of input images that are images of an event captured by a plurality of imaging devices, in which a first area that is a main image area and a second area that is an image area different from the first area are formed on the screen, and a control unit that controls the image processing unit so that control can be performed using a plurality of switching control modes for switching the input images to be included in the distribution image, and the captured image displayed in the second area can be switched based on one of a plurality of switching control modes. This allows a plurality of switching control modes to be applied to the image in the second area. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is an explanatory diagram illustrating an example of an arrangement of imaging devices for distribution according to an embodiment of the present technology; [Figure 2] 1 is an explanatory diagram of a distribution system including an image processing apparatus according to an embodiment; [Figure 3] FIG. 2 is a block diagram of a processing function of an image processing unit according to an embodiment. [Figure 4] FIG. 2 is a block diagram of a control unit according to the embodiment. [Figure 5] FIG. 10 is an explanatory diagram of image switching according to an embodiment. [Figure 6] FIG. 2 is an explanatory diagram of an example of a GUI (Graphical User Interface) according to the embodiment. [Figure 7] 10 is a flowchart of an example of processing corresponding to a user operation according to the first, second, and third embodiments. [Figure 8] 10 is a flowchart of an example of a process related to mode selection according to the first embodiment. [Figure 9] 10 is a flowchart of an example of a process related to mode selection according to the second embodiment. [Figure 10] 13 is a flowchart of an example of a process related to mode selection according to the third embodiment. [Figure 11] FIG. 2 is an explanatory diagram of an example of a GUI according to an embodiment. [Figure 12] FIG. 2 is an explanatory diagram of an example of a GUI according to an embodiment. [Figure 13] 10 is an explanatory diagram of an example of a GUI for fixing a recommended area according to an embodiment. FIG. [Figure 14] 10 is a flowchart of an example of a process for fixing a recommended area according to an embodiment. [Figure 15] FIG. 2 is an explanatory diagram of an example of a GUI according to an embodiment. [Figure 16] 13 is a flowchart of a processing example of the fourth embodiment. [Figure 17] 13 is a flowchart of a processing example according to the fifth embodiment. [Figure 18] FIG. 10 is an explanatory diagram of an example of association according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] The embodiments will be described below in the following order. <1. System configuration> <2. Streaming image and switching control mode> 3. First Embodiment 4. Second Embodiment 5. Third Embodiment <6.GUI example> 7. Fourth Embodiment 8. Fifth Embodiment <9. Examples of applicable processing> <10. Summary and Variations>

[0009] <1. System configuration> 1 shows an example of the arrangement of imaging devices (cameras) 1 in a distribution system according to an embodiment. The imaging device 1 may be a video camera or a still camera. The imaging device 1 shown in the figure is an example of one that captures video of performers on a stage.

[0010] In this disclosure, "image" refers to both moving images and still images. While the description will be primarily focused on video distribution, the images to be distributed are not limited to moving images; they may also be still images or slide shows of multiple still images. Furthermore, "image" refers to the image that is actually displayed on the screen, but "image" in the signal processing process and transmission path leading up to its display on the screen refers to image data.

[0011] The example in FIG. 1 shows a schematic example in which a plurality of imaging devices 1 are arranged on a live music stage with six performers. In this case, the three main band members are lined up in the front row, and the three support members are lined up in the back row. There are various types of imaging devices 1, such as those arranged to capture the entire stage, those arranged to capture close-up images of individual band members, and those arranged to capture images of the band members' hands and equipment.

[0012] When multiple imaging devices 1 are arranged on a live stage or the like in this way to stream live images of an event, the images from each imaging device 1 are typically switched sequentially using a switcher, and one system of images from each imaging device 1 is generated and distributed as the images to be distributed to users (so-called main line images). In this case, typical image switching tends to involve sequentially switching between the overall image and images of each of the three band members, with images of the support members occasionally inserted. In this embodiment, as will be described in detail later, in addition to a main image such as a main line image or a specific image of the entire stage, an image in a certain switching control mode is presented as a recommended image. In this case, the switching control mode is selected according to the preferences of each user, so that image switching suitable for each user can be easily performed. In this disclosure, the term "user" refers to a destination user who watches a broadcast. In addition, in this disclosure, "live" refers to a "live show" and includes "live music (concert)", "live event", etc.

[0013] 2 shows the configuration of a distribution system according to an embodiment, which includes imaging devices 1 (1A to 1(n)), a server device 2, a user terminal 3, a transceiver 5, a network 6, and an image processing device 10.

[0014] Imaging devices 1A, 1B, 1C, . . . 1(n) indicate imaging devices 1 arranged at various locations as shown in Fig. 1. For the sake of explanation, when there is no need to distinguish between imaging devices 1A, 1B, . . . 1(n), they will be referred to as "imaging device 1."

[0015] Captured images are input from the imaging devices 1A, 1B, 1C, . . . 1(n) to the image processing device 10. These are referred to as input images PinA, PinB, PinC, . . . Pin(n). When these input images PinA, PinB, PinC, . . . Pin(n) are collectively referred to as "input image Pin."

[0016] In the distribution system of this embodiment, the input image Pin is considered to be a real-time captured image captured by each imaging device 1. For example, the image processing device 10 is installed at an event venue such as a live event, and captured images from each imaging device 1 are input via a cable or the like.

[0017] However, the present invention is not limited to this configuration. The image processing device 10 may be located at a location remote from the event venue, and each input image Pin may be transmitted via wired or wireless communication. Furthermore, the distribution in this embodiment is not limited to real-time distribution. For example, images captured by each imaging device 1 may be recorded on a recording medium. Then, at a later point in time, the images captured by each imaging device 1 may be synchronously read out from the recording medium, and the read images may be input to the image processing device 10 as input images PinA, PinB, PinC,..., Pin(n).

[0018] The image processing device 10 includes a control unit 11 and an image processing unit 12 . The image processing unit 12 generates a distribution image PO having an image of a first area, which is the main image area on the screen, and an image of a second area, which is an image area different from the first area, based on multiple input images Pin, which are images of an event captured by multiple imaging devices 1. In this embodiment, the first area is assumed to be a main area AR1 in Fig. 6, which will be described later, and the second area is assumed to be a recommended area AR2 in Fig. 6. Furthermore, a camera area AR3 in Fig. 6 may be provided as a third area.

[0019] The control unit 11 controls the image processing unit 12 so that control can be performed using multiple switching control modes for switching the input image Pin to be included in the distribution image PO, and the image displayed in the second area is switched based on one of the multiple switching control modes. For this reason, the control unit 11 includes a processing instruction unit 11a. The processing instruction unit 11a instructs the image processing unit 12 to use an input image Pin to generate a delivery image PO. In this case, the instruction can be given by applying various switching control modes, which will be described later. The control unit 11 also includes a DB (database) 11b, which is configured to store information for each user, such as the operation history and attribute information for each distribution destination user.

[0020] An example of processing by the image processing unit 12 is shown in FIG. The image processing unit 12 is configured as an image processor, for example, using a DSP (Digital Signal Processor), etc. In Fig. 3, the processing performed in the image processing unit 12 is shown in blocks.

[0021] The image processing unit 12 performs image size processing 15A, 15B, 15C, . . . 15(n) on each of the input images PinA, PinB, PinC, . . . Pin(n). When these are collectively referred to, they are referred to as "image size processing 15."

[0022] The image size processing 15 includes processing for converting the input image Pin into an image S1 of a size to be used in the first area, processing for converting the input image Pin into an image S2 of a size to be used in the second area, and processing for converting the input image Pin into an image S3 of a size to be used in the third area. The first area (main area AR1) is, for example, the largest area on the screen. The second area (recommended area AR2) is smaller than the first area. The third area (camera area AR3) is an area in which images from each imaging device 1 are arranged, and is the area in which each image is displayed in the smallest size. Note that this relationship between the areas is just an example.

[0023] To obtain image S1 displayed in the first area, image S2 displayed in the second area, and image S3 displayed in the third area, image size processing 15 performs a data size conversion process on the input image Pin. Data size conversion is, for example, a resolution conversion process. The resolution is set to image S1 > image S2 > image S3. Note that image S1 may have the highest resolution, and images S2 and S3 may have approximately the same resolution. In other words, the resolution may be set to image S1 > image S2 ≧ image S3. However, there may be cases where it becomes difficult to maintain the resolution of image S1 due to changes in the communication status or processing load of network 6. Taking this into consideration, the resolution may be set to image S1 ≧ image S2 ≧ image S3. Image cropping may also be performed in the image size processing 15. For example, image S1 may be a full-size image, image S2 may be an image cropped by cutting out the periphery of the image, and image S3 may be an image cropped by cutting out the center of the image.

[0024] The compression rate may also be adjusted in the image size processing 15. For example, processing is performed so that the compression rate increases stepwise for image S1, image S2, and image S3.

[0025] Furthermore, frame rate conversion may be performed in the image size processing 15. Image S1 of the first area may have the highest frame rate, image S2 may have a lower frame rate, and image S3 may have an even lower frame rate. Note that image S1 may have the highest frame rate, and images S2 and S3 may have approximately the same frame rate. In other words, the frame rate may be set to image S1 > image S2 ≥ image S3. In this case, too, taking into consideration temporary changes due to communication conditions, etc., the frame rate may be set to image S1 ≥ image S2 ≥ image S3.

[0026] Furthermore, for example, image S3 may be a so-called thumbnail image. In the case of a moving image, it is possible to extract an image from a long span, for example, one frame per minute, and convert it to a thumbnail image with low resolution.

[0027] As in the above example, in the image size processing 15, image S1 is generated as a high-resolution image with the largest amount of data, and images S2 and S3 are generated as data with reduced amounts of data compared to image S1. Note that image S1 only needs to have the largest amount of data, and images S2 and S3 may have approximately the same amount of data. In other words, the data amount may be such that image S1 > image S2 ≥ image S3. Furthermore, the processing in the image size processing 15 described above is just one example, and the resolution or frame rate of image S1 may be substantially the same as the resolution or frame rate of image S2 depending on the state of the network 6, settings by the destination user, etc. In this sense, it is conceivable to make the data amount equal to image S1 ≧ image S2 ≧ image S3.

[0028] In this disclosure, the terms "image S1," "image S2," and "image S3" are used to indicate differences in the amount of data for the input image Pin. These images S1, S2, and S3 are collectively referred to as "input image Pin."

[0029] The image processing unit 12 performs image size processing 15 on each of the input images PinA, PinB, ..., Pin(n) to obtain image data as images S1, S2, and S3, respectively, and then performs distribution image generation processing 16. In the distribution image generation process 16, image S1, image S2, and image S3 are selectively synthesized to generate distribution images PO1, PO2, PO3, PO(m) for multiple channels ch1, ch2, ch3, ch(m). Note that the distribution images PO1, PO2, PO3, PO(m) are collectively referred to as "distribution images PO."

[0030] The distribution image PO of one channel is generated corresponding to, for example, one distribution destination user (a viewer who watches the distribution image at the distribution destination; hereinafter simply referred to as "user"). Therefore, the number of channels "m" is not fixed, but varies depending on the number of users at that time. The number of users can be considered to be the number of channels, but since it is also assumed that there are users to whom distribution can be made on a common channel, the number of users does not necessarily equal the number of channels. The distribution image PO of each channel has different images that can be switched between the first area and the second area, and the upper limit of the number of channels can be said to be determined by the number of combinations based on the number of imaging devices 1 and the number of switching control modes, which will be described later.

[0031] Each delivery image PO is image data including an image S1 of the first area, an image S2 of the second area, and an image S3 of the third area, but the combination of images S1, S2, and S3 differs depending on which imaging device 1 captured them. The combination of images S1, S2, and S3 in each delivery image PO also changes over time. This is to ensure that the images displayed in the first and second areas are switched for each scene to suit the preferences of the user at the delivery destination.

[0032] Furthermore, the distribution image generation process 16 selectively combines images S1, S2, and S3 to generate one distribution image PO, but in this case, "combining" may mean combining images S1, S2, and S3 to generate one frame of image data so that they are displayed in a split format on the screen, or it may be a process of associating each frame of data as image S1, image S2, and image S3 of each imaging device 1 as one set of data (i.e., data for one frame timing on one channel) that is displayed simultaneously.

[0033] 2 to the network 6 and is supplied to, for example, the server device 2. Note that the transceiver 5 may supply the distribution images PO directly to the server device 2 via wired or wireless communication without going through the network 6. The image processing unit 12 may also be configured within the server device 2.

[0034] The server device 2 is a server that provides a distribution service to users, and distributes, via a network 6, for example, a distribution image PO of a channel generated for a user who has logged in to the distribution service to the user terminal 3 of the user.

[0035] The user terminal 3 is an information processing device for each user, and is configured, for example, by a mobile terminal device such as a smartphone, a personal computer device, a display device capable of network communication, or the like. The user terminal 3 is equipped with a display device, and can view the delivered images. The user can also perform various operations on the delivered images displayed in a predetermined GUI. The operation information is transmitted to the control unit 11 of the image processing device 10 via the network 6.

[0036] The network 6 may be, for example, the Internet, a home network, a LAN (Local Area Network), a satellite communication network, or any other type of network. In FIG. 2, the server device 2, the user terminal 3, and the transceiver 5 communicate with each other via the network 6, but they may also be directly connected to each other for communication without the network 6 being used.

[0037] 2 is configured by, for example, a microprocessor and a storage device, etc. Alternatively, it may be configured as a computer system including a plurality of chips, interfaces, various devices, etc. An example of the configuration of such a control unit 11 is shown in FIG.

[0038] The control unit 11 has a CPU (Central Processing Unit) 71. The CPU 71 executes various processes according to programs stored in a ROM (Read Only Memory) 72 or a nonvolatile memory unit 74 such as an EEPROM (Electrically Erasable Programmable Read-Only Memory), or programs loaded from a storage unit 79 to a RAM (Random Access Memory) 73. The RAM 73 also stores data necessary for the CPU 71 to execute various processes as needed.

[0039] Instead of the CPU 71, or together with the CPU 71, a GPU (Graphics Processing Unit), a GPGPU (General-purpose computing on graphics processing units), an AI (Artificial Intelligence) processor, or the like may be provided.

[0040] The CPU 71, ROM 72, RAM 73, and nonvolatile memory unit 74 are interconnected via a bus 83. The bus 83 is also connected to an input / output interface 75.

[0041] The input / output interface 75 may be connected to an input unit 76 that includes an operator or an operating device. For example, the input unit 76 may be various types of operators or operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, or a remote controller. The input unit 76 detects an operation by the operator, and the CPU 71 interprets a signal corresponding to the input operation.

[0042] The input / output interface 75 may be integrally provided with a display unit 77 consisting of a liquid crystal panel (LCD: Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, or an audio output unit 78 consisting of a speaker, or these may be connected separately. The display unit 77 displays images for various image processing, moving images to be processed, etc. on the display screen based on instructions from the CPU 71. Furthermore, the display unit 77 displays various operation menus, icons, messages, etc., i.e., GUI (Graphical User Interface), based on instructions from the CPU 71.

[0043] The input / output interface 75 may be connected to a storage unit 79 configured with a hard disk or solid-state memory, or a communication unit 80 configured with a modem or the like.

[0044] The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, and the like.

[0045] A drive 81 is also connected to the input / output interface 75 as required, and a removable recording medium 82 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon. Drive 81 allows data files such as image files and various computer programs to be read from removable recording medium 82. The read data files are stored in storage unit 79, and images and sounds contained in the data files are output on display unit 77 and audio output unit 78. Furthermore, computer programs and the like read from removable recording medium 82 are installed in storage unit 79 as needed.

[0046] In the control unit 11 configured as above, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 80 or via a removable recording medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, etc.

[0047] In the control unit 11 having the above configuration, the CPU 71 performs the function of, for example, the processing instruction unit 11a based on a program. Furthermore, DB 11b can be constructed in the storage unit 79.

[0048] The configuration example shown in FIG. 4 is also assumed as an example of the configuration of the information processing device as the server device 2 or the user terminal 3 in FIG.

[0049] <2. Streaming image and switching control mode> As shown in FIGS. 1 and 2, when input images Pin are input to the image processing unit 12 by a plurality of imaging devices 1, the image processing unit 12 can generate distribution images PO with diverse content by switching between images.

[0050] 5A, 5B, 5C, 5D, and 5E show examples of input images Pin obtained by, for example, five imaging devices 1. FIG. Fig. 5A is an image taken by imaging device 1 capturing a wide-angle shot of the entire stage. Figs. 5B, 5D, and 5E are close-up images of individual band members, and Fig. 5C is an example of a close-up image of the hands of one band member.

[0051] By switching between images of such diverse content as the music progresses, users can view them from a wider variety of perspectives than when watching a live concert at a venue. However, the type of images that users want to see varies from user to user.

[0052] Therefore, in this embodiment, for example, the GUI on the screen displayed on the user terminal 3 is configured to display a main area AR1, a recommended area AR2, and a camera area AR3 as shown in Figure 6, allowing the user to perform any operation.

[0053] The main area AR1 is configured to occupy the largest area on the display screen and to display the image with the highest resolution. This main area AR1 has the meaning of a so-called main image, and displays the image that the user primarily views. The image in main area AR1 may be fixed as an image showing the entire stage as shown in Figure 5A, or may be switched to each of the images shown in Figures 5B, 5C, 5D, and 5E as the music progresses. Furthermore, in this embodiment, when the user operates recommended area AR2 or camera area AR3, an image of that area is displayed in main area AR1.

[0054] The recommendation area AR2 is an area where images are switched based on a certain switching control mode. Switching images means that the displayed image is switched among the input images PinA...Pin(n). For example, as the music progresses, one of the input images PinA...Pin(n) is selected sequentially and becomes the image displayed in the recommendation area AR2.

[0055] In this embodiment, a plurality of switching control modes are provided for switching images, and the images displayed in the recommendation area AR2 are switched depending on which switching control mode is selected. The selection of this switching control mode and the switching control of the images displayed in the recommendation area AR2 according to the selected switching control mode are performed by the control unit 11. In response to an instruction from the control unit 11 to switch images based on the selected switching control mode, the image processing unit 12 selects an image S2 from one of the input images PinA...Pin(n) as the image S2 to be displayed in the recommended area AR2, and generates a distribution image PO for one channel. In addition, the image processing unit 12 also switches the image displayed in the main area AR1 in response to an instruction from the control unit 11.

[0056] Here, the control unit 11 selects the switching control mode individually for each user. That is, the switching control mode is selected / updated for each channel. This allows the image of the recommended area AR2 to be switched to an image suitable for each user.

[0057] The camera area AR3 in Fig. 6 is an area that displays images captured by each of a plurality of imaging devices 1. The figure shows an example in which the camera area AR3 is divided into 12 frames W, and each frame W is an area in which an image captured by each imaging device 1A, 1B, etc. is displayed. Therefore, the division indicated by the dashed lines in the camera area AR3 (the number of frames W) may be made according to the number of imaging devices 1. Furthermore, if a larger number of imaging devices 1 are used and they cannot all be displayed, the frames W in the camera area AR3 may be scrolled or some of the imaging devices 1 may be selectively displayed. Furthermore, the imaging device 1 whose captured image is displayed may be switched at regular intervals. Furthermore, an image captured by an imaging device 1 that is not displayed in the main area AR1 or the recommended area AR2 may be assigned to each frame W of the camera area AR3 for each time point.

[0058] In the user terminal 3, for example, the recommended area AR2 or any frame W within the camera area AR3 can be specified by using a touch panel on a display displaying such a GUI or by operating a cursor.

[0059] The above-mentioned switching control modes are exemplified by the following modes M1, M2, M3, M4, and M5. Note that in the present disclosure, the switching control modes are also simply referred to as "modes."

[0060] Mode M1 is a mode based on advance learning, specifically, a mode in which image switching control is performed based on progress information about an event that is obtained in advance. Events to which the distribution system of this embodiment can be applied include live music performances, plays, operas, ballets, dance performances, etc., where the progression is somewhat predetermined. In other words, these are events where progress information such as sheet music for each song, live set lists, scripts, and scenarios can be obtained. This may be part of an event. Progress information may also be obtained by analyzing videos taken during live rehearsals, videos of other live performances related to the target song, or promotional videos (including audio) for the target song. For example, in the case of a live music performance, if there is sheet music information as progression information for a certain song, the progression of the song during its performance can be known in advance, such as the main instruments in the intro period (the introductory part of the song), when the main vocals begin and when the main melody of the instrument begins, the timing of the rhythm section's "decisive moments," the period when you want to pay attention to the backing chorus, the lead instruments and duration of interludes such as guitar solos, and the outro period (the final part of the song). Based on such progress information, it is possible to know from the start timing of a song, for example, which image capturing device 1 should be used to select an image captured. Mode M1 is a mode in which image switching is controlled based on such progress information.

[0061] An example of mode M1 will be described more specifically. For example, the control unit 11 analyzes and stores in advance the captured images to be provided to the user in accordance with the "melody perspective" and "change perspective" at each scene during the progression of a specific song at a live performance, based on the sheet music, video content, etc. In other words, it analyzes which of the captured images by the imaging device 1 is the image that the user should most look at as the song progresses. For example, from the "melody viewpoint" and "variation viewpoint," an image showing a specific part of a specific performer is selected in the following cases.

[0062] First, here is an example from a "melody perspective." When the main melody is being played, which is the most noticeable part, the image switches to the main melody performer and their hands. When the counter melody, which is the second most prominent part, is played, the image switches mainly to the counter melody performer and their hands. The third most prominent element is the backing, which switches images to show the performers of each instrument in turn. Backing is a performance that maintains a certain pattern or rhythm to support the solos of other performers. During a solo performance that stands out in a particular scene, the image will switch to the image of the soloist and their hands.

[0063] Examples of progression information for analyzing melodic aspects include piano scores, band scores, orchestra scores, and scores for solo instruments. According to the piano score, the parts played by the left hand include bass notes, chords, counter melodies, etc., and the parts played by the right hand include the main melody and chords. Band scores show the playing status of each instrument, such as keyboard scores, guitar scores (standard scores, tablature), bass scores (standard scores, tablature), and drum scores. Tablature is a score that indicates which strings to press and where to press them. Drum scores are scores that show the snare, hi-hat, bass drum, toms, cymbals, etc., in correspondence with specific scales. Orchestral scores include full scores (musical scores for each instrument in the orchestra) and condensed scores (simple scores that mainly represent orchestral and wind band music), which show the playing status of each instrument as the music progresses. Scores for solo instruments include string instruments, woodwind instruments, brass instruments, percussion instruments, Japanese instruments (such as the koto, shamisen, and shakuhachi), and other folk instruments from around the world, allowing you to see the performance of each instrument as the piece progresses.

[0064] The control unit 11 can input the above-mentioned musical score data in advance and perform analysis processing to determine the timing and duration of the main melody, counter melody, solo, backing, etc. in the music piece.

[0065] Next, the following are examples of "perspectives of change." ·Light and shade in major / minor keys When the music changes from a bright atmosphere (major key) to a dark atmosphere (minor key), or vice versa, the image is switched to that of the performer of each instrument whose volume is the loudest before and after the change. ·Slow and fast When there is a change in tempo or beat, from slow to fast or from fast to slow, the image switches to that of the performer of each instrument whose volume is the loudest before and after the change. ·Volume volume When there is a sudden increase or decrease in volume, the image switches to that of the performer of the main instrument that makes up the music.

[0066] The presence or absence of changes and the timing of changes can be determined by focusing on the following points. The main key and key changes of the music piece are determined, and a performer of a melody that gives an impression of a bright atmosphere in a major key or a dark atmosphere in a minor key is determined. The presence or absence of repeat signs is also a factor in the extraction. When there are repeat signs, the volume and changes are generally made larger the second time it is played than the first time to impress the audience. Therefore, it is possible to determine whether the performer of the main melody will play for a longer period of time. The overall number of notes is also a factor in the judgment. If there are many whole notes and half notes, the song has little variation, and if there are many eighth notes and sixteenth notes, the song has much variation. The presence or absence of tempo markings and fermatas can be used to determine changes in beat and tempo during a piece. Volume symbols and volume change symbols (crescendo, diminish) are used to determine volume changes. The title of a song can be used to determine what the song is trying to convey as a whole. For example, it can roughly tell whether the song is generally bright or dark, or whether there are likely to be changes along the way.

[0067] By determining the brightness, tempo, and volume of the music through these determinations, the timing of image switching as a change point can be set.

[0068] Mode M2 ​​is a mode in which image switching control is performed for a delivery image PO(x) for a specific delivery destination user based on the viewing information of delivery destination users other than the specific delivery destination user. Note that delivery image PO(x) means any of delivery images PO1 to PO(m). That is, in mode M2, when an image is delivered to a user, image switching control is performed according to the viewing status of other users, for example, switching control is performed to select an image that is being viewed by many other users from among multiple input images Pin.

[0069] The control unit 11 detects operation information of each user at the distribution destination, and can therefore determine which images of the input image Pin are being viewed in the main area AR1 by most of the other users as seen from the user U1. Mode M2 ​​is a mode in which images are switched sequentially so as to provide such determination results as images in the recommended area AR2 for the user U1.

[0070] Mode M3 is a mode in which image switching control is performed for a distribution image PO(x) for a specific distribution destination user based on viewing information of other distribution destination users having the same or similar attributes as the specific distribution destination user.

[0071] The control unit 11 collects user attribute information for each user. For example, the user terminal 3 is assumed to have the functions and shape of a tablet terminal, that is, it has a display device with a touch panel, is capable of accessing the Internet, and has a built-in ultra-small camera.

[0072] In such cases, the following items may be collected automatically: For example, the user is photographed using an ultra-small camera on the user terminal 3. By acquiring this image, the control unit 11 can determine the user's gender, age, skin color, and race, as well as other characteristics, such as whether the user has long or short hair and whether or not the user has face paint. From such appearance determinations, the user's preferences can be inferred. For example, if the user is male and has long hair, it can be inferred that the user "likes rock music" or "likes electric guitar." If the user has face paint, it can be inferred that the user "is a soccer or baseball fan, or likes cheerful music with musical instruments." Such inferred information is stored as the user's attributes in, for example, DB 11b. The user may be required to input information such as age, sex, and preferences.

[0073] The control unit 11 also acquires location information from a GPS receiver or the like within the user terminal 3, and recognizes the location where the user terminal 3 is being used. If the user terminal 3 is in the same location for a certain period of time, the control unit 11 stores the location as user attribute information in the DB 11b. The control unit 11 recognizes the location most frequently recorded over a certain period, such as a week or a month, as "home" or "long-term stay hotel," and can estimate the user's preferences from information such as descriptions of videos that the user manually searches for and watches on the user terminal 3, titles displayed therein, analysis of the videos themselves, and frequency of repeated viewing. For example, it can estimate the user's favorite artists (singers or performers), favorite songs, favorite scenes related to live performances, and specific parts of songs. Furthermore, by obtaining property information of videos downloaded to the user terminal 3, the control unit 11 can estimate the user's favorite content, artists, etc. Such estimated information is stored as attributes of the user in, for example, DB 11b.

[0074] For example, by managing attribute information such as the above example for each user in DB 11b, it is possible to switch images for a delivery image PO(x) for a certain user by referring to images selected in the main area AR1 among delivery images PO of other users who have common or similar attributes. Mode M3 controls image switching based on such common attributes.

[0075] Mode M4 is a mode in which image switching control is performed for a plurality of input images relating to a specific subject. For example, this mode focuses on a specific artist, especially one of the band members, and switches between the face, hands, instrument, etc. A user who is a fan of a specific artist or a user who watches the video as a reference for practicing an instrument may want to focus on only a specific member. In such cases, an image captured by the imaging device 1 showing that specific member is distributed.

[0076] Mode M5 is a mode that randomly switches between images. Some users may want to see a variety of members and performance scenes without being tied to a specific scene. Mode M5 is the ideal switching control mode for such cases.

[0077] 3. First Embodiment A processing example of the first embodiment will be described with reference to Figures 7 and 8. The processing described below is an example of control processing by the control unit 11, and a delivery image PO is generated by the image processing unit 12 in response to this control processing. In the first embodiment, an example is taken in which either the above-mentioned mode M1 or M2 is selectively applied for image switching in the recommended area AR2.

[0078] FIG. 7 shows the process of image switching control in response to user operations, and FIG. 8 shows the process related to the switching control mode applied to the display of the recommended area AR2 and the main area AR1. The control unit 11 starts the process of Fig. 7 and the process of Fig. 8 in parallel at a certain start timing. For example, in the case of a live music streaming event, the start timing of the process may be when the streaming starts or when a specific song starts to be played. The period during which these processes are performed can be the period during which image switching control based on progress information is possible, for example, when using the above-mentioned mode M1 as one of the modes. Of course, this is just one example. Various periods can be set, such as the entire period within the distribution period or a part of the period.

[0079] First, the processing in FIG. 7 will be described. In step S100, the control unit 11 performs image setting for the main area AR1 and the recommended area AR2. This image setting is a process of determining which input image Pin is initially applied to the main area AR1 and the recommended area AR2, or in which mode switching control is performed.

[0080] For example, initially, the main area AR1 is set to be fixed to the input image Pin that captures the entire stage. For example, if the imaging device 1A captures the entire stage, the image of the main area AR1 is fixed to the image of the input image PinA. In addition, the mode selected in step S200 of Figure 8 described below is applied preferentially to the recommended area AR2, image switching control is applied, and distribution proceeds while switching between input images PinA...Pin(n). However, in this first embodiment, due to the processing of Figure 8 described below, in the recommended area AR2, the switching control mode is not fixed to a specific mode and image switching is performed, but rather, for example, one of modes M1 and M2 is applied preferentially, while image switching is also performed using the other mode.

[0081] In addition, if a switching control mode suitable for the destination user has been determined in the past, or if the same song, artist, etc. has been distributed in the past and a switching control mode has been selected, in step S100 of Figure 7, the setting may be made so that switching control is initially performed in the corresponding mode in the main area AR1.

[0082] In step S101, the control unit 11 instructs the image processing unit 12 to generate a delivery image PO in accordance with the settings in step S100. As a result, an initial delivery image PO for a certain user is generated and delivered to the user terminal 3.

[0083] On the user terminal 3, images are displayed in the main area AR1, the recommended area AR2, and the camera area AR3, for example, as shown in FIG. For example, the main area AR1 initially displays an image of the entire stage, such as image S1 based on input image PinA, while the recommended area AR2 switches between images S2 based on input images PinA...Pin(n) in mode M1 or mode M2. The camera area AR3 displays images S3 based on input images PinA...Pin(n).

[0084] During distribution, the control unit 11 monitors user operations in steps S102 and S104. The user can perform some operations such as tapping, pinching, moving the cursor, or pressing the enter key on the GUI screen of the user terminal 3 shown in FIG. 6, and these operations are transmitted to the control unit 11 via the network 6.

[0085] When it is detected that the user has performed an operation to specify the recommended area AR2, the control unit 11 advances the process from step S102 to step S103. In this case, the control unit 11 controls the image processing unit 12 to apply the switching control mode that was applied to the recommended area AR2 at that time to the main area AR1. For example, if image switching was performed in mode M1 in the recommended area AR2 at the time of the user operation, the control unit 11 instructs the image processing unit 12 to also perform image switching in mode M1 in the main area AR1 in response to the operation.

[0086] When it is detected that the user has performed an operation to designate the camera area AR3, the control unit 11 advances the process from step S104 to step S105. In this case, the control unit 11 controls so that an image captured by the imaging device 1 corresponding to the frame W designated by the user within the camera area AR3 is displayed in the main area AR1. For example, when the user performs an operation to designate the frame W in which the image captured by the imaging device 1C is displayed, the control unit 11 instructs the image processing unit 12 to display an image S1 of the input image PinC in the main area AR1. It is possible that the image within the camera area AR3 will not be changed.

[0087] The control unit 11 repeats the above-described processes corresponding to the user operations until it is determined in step S106 that the control is to be ended. The end timing may be, for example, when the distribution ends or when the performance of a specific piece of music ends. When the end timing arrives, the control unit 11 performs the end process in step S110 to end the series of processes. In this end process, the number of operations performed by the user and history information on the operation contents may be stored in DB 11b in association with the user. In addition, the final display frequency value for each mode, which will be described next with reference to FIG. 8, may be stored in DB 11b in association with the user.

[0088] In addition to the above-described processing of FIG. 7, the control unit 11 performs processing of FIG. In step S200 of FIG. 8, a mode is selected in response to a user operation. For example, the control unit 11 controls the user terminal 3 to request the user to select a mode at an appropriate timing, such as before the start of distribution or before the start of playing a specific song. For example, modes M1 and M2 are presented and the user is asked to select one. In step S200, the control unit 11 obtains the corresponding user operation and initially selects the mode specified by the user.

[0089] In step S201, the control unit 11 sets the display frequency value for each mode in accordance with the selected mode. For example, if the user selects mode M1, the display frequency value for mode M1 is set to "90" and the display frequency value for mode M2 ​​is set to "10." This means that, for example, within 100 seconds, images selected by the algorithm of mode M1 are displayed for a total of 90 seconds, and images selected by the algorithm of mode M2 ​​are displayed for a total of 10 seconds.

[0090] In step S202, the control unit 11 selects a mode to be displayed in the recommendation area AR2 according to the display frequency value at that time, and instructs the image processing unit 12 to select the mode. For example, suppose the display frequency values ​​of modes M1 and M2 are set at a ratio of 90:10 as described above, and the process in Fig. 8 proceeds to step S202 for each unit time. In this case, control unit 11 selects mode M1 9 times out of 10 times when step S202 is reached, and instructs image processing unit 12 to display the image selected in mode M1 at that time in recommendation area AR2. Control unit 11 also selects mode M2 ​​1 time out of 10 times when step S202 is reached, and instructs image processing unit 12 to display the image selected in mode M2 ​​at that time in recommendation area AR2.

[0091] Note that the timing of image switching between modes M1 and M2, i.e., the timing of image S2 switching based on switching of input images PinA...Pin(n), is not necessarily constant but depends on the progress of the music, etc. Therefore, the above does not mean that the image of the recommended area AR2 is switched every time the processing of step S202 is performed, for example, per unit time, but rather that the mode applied to the recommended area AR2 at the time of processing step S202 is selected. For example, when mode M2 ​​is applied, if the image displayed in mode M2 ​​is different from the image of the recommended area AR2 at that time, the image is switched. Furthermore, during mode M2, when the progress of the music reaches the point at which the image is switched according to the algorithm of mode M2, the image is switched.

[0092] That is, for the mode applied to the recommended area AR2, step S202 of Through processing, modes M1 and M2 are selectively applied for each unit time at a ratio according to the display frequency value, for example. During the period in which mode M1 is applied, images that should be selected based on the algorithm of mode M1 are displayed in the recommendation area AR2, and during the period in which mode M2 ​​is applied, images that should be selected based on the algorithm of mode M2 ​​are displayed in the recommendation area AR2. Therefore, if the display frequency value is 90:10 as described above, in the recommended area AR2, image switching is mainly performed in mode M1, and image switching is occasionally performed in mode M2.

[0093] In this example, mode M1 is given priority in order to respect the user's selection. In other words, if the user first selects mode M1, images based on mode M1 are basically displayed in the recommendation area AR2. However, images based on mode M2 ​​are also occasionally displayed so that the user is also presented with image switching in modes that the user did not select.

[0094] As explained in Fig. 7, during distribution, the user can arbitrarily operate the recommended area AR2 or the camera area AR3, and the image displayed in the main area AR1 is switched. In the process of Fig. 8, the display frequency value is updated in response to such user operations.

[0095] In step S203, the control unit 11 determines whether or not the user has selected an image other than the main area AR1. The image selection operation here refers to an operation of specifying the recommended area AR2 or the camera area AR3, and determines whether or not such an operation has been performed.

[0096] Upon detecting a user operation, the control unit 11 proceeds from step S203 to step S207, and if the user operation is an operation to designate the recommended area AR2, the control unit 11 proceeds to step S208. In step S208, the control unit 11 determines the mode applied to the recommended area AR2 at the time of the operation, and updates the display frequency value of that mode to increase it. For example, if an operation is performed while mode M1 is applied to recommended area AR2, the display frequency value of mode M1 is increased and the display frequency value of mode M2 ​​is decreased. The increase amount may be a fixed value or a variable value. For example, if an increase of 5 points is desired, if the ratio of the display frequency values ​​immediately before the user selected an image other than the main area in S203 was M1:M2 = 90:10, the ratio of the display frequency values ​​after the user performed an operation while mode M1 was applied to recommended area AR2 is changed to M1:M2 = 95:5. Also, if an operation is performed while mode M2 ​​is applied to recommended area AR2, the display frequency value of mode M2 ​​is increased. At the same time, the display frequency value of mode M1 may be decreased. For example, if M1:M2 = 90:10, change it to M1:M2 = 85:15.

[0097] For the sake of explanation, the display frequency value is assumed to increase or decrease the ratio points, but it may also be to simply increase one of the points, as long as the update results in a change in the display ratio for each mode.

[0098] On the other hand, if the user operation is not an operation for specifying the recommended area AR2, the control unit 11 proceeds from step S207 to step S209, which is the case when an operation for selecting a certain frame W in the camera area AR3 is performed. In this case, the control unit 11 checks whether the selected image is an image that is displayed in the mode selected in step S200.

[0099] For example, suppose the mode selected in step S200 is mode M1. Suppose the user selects input image PinB in mode M1 at the timing of the operation. If the user then specifies frame W of input image PinB, which is the image of imaging device 1B, in camera area AR3, then the image selected from camera area AR3 can be said to be the image displayed in the mode selected in step S200. In such a case, the control unit 11 proceeds to step S210 and increases the display frequency value for the mode selected by the user in step S200. For example, if the user selects mode M1 and the image selected from camera area AR3 is an image that matches mode M1, the control unit 11 increases the display frequency value for mode M1. At this time, the control unit 11 may decrease the display frequency value for mode M2. For example, M1:M2 = 90:10 is changed to M1:M2 = 95:5.

[0100] On the other hand, if the image selected by the user from camera area AR3 is not an image corresponding to the mode selected by the user at the time of the operation, control unit 11 proceeds to step S211 and increases the display frequency value corresponding to the mode other than the mode selected by the user in step S200. For example, if the user has selected mode M1 and the image selected from camera area AR3 is not an image corresponding to mode M1, the display frequency value of mode M2 ​​is increased. The display frequency value of mode M1 may also be decreased. For example, M1:M2 = 90:10 is changed to M1:M2 = 85:15.

[0101] During the period when no user operation for the recommended area AR2 or the camera area AR3 is detected, the control unit 11 proceeds from step S203 to step S204, and checks whether a predetermined time has elapsed since the user's previous image selection (i.e., operation for the recommended area AR2 or the camera area AR3). For example, the predetermined time may be 10 minutes. If the predetermined time has not elapsed, the control unit 11 returns to step S202.

[0102] If an image selection operation has not been performed for a predetermined time, the control unit 11 proceeds to step S205 to check whether a mode selected by the user is currently being applied to the main area AR1. If a mode is currently being applied, the control unit 11 proceeds to step S212 to increase the display frequency value for the mode selected by the user in step S200. For example, if the user has selected mode M1 and mode M1 is currently being applied to the main area AR1, the display frequency value for mode M1 is increased and the display frequency value for mode M2 ​​is decreased.

[0103] For example, the main area AR1 may initially display a fixed image, or an image based on a mode set based on past history. Alternatively, the mode of the recommended area AR2 may be applied in step S103 of FIG. 7, causing the main area AR1 to display a certain mode. The process proceeds to step S212 of FIG. 8 when the mode selected by the user in step S200 has been applied to the main area AR1 for a predetermined period of time. In other words, the user is viewing the content and considers the image switching control in that mode appropriate. Therefore, the display frequency value for the mode selected by the user in step S200 is increased.

[0104] On the other hand, if an image selection operation has not been performed for a predetermined time and it is determined in step S205 that the mode selected by the user is not being applied to the main area AR1 at that time, the control unit 11 proceeds to step S206 and increases the display frequency values ​​of modes other than the mode selected by the user in step S200. For example, if the user selects mode M1 and mode M2 ​​is applied to the main area AR1 at that time, but no operation has been performed for a predetermined time, it can be assumed that the user's interest in mode M1 is waning. Therefore, the control unit 11 increases the display frequency value of mode M2 ​​and decreases the display frequency value of mode M1, for example.

[0105] As described above, the display frequency value is updated in response to a user operation or in response to no operation for a predetermined period of time or longer, and the process returns to step S202 via step S220. In step S202, as described above, the mode is selected and displayed in the recommendation area AR2 according to the display frequency value, but the frequency at which modes M1 and M2 are applied changes as the display frequency value is updated. For example, if the display frequency value of mode M2 ​​is high, there will be more opportunities for image switching in mode M2 ​​in the recommended area AR2.

[0106] When the same end timing as in FIG. 7 arrives, the control unit 11 ends the processing in FIG. 8 from step S220.

[0107] As shown in FIG. 8, the display frequency value is updated and the mode application control of the recommended area AR2 is performed in accordance with the display frequency value. Therefore, the switching control of the images displayed in the recommendation area AR2 will converge to a state suitable for the user depending on the user's operations and viewing state.

[0108] Note that due to the control shown in FIGS. 7 and 8, there may be a period in which the same image is displayed in the main area AR1 and the recommended area AR2. However, in the recommended area AR2, one mode is not continuously applied, and the user's operations are performed irregularly, so the situation in which the same image is displayed in the main area AR1 and the recommended area AR2 is gradually eliminated.

[0109] Although the above examples show the use of modes M1 and M2, three or more modes may be used. For example, modes M1, M2, and M3 may be used, or modes M1, M2, and M4 may be used. Furthermore, modes M1, M2, M3, and M4 may also be used.

[0110] In such cases, there will be multiple modes other than the mode selected by the user in step S200, and the display frequency value can be updated accordingly. For example, if modes M1, M2, and M4 are selectable and mode M1 is the mode selected by the user, the initial display frequency value can be set as follows: modes M1:M2:M4=80:10:10. In addition, in steps S211 and S206, the corresponding mode is determined, and the display frequency value of that mode is increased. If none of the modes apply, the display frequency values ​​of multiple modes other than the mode selected by the user are increased equally.

[0111] In the first embodiment described above, it is also possible to limit the update of the display frequency value so that the display frequency value of all modes except one mode will never be "0." In this way, it is possible to ensure that multiple modes are always applied to the recommended area AR2 in a time-division manner, even if the frequency differs. This type of control is also possible in the second and third embodiments described below.

[0112] 4. Second Embodiment An example of processing by the control unit 11 in the second embodiment will be described with reference to Fig. 9. Note that the processing in Fig. 7 is performed in the same manner in the second embodiment. The second embodiment is an example in which the control unit 11 performs the processing in Fig. 7 and also performs the processing in Fig. 9 instead of Fig. 8. In Fig. 9, the same processing as in Fig. 8 is assigned the same step number to avoid redundant explanation.

[0113] The processing example in Fig. 9 differs from Fig. 8 in that the control unit 11 automatically selects a mode in step S200A. That is, in Fig. 9, the control unit 11 selects a mode that is estimated to be suitable for the user who is the distribution destination in step S200A, rather than a user operation.

[0114] For example, the control unit 11 automatically selects a mode that is estimated to be applied to the content of the current distribution by the user based on the user's past selection history, the history of modes applied to the main area AR1 in distributions to that user, the history of modes selected by the user for the same song or the like in the past, or the history of the display frequency values ​​of each mode in those past distributions. Alternatively, a mode that is selected by many users, not just the user, may be selected, or a mode that is often applied depending on the event or content. Furthermore, the control unit 11 may randomly select a mode without using the user's history or the selection status of other users.

[0115] The subsequent processes of steps S201, S202, S203, S204, S207, S208, and S220 are the same as those in FIG.

[0116] If the user performs an operation to select an image in camera area AR3, control unit 11 determines in step S209A whether or not the image is an image corresponding to the mode automatically selected in step S200A. As explained in step S209 of FIG. 8, "corresponding image" means that the selected image is an image displayed in the mode selected in step S200A. If the image corresponds to the automatically selected mode, the control unit 11 increases the display frequency value of the automatically selected mode in step S210A. On the other hand, if the image does not correspond to the automatically selected mode, the control unit 11 increases the display frequency value of the mode other than the automatically selected mode in step S211A.

[0117] Also, if no operation to specify the recommended area AR2 or the camera area AR3 is performed for a certain period of time or longer, the control unit 11 determines in step S205A whether the image in the main area AR1 at that time is an image corresponding to the mode automatically selected in step S200A. If the image corresponds to the automatically selected mode, the control unit 11 increases the display frequency value of the automatically selected mode in step S212A. On the other hand, if the image does not correspond to the automatically selected mode, the control unit 11 increases the display frequency value of the mode other than the automatically selected mode in step S206A.

[0118] As shown in Figure 9 above, there may be cases where the user does not initially select a mode. In that case, image switching in the recommended area AR2 is initially prioritized based on the automatically selected mode. Depending on the user's operation, a transition to a state in which another mode is prioritized may occur.

[0119] 5. Third Embodiment An example of processing by the control unit 11 in the third embodiment will be described with reference to Fig. 10. Note that the processing in Fig. 7 is performed in the same manner in the third embodiment. That is, the control unit 11 executes the processing in Fig. 7, and also performs the processing in Fig. 10 instead of Fig. 8. In Fig. 10, the same processing as in Fig. 8 is assigned the same step numbers to avoid redundant explanation.

[0120] The third embodiment of FIG. 10 is an example in which an initial mode selection such as that in step S200 of FIG. 8 or step S200A of FIG. 9 is not performed.

[0121] In step S201B, the control unit 11 sets the initial display frequency values ​​of the respective modes to be equal. For example, when modes M1 and M2 are used, the display frequency values ​​are set to 50:50. Therefore, in step S202, initially, control is performed so that image switching in mode M1 and image switching in mode M2 ​​are performed at approximately the same frequency in the recommended area AR2.

[0122] Thereafter, the control unit 11 performs the processes of steps S203, S204, S207, S208, and S220 in the same manner as in FIG.

[0123] If the user performs an operation to select an image in camera area AR3, control unit 11 proceeds from step S207 to step S230, where it determines whether the selected image corresponds to a certain mode at that time, i.e., whether the image is selected by a certain mode at that time. If a certain mode exists, control unit 11 increases the display frequency value of the certain mode at step S231.

[0124] Also, if no operation to specify the recommended area AR2 or the camera area AR3 is performed for a certain period of time or longer, the control unit 11 proceeds from step S204 to S232 and determines whether any mode is applied to the image in the main area AR1 at that time. If it is applied, the process proceeds to step S233, where the control unit increases the display frequency value of the mode applied to the main area AR1.

[0125] As described above, initially, each mode is applied approximately equally in the recommended area AR2 to switch images, but as the display frequency value is updated in response to user operation or a certain period of inactivity, the frequency with which the mode desired by the user is applied gradually increases.

[0126] <6.GUI example> An example of a GUI that can be applied to each of the embodiments described above is shown in FIG. 6, but other examples of the GUI will now be described.

[0127] 11 shows an example having a main area AR1, recommended areas AR2-1 and AR2-2, and a camera area AR3. By providing two recommended areas AR2-1 and AR2-2, images based on various modes can be displayed with different frequency values, allowing images based on a variety of modes to be presented to the user.

[0128] For example, suppose that modes M1, M2, M3, and M4 are used, and mode M1 is selected first in step S200 in FIG. 8 or step S200A in FIG. In this case, in step S201 of Figures 8 and 9, an example can be considered in which mode M1 is applied frequently and mode M4 is applied infrequently in recommended area AR2-1, and modes M2 and M3 are applied equally frequently in recommended area AR2-2. For example, when using modes M1, M2, M3, and M4, in the case of Figure 10, in step S201B, modes M1 and M2 may be applied with equal frequency in recommended area AR2-1, and modes M3 and M4 may be applied with equal frequency in recommended area AR2-2.

[0129] FIG. 12 is an example of a GUI in which a setting area AR4 is provided in addition to a main area AR1, a recommended area AR2, and a camera area AR3. The setting area AR4 displays the following settings that can be operated.

[0130] Mode selection For example, when the user is asked to select a mode in step S200 of FIG. 8, a GUI that the user can operate is displayed.

[0131] Screen layout settings For example, a GUI display is provided that allows the main area AR1 to be displayed larger, the camera area AR3 to be hidden and only the main area AR1 and recommended area AR2 to be displayed, and the layout of each area to be set / changed.

[0132] Audio settings: A GUI display is provided for easy operation, such as adjusting the volume or increasing or decreasing the volume from a specific microphone. A display for adjusting sound quality, such as an equalizer display, may also be provided.

[0133] - Display of time information For example, it will be possible to check the elapsed time from the start of a live music concert, a specific song, or the elapsed time from the start of listening.

[0134] FIG. 13A shows an example in which a check box 30 for fixing the display is provided in the recommendation area AR2. For example, when the user wants to continue viewing the content displayed in the recommended area AR2 in the main area AR1, the user checks the check box 30. Accordingly, the main area AR1 and the recommended area AR2 are made common. For example, as shown in Fig. 13B, the GUI screen is made up of a common area AR10 in which the main area AR1 and the recommended area AR2 are integrated into one area, and a camera area AR3. Alternatively, as shown in FIG. 13C, the main area AR1 and the recommended area AR2 are integrated into one area, the camera area AR3 is hidden, and the common area AR10 is displayed in full screen. In these FIGS. 13B and 13C, the check box 30 is displayed as checked, which indicates that the main area AR1 and the recommended area AR2 are common areas.

[0135] In the common area AR10 of FIGS. 13B and 13C, it is conceivable that an image display is performed in which the mode applied to the recommended area AR2 is fixedly applied at the time of check-on. That is, if the user finds the image related to the switching control in the recommended area AR2 to be preferable, the user can check the check box 30 to enable the image in that mode to be continuously viewed in the shared area AR10.

[0136] When such an area is shared, the control unit 11 performs the process shown in FIG. 14 instead of the process shown in FIG. Steps S100 to S106 and step S110 in Fig. 14 are the same as those in Fig. 7. In the process in Fig. 14, the control unit 11 monitors the operations in steps S102 and S104, and also monitors the recommended area fixing operation in step S150.

[0137] The recommended area fixing operation in step S150 is an operation for fixing the main area AR1 in the mode of the recommended area AR2, and is, for example, an operation for checking the check box 30 in the state of FIG. 13A. When this operation is detected, the control unit 11 proceeds to step S151 and performs control to display a shared area AR10 that combines the main area AR1 and the recommended area AR2, as shown in Figure 13B or 13C. In this case, the mode applied to the recommended area AR2 is applied to the shared area AR10, and this state continues thereafter. As a result, if the user performs a check-on operation when a desired image is displayed in the recommended area AR2, the image in the current mode can be viewed in the shared area AR10, which is a large area on the screen. When a check-on operation is performed, the mode that is preferentially applied in the recommended area AR2 at the time of the operation, that is, the mode with the highest display frequency value in the first, second, and third embodiments, may be applied to the shared area AR10, rather than the mode that was applied to the recommended area AR2 at the time of the operation. This allows the user to perform a check-on operation in the shared area AR10 to view content in the mode that is most desirable to the user, if the user often finds that the images in the recommended area AR2 up to that point suit his or her preferences.

[0138] In step S155, the control unit 11 determines the end timing in the same manner as in step S106. After starting to display the shared area AR10, the control unit 11 monitors the operation of releasing the fixation in step S152 until it determines to end the display in step S155. The operation of releasing the fixation is, for example, the operation of checking off the check box 30 in the state of FIG. 13C.

[0139] If the control unit 11 detects an operation to release the fixation, in step S153, the control unit 11 releases the commonality between the main area AR1 and the recommended area AR2, returning to the display state shown in Fig. 13A, for example. Then, the control unit 11 proceeds to the operation monitoring loop of steps S102 and S104. This returns the image in the main area AR1 to a state in which the user can change it. 14, if the camera area AR3 is left as is when the shared area AR10 is displayed as in FIG. 13B, the operation of the camera area AR3 may be monitored in addition to the monitoring of the unlocking operation in step S152. In the case of the operation of the camera area AR3, an image of the imaging device 1 corresponding to the operation may be displayed in the shared area AR10. Alternatively, the operation of the camera area AR3 may also proceed to step S153, where sharing is released, and an image of the imaging device 1 corresponding to the operation may be displayed in the main area AR1. Furthermore, when the shared area AR10 is displayed, the operation of the camera area AR3 may be invalid.

[0140] For example, with the above processing, the user can arbitrarily fix the mode of the main area AR1 to the mode applied to the recommended area AR2, and can view images in a wide area with the mode fixed.

[0141] Next, FIGS. 15A and 15B show an example in which one of the frames W of the camera area AR3 is selected as the image of the main area AR1. 15A, for example, in the main area AR1, an image is displayed in a state where switching control is performed in a certain mode. In each frame W of the camera area AR3, an image (image S3) captured by each of the imaging devices 1A...1(n) is displayed.

[0142] In this case, it is assumed that the user performs an operation to select the frame W1 in the camera area AR3. In this case, in the process of step S105 in FIG. 7, an image captured by the imaging device 1 corresponding to the selected frame W1 is displayed in the main area AR1 as shown in FIG. 15B.

[0143] In the above embodiment, it was stated that the image in the camera area AR3 would not be changed in this case, but in the example of Fig. 15B, the image content that had been displayed in the main area AR1 until then is displayed in the frame W1. In other words, this is an example in which the image content of the frame W1 and the main area AR1 are swapped.

[0144] In this way, when an image of the main area AR1 is to be displayed in the frame W1 within the camera area AR3, this is clearly indicated. The image in the main area AR1 before the replacement is an image that has been switched in a certain mode, and therefore the image captured by one imaging device 1 is not continuous. 15B, each frame of the camera area AR3 displays an image captured by a certain imaging device 1, but only frame W1 displays images captured by a plurality of imaging devices 1 that can be switched in a certain mode in a time-division manner. In other words, the image display has a different meaning from the other frames.

[0145] Therefore, the frame W1 is displayed in a manner different from the other frames W by, for example, making the frame display a specific color, making it a thick frame, or displaying it at high brightness, so that the user is made aware that it has been replaced with the main area AR1.

[0146] This makes it easy to see that only the frame W1 in the camera area AR3 is not an image taken by a specific imaging device 1 like the other frames W. If the user wishes to return to the original image in the main area AR1, he or she can simply select the frame W1 displayed in that particular display mode, thereby returning to the state shown in Figure 15A.

[0147] 7. Fourth Embodiment A fourth embodiment will be described below, taking the example of the GUI screen shown in FIG. In the fourth embodiment, the mode applied in the recommended area AR2 is switched by user operation, etc., but this is different from the first, second, and third embodiments described above, in which multiple modes are applied while the frequency is adjusted. As the switching control modes, modes M1, M2, M3, M4, and M5 are used, but initially, some of the modes, for example, modes M1, M2, and M3, are selectable.

[0148] In general, the process transitions as shown in the following first to fourth stages.

[0149] First step: Reference mode selection The user is prompted to select a preferred mode from modes M1, M2, and M3, which is set as the reference mode.

[0150] Phase 2: Applying standard mode and handling operations During distribution, the standard mode selected in the first stage is applied to the main area AR1, and image switching is performed. In the recommended area AR2, a mode that was not selected, for example, a mode with a lower mode number, is applied to the recommended area AR2, and image switching is performed. Furthermore, when the user operates the recommended area AR2, the modes applied to the main area AR1 and the recommended area AR2 are switched. When the user operates the camera area AR3, an image of the frame W selected in the camera area AR3 (an image captured by a certain imaging device 1) is displayed in the main area AR1.

[0151] Third stage: Judgment process When a certain determination timing occurs during distribution, a determination process is performed to determine the mode that is appropriate for the user based on the user's selection operations and viewing status. In this case, modes M4 and M5 are added to modes M1, M2, and M3, and the appropriate mode is determined from among them. Then, if necessary, the reference mode is updated depending on the determination result. The updated reference mode is applied to the main area AR1.

[0152] 4th stage: Re-determination process During distribution after the determination, if a certain determination timing occurs, the determination process is performed again and the reference mode is updated according to the determination result. The updated reference mode is applied to the main area AR1. The re-determination process may be performed multiple times thereafter until the process is completed.

[0153] For example, an example of the control process at each stage as described above will be explained with reference to FIG.

[0154] In step S300, the control unit 11 sets the reference mode in response to a user operation. For example, the control unit 11 controls the GUI screen on the user terminal 3 to prompt the user to select one of modes M1, M2, or M3, and detects the user's corresponding operation input. The mode selected by the user is then set as the reference mode. Setting the reference mode in this manner is the first stage of processing. Note that, instead of relying on user selection, a mode that is preferred by the user may be automatically selected and set as the reference mode, as described in the second embodiment, for example. In the following description, it is assumed that the user has selected mode M1 and initially set mode M1 as the reference mode.

[0155] Next, the second stage of processing is carried out from step S301 onwards. In step S301, the control unit 11 sets the display control for each area, and controls the image processing unit 12 so that a delivery image PO is generated in accordance with the settings. The main area AR1 is an image to which a reference mode is applied, for example mode M1. The recommended area AR2 is an image to which a mode other than the standard mode is applied. For example, a mode other than the standard mode with the smallest mode number is selected, for example, mode M2. As for the camera area AR3, an image captured by each imaging device 1 is displayed in each frame W, similarly to the above-described embodiment.

[0156] In step S302, the control unit 11 monitors the end timing, which is the same as step S110 in FIG. In step S303, the control unit 11 checks whether or not it is time to make a determination, which is the process of checking whether or not it is time to perform the above-mentioned third or fourth stage processing.

[0157] If none of these timings apply, the second stage of processing continues. In step S304, the control unit 11 checks whether the user has performed an image selection operation. The image selection operation here refers to the operation of specifying the recommended area AR2 or the operation of specifying a certain frame W in the camera area AR3, as in the above-described embodiment.

[0158] If an image selection operation is detected, the control unit 11 branches the process in step S305 depending on whether the operation is for the recommended area AR2 or the camera area AR3.

[0159] If an operation is performed on the recommended area AR2, the control unit 11 checks in step S306 whether or not the standard mode is currently being applied to the recommended area AR2. If the standard mode is applied to the main area AR1, the control unit 11 proceeds to step S307 and switches the modes applied to the main area AR1 and the recommended area AR2. For example, if mode M1 has been applied to the main area AR1 and mode M2 ​​to the recommended area AR2, the mode is switched to a state in which mode M2 ​​is applied to the main area AR1 and mode M1 is applied to the recommended area AR2.

[0160] This is because the user has performed an operation to view the image in the recommended area AR2, and the display that had been displayed in the recommended area AR2 up until then is now displayed in the main area AR1. On the other hand, the main area AR1 initially displays images in the standard mode, which is the mode selected by the user, so rather than hiding the images in the standard mode, the images will continue to be displayed in the recommended area AR2 in the standard mode.

[0161] As described above, since the mode application of the main area AR1 and the recommended area AR2 has been switched, when the standard mode is applied to the recommended area AR2 and the user performs an operation to specify the recommended area AR2 again, the control unit 11 will proceed from step S306 to step S308. In this case, the control unit 11 returns to the mode settings before the switching. For example, in step S308, the control unit 11 returns to a state in which the main area AR1 displays based on the reference mode (e.g., mode M1) and the recommendation area AR2 displays based on the mode (e.g., mode M2) that was applied before the switching in the previous step S307.

[0162] When an image selection operation by the user is detected and the operation is an operation to specify a certain frame W in the camera area AR3, the control unit 11 proceeds to step S309, displays the specified image in the main area AR1, and applies the standard mode to the recommended area AR2. For example, when an image (low-resolution image S3) based on input image PinB of image capture device 1B is selected in camera area AR3, an image (high-resolution image S1) based on input image PinB of image capture device 1B is displayed in main area AR1. Then, by applying standard mode M1 to recommended area AR2, the image switching that was previously performed in main area AR1 is performed in recommended area AR2.

[0163] This is particularly the case when the user wants to focus on an image captured by a particular imaging device 1, and the image captured by that particular imaging device 1 is displayed at high resolution in the main area AR1. On the other hand, the main area AR1 initially displays images in the standard mode, which is the mode selected by the user, so rather than hiding the images in the standard mode, the images will continue to be displayed in the recommended area AR2 in the standard mode.

[0164] It should be noted that before the operation to specify a certain frame W of the camera area AR3 is performed, the standard mode may have been applied to the recommended area AR2 in the process of step S307. For example, if the user selects an image based on input image PinB of imaging device 1C in camera area AR3 in this state, in step S309, an image based on input image PinC of imaging device 1C is displayed in main area AR1, while the recommended area AR2 is in the standard mode, so this mode can be continued. In this way, the user can display any image in the main area AR1 by operating the camera area AR3.

[0165] As described above, when an image captured by a certain imaging device 1 is displayed in the main area AR1, if the user performs an operation to designate the recommended area AR2 again, the control unit 11 will proceed from step S306 to step S308. In this case, the control unit 11 returns the main area AR1 and the recommended area AR2 to the mode settings before the image of the frame W in the main area AR1 and the camera area AR3 was displayed. For example, in step S308, the control unit 11 returns to a state in which the main area AR1 displays based on the standard mode (e.g., mode M1) and the recommended area AR2 displays based on the originally applied mode (e.g., mode M2).

[0166] As can be seen from the above process, the image in the main area AR1 is basically an image that is switched and controlled by the standard mode, and the image in the recommended area AR2 is an image that is switched and controlled by a mode other than the standard mode. By user operation, an image that can be switched depending on the mode initially applied to the recommended area AR2 is displayed in the main area AR1. Furthermore, when the user specifies an image in the camera area AR3, the image captured by the corresponding imaging device 1 is displayed in the main area AR1. However, when the main area AR1 is not displayed in the standard mode, the standard mode is applied to the recommended area AR2, and the user can perform an operation to specify the recommended area AR2 to return to the original state, i.e., a state in which the main area AR1 is displayed based on the standard mode and the recommended area AR2 is displayed based on another mode.

[0167] If no user operation is detected, the control unit 11 proceeds from step S304 to step S310 and checks whether the standard mode is applied to the recommended area AR2. If the standard mode is not applied to the recommended area AR2, the process returns to step S302.

[0168] If the standard mode is applied to the recommended area AR2, the control unit 11 proceeds to step S311 and checks whether or not this state has continued for a predetermined time period. If the time has not expired, the control unit 11 returns to step S302, but if the time has expired, the control unit 11 proceeds to step S312.

[0169] If the standard mode is applied to the recommended area AR2 for a predetermined period of time, it can be assumed that the user does not consider the image switching based on the standard mode appropriate. Therefore, in step S312, the control unit 11 changes the mode applied to the recommended area AR2 at this time and changes the "recommended" switching control method.

[0170] For example, if the reference mode up to this point has been mode M1 and mode M2 ​​has been applied to the initially recommended area AR2, then at the time of step S312, mode M2 ​​will be applied to the main area AR1, or an image captured by a certain imaging device 1 will be displayed, and mode M1, which is the reference mode, will be applied to the recommended area AR2. In this case, the control unit 11 changes the mode so that, for example, mode M3 is applied to the recommended area AR2. For example, the mode M2 ​​is applied to the main area AR1, and the mode M3 is applied to the recommended area AR2. Alternatively, for example, an image captured by a certain imaging device 1 may be displayed in the main area AR1, and mode M3 may be applied to the recommended area AR2.

[0171] By switching the mode applied to the recommendation area AR2 in this way, it becomes possible to present images in a variety of switching control modes to the user. Thereafter, for example, if the user performs an operation to specify the recommended area AR2, the process proceeds to step S307. For example, if mode M2 ​​is applied to the main area AR1 and mode M3 is applied to the recommended area AR2, the mode switching in step S307 results in mode M3 being applied to the main area AR1 and mode M2 ​​being applied to the recommended area AR2. Furthermore, even if, during operation, an image captured by a certain imaging device 1 is displayed in the main area AR1 and mode M3 is applied to the recommended area AR2, step S307 is a mode switching process, so the mode that was most recently applied before the image captured by a certain imaging device 1 was displayed in the main area AR1 can be switched to be applied to the recommended area AR2.

[0172] The above steps S310, S311, and S312 are performed to obtain a period during which various modes are applied to the recommended area AR2, but as a modified example, the determination in step S310 may be omitted. That is, even if, for example, mode M1 is applied to the main area AR1 and mode M2 ​​is applied to the recommended area AR2 in the initial state, when the time expires, the mode applied to the recommended area AR2 may be changed to, for example, mode M3 in step S312.

[0173] Also, if the time runs out while the standard mode is applied to the recommended area AR2, in step S312, the standard mode itself may be changed instead of simply changing the mode applied to the recommended area AR2.

[0174] This is the second stage of processing. In other words, the display of each area is performed in response to the user's operations and viewing status. While the second stage processing continues, the control unit 11 determines the timing of the determination in step S303 according to a certain condition, and performs the third stage processing.

[0175] Whether or not the timing for determination has come is determined based on the following conditions, for example. The timing for determination is when the number of user operations on the recommended area AR2 or the camera area AR3 reaches a predetermined number of times or more. The timing for determination is when the operation frequency per unit time of the user's operation on the camera area AR3 or the camera area AR3 becomes equal to or greater than a predetermined value. The timing for judgment is when a predetermined time has passed since the start of distribution. The timing for determination is when the time during which the reference mode is not applied to the main area AR1 exceeds a predetermined time, for example, a predetermined time longer than the time-out in step S311.

[0176] In step S303, the control unit 11 checks whether or not it is time to make a decision based on one of these conditions, or a combination of two or more of these OR conditions, or a combination of two or more of these AND conditions. When the timing for determination arrives, the control unit 11 proceeds to step S320 to determine the appropriate mode.

[0177] The suitable mode determination is to determine which of the modes M1, M2, M3, and M4 the user's viewing tendency is closest to. If it cannot be determined that the viewing tendency is closest to any of the modes, the random mode M5 is used.

[0178] The determination of which of the modes M1, M2, M3, and M4 the user's viewing tendency is closest to is made taking into account the following factors. The length of time that the mode is applied to the main area AR1 is reflected in the suitability. If there is a mode to select from the image selected by the user from the camera area AR3 at the time of the user's selection, the mode is deemed to be highly suitable. If the user does not perform any operation for a certain period of time after performing an operation, the mode selected by the user through an operation or the mode that selects the image selected by the user is deemed to have high compatibility. - The suitability is determined based on the tendency of the content of the images selected by the user. For example, if a specific subject is selected many times, it is deemed to be highly suitable for mode M4.

[0179] For example, after determining the mode suited to the user based on the above-mentioned determination factors, the control unit 11 sets the reference mode in accordance with the determination in step S321. If the mode determined to be suitable is the same as the reference mode initially set in step S300, the reference mode setting is maintained. If the mode determined to be suitable is different from the reference mode initially set in step S300, the setting is updated so that the mode determined to be suitable becomes the reference mode.

[0180] Then, in step S301, the image processing unit 12 performs setting control of the image to be delivered to be output. The reference mode is applied to the main area AR1. In the recommended area AR2, another mode is applied. In this case, the mode with the second highest compatibility as a result of the determination may be applied to the recommended area AR2.

[0181] Therefore, from then on, in the main area AR1, one of the modes M1, M2, M3, M4, and M5 is applied to display an image. In the recommended area AR2, images are displayed in a mode different from that in the main area AR1. There are no changes to the camera area AR3.

[0182] Thereafter, the processes from step S302 onwards are carried out in the same manner, and images are displayed in each area according to the user's operation, viewing state (no operation), etc.

[0183] If the control unit 11 determines again in step S303 that it is time to make a determination, the process of steps S320 and S321 as the fourth stage, that is, a re-determination is performed. The determination timing for the fourth stage in step S303 may be determined under the same conditions as those for the above-described third stage, or may be determined under different conditions. The elements for determining the suitable mode in step S320 may be the same as the elements for determining the suitable mode in the third stage described above, or may be different elements. The reference mode may also be updated by re-determination.

[0184] When the timing for ending the process reaches a certain point in time during the above process, the control unit 11 proceeds from step S302 to step S330, performs an ending process, and ends the series of processes. In this case, in the termination process, it is conceivable that the result information of the suitable mode determination for the user is stored in the DB 11b in association with the user. In this case, when the same music piece or the like is distributed to the user, the reference mode setting in step S300 may be performed according to the result of the previous suitable mode determination.

[0185] Although the fourth embodiment has also been described using the GUI example in FIG. 6, GUI examples such as those in FIGS. 11, 12, 13, and 15 can also be applied. Initially, three modes, M1, M2, and M3, are used, but if four or more modes are used by the second stage, images in some of the modes other than the standard mode are displayed in the recommended areas AR2-1 and AR2-2, as shown in Figure 11. Furthermore, as explained in Figures 13 and 14, if the mode in the recommended area AR2 is fixedly applied to the main area AR1 and the areas are made common, it is possible to make the suitability of the fixed mode highly evaluated in the suitable mode determination in step S320 of Figure 16.

[0186] 8. Fifth Embodiment A processing example of the fifth embodiment will be described with reference to Fig. 17. In this processing example, the reference mode is not set by user selection or the like as in the example of Fig. 16 above. As the switching control modes, modes M1, M2, M3, M4, and M5 are used, but initially, some of the modes, for example, modes M1, M2, and M3, are selectable.

[0187] In general terms, the process transitions between the first and second stages below. Phase 1: Support for operations after distribution begins During distribution, a certain image, for example, the entire image, is displayed in the main area AR1, and a certain mode, for example, a mode with a lower mode number, is applied to the recommendation area AR2, and image switching is performed. Furthermore, when the user operates the recommended area AR2, the modes applied to the main area AR1 and the recommended area AR2 are switched. When the user operates the camera area AR3, an image of the frame W selected in the camera area AR3 (an image captured by a certain imaging device 1) is displayed in the main area AR1.

[0188] 2nd stage: Judgment process and mode setting When a certain determination timing occurs during distribution, a determination process is performed to determine the mode that is appropriate for the user based on the user's selection operation and viewing situation. In this case, modes M1, M2, M3 as well as modes M4 and M5 are added, and the appropriate mode is determined from among them. Then, the mode to be applied to the main area AR1 and the recommended area AR2 is set according to the determination result. The determination process may be performed again.

[0189] For example, the control process at each stage as described above is executed according to FIG. First, in the first stage of processing, in step S401, the control unit 11 sets the output image for each area and controls the image processing unit 12 to start outputting the distribution image PO. In this case, the main area AR1 initially displays the entire image. For example, the image captured by the imaging device 1 capturing an overhead image of the entire stage is set as the image in the main area AR1. For the recommended area AR2, one of the modes is automatically selected. For example, mode M1 with the smallest mode number is selected. In the camera area AR3, images captured by the individual imaging devices 1 are displayed.

[0190] In step S402, the control unit 11 monitors the end timing, which is the same as step S302 in FIG. In step S403, the control unit 11 checks whether or not it is time to make a determination, which is the process of checking whether or not it is time to perform the second-stage determination process in the fifth embodiment.

[0191] If none of these timings apply, the first stage of processing continues. In step S404, the control unit 11 checks whether the user has performed an image selection operation. The image selection operation is an operation to specify the recommended area AR2 or an operation to specify a certain frame W in the camera area AR3.

[0192] If an image selection operation is detected, the control unit 11 branches the process in step S405 depending on whether the operation is for the recommended area AR2 or the camera area AR3.

[0193] If an operation is performed on the recommended area AR2, the control unit 11 applies the mode currently applied to the recommended area AR2 to the main area AR1 in step S406, and also applies another mode to the recommended area AR2. For example, if the entire image has been displayed in the main area AR1 and mode M1 has been applied to the recommended area AR2, the display is switched to a state in which mode M1 is applied to the main area AR1 and mode M2 ​​is applied to the recommended area AR2. This allows the user to change the main area AR1 from a fixed display state of the entire image to a display state in which image switching control is performed according to a predetermined algorithm.

[0194] Furthermore, after the state in which, for example, mode M1 is applied to the main area AR1 and mode M2 ​​is applied to the recommended area AR2 as described above, if the user performs an operation to specify the recommended area AR2 again, the control unit 11 switches to a state in which mode M2 ​​is applied to the main area AR1 and, for example, mode M3 is applied to the recommended area AR2 in step S406.

[0195] Further, if the user subsequently performs an operation to specify the recommended area AR2, the control unit 11 proceeds to step S 4 In step 06, the mode M3 is applied to the main area AR1, and the mode M1, for example, is applied to the recommended area AR2. In this way, the application modes of the main area AR1 and the recommended area AR2 are changed sequentially. For example, the user can change the application of the modes M1, M2, and M3 each time he operates the recommended area AR2.

[0196] If an image selection operation by the user is detected and the operation is an operation to specify a frame W in the camera area AR3, the control unit 11 proceeds to step S407 and displays the specified image in the main area AR1.

[0197] At this time, the display state of the main area AR1 may be displayed in the camera area AR3 in a specified frame W. For example, this is an example in which the image is switched from the state of Fig. 15A to the state of Fig. 15B.

[0198] Furthermore, when an image captured by a certain imaging device 1 is displayed in the main area AR1 as described above, if the user performs an operation to again specify the recommended area AR2, the control unit 11 applies the mode applied to the recommended area AR2 to the main area AR1 in step S406, and applies another mode to the recommended area AR2.

[0199] If no user operation is detected in step S404, the control unit 11 proceeds to step S408 to check whether a certain mode has been continuously applied to the recommended area AR2 for a predetermined time or more. If a certain mode has not been continuously applied to the recommended area AR2 for a predetermined time or more, the control unit 11 returns to step S402.

[0200] If a certain mode has been applied to the recommended area AR2 continuously for a predetermined time or longer, at which time the timeout occurs, the control unit 11 proceeds to step S409 and changes the mode to be applied to the recommended area AR2. In other words, the mode applied to the recommended area AR2 can be switched even if no operation is performed for a certain period of time. By switching the mode applied to the recommended area AR2 in this way, it is possible to present images in a variety of switching control modes to the user.

[0201] While the first stage processing continues as described above, the control unit 11 determines that it is time to make a determination in step S403 due to a certain condition, and performs the second stage processing. The determination timing is determined based on the conditions described in the fourth embodiment, for example. When the timing for determination arrives, the control unit 11 proceeds to step S420 and performs a suitable mode determination. Similar to step S320 in Fig. 16, the suitable mode determination determines which of modes M1, M2, M3, and M4 the user's viewing tendency is closest to. If it cannot be determined that the viewing tendency is closest to any of the modes, the random mode, mode M5, is selected.

[0202] After determining the mode suited to the user, the control unit 11 sets the mode to be applied to the main area AR1 and the recommended area AR2 in accordance with the determination in step S421. Thereafter, in the main area AR1, one of modes M1, M2, M3, M4, and M5 is applied to display an image. In the recommended area AR2, images are displayed in a mode different from that in the main area AR1. There are no changes to the camera area AR3.

[0203] Thereafter, the processes from step S402 onwards are carried out in the same manner, and images are displayed in each area according to the user's operation, viewing state (no operation), etc. It may also happen that step S403 determines that it is time for a further determination, and the determination process of step S420 and the mode setting of step S421 are performed again.

[0204] When the end timing is reached at a certain point in the process of the above processing, the control unit 11 executes step S 4 The process proceeds from 02 to step S330, where an end process is performed to complete the series of processes.

[0205] <9. Examples of applicable processing> An example of applicable processing will be described. This is an example of processing that can be additionally adopted in the example of processing in which some switching control mode is applied to the main area AR1, as in the first to fifth embodiments.

[0206] When the application to the main area AR1 is changed as in each embodiment, or when an image captured by a specific imaging device 1 is displayed, the elapsed time of the music being distributed, the selected mode, and the image are continuously recorded as viewing tendency information RC for each user.

[0207] For example, the following listening tendency information RC1, RC2, . . . RC(x) is recorded as listening tendency information RC for a song N1 of a certain user U1.

[0208] Listening trend information RC1: 0 minutes 00 seconds: Start of listening to song Since listening does not necessarily start from the beginning of a song, it is desirable to record a time code within the entire live performance or song so that it is clear at what time point in the entire live performance or song this "song listening start" occurs. For example, the live image can be recognized, and the fact that user U1 started listening from a specific scene (specific shape) can be recorded along with that scene. Alternatively, the live image can be recorded as metadata for user U1.

[0209] Viewing trend information RC2: X minutes XX seconds: Select to apply mode M2 Viewing trend information RC3: Y minutes YY seconds: Select input image PinC (image from imaging device 1C) Viewing trend information RC4: Z minutes ZZ seconds: Select to apply mode M1 … · Listening trend information RC(x): A minutes AA seconds: End of song listening

[0210] As described above, by recording the viewing tendency information RC for a combination of, for example, a certain song N1 and a user U1, the control unit 11 can perform control based on the viewing tendency information RC the next time that the user U1 watches a broadcast of the same song N1.

[0211] For example, it is possible to change the mode applied to the main area AR1 at a timing according to the viewing tendency information RC, and to switch the displayed image to an image captured by a specific imaging device 1. In other words, this is a control that reproduces the previous "viewing style" of the user. In this case, the timing at which the previous viewing tendency information RC was acquired is in accordance with the user's operation, and considering that this may be slightly later than the timing the user truly desires, it is conceivable to make the switching take place at a timing slightly earlier than the time information in the viewing tendency information RC, thereby enabling control that is more satisfying to the user.

[0212] On the other hand, even if control is performed to reproduce previous viewing patterns, the user may not be satisfied. For example, when the viewing trend information RC was acquired, the user may not have decided how they wanted to view the content, and may have been trying out various operations. Reproducing the viewing pattern in such a case may feel unnatural to the user. Therefore, even when reproduction control is performed, it is expected that the user will perform an operation to select the recommended area AR2 or the camera area AR3.

[0213] Therefore, when performing reproduction control, the control unit 11 displays in the recommended area AR2 an image based on a frequently applied mode determined from the viewing tendency information RC, or an image captured by a specific imaging device 1 with a high display frequency. This makes it easier for the user to select the recommended area AR2 and be provided with a viewing state similar to the previous viewing state.

[0214] Alternatively, control based on learning may be performed instead of reproduction control. The viewing tendency information RC can determine the type of image desired by the user. Therefore, the viewing tendency information RC may be used to perform the suitable mode determination process as described in the fourth and fifth embodiments, and the mode determined to be suitable for the main area AR1 may be applied at the start of distribution at the next distribution opportunity.

[0215] Furthermore, control based on the viewing tendency information RC can also be performed when the user U1 views another live broadcast of the same song N1. For example, if user U1 is a fan of a certain artist, it is expected that he or she will watch live broadcasts of the same song N1 many times at various times and places. Therefore, if viewing tendency information RC for user U1 and song N1 is acquired during distribution of a certain live event LV1, it is conceivable that control unit 11 will perform control using the viewing tendency information RC when distributing song N1 to user U1 at another live event LV2. For example, the control unit 11 may perform the reproduction control described above using the viewing tendency information RC, or may perform mode application control according to the result of a suitable mode determination based on the viewing tendency information RC.

[0216] However, in this case, the relationship between the image content and the imaging device 1 is not the same between the live event LV1 and the live event LV2.

[0217] For example, suppose that live event LV1 uses imaging devices 1A, 1B, . . . 1(n) as shown in Fig. 2 and arranges them on stage. At this time, suppose that imaging device 1A is arranged to capture an image of a keyboard player, imaging device 1B is arranged to capture an image of the keyboard player's hands, imaging device 1C is arranged to capture an image of a drummer, and so on.

[0218] On the other hand, at live event LV2, imaging devices 100A, 100B, etc. 100(m) were used, with imaging device 100A positioned to capture the drummer, imaging device 100B positioned to capture the keyboard player, etc.

[0219] In this way, it is generally assumed that the number and setting of the imaging devices 1 will differ. Therefore, even if viewing tendency information RC obtained when images captured by imaging devices 1A, 1B,..., 1(n) are used as input images PinA...Pin(n), if images captured by imaging devices 100A, 100B..., 100(m) are used as input images PinA, PinB...Pin(m), the same viewing experience cannot be reproduced and the suitable mode determination result cannot be reflected.

[0220] Therefore, the image capture devices 1 and 100 used in the live events LV1 and LV2 are associated as shown in FIG. For example, the control unit 11 compares the images captured by the image capture devices 1 and 100 at the live events LV1 and LV2, and performs processing to associate the image capture devices that are considered to have the most similar roles. For example, image analysis is performed to determine the subjects and determine the correspondence relationship of the most similar roles.

[0221] The process of determining the "closest role" uses the following factors: -Is the subject the same person, the equipment in the same category (e.g. electric guitar), or the same equipment? ·Is the number of subjects equal or the same? Is the subject facing the same direction? -Are the subjects roughly the same size or are they in a consistent size ratio?

[0222] Information about the person (artist, etc.) in the photo and the musical instrument lineup can also be used to make an individual or equipment determination by referring to information about performers at a live event, etc.

[0223] Based on the result of such a determination, information for associating the image capture devices 1 and 100 with the most similar roles is generated as shown in FIG. 18 and stored in the DB 11b.

[0224] For example, when delivering archive images of both live events LV1 and LV2, the above-described association process can be performed in advance at a timing unrelated to the user's viewing. Furthermore, if live event LV1 is an archive and live event LV2 is currently being performed and distributed in real time, the control unit 11 can perform the association by determining the image content of each imaging device 100 while live event LV2 is in progress. Once the imaging devices 1 and 100 have been associated, control can be performed using the viewing tendency information RC. Until the association is complete, it is possible to present an image of the entire stage in the main area AR1.

[0225] <10. Summary and Variations> According to the above embodiment, the following effects can be obtained. The image processing device 10 according to the embodiment includes an image processing unit that generates a delivery image based on a plurality of input images Pin, which are images of an event captured by a plurality of imaging devices 1, in which a main area AR1 (first area) serving as a primary image area and a recommended area AR2 (second area) serving as an image area separate from the main area AR1 are formed within the screen. Here, the main area AR1 serving as the primary image area can be considered, for example, as the area having the largest surface area within the screen. Alternatively, the main area AR1 serving as the primary image area can be considered, for example, as the area in which the highest-resolution image is displayed within the screen. Therefore, for example, the image processing unit 12 generates a delivery image PO based on the plurality of input images Pin, the delivery image PO including an image of the main area AR1 (first area) within the screen and an image of the recommended area AR2 (second area), an image area smaller than the main area AR1. Alternatively, for example, the image processing unit 12 generates a delivery image PO based on the plurality of input images Pin, the delivery image PO including an image of the main area AR1, which has the highest resolution within the screen, and an image of the recommended area AR2, which has a lower resolution than the image of the main area AR1. The image processing device 10 is also capable of controlling the switching of the input image Pin to be included in the distribution image PO using a plurality of switching control modes (modes M1, M2, etc.), and is equipped with a control unit 11 that controls the image processing unit 12 so that the captured image displayed in the recommended area AR2 is switched based on one of the plurality of switching control modes.

[0226] This makes it possible to apply a plurality of switching control modes to the image of the recommended area AR2. For example, in the streaming of a live music performance, each user will have different viewing preferences. Considering the following, even in the streaming of a single live performance, there will be people who want to see images that sync with the progress of the music performed by the performers, people who are interested in the techniques of playing musical instruments, people who are interested in the musical instruments and other equipment themselves, and fans of individual artists, each user will want to see different scenes. Therefore, it is conceivable that users will be able to select from images captured by a number of imaging devices 1 (input images Pin) when viewing. However, it is more desirable to automatically switch between images according to the desires of each viewer, rather than requiring the user to select all of them. Therefore, in this embodiment, multiple switching control modes are provided, and the image in the recommended area AR2 can be switched by control based on one of the switching control modes. If the switching of images in the recommended area AR2 is performed to match the scene that the user wants to see, the switching control mode applied to the recommended area AR2 will be suitable for the user.

[0227] In the embodiment, an example is given in which the control unit 11 controls the image processing unit 12 so that, in response to detecting a selection operation of the recommended area AR2 by the destination user, image switching based on the switching mode applied to the recommended area AR2 is applied to the image of the main area AR1 (see steps S102, S103, etc. in Figure 7). That is, in response to a user's selection operation on the recommended area AR2, the image switching control mode of the recommended area AR2 is applied to the high-resolution main area AR1 for the image that the user views. Therefore, if the user finds the image change in the recommended area AR2 to be preferable, the user can select the recommended area AR2 and view the image that was previously displayed in the recommended area AR2 in high resolution in the main area AR1. In other words, by responding to the recommendation in the recommended area AR2, the user can easily enjoy the delivered image change that best suits their preferences.

[0228] For example, a user at the distribution destination can view the entire stage in high resolution at the beginning of a song, and then at a certain timing can view in high resolution an image of a specific performer or a specific part of that performer captured by the imaging device 1. The user can also request that the image they want be distributed by performing operations such as touching the screen or pinching in / out. This makes it possible to learn about various aspects of the performer. By installing a large number of imaging devices 1, users can obtain more information than they would by attending a real live performance (by watching a live show at a concert venue). Also, even at real live performances, depending on the seat, it may be difficult to see the performers or the instruments being used, so by combining this embodiment of streaming at live venues, the added value of real live performances can be increased.

[0229] Note that the operation of selecting the recommended area AR2 changes the display of the main area AR1, but the user may accidentally touch it and change the display of the main area AR1. Therefore, the operation of specifying the recommended area AR2 may be performed by long pressing or touching twice (double tapping). The same applies to the operation of specifying the frame W in the camera area AR3.

[0230] In the embodiment, the image processing unit 12 generates a delivery image PO in which a camera area AR3 (third area) is formed on the screen, in addition to a main area AR1 and a recommended area AR2, and in which all or some of a plurality of input images Pin are individually displayed. Then, in response to detecting a selection operation of an input image Pin in the camera area AR3 by a delivery destination user, the control unit 11 controls the image processing unit 12 to display the input image Pin related to the selection operation in the main area AR1 (see steps S104, S105, etc. in FIG. 7). By providing the camera area AR3, the user can recognize the input images Pin captured by each imaging device 1 and select a desired image. For example, when the image in the recommended area AR2 is not a scene that the user wants to see, the user can select a captured image from the camera area AR3 and display it in the main area AR1, allowing the user to view the desired image with a high degree of freedom.

[0231] Furthermore, such selection operations of the camera area AR3 become learning data for later selection of an appropriate switching control mode for the user. By being able to determine the image content of the input image Pin that is easy for the user to select by accumulating operations of the camera area AR3, it is possible to improve the accuracy of selection of the switching control mode to be applied to the recommended area AR2 and the main area AR1, for example, the accuracy of the suitable mode determination described in Figures 16 and 17. It is also possible to consider an example in which the camera area AR3 as the third area is not provided.

[0232] In the embodiment, one of the plurality of switching control modes is mode M1 in which image switching control is performed based on progress information about an event that is obtained in advance. The above-mentioned mode M1 is a mode in which image switching control is performed based on progression information such as musical scores. Events to which the system of this embodiment can be applied include live music performances, plays, operas, ballets, dances, etc., where the progression is somewhat predetermined. In other words, these are events (including parts of events) for which progression information such as musical scores for each song, live set lists, scripts, and scenarios can be obtained.

[0233] For example, in the case of a live music performance, if there is sheet music information as progression information for a certain song, it is possible to know in advance during the performance of the song the periods in which the vocals are the main focus, the periods in which the lead instruments are the main focus, the periods in which the drums are the highlight, the periods in which you can focus on the backing, etc. Based on this progression information, in mode M1, it is possible to perform image switching control so that the distributed image PO of the song performance successively switches to an image of the main performer during the song.

[0234] The progression information may be the data itself, such as the musical score or scenario, or may be information generated by analyzing the musical score, etc., such as information that specifies the timing of image switching over time and the designated values ​​of the images to be switched.

[0235] In the embodiment, one of the plurality of switching control modes is mode M2 ​​in which image switching control is performed for a distribution image for a specific distribution destination user based on viewing information of distribution destination users other than the specific distribution destination user. The above-mentioned mode M2 ​​is a mode in which image switching control is performed based on the viewing information of other distribution destination users. For example, it is a mode in which control is performed to sequentially switch to the image that is being viewed by the most people (the image that has been viewed) among many other users. For users who want to enjoy images that attract the attention of many other people, such mode M2 ​​control is suitable.

[0236] The viewing information may be the number of selection operations to instruct the display of each input image Pin, the number of people actually viewing each input image Pin, etc. Even for a distributed image that is only an image and has no audio, the viewing information here may be considered to be information corresponding to the number of people viewing it.

[0237] In the embodiment, one of the multiple switching control modes is mode M3, which performs image switching control for a distribution image for a specific distribution user based on viewing information of other distribution destination users who have attributes that are the same as or similar to the attributes of the specific distribution destination user. The above-mentioned mode M3 is a mode that performs image switching control based on viewing information of other distribution destination users who have the same or similar attributes. For example, this mode is a control mode that sequentially switches to images that are viewed by many people (images that have been viewed) among other users who are estimated to have similar tastes to the user. For users who want to enjoy images that attract the attention of people with similar tastes, such mode M3 control is suitable.

[0238] In the embodiment, one of the plurality of switching control modes is mode M4, which performs image switching control for a plurality of input images relating to a specific subject. The above-mentioned mode M4 is a mode that switches between images of a certain subject, for example, a particular artist, so that the viewer can focus on that subject. This mode M4 control is suitable for users who are fans of a particular artist or who want to focus on their performance.

[0239] In the first and second embodiments, an example has been given in which the control unit 11 performs control so that the input image Pin that is switched by processing in the switching control mode selected corresponding to the destination user is displayed in the recommended area AR2 more frequently than the input image Pin that is switched by processing in other switching control modes. The switching control mode applied to the recommended area AR2 is the mode selected in step S200 of Fig. 8 or step S200A of Fig. 9 with priority. However, other modes may also be applied. At this time, the selected mode is applied with priority, and the input image Pin in the selected mode is displayed more frequently than images in other modes. This allows the image in the recommended area AR2 to be based on the selected mode as much as possible, while occasionally presenting the user with input images Pin in other modes, giving the user an opportunity to make a selection. It is conceivable that the user may prefer image switching in other modes, so it may be possible to present the user with a more desirable image switching.

[0240] Furthermore, by selecting a switching control mode corresponding to a specific distribution destination user, that is, by providing images of recommended areas corresponding to each individual user, the image content can be more likely to be suitable for each user.

[0241] Furthermore, switching between multiple modes in the recommendation area AR2 means that there is no need to prepare a display area for each mode, which has the advantage of not needing to unnecessarily increase the display area according to the mode.

[0242] In the first embodiment, the switching control mode selected corresponding to the destination user is the switching control mode designated by the selection operation of the destination user (see step S200 in FIG. 8). As for the switching control mode to be applied to the recommended area AR2, the mode selected by the user operation is applied first. For example, if mode M1 is selected, the frequency ratio of mode M1 to mode M2 ​​is set to 90:10. This allows the user to be provided with images that can be switched to other modes from time to time while respecting the user's wishes, thereby making the user aware of images in other modes.

[0243] In the second embodiment, the switching control mode selected corresponding to the destination user is the switching control mode designated in the automatic selection process (see step S200A in FIG. 9). The switching control mode to be preferentially applied to the recommended area AR2 may be automatically selected by the control unit 11. For example, the selection may be made according to the user's attributes, or based on learning results using the past selection status and image switching operation history. Alternatively, a mode that is frequently applied to other recipient users may be selected. By automatically selecting a mode that is deemed suitable for the user, the images in the recommended area AR2 are more likely to be desirable for the user. Furthermore, by applying other modes at a lower frequency, the user can be made aware of images in other modes.

[0244] In the first, second and third embodiments, an example has been given in which the control unit 11 variably sets, in the recommended area AR2, the display frequency values ​​of the input images Pin that are switched by processing in the switching control mode selected corresponding to the destination user and the display frequency values ​​of the input images Pin that are switched by processing in other switching control modes in accordance with the operation of the destination user, and controls the input images Pin based on each switching control mode to be displayed at a frequency according to the display frequency values ​​(see FIGS. 8, 9 and 10). By varying the display frequency value for the switching control mode applied to the recommendation area AR2 in response to a user operation, it is possible to increase the display frequency of images in a mode that the user considers desirable. Therefore, when a mode is selected first as in the first and second embodiments, if another mode is more suitable for the user, the frequency with which the other mode is applied increases. Furthermore, even if a mode is not selected in advance as in the third embodiment, the frequency of images in a mode preferred by the user gradually increases. Therefore, an image that is desirable to the user is more likely to be displayed.

[0245] In the fourth embodiment, an example was given in which the control unit 11 performs control such that an input image Pin that can be switched by processing in a reference mode (reference switching control mode) selected corresponding to the destination user is displayed in the main area AR1, and an input image Pin that can be switched by processing in a switching control mode other than the reference mode is displayed in the recommended area AR2 (see Figure 16). In the main area AR1, a reference mode selected for the user, for example, by a user selection operation or automatic selection based on learning, is applied, thereby increasing the likelihood that an image desired by the user will be switched in the main area AR1. Then, an image display in which another mode is applied can be presented to the user in the recommendation area AR2.

[0246] In the fourth embodiment, the control unit 11 performs control such that, in response to a selection operation on the recommended area AR2, the input image Pin that can be switched by the processing of the switching control mode applied to the recommended area AR2 is displayed in the main area AR1, and the input image Pin that can be switched by the processing of the standard mode is displayed in the recommended area AR2 (see step S307 in FIG. 16). Also, an example has been described in which the control unit 11 performs control such that, in response to the lapse of a predetermined time, an input image Pin that can be switched by the processing of another switching control mode is displayed in the recommended area AR2 (see step S312 in FIG. 16). If the user wishes, the mode applied to the recommended area AR2 can be applied to the main area AR1 by operation. In other words, if the user finds the image content that can be switched in the recommended area AR2 to be suitable, they can view it in the main area AR1. Furthermore, if the standard mode remains applied to the recommended area AR2 for a predetermined period of time, it can be assumed that the user has little interest in the images in that standard mode. Therefore, by displaying another mode in the recommended area AR2, it is possible to present another switching control mode.

[0247] In the fourth embodiment, an example has been given in which the control unit 11 performs a determination process of which of a plurality of switching control modes is suitable for the user based on the history of the user's selection operations, and updates the reference mode according to the determination result (see steps S320, S321, and S301 in FIG. 16). By using the user's operation history for the recommended area AR2 and the camera area AR3, it is possible to determine which mode is suitable for the user. Then, by performing a determination process and updating the reference mode, it is possible to set the reference mode suitable for the user.

[0248] In the processing example of FIG. 16, the user first selects a preferred mode from modes M1, M2, and M3, which is set as the reference mode, but the determination process determines the appropriate mode from modes M1, M2, M3, M4, and M5. In other words, the determination process provides a greater variety of modes than when the user selects. Of course, the appropriate mode determination may be performed using five or more different modes. This prevents the user from being overwhelmed by the selection operation, and ultimately provides the user with a more appropriate mode through the determination process.

[0249] In the fifth embodiment, an example was given in which the control unit 11 performs control such that, as an initial state, an input image Pin from one imaging device 1 is fixedly displayed in the main area AR1, and an input image Pin that can be switched by processing in a predetermined switching control mode is displayed in the recommended area AR2 (see step S401 in Figure 17). In the initial state, a specific input image Pin, such as an overall image of the stage, is fixedly displayed in the main area AR1. Initially, the main area AR1 displays an input image Pin captured by a specific imaging device 1, such as an overall image, and no image switching is performed. On the other hand, an image in which a certain switching control mode is applied is displayed in the recommended area AR2, and if the user likes the image, they can select the recommended area AR2 to view an image based on that switching control mode in the main area AR1. This allows the user to select a display method for switching the input image Pin according to their preference. This type of processing can also be applied to the first, second, and third embodiments.

[0250] In this embodiment, the image processing unit 12 performs image size conversion processing (image size processing 15A, 15B, . . . 15(n)) on a plurality of input images Pin to convert the data size of the input images Pin for use in display in the main area AR1 and the recommended area AR2. Then, the delivery image generation processing 16 generates a delivery image PO including the converted input images Pin (i.e., images S1, S2) for each individual delivery destination user (see FIG. 3). The image processing unit 12 generates an image S1 sized for the main area AR1 and an image S2 sized for the recommended area AR2 for each input image Pin, selects and combines the input images Pin to be displayed at each time, and generates a distribution image PO for one channel for one user. Such a distribution image PO is generated for each distribution destination user, that is, for multiple channels. This makes it possible to generate distribution images PO with content tailored to individual users with various preferences. Furthermore, by using the combined image data as the distribution image PO, each distribution image PO does not require multiple images to be distributed in parallel, so the amount of communication data is not excessive, and no particular processing burden is imposed on the user terminal 3.

[0251] In addition, the image processing device 10 may deliver the image S1 as high-resolution image data and the image S2 as relatively low-resolution data without combining them, and display them separately in the main area AR1 and the recommended area AR2 on the user terminal 3.

[0252] It is also conceivable that the image processing unit 12 constantly generates distribution images PO for a plurality of channels assuming combinations of each area, and selects a channel according to the user to distribute the images. For example, for input images PinA...Pin(n), all composite images to be displayed in the main area AR1 and the recommended area AR2 are generated for each channel. Then, depending on the operation and application mode of each user, the image PO to be delivered for the corresponding channel is selected and delivered to the user.

[0253] In this embodiment, the image processing unit 12 further performs a data size conversion process (image size processes 15A, 15B, . . . 15(n)) of the input images Pin so that the data size is smaller than that of the images S2 in the recommended area AR2, in order to use the input images Pin for display in the camera area AR3 where all or some of the input images Pin are individually displayed. Then, in the delivery image generation process 16, a delivery image PO including the converted input images Pin (i.e., images S1, S2, S3) is generated for each individual delivery destination user (see FIG. 3). For each input image Pin, the image processing unit 12 generates an image S1 sized for the main area AR1, an image S2 sized for the recommended area AR2, and an image S3 sized for the camera area AR3, selects and combines the input images Pin to be displayed at each time, and generates a distribution image PO for one channel for one user. This makes it possible to generate distribution images PO that correspond to individual users with various preferences and that allow selection of input images Pin from each imaging device 1. Furthermore, by using the combined image data as the distribution image PO, each distribution image PO does not involve the distribution of multiple images in parallel, so the amount of communication data is not excessive, and no particular processing burden is imposed on the user terminal 3.

[0254] Even when a camera area AR3 is assumed, the image processing device 10 may deliver image S1 as high-resolution image data, image S2 as relatively low-resolution data, and image S3 as even lower-resolution data without combining them, and display them on the user terminal 3 as a main area AR1, a recommended area AR2, and a camera area AR3.

[0255] In the embodiment, an example is given in which the control unit 11 controls the image processing unit 12 to eliminate the screen division between the main area AR1 and the recommended area AR2 in the delivery image PO for the destination user and make them a common area in response to the operation of the destination user, and to perform image switching in the common area based on the switching mode applied to the recommended area AR2. 13 and 14, the recommended area AR2 and the main area AR1 are made common by operating the check box 30 in the recommended area AR2, which allows the user to enjoy the delivered images on a wide screen with the images switched in the recommended mode, if desired.

[0256] The program of the embodiment is a program that causes the processing of the control unit 11 described above to be executed by, for example, a CPU, a DSP, or a device including these. That is, the program of the embodiment is a program that causes a control unit, which controls an image processing unit, to execute processing to generate a delivery image in which a main area AR1 (first area) that is a primary image area and a recommended area AR2 (second area) that is an image area separate from the main area AR1 are formed on the screen, based on a plurality of input images Pin that are images of an event captured by a plurality of imaging devices 1. The program causes the control unit 11 to execute processing to control the image processing unit 12 so that control can be executed using a plurality of switching control modes for switching the input images Pin to be included in the delivery image PO, and the captured image displayed in the recommended area AR2 can be switched based on one of the plurality of switching control modes. By using such a program, the image processing device 10 described above can be realized by various computer devices.

[0257] Such a program can be pre-recorded on a hard disk drive (HDD) as a recording medium built into a computer or other device, or on a ROM within a microcomputer having a CPU. Furthermore, such a program can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disc, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called packaged software. Such a program can be installed onto a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0258] Furthermore, such a program is suitable for widely providing the image processing device 10 of the embodiment. For example, by downloading the program to a personal computer, a communication device, a mobile terminal device such as a smartphone or a tablet, a mobile phone, a game device, a video device, a PDA (Personal Digital Assistant), or the like, these devices can function as the image processing device 10 of the present disclosure.

[0259] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0260] The present technology can also be configured as follows. (1) an image processing unit that generates a distribution image based on a plurality of input images that are images of an event captured by a plurality of imaging devices, in which a first area that is a main image area and a second area that is an image area different from the first area are formed within the screen; a control unit that controls the image processing unit so that control can be executed in a plurality of switching control modes for switching the input image included in the delivery image, and the captured image displayed in the second area can be switched based on one of the plurality of switching control modes. Image processing device. (2) The control unit In response to detecting a selection operation of the second area by a distribution destination user, the image processing unit is controlled so that image switching based on the switching control mode applied to the second area is applied to the image of the first area. The image processing device according to (1) above. (3) The image processing unit generating a delivery image in which a third area is formed on the screen in addition to the first area and the second area, in which all or part of the plurality of input images are individually displayed; The control unit In response to detecting a selection operation of the input image in the third area by the distribution destination user, the image processing unit is controlled so that the input image related to the selection operation is displayed in the first area. The image processing device according to (1) or (2) above. (4) One of the multiple switching control modes is a mode in which image switching control is performed based on progress information about an event that is obtained in advance. The image processing device according to any one of (1) to (3) above. (5) One of the plurality of switching control modes is a mode in which image switching control is performed for the delivery image for a specific delivery destination user based on viewing information of delivery destination users other than the specific delivery destination user. The image processing device according to any one of (1) to (4) above. (6) One of the plurality of switching control modes is a mode in which image switching control is performed for the delivery image for a specific delivery destination user based on viewing information of other delivery destination users having attributes that are the same as or similar to the attributes of the specific delivery destination user. The image processing device according to any one of (1) to (5) above. (7) One of the plurality of switching control modes is a mode in which image switching control is performed for a plurality of input images relating to a specific subject. The image processing device according to any one of (1) to (6) above. (8) The control unit In the second area, control is performed so that the input image switched by the processing of the switching control mode selected corresponding to the distribution destination user is displayed more frequently than the input images switched by the processing of other switching control modes. An image processing device according to any one of (1) to (7) above. (9) The switching control mode selected in correspondence with the destination user is the switching control mode designated by the selection operation of the destination user. The image processing device according to (8) above. (10) The switching control mode selected for the destination user is the switching control mode specified in the automatic selection process. The image processing device according to (8) above. (11) The control unit In the second area, a display frequency value of the input image that is switched by processing in a switching control mode selected corresponding to a distribution destination user and a display frequency value of the input image that is switched by processing in another switching control mode are variably set in accordance with an operation by the distribution destination user; The input image based on each switching control mode is controlled so as to be displayed at a frequency according to the display frequency value. The image processing device according to any one of (8) to (10) above. (12) The control unit Control is performed so that the input image that can be switched by processing of a standard switching control mode selected corresponding to a distribution destination user is displayed in the first area, and the input image that can be switched by processing of a switching control mode other than the standard switching control mode is displayed in the second area. An image processing device according to any one of (1) to (7) above. (13) The control unit In response to a selection operation being performed on the second area, control is performed such that the input image to be switched by processing in the switching control mode applied to the second area is displayed in the first area, and the input image to be switched by processing in the reference switching control mode is displayed in the second area; Furthermore, in response to the lapse of a predetermined time, control is performed so that the input image that is switched by processing in another switching control mode is displayed in the second area. The image processing device according to (12) above. (14) The control unit Based on the history of the user's selection operation, a process of determining which of the multiple switching control modes is suitable for the user is performed, and the reference switching control mode is updated according to the determination result. The image processing device according to (12) or (13) above. (15) The control unit As an initial state, the input image from one imaging device is fixedly displayed in the first area, and the input image that can be switched by processing in a predetermined switching control mode is displayed in the second area. An image processing device according to any one of (1) to (7) above. (16) The image processing unit For the plurality of input images, a conversion process is performed on the data size of the input images to be used for display in the first area and the second area, and the distribution image including the input images after the conversion process is generated for each individual distribution destination user. An image processing device according to any one of (1) to (15) above. (17) The image processing unit A data size of the input image is converted for use in display in the first area and the second area, and the data size of the input image is further converted so that the data size is smaller than that of the image in the second area for use in display in a third area in which all or part of the plurality of input images are individually displayed, and the delivery image including the converted input image is generated for each individual delivery destination user. An image processing device according to any one of (1) to (16) above. (18) The control unit, in response to an operation by a distribution destination user, controls the image processing unit so that the screen division of the first area and the second area in the distribution image for the distribution destination user is eliminated to make the first area and the second area a common area, and image switching is performed in the common area based on the switching mode applied to the second area. An image processing device according to any one of (1) to (17) above. (19) The image processing device A distribution image is generated based on a plurality of input images that are images of an event captured by a plurality of imaging devices, in which a first area that is a main image area and a second area that is an image area different from the first area are formed on the screen; and The switching of the input image included in the delivery image can be controlled by a plurality of switching control modes, and the captured image displayed in the second area can be switched based on one of the plurality of switching control modes. Image processing methods. (20) A program for causing a control unit to execute processing, the program controlling an image processing unit that generates a delivery image in which a first area that is a main image area and a second area that is an image area different from the first area are formed on a screen based on a plurality of input images that are images of an event captured by a plurality of imaging devices, A process of controlling the image processing unit so that switching of the input image included in the delivery image can be controlled using a plurality of switching control modes and the captured image displayed in the second area can be switched based on one of the plurality of switching control modes is executed. program. [Explanation of symbols]

[0261] 1,100 imaging devices (cameras) 2. Server device 3. User terminal 5 Transceiver 6 Network 10 Image processing device 11 Control section 11a Processing instruction section 11b DB 12 Image processing section 15, 15A···15(n) Image size processing 16 Delivery image generation process 30 Checkboxes 71 CPU

Claims

1. an image processing unit that generates a distribution image based on a plurality of input images that are images of the event captured by a plurality of imaging devices, in which a first area that is a main image area and a second area that is an image area different from the first area are formed within the screen; a control unit that controls the image processing unit so that control can be executed in a plurality of switching control modes for switching the input image included in the delivery image, and the input image displayed in the second area can be switched based on one of the plurality of switching control modes; The control unit In the second area, control is performed so that the input image switched by the processing of the switching control mode selected corresponding to the distribution destination user is displayed more frequently than the input images switched by the processing of other switching control modes. Image processing device.

2. An image processing unit that generates a distribution image based on a plurality of input images that are images of an event captured by a plurality of imaging devices, in which a first area that is a main image area and a second area that is an image area different from the first area are formed on the screen; a control unit that controls the image processing unit so that control can be executed in a plurality of switching control modes for switching the input image included in the delivery image, and the input image displayed in the second area can be switched based on one of the plurality of switching control modes; The control unit performing control so that the input image to be switched by processing in a standard switching control mode selected corresponding to a distribution destination user is displayed in the first area, and the input image to be switched by processing in a switching control mode other than the standard switching control mode is displayed in the second area; In response to a selection operation on the second area, control is performed so that the input image to be switched by processing in the switching control mode applied to the second area is displayed in the first area, and the input image to be switched by processing in the reference switching control mode is displayed in the second area; Furthermore, as a predetermined time elapses, control is performed so that the input image that is switched by processing in another switching control mode is displayed in the second area. Image processing device.

3. An image processing unit that generates a distribution image based on a plurality of input images that are images of an event captured by a plurality of imaging devices, in which a first area that is a main image area and a second area that is an image area different from the first area are formed on the screen; a control unit that controls the image processing unit so that control can be executed in a plurality of switching control modes for switching the input image included in the delivery image, and the input image displayed in the second area can be switched based on one of the plurality of switching control modes; The control unit a control for displaying, in the first area as an initial state, the input image captured by one imaging device in a fixed manner, and, in the second area, the input image that can be switched by processing in a predetermined switching control mode; In response to the lapse of a predetermined time, control is performed so that the input image to be switched by processing in another switching control mode is displayed in the second area. Image processing device.

4. The control unit In response to detecting a selection operation of the second area by a distribution destination user, the image processing unit is controlled so that image switching based on the switching control mode applied to the second area is applied to the image of the first area.

4. The image processing device according to claim 1.

5. The image processing unit generating a delivery image in which a third area is formed on the screen in addition to the first area and the second area, in which all or part of the plurality of input images are individually displayed; The control unit In response to detecting a selection operation of the input image in the third area by a distribution destination user, the image processing unit is controlled so that the input image related to the selection operation is displayed in the first area.

4. The image processing device according to claim 1.

6. One of the plurality of switching control modes is a mode in which image switching control is performed based on progress information regarding an event that is obtained in advance.

4. The image processing device according to claim 1.

7. One of the plurality of switching control modes is a mode in which image switching control is performed for the delivery image for a specific delivery destination user based on viewing information of delivery destination users other than the specific delivery destination user.

4. The image processing device according to claim 1.

8. One of the plurality of switching control modes is a mode in which image switching control is performed for the distribution image for a specific distribution destination user based on viewing information of other distribution destination users having attributes that are the same as or similar to the attributes of the specific distribution destination user.

4. The image processing device according to claim 1.

9. One of the plurality of switching control modes is a mode in which image switching control is performed for a plurality of input images relating to a specific subject.

4. The image processing device according to claim 1.

10. The switching control mode selected in correspondence with the destination user is the switching control mode designated by the selection operation of the destination user. The image processing device according to claim 1 .

11. The switching control mode selected for the destination user is the switching control mode specified in the automatic selection process. The image processing device according to claim 1 .

12. The control unit In the second area, a display frequency value of the input image that is switched by processing in a switching control mode selected corresponding to a distribution destination user and a display frequency value of the input image that is switched by processing in another switching control mode are variably set in accordance with an operation by the distribution destination user; The input image based on each switching control mode is controlled so as to be displayed at a frequency according to the display frequency value. The image processing device according to claim 1 .

13. The control unit Based on the history of the user's selection operation, a process of determining which of the multiple switching control modes is suitable for the user is performed, and the reference switching control mode is updated according to the determination result. The image processing device according to claim 2 .

14. The image processing unit For the plurality of input images, a conversion process is performed to convert the data size of the input images to be used for display in the first area and the second area, and the distribution image including the converted input images is generated for each individual distribution destination user.

4. The image processing device according to claim 1.

15. The image processing unit A data size of the input image is converted for use in display in the first area and the second area, and the data size of the input image is further converted so that the data size is smaller than that of the image in the second area for use in display in a third area in which all or part of the plurality of input images are individually displayed, and the delivery image including the converted input image is generated for each individual delivery destination user.

4. The image processing device according to claim 1.

16. The control unit controls the image processing unit in response to an operation by a destination user to eliminate screen division between the first area and the second area in the delivery image for the destination user and to make the first area and the second area common to each other, and to perform image switching in the common area based on the switching mode applied to the second area.

4. The image processing device according to claim 1.

17. The image processing device A distribution image is generated based on a plurality of input images that are images of the event captured by a plurality of imaging devices, in which a first area that is a main image area and a second area that is an image area different from the first area are formed on the screen; and A control can be executed in a plurality of switching control modes for switching the input image included in the delivery image, and the input image displayed in the second area can be switched based on one of the plurality of switching control modes; In the second area, the input image switched by the process of the switching control mode selected corresponding to the distribution destination user is displayed more frequently than the input images switched by the process of other switching control modes. Image processing methods.

18. A program for causing a control unit to execute processing, the program controlling an image processing unit that generates a delivery image in which a first area that is a main image area and a second area that is an image area different from the first area are formed on a screen based on a plurality of input images that are images of an event captured by a plurality of imaging devices, a process of controlling the image processing unit so that control can be executed in a plurality of switching control modes for switching the input image included in the delivery image, and the input image displayed in the second area can be switched based on one of the plurality of switching control modes; In the second area, a process is executed to perform control such that the input image switched by the process of the switching control mode selected corresponding to the distribution destination user is displayed more frequently than the input images switched by the process of other switching control modes. program.

Images