Information processing device, information processing procedure and program

Abstract

[Problem] Providing an information processing device, an information processing procedure and a program that can reduce the effort required to capture a recorded image of a subject. [Solution] This information processing device is equipped with a control unit that analyzes a captured image taken by one or more imaging devices that capture a target space, determines one or more subjects as cut-out targets from the captured image, and performs a control to cut out the determined subjects.

Description

TECHNICAL AREA

[0001] The present disclosure relates to an information processing device, an information processing method and a program. STATE OF THE ART

[0002] Typically, pre-recorded video distribution and live distribution (real-time distribution) are used for events such as music concerts and sporting events. Viewers can access the content via a smartphone, tablet, television, PC, or similar device.

[0003] With regard to such video distribution, for example, the patent document 1 below discloses a technique for the appropriate processing of content that was distributed live. LIST OF EQUATIONS PATENT DOCUMENT

[0004] Patent document 1: WO 2018 / 173876 A SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION

[0005] In the traditional method of distributing images of a subject at an event venue, the selection of the subject and the adjustment of the viewing angle were done manually, which cost time and effort.

[0006] Therefore, the present disclosure proposes an information processing device, an information processing method and a program that make it possible to reduce the effort required to capture a depicted image of a subject. SOLUTIONS TO THE PROBLEMS

[0007] According to the present disclosure, an information processing device is provided which includes a control unit which analyzes an image captured by one or more imaging devices mapping a target space, determines one or more subjects as cut-out targets from the imaged image and performs a control to cut out a specific subject.

[0008] Furthermore, according to the present disclosure, an information processing method is provided which is carried out by a processor, including the analysis of an image taken by one or more imaging devices mapping a target space, the determination of one or more motifs as cut-out targets from the imaged image and the execution of a control to cut out a specific motif.

[0009] Furthermore, according to the present disclosure, a program is provided that causes a computer to act as a control unit that analyzes an image captured by one or more imaging devices mapping a target space, determines one or more subjects as cut-out targets from the imaged image, and performs a control to cut out a specific subject. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a diagram showing the outline of a distribution system according to an embodiment of the present disclosure. Fig. Figure 2 is a block diagram showing an example of a configuration of a content generation device according to the present embodiment. Fig. Figure 3 is a diagram illustrating an example of a position setting screen 400 displayed on a display unit of the content generation device according to the present embodiment. Fig. Figure 4 is a diagram showing an example of a cropped image display screen according to the present embodiment. Fig. Figure 5 is a diagram describing the cutting out of a motif located in a region of interest, according to the present embodiment. Fig. Figure 6 is a diagram describing a section area according to the present embodiment. Fig. Figure 7 is a diagram showing a section of a case containing a multitude of motifs according to the present embodiment. Fig. Figure 8 is a diagram illustrating the switching of a displayed image from a crop source by the movement of a subject according to the present embodiment. Fig. Figure 9 is a diagram describing the designation of a detection area according to the present embodiment. Fig. Figure 10 is a block diagram showing an example of the configuration of a distribution switching device according to the present embodiment. Fig. Figure 11 is a flowchart that illustrates an example of the operational processing sequence of the content generation device according to the present embodiment. Fig. Figure 12 is a diagram showing another method for using a cut-out image according to an application example of the present embodiment. MODE FOR EXECUTING THE INVENTION

[0010] A preferred embodiment of the present disclosure is described in detail below with reference to the accompanying drawings. It should be noted that in the present description and in the drawings, components having essentially the same functional configuration are identified by the same reference numerals, and redundant descriptions are omitted.

[0011] Furthermore, a description is given in the following order. 1. Outline of a distribution system according to an embodiment of the present disclosure 2. Configuration example 2-1. Configuration example for the content generation device 20 2-2. Configuration example for the distribution switching device 30 3. Operational processing 4. Application examples 5. Additional Information <<1. Outline of the distribution system according to an embodiment of the present disclosure>>

[0012] Fig. Figure 1 is a diagram showing the outline of a distribution system according to one embodiment of the present disclosure. As in Fig. Figure 1 illustrates a case in which the state of a venue V, where a music concert, a musical, or the like is being performed, is distributed live. Specifically, the distribution system according to the present embodiment comprises cameras 10a to 10d (example of imaging devices) that image a stage S (example of a target room) of the venue V, a content generation device 20 (example of an information processing device) that generates content (in particular, images) as distribution candidates, and a distribution switching device 30 that switches the content to be distributed.

[0013] Venue V can be a facility with stage S and seating for spectators, or a recording room (sound studio).

[0014] Cameras 10a to 10c are installed in venue V and can capture any area of ​​stage S. Although the viewing angles of cameras 10a to 10c differ, the imaging is performed in a state where the cameras partially overlap, as shown in Figure 1. Fig. Figure 1 illustrates this. The images captured by cameras 10a to 10c are output to the content generation device 20 and used to crop a subject within the content generation device 20. Cameras 10a to 10c can, for example, be 4K, 8K, or 16K cameras. The resolution of cameras 10a to 10c is preferably sufficient to produce a viewable image section when a subject is cropped from a displayed image, and this resolution is not particularly limited. Furthermore, cameras 10a to 10c can be arranged and installed side-by-side on the seating side of the stage S. The number of cameras 10 is also arbitrary; there can be one or more cameras 10.

[0015] Furthermore, a camera 10d can be used that captures the entire stage S within its field of view. An image captured by camera 10d (overview image of stage S) is not used for cropping in the content generation device 20, but is output to the distribution switching device 30. Camera 10d can, for example, be a high-resolution (HD) camera. The resolution of camera 10d can be, for example, lower than that of cameras 10a to 10c, which capture images for cropping a subject, but is not limited to this. In addition, a variety of cameras can be installed that capture images not used for cropping a subject. For example, a camera can be installed that images the entire stage S from a different direction than that of camera 10d.

[0016] The content generation device 20 is an information processing device that extracts one or more subjects from each of the images captured by the cameras 10a to 10c and performs a control operation to generate one or more of the extracted images of the subjects as the content of distribution candidates. The content generation device 20 transmits the extracted images to the distribution switching device 30. For example, a serial digital interface (SDI) output is used for the image output from the content generation device 20 to the distribution switching device 30. The content generation device 20 extracts images according to the number of outputs (in particular, the number of SDI outputs).

[0017] The distribution switching device 30 is a device that controls the switching (selection) of an image to be distributed to a distribution destination (in particular, a viewer terminal). A multitude of images, such as cropped images output by the content generation device 20 and images captured by the camera 10d, can be input into the distribution switching device 30. The distribution switching device 30 selects an image to be output (distributed) from the multitude of input images and outputs the image to the distribution destination. The distribution switching device 30 also switches (reselects) an image to be distributed accordingly. The switching (selection) can be performed manually by the operator (e.g., a switch) or automatically. (Reviewing the problems)

[0018] In traditional broadcasting, a large number of cameras were set up at a venue, camera operators were assigned to each camera, and camera operation, including adjusting the camera angle (zoom, changing the image direction, etc.) to a subject, was done manually. For example, in a case where a large number of performers, such as a group of idols, are on a stage, it was usually determined in advance, based on the song schedule and the like, which subject would be tracked by which camera at what time, and a rehearsal of the camera work was conducted. As described above, in traditional broadcasting, in a case where a subject is being filmed at a venue, the selection of the subject to be filmed and the camera angle to be set were done manually, which was time-consuming and laborious.

[0019] Therefore, in the distribution system according to the present disclosure, the effort required to capture an image of a subject can be reduced, and the number of people at the time of capture can be reduced. For example, by automatically cropping any subject from the images captured by the recordings of the in Fig. The numerous cameras 10a to 10c shown in Figure 1, located at venue V, allow for the adequate capture of an image of a subject without the need for a cameraman. Even with a large number of subjects on stage, the workload can be reduced by automatically selecting one subject as the cutout.

[0020] The outline of the distribution system according to the embodiment of the present disclosure has been described above. The configuration of the individual devices of the distribution system according to the present embodiment is described below with reference to the drawings. <<2. Configuration Example>><2-1. Configuration Example for the Content Generation Device 20>

[0021] Fig. Figure 2 is a block diagram showing an example of a configuration of the content generation device 20 according to the present embodiment. As shown in Fig. As shown in Figure 2, the content generation device 20 comprises a communication unit 210, a control unit 220, an operator input unit 230, a display unit 240, and a storage unit 250. The content generation device 20 is used, for example, by a director who manages an entire event. (Communication Unit 210)

[0022] The communication unit 210 comprises a transmitter that sends data to an external device via a wired or wireless connection, and a receiver that receives data from the external device. The communication unit 210 is communicatively connected to the cameras 10a to 10c and the distribution switching device 30, for example, via a wired / wireless local area network (LAN), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communications network (Long Term Evolution (LTE), a fourth-generation mobile communications system (4G), or a fifth-generation mobile communications system (5G)), or the like.

[0023] Furthermore, the communication unit 210 can also function as a transmitter, transmitting (outputting) a cropped image of the subject to the distribution switching device 30. SDI output can be used as a special output method. The output of an image can be separate from the data transmission via LAN or similar. (Control unit 220)

[0024] The control unit 220 functions as an arithmetic processing device and as a control device, managing the overall operation of the content generation device 20 according to various programs. The control unit 220 is implemented, for example, by an electronic circuit such as a central processing unit (CPU), a microprocessor, or the like. Furthermore, the control unit 220 may also contain read-only memory (ROM) in which programs to be used, arithmetic parameters, etc., are stored, as well as random-access memory (RAM) in which parameters, etc., that may change are temporarily stored. The control unit 220 may also include a graphics processing unit (GPU).

[0025] In addition, the control unit 220 also functions as a display position setting unit 221, as a cutout processing unit 222 and as an output control unit 223.

[0026] The display position setting unit 221 processes the arrangement and display of a multitude of images with partially overlapping viewpoints, captured by cameras 10a to 10c, which are multiple imaging devices arranged side-by-side in a partially overlapping configuration on the seating side of stage S on the display unit 240. It also processes the receipt of the setting for an overlapping position of the multitude of images. This setting can be made by an operator (e.g., a director) during a preparation phase before the start of the event. In the preparation phase, cameras 10a to 10c are first positioned on the seating side so that the entire stage S can be imaged simultaneously. In the Fig. In the example shown, the left side of stage S is mainly captured by camera 10a, the center of stage S mainly by camera 10b, and the right side of stage S mainly by camera 10c. At this point, the viewing angle (image area) of each camera 10 can be adjusted so that the viewing angle (image area) of the neighboring camera 10 partially overlaps. Fig. For example, the left end of the imaging area of ​​the centrally positioned camera 10b overlaps the right end of the imaging area of ​​the left-hand camera 10a, and the right end of the imaging area of ​​the centrally positioned camera 10b overlaps the left end of the imaging area of ​​the right-hand camera 10c. Next, the display position adjustment unit 221 arranges the images from cameras 10a to 10c side by side and displays them on the display unit 240. A specific description follows with reference to... Fig. 3 given.

[0027] Fig. Figure 3 is a diagram showing an example of a position setting screen 400 displayed on the display unit 240 of the content generation device 20 according to the present embodiment. As shown in Fig. As shown in Figure 3, the position setting screen 400 displays an image 401 captured by camera 10a, an image 402 captured by camera 10b, and an image 403 captured by camera 10c, arranged and displayed side by side. The position setting screen 400 also includes an operation screen for adjusting the display position, size, and transparency of each of the displayed images 401 to 403. An operator (e.g., a director) of the content generation device 20 moves the display position of each of the displayed images 401 to 403 up and down, left and right, increases / decreases the display size, and adjusts the overlap position while verifying the overlap of a subject by transmitting the displayed images. More precisely, the operator adjusts the display position of an image so that the subjects in the overlap area coincide.The display position setting unit 221 receives the input of a setting operation for a display position and stores the setting result (display position and display size of each of the displayed images) in the memory unit 250. The result of the setting can be at least information about the overlap position of each of the recorded images (which area of ​​the image field overlaps which area of ​​the image field of which camera).

[0028] It should be noted that in the present embodiment, the operator makes the setting manually via the position setting screen 400, but the present disclosure is not limited to this, and the setting can be made automatically by the display position setting unit 221. Furthermore, the operator can be prompted to check the result of the automatic setting.

[0029] The cropping processing unit 222 analyzes images captured by one or more imaging devices (e.g., cameras 10a to 10c) depicting a target space (e.g., stage S), identifies one or more subjects as cropping targets from the images, and executes a control to crop these selected subjects. This cropping processing can be performed continuously from the beginning of the event distribution (start of imaging). Specifically, the processing is performed for each image.

[0030] First, the cropping processing unit 222 performs an image analysis of the captured images 401 to 403 and identifies the subjects by subject recognition. Examples of a subject include a person, an animal, an object, and the like, but in the present embodiment, a person performing on a stage is assumed. The cropping processing unit 222 can perform facial recognition to identify a subject. Next, the cropping processing unit 222 determines a subject that fulfills a predetermined condition among the identified subjects as the cropping target and performs the cropping.

[0031] An image cut out by the cropping processing unit 222 (cut-out image; captured image of the subject) is output by the output control unit 223 to the distribution switching device 30 and the display unit 240. The output control unit 223 can control the output (transmission) of one or more cut-out images from the communication unit 210 to the distribution switching device 30 and the output (display) of the cut-out images to the display unit 240. Furthermore, the output control unit 223 can output the cut-out images to the distribution switching device 30 and transmit a distribution switching control signal to the distribution switching device 30.For example, a signal can be transmitted that indicates a cut-out image with high distribution priority (the signal is information used to control the distribution switching in the distribution switching device 30), e.g. a singing motif or a motif in a region of interest.

[0032] With reference to Fig. Section 4 describes an example of how to display cropped images. Fig. Figure 4 is a diagram showing an example of a cropped image display screen 410 according to the present embodiment. The one in Fig. The cropped image display screen 410, shown in Figure 4, is displayed on the display unit 240 of the content generation device 20 during event distribution. By visually recognizing the cropped image display screen 410, the director can intuitively identify the themes and images (cropped images) identified by the system that were preferentially cropped and output to the distribution switching device 30 (SDI output).

[0033] As in Fig. As shown in Figure 4, the images 401 to 403 captured by the respective cameras 10a to 10c and the images 501 to 505 extracted from the respective images 401 to 403 are displayed on the cropped image display screen 410. Corresponding SDI output numbers are assigned to the extracted images 501 to 505. The extracted images 501 to 505 are output via SDI to the distribution switching device 30.

[0034] Furthermore, the images 401 to 403 displayed on the cropped image display screen 410 are arranged and displayed side by side in a partially overlapping state, depending on the result of a previous setting by the display position adjustment unit 221. The in Fig. Images 401 to 403 contain subjects P1 to P9, and the result of the face detection for each subject is clearly indicated by a frame line (the frame line surrounding the face). This allows the director to intuitively see that the subjects have been detected by the system. Furthermore, the frame line of a subject identified as a cutout target can be highlighted and displayed. Additionally, the SDI output number associated with the cutout image of a subject identified as a cutout target is also displayed in the subject's frame line. This allows the director to intuitively identify which subject has been identified by the system as a cutout target and to view the cutout image of that specific subject.

[0035] The following section describes in more detail the section processing described above by the section processing unit 222.

[0036] The cropping processing unit 222 identifies a subject that fulfills a predetermined condition as the cropping target and performs the cropping, wherein the "predetermined condition" is, for example, the execution of a predetermined action. The cropping processing unit 222 preferably identifies a subject that is recognized as performing a predetermined action as the cropping target. The cropping processing unit 222 can recognize a predetermined action by analyzing a captured image. Furthermore, the cropping processing unit 222 can recognize a predetermined action based on sensor data other than a captured image.

[0037] An example of a predefined action is a singing action. The clipping processing unit 222 identifies a singing subject as the clipping target that meets a predefined condition. In a case where a group with a large number of idols or similar subjects are the clipping subjects, the clipping processing unit 222 preferentially identifies a singing subject as the clipping target. This is because, at a music concert, it is important to track a singing person with a camera.

[0038] Examples of a procedure for determining whether or not singing is taking place are the following. For instance, the Clipping Processing Unit 222 analyzes a captured image to estimate a person's skeleton and determines that singing is occurring in a case where the subject raises a hand holding a handheld microphone. Furthermore, the Clipping Processing Unit 222 determines that singing is occurring in a case where a sound source is active (in a case where a microphone is active) based on information about the subject's microphone (handheld microphone held by the subject, headset microphone attached to the subject, stand microphone positioned in front of the subject, or the like).Furthermore, the clipping processing unit 222 determines that singing is performed in a case where microphone movement is detected based on information from an accelerometer or similar device included in the subject's microphone. Additionally, the clipping processing unit 222 performs image recognition of the captured image and determines that singing occurs when the subject's mouth is open. Furthermore, the clipping processing unit 222 determines that singing is performed in a case where the subject is in a predetermined position at a predetermined time (which is predefined based on a song outline and a standing position) based on positional information of the subject on stage. The subject's positional information on stage is obtained from a sensor (e.g., a microphone).an ultra-wideband (UWB) position information tag worn by the subject, or determined by image recognition.

[0039] Examples of a "predetermined condition" include positioning within a region of interest. The clipping unit 222 identifies a region of interest and selects a subject located within that region as a clipping target that satisfies a predetermined condition. This occurs because, at a music concert or similar event, a region of interest (a region to which attention is desired during production) can be temporarily created. The clipping unit 222 detects the movement of each subject, for example, through skeletal estimation or similar methods, and identifies a region of movement (a region with a greater degree of movement than other regions). For instance, if only one person or a specific group (a group of several people) begins to move, the clipping unit 222 will preferentially select that person or group as the clipping target. Fig. Figure 5 is a diagram describing the cutting out of a motif located in a region of interest, according to the present embodiment. As in Fig. As shown in Figure 5, the cropping processing unit 222, in an image 404 captured by a camera 10 (one of 10a to 10c), in a case where only a certain group (subjects P10 and P11) is moving and the other subjects P12 and P13 are stationary, determines the subjects P10 and P11 as a group as cropping targets and performs a cropping of the captured image 404 (a cropped image 506 is produced).

[0040] Examples of a "predetermined condition" include positioning in the center of the stage. This is because, in a music concert or similar event, a subject that is to be highlighted is often located in the center of the stage. The clipping processing unit 222 identifies a subject positioned in the center of the stage as a clipping target as one that fulfills a predetermined condition.

[0041] Furthermore, the 222 cutout processing unit can perform cutouts in an area containing a single motif (single cutout) or cutouts in an area containing multiple motifs (group cutout). As shown in Fig. As described in section 5, group excerpting can be carried out, for example, in a case where the excerpting is based on a region of interest.

[0042] Furthermore, the cropping processing unit 222 cuts out a motif (produces a cropped image) by the number of cutouts corresponding to the number of image outputs to the distribution switching device 30. The number of image outputs is, for example, the number of SDI outputs and can be preset.

[0043] Furthermore, the cutout processing unit 222 can preferably determine a motif identified from an image as the cutout target. In a case where the number of identified motifs is equal to or greater than the number of cutouts, the cutout processing unit 222 preferably cuts out a motif that satisfies a condition in accordance with each predetermined condition described above. Moreover, the cutout processing unit 222 can determine a motif as the cutout target by combining all the conditions described above. For example, if the number of identified motifs is equal to or greater than the number of cutouts and all motifs are singing, the cutout processing unit 222 can preferably determine a motif near the center as the cutout target.Furthermore, in a case where motifs can be identified and information about the popularity of each of the motifs is entered, the cutout processing unit 222 can preferentially determine a popular motif as the cutout target.

[0044] If, however, the number of identified motifs is less than the number of clips, the clipping processing unit 222 can define a fixed position on the stage as the clipping target. For example, at the beginning, during a transition, or at the end of a music concert, a motif might appear on stage after a certain time. In this case, the clipping processing unit 222 preferentially cuts the video at a fixed position, such as the center of the stage or a (pre-configurable) appearance position of a motif on the stage.

[0045] The selection of a cutout target has already been described above. It should be noted that a subject can be freely selected as a cutout target by an operator (e.g., a director) of the content generation device 20. The operator selects a subject as the cutout target on the cutout image display screen 410, as shown, for example, in Fig. Figure 4 illustrates this. The identification method is arbitrary, but for example, identification can be performed by a touch operation on a motif appearing in the images 401 to 403 displayed on screen 410. Furthermore, identification can be performed by dragging and dropping the display of the frame line surrounding the face of one motif onto the face of another.

[0046] Next, an area of ​​the cutout is described in more detail by the cutout processing unit 222.

[0047] The cropping unit 222 crops a subject in an area that includes at least the subject's face. Furthermore, the cropping unit 222 can crop an area that includes at least the subject's face, enlarging the area to a resolution limit (resolution on a plane suitable for viewing). The resolution limit can be preset. Additionally, the cropping unit 222 can crop a subject in an area that includes at least one of the subject's hands. Considering the subject's choreography, it may be desirable to crop a subject in an area that includes at least the face and one hand.

[0048] Furthermore, the cropping processing unit 222 can determine a crop area (whether it includes only the face, a hand, the torso, the entire body, or the like) based on the skeletal estimation of a subject. For example, if the skeletal estimation detects that a hand is being moved extensively due to choreography or the like, the cropping processing unit 222 can define a crop area that includes the hand.

[0049] Furthermore, the cutout processing unit 222 can perform a cutout in an area that includes a predetermined border above the top part of the subject's body (cutout target). The top part of the body is the part located at the highest point of the person; this is usually assumed to be the head or a hand if the hand is raised. Fig. Figure 6 is a diagram describing a section of the diagram according to the present embodiment. As in Fig. As shown in Figure 6, the cutout processing unit 222, for example, captures (generates) a cutout image 507 in an area that includes a border h above the head, which represents the top part of a motif P.

[0050] Furthermore, it is assumed that in a case where the cropping unit 222 crops a subject as a crop target in an area magnified to the resolution limit, which includes at least the face, another subject appears nearby in the crop area. In this case, the cropping unit 222 temporarily includes in a crop target a subject whose body half or more appears in the crop area, or a subject that appears in the crop area to such an extent that recognition by skeletal estimation can be performed, and performs the crop in an area corresponding to the heights of all subjects. A specific example is given with reference to Fig. 7 described.

[0051] Fig. Figure 7 is a diagram showing a section of a case containing a multitude of motifs according to the present embodiment. Fig. 7. The case is assumed where, in a case where motif P15 is specified as the cutout target, motifs P16 and P17 appear nearby within a cutout area. In this case, the cutout processing unit 222 captures (generates) a cutout image 508 in an area that includes a border h above the topmost part of the bodies of all motifs (the head of motif P17). This avoids cutting out an image in which a head is unnaturally cut out. Such an adaptation of a cutout area in a case containing a multitude of motifs can also be applied to a case of group cutout described above.

[0052] It should be noted that even if the number of motifs in a cropping area increases when a cropped image is selected (programmed) for distribution by the distribution switching device 30, the cropping processing unit 222 can maintain the height of the cropping area corresponding to a motif that is designated as the cropping target in a case where the image is selected for distribution. Furthermore, in a case where a cropped image is selected (programmed) for distribution by the distribution switching device 30 and motifs in the cropping area are reduced (in a case where a motif temporarily designated as the cropping target lies outside the cropping area), the cropping processing unit 222 cannot change the height of the cropping area. In this way, the quality of a programmed image is preserved.

[0053] Although the setting of a cutout area in a case where a motif appears has been described above, the present embodiment is not limited to this, and a cutout area can be set according to a motif that has been determined as the cutout target, without taking a motif into account, even in a case where the motif appears.

[0054] Furthermore, the Crop Processing Unit 222 can smooth the motion direction of a cropped area between frames, so that the movement of a subject appears natural in continuously cropped images (cropped video with a large number of frames). Examples of smoothing methods include averaging, weighting, and similar methods for the amount of movement of the images in a given section. The Crop Processing Unit 222 can use the average of a subject's coordinate positions as the cropping target and smooth the amount of movement within the cropped area (this does not affect subtle movements of the subject).

[0055] Furthermore, in cases where the eye line of a subject is directed left and right as the cutout target (in a case where the face is turned sideways), the cutout processing unit 222 can perform the cutout in an area that includes a large border in the direction of the eye line (face direction). With this arrangement, a cutout image with a sophisticated composition can be achieved, providing the viewer with a sense of depth or line of sight.

[0056] Furthermore, the cutout processing unit 222 can perform cutouts in an area containing a multitude of motifs (group cutout) or in an area containing only one motif from the multitude of motifs (single cutout). This means that both group cutouts and single cutouts can be performed simultaneously on the same motif as the cutout target. With this arrangement, it can be expected, for example, that in a case where a group cutout image and a single cutout image are switched in the distribution switching device 30, a dynamic feeling can be evoked in the viewer, conveying a realistic sense of a music concert or the like.

[0057] Next, an image of a cropping source is described when the cropping is performed by the cropping processing unit 222. If a subject is contained within an overlapping area that has been preset by the display position setting unit 221, the cropping processing unit 222 crops the subject from each captured image. It is also assumed that a subject is performing vigorous movement, such as walking around on stage, particularly at a concert or similar event with a large number of performers. Even in such a case, the cropping processing unit 222 must continue to track (and crop) a subject as a cropping target.Therefore, in a case where a subject moves across a multitude of images as a clipping target (also called a tracking target), the clipping processing unit 222 can switch the clipping source image at the point where the subject enters an overlapping area and continue tracking. That is, in a case where a subject moves from a first image to a second image in a multitude of adjacent images, the clipping processing unit 222 switches the clipping source image at a point where the first and second images overlap. A specific description follows with reference to [reference missing]. Fig. 8 given.

[0058] Fig. Figure 8 is a diagram illustrating the switching of a displayed image from a cropped source by the movement of a subject according to the present embodiment. As in Fig. As shown in Figure 8, for example, in a case where images 401 to 403 are arranged side by side with partial overlap, it is assumed that the subject P1, as the cropping target contained only in the area of ​​image 402, moves in the left direction (area of ​​the captured image 401). If, in this case, subject P1 enters an overlap area E between the captured image 402 and the captured image 401, the cropping processing unit 222 switches the cropping source of the subject from the captured image 402 to the captured image 401. Therefore, even if subject P1 moves to a position that lies only within the area of ​​the captured image 401, smooth tracking (with continued cropping of subject P1) can be performed.

[0059] It should be noted that if the viewing angle of the switched recorded image differs from the source image, the zoom factor on the cropped image output to the distribution switching device 30 appears to be changed. As a countermeasure against confusion in cases where people overlap in an overlapping area (front and back), it is also conceivable to distinguish and identify a feature (clothing color, hairstyle, or the like) of a subject as a tracking target, or to detect and identify the subject's direction of movement by combining it with a depth sensor. Furthermore, it is also possible to detect and identify the subject's position by combining it with a position measurement sensor (e.g., by having the subject wear an identifiable tag).

[0060] Furthermore, the Cutout Processing Unit 222 is not limited to tracking a subject, but can also perform cutouts of a (predefined) area on the stage (fixed-position cutout). Specifically, the Cutout Processing Unit 222 designates one or more subjects within a predetermined area on the stage as cutout targets and performs the cutout within an area that includes the subjects. The Cutout Processing Unit 222 then performs no tracking, even if the subjects move outside the predetermined area.

[0061] Next, the determination of a subject recognition area by the cropping processing unit 222 in a captured image is described. For example, it is possible to define an area in which image recognition is performed to avoid mistakenly identifying an audience or a person appearing on a stage backdrop as the subject (performer). Fig. Figure 9 is a diagram describing the designation of a detection area according to the present embodiment. On a Fig. In the recognition area determination screen 420 shown in Figure 9, the captured images 401 to 403 are arranged side by side and displayed partially overlapping. Rectangular recognition frames D are also displayed on the captured images 401 to 403. An operator (e.g., a director) of the content generation device 20 can define the recognition area by adjusting the position and size of a recognition frame D (e.g., excluding the audience or the background screen). The cropping processing unit 222 calculates the coordinate position of the intended recognition frame D and defines the recognition area (image analysis area) in each of the captured images 401 to 403, as shown in the lower part of Figure 9. Fig. Figure 9 illustrates this. The cropping processing unit 222 performs an image analysis within such a recognition area and identifies a subject. It should be noted that the setting of a recognition frame D is not limited to manual adjustment, but can also be performed automatically by the content generation device 20.

[0062] The section processing by the section processing unit 222 has already been described in detail above. The following section describes the individual configurations with reference to... Fig. 2 continued. (Operating input unit 230 and display unit 240)

[0063] The operating input unit 230 receives an operator input and outputs input information to the control unit 220. In addition, the display unit 240 shows various operating screens, each with reference to... Fig. 3, Fig. 4 and Fig. 9. The display unit 240 can be a display panel such as a liquid crystal display (LCD) or an organic electroluminescent display (EL). The operating input unit 230 and the display unit 240 can be contained in one piece. The operating input unit 230 can, for example, be a touch sensor laminated onto the display unit 240 (e.g., a panel display). (Storage unit 250)

[0064] The memory unit 250 consists of a read-only memory (ROM) in which programs, arithmetic parameters and the like are stored for processing by the control unit 220, and a random access memory (RAM) in which parameters and the like are temporarily stored that change as needed.

[0065] Although the configuration of the content generation device 20 has been specifically described above, the configuration of the content generation device 20 according to the present disclosure is not limited to the one described in Fig. The example shown in Figure 2 is limited. For instance, the content generation device 20 may not have an input unit 230 or a display unit 240. Furthermore, the content generation device 20 can be implemented by a variety of devices. In addition, at least some of the functions of the content generation device 20 can be performed by a server. <2-2. Configuration example for the distribution switching device 30>

[0066] Fig. Figure 10 is a block diagram showing an example of a configuration of the distribution switching device 30 according to the present embodiment. As shown in Fig. As shown in Figure 10, the distribution switching device 30 comprises a communication unit 310, a control unit 320, an operator input unit 330, a display unit 340, and a storage unit 350. An operator of the distribution switching device 30 can be a switch who switches a distribution image. (Communication Unit 310)

[0067] The communication unit 310 comprises a transmitting unit that sends data to an external device via a wired or wireless connection, and a receiving unit that receives data from the external device. The communication unit 310 is communicatively connected to the content generation device 20 and a distribution destination, for example, via a wired / wireless local area network (LAN), Wi-Fi (registered trademark), Bluetooth (registered trademark), a cellular network (Long Term Evolution (LTE), a fourth-generation (4G) mobile communication system, or a fifth-generation (5G) mobile communication system), or the like.

[0068] More precisely, SDI can be used to input a cropped image of a subject from the content generation device 20 via the communication unit 210. Furthermore, the internet can be used for the transmission (distribution) of an image to a distribution destination via the communication unit 210. (Control unit 320)

[0069] The control unit 320 functions as an arithmetic processing and control device, controlling the overall operation of the distribution switching device 30 according to various programs. The control unit 320 is implemented, for example, by an electronic circuit such as a central processing unit (CPU), a microprocessor, or the like. Furthermore, the control unit 320 can also contain a read-only memory (ROM) in which programs to be used, arithmetic parameters, etc., are stored, as well as a random-access memory (RAM) in which parameters, etc., that may change are temporarily stored.

[0070] The control unit 320 also functions as a switching unit 321 and as a distribution control unit 322.

[0071] The switching unit 321 switches (selects) a programmed image to be distributed to a distribution target (viewer terminal). Specifically, the switching unit 321 selects an image to be distributed from a multitude of images cut out via SDI, which are output by the content generation device 20. The distribution control unit 322 then controls the distribution of the selected image from the communication unit 310 to the distribution target.

[0072] The switching unit 321 can automatically select an image to be distributed according to a control signal from the content generation device 20. For example, the content generation device 20 receives five cut-out images obtained by cutting out five subjects and a signal to determine, from among the five cut-out images, cut-out images of each pair of subjects whose singing action has been recognized as high-priority images. The switching unit 321 randomly selects one of the two cut-out images designated as high-priority images (images of the singing subjects). Note that in a case where there are many singing subjects, the content generation device 20 sets the distribution priority for a subject near the center to high, and the switching unit 321 can make the selection accordingly.Furthermore, the distribution priority for a motif in a region of interest can also be set high. For production reasons, if a cut-out image of a motif exists in a region of interest, the cut-out image can always be selected (as an image to be distributed) by switching unit 321.

[0073] Furthermore, when the singing motif changes, the switching unit 321 also switches the image to be distributed (switches the image to a cut-out image of a motif that then sings).

[0074] Furthermore, the switching (selection) of the distribution image is performed automatically by the switching unit 321 as described above, but the present invention is not limited thereto, and the switching unit 321 can receive a switching operation from an operator (e.g., a switcher) of the distribution switching device 30. For example, the control unit 320 can display a plurality of cut-out images (distribution image candidates) output by the content generation device 20 on the display unit 340 and prompt the operator to make a free selection. At this time, the display unit 340 can also display information about the cut-out images (popularity, number of followers, center, etc.) and provide recommendations to the operator.

[0075] Furthermore, the switching unit 321 can adjust the switching point of the distribution pattern to the tempo (beats per minute (BPM)) of the music sung by the subjects. The switching unit 321 can extract the BPM from an input sound source (e.g., from a subject's voice captured by a microphone). Additionally, the switcher can input the BPM by touching a touch panel display (which integrates the operating input unit 330 and the display unit 340) in accordance with the rhythm (touching at regular intervals in time with the melody).

[0076] Furthermore, the 321 switching unit can perform the switching in accordance with the operator's press of a switch button. The image to be switched can be automatically selected by the 321 switching unit. Neither a director nor a switcher is required on-site, as the number of personnel involved in distribution is reduced under the current system, and it is assumed that only one manager is operating. However, even if the manager lacks the necessary operating skills, such as those of a switcher, they can easily switch the distribution image by pressing the switch button at any time, in accordance with the melody.

[0077] Among the candidates for the distribution image is an overview image taken by the 10d camera, as shown in Fig. As described in section 1, the overview image has a low priority. Therefore, the overview image from camera 10d can be selected as the distribution image, for example, when no one is singing or when there is no subject on stage (at the beginning, end, or similar point of a song). (Operating input unit 330 and display unit 340)

[0078] The operator input unit 330 receives input from an operator and outputs input information to the control unit 220. The display unit 340 also shows various operator screens and candidates of the distribution image (cut-out image). The display unit 340 can be a display panel such as a liquid crystal display (LCD) or an organic electroluminescent display (EL). The operator input unit 330 and the display unit 340 can be integrated into a single unit. For example, the operator input unit 330 could be a touch sensor laminated onto the display unit 340 (e.g., a panel display). (Storage unit 350)

[0079] The memory unit 350 consists of a read-only memory (ROM) in which programs, arithmetic parameters and the like are stored for processing by the control unit 320, and a random access memory (RAM) in which parameters and the like are temporarily stored that change as needed.

[0080] Although the configuration of the distribution switching device 30 has been specifically described above, the configuration of the distribution switching device 30 according to the present disclosure is not limited to the one described in Fig. The example shown in Figure 10 is limited. For instance, the distribution switching device 30 can do without the operating input unit 330 and the display unit 340. Furthermore, the distribution switching device 30 can be implemented using a variety of devices. <<3. Operational Processing>>

[0081] The following describes in more detail the operational process of the content generation device 20 according to the present embodiment with reference to the drawings. Fig. Figure 11 is a flowchart showing an example of the operational processing sequence of the content generation device 20 according to the present embodiment.

[0082] First, it controls, as in Fig. Figure 3 shows the control unit 220 of the content generation device 20 controlling the cameras 10 (10a to 10c) so that imaging is started (step S103). Distribution can be started by the cameras 10 that initiate imaging. Next, the

[0083] Content generation device 20 recorded images from the respective cameras 10a to 10c (step S106).

[0084] Next, the cropping processing unit 222 of the content generation device 20 analyzes each of the captured images (step S109) and identifies subjects.

[0085] Next, the cropping processing unit 222 determines subjects as cropping targets based on the number of crops from each of the captured images (step S112). Note that a group containing multiple subjects (subject group as cropping target) is added as 1.

[0086] The cropping unit 222 then cuts out the subjects by the specified number of cutouts (step S115). This means that the cropping unit 222 captures (generates) cut-out images from the captured images.

[0087] The output control unit 223 then displays one or more clipped images on the display unit 240 (step S118). The output control unit 223 also transmits one or more clipped images (SDI outputs) to the distribution switching device 30 (step S121). The distribution switching device 30 selects one image to be distributed from among one or more clipped images.

[0088] The processing described above (steps S106 to S121) is performed for each image until imaging (distribution) is complete (step S124). Distribution can be carried out in real time from the distribution switching device 30.

[0089] An example of the operational processing sequence of the content generation device 20 according to the present embodiment has been described above. It should be noted that the in Fig. The operational processing shown in Figure 11 is an example and shows that part of the processing can be carried out in different sequences or in parallel, or may not be carried out at all. <<4. Application Examples>>

[0090] The following are application examples of the present embodiment.

[0091] Fig. Figure 12 is a diagram showing another method for using a cut-out image according to an application example of the present embodiment. As in Fig. As shown in Figure 12, the output control unit 223 of the content generation device can arrange and display 20 cut-out images side by side on a rear screen 600 on stage using a multi-screen setup. The display is not limited to the rear screen 600 but can also be another large display installed in the venue. The priority order for the display can be determined based on the singing, a region of interest, a center, and the like, as described above.

[0092] In a scenario where cutout images of all subjects on stage can be obtained, the Output Control Unit 223 can always display these cutout images using a multi-screen setup. Furthermore, after a subject is lost (if tracking fails or the subject is lost), the Output Control Unit 223 can display a cutout image of a newly identified subject in the same display position, ensuring that the display positions of the respective subjects are not spread across the multi-screen. It is important to note that the output resolution should not be relied upon. The resolution of an LED display installed at the venue may vary, for example, HD, 4K, or 8K.

[0093] As a further application example, the output control unit 223 can acquire information about a selected (programmed) cut-out image from the distribution switching device 30 and display the selected cut-out image in real time on the in Fig. Highlight and display the screen shown in section 4. This allows the director to easily see which videos are currently being distributed.

[0094] Furthermore, the embodiment described above assumes real-time distribution, but the present disclosure is not limited to this. The present system can also be used for distribution during recording.

[0095] Furthermore, in the embodiment described above, a group consisting of a large number of idols was described primarily as an example, but the present disclosure is not limited to this, and a performer and a player are broadly included. Moreover, an event to be depicted is not limited to a music concert, and a musical, a theatrical performance, a sporting event, and the like are also included. <<5. Additional Information>>

[0096] The preferred embodiment of the present disclosure has been described in detail above with reference to the accompanying drawings, but the present technology is not limited to such an example. It is obvious that persons with normal knowledge in the technical field of the present disclosure may devise various modifications or corrections within the scope of the technical idea set out in the claims, and it is self-evident that these also fall within the technical scope of the present disclosure.

[0097] Furthermore, one or more computer programs can be created to cause hardware such as a CPU, ROM, and RAM, which are built into the content generation device 20 or the distribution switching device 30 described above, to perform functions of the content generation device 20 or the distribution switching device 30. A computer-readable storage medium is also provided to store the one or more computer programs.

[0098] Furthermore, the effects disclosed in this description are merely illustrative or exemplary, but not limiting. That is to say, the technology according to this disclosure can achieve other effects, in addition to or instead of those described above, which will be obvious to a person skilled in the art from the description in this statement.

[0099] It should be noted that the present technology can also have the following configurations. (1) Information processing device comprising a control unit which performs a control operation which analyzes a captured image which has been taken by one or more imaging devices which map a target space, identifies one or more subjects as cut-out targets from the captured image and cuts out a specific subject. (2) Information processing device according to point (1) wherein the control unit performs the cutting in an area which includes at least one face of the subject. (3) Information processing device according to point (2) wherein the control unit preferably determines a motif that satisfies a predetermined condition as a cut-out target. (4) Information processing device according to point (3) wherein the control unit determines a singing motif as a cut-out target as a motif that satisfies the predetermined condition. (5) Information processing device according to point (3) wherein the control unit determines a motif located in a region of interest as a cut-out target as a motif that satisfies the predetermined condition. (6) Information processing device according to point (3) wherein the control unit determines a motif located in the center of a stage, which is the target space as a cut-out target, as a motif that satisfies the predetermined condition. (7) Information processing device according to point (1) wherein the control unit determines a fixed position on a stage as a cut-out target in a case where a number of motifs is less than a predetermined number of cut-outs. (8) Information processing device according to any of points (2) to (7) wherein the control unit performs the cutting by a number of cutouts corresponding to a number of outputs of an image. (9) Information processing device according to any of points (2) to (8) wherein the control unit performs a limitation in an area having one motif or a limitation in an area having a plurality of motifs. (10) Information processing device according to point (9) wherein the control unit performs the cutting in an area which includes a predetermined border over a top part of a body of a motif as the cutting target. (11) Information processing device according to point (10) wherein the control unit further performs a cut-out in an area which includes a border in an eye line direction when an eye line of a subject is directed to the left and right as the cut-out target. (12) Information processing device according to any of points (2) to (11) wherein the control unit also makes a cut-out in an area which includes at least one hand of the subject. (13) Information processing device according to any of points (1) to (12) wherein the control unit performs cutting in an area having a plurality of motifs and cutting in an area having one motif from the plurality of motifs. (14) Information processing device according to any of points (1) to (13) wherein the control unit performs the cutting in an area which includes one or more motifs in a predetermined area on a stage. (15) Information processing device according to any of points (1) to (14) wherein the captured image is a plurality of captured images with partially overlapping viewpoints, captured by a plurality of imaging devices arranged on a seating side of a stage, and the control unit arranges and displays the plurality of captured images side by side in a partially overlapping state and receives the setting of an overlap position. (16) Information processing device according to point (15) wherein the control unit performs a control for outputting a plurality of cut-out images obtained by cutting to a device that switches a distribution image and performs a control for displaying the plurality of cut-out images on a display unit together with the plurality of captured images arranged side by side. (17) Information processing device according to point (15) or (16) wherein, in a case where a subject as the cropping target moves from a first captured image to a second captured image of the plurality of adjacent captured images, the control unit switches a captured image of a cropping source in a section in which the first captured image and the second captured image overlap. (18) Information processing procedure performed by a processor which includes: performing a control operation which analyzes an image acquired by one or more imaging devices mapping a target space, identifies one or more subjects as cut-out targets from the acquired image and cuts out a specific subject. (19) Program that causes a computer to function as follows: a control unit that performs a control operation which analyzes an image captured by one or more imaging devices mapping a target space, identifies one or more subjects as cut-out targets from the captured image, and cuts out a specific subject. LIST OF REFERENCE MARKS 10 Camera (imaging device) 20 Content generation device 210 Communication unit 220 control unit 221 Display position adjustment unit 222 Cutout processing unit 223 Output control unit 230 Operating input unit 240 display unit 250 storage units 30 Distribution switching device 310 Communication unit 320 control unit 321 Switching unit 322 Distribution control unit 330 Operating input unit 340 display unit 350 storage units QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] WO 2018 / 173876 A

[0004]

Claims

[1] Information processing device comprising: a control unit that performs a control that analyses an image captured by one or more imaging devices that image a target space, determines one or more subjects as clipping targets from the captured image, and cuts out a specific motif. [2] The information processing apparatus according to claim 1, wherein the control unit performs the cutting in an area including at least one face of the subject. [3] The information processing apparatus according to claim 2, wherein the control unit preferably determines a subject satisfying a predetermined condition as a cutting target. [4] The information processing apparatus according to claim 3, wherein the control unit determines a singing motif as a cutout target as a motif satisfying the predetermined condition. [5] The information processing apparatus according to claim 3, wherein the control unit determines a subject located in a region of interest as a clipping target as a subject satisfying the predetermined condition. [6] The information processing apparatus according to claim 3, wherein the control unit determines a subject positioned at the center of a stage constituting the target space as a cutout target as a subject satisfying the predetermined condition. [7] The information processing apparatus according to claim 1, wherein the control unit determines a fixed position on a stage as a cutting target in a case where a number of motifs is less than a predetermined number of cutouts. [8] The information processing apparatus according to claim 2, wherein the control unit performs the cutting by a number of cutouts corresponding to a number of outputs of an image. [9] The information processing apparatus according to claim 2, wherein the control unit performs the cutting out in an area including one motif or the cutting out in an area including a plurality of motifs. [10] The information processing apparatus according to claim 9, wherein the control unit performs the cutting out in a range including a predetermined edge above an uppermost part of a body of a subject as a cutting out target. [11] The information processing apparatus according to claim 10, wherein the control unit further performs the clipping in a range including an edge in an eye line direction when an eye line of a subject as the clipping target is directed left and right. [12] The information processing apparatus according to claim 2, wherein the control unit further cuts out an area including at least one hand of the subject. [13] The information processing apparatus according to claim 1, wherein the control unit performs the cutting in an area having a plurality of motifs and the cutting in an area having one motif among the plurality of motifs. [14] The information processing apparatus according to claim 1, wherein the control unit performs the cutting in an area including one or more motifs in a predetermined area on a stage. [15] Information processing device according to claim 1, wherein the captured image is a plurality of captured images with partially overlapping view angles captured by a plurality of imaging devices arranged on a seat side of a stage, and the control unit arranges and displays the plurality of captured images side by side in a partially overlapped state, and receives the setting of an overlapping position. [16] The information processing apparatus according to claim 15, wherein the control unit performs control for outputting a plurality of cut-out images obtained by cutting to a distribution image switching device, and performs control for displaying the plurality of cut-out images on a display unit together with the plurality of captured images arranged side by side. [17] The information processing apparatus according to claim 15, wherein, in a case where a subject as a clipping target moves from a first captured image to a second captured image of the plurality of juxtaposed captured images, the control unit switches a captured image of a clipping source at a portion where the first captured image and the second captured image overlap. [18] Information processing method performed by a processor and comprising: Performing a control that analyzes an image captured by one or more imaging devices imaging a target space, determines one or more subjects as clipping targets from the captured image, and clips a specific subject. [19] A program that causes a computer to act as: a control unit that performs control that analyzes an image captured by one or more imaging devices that image a target space, determines one or more subjects as clipping targets from the captured image, and clips a specific subject.

Images