Control device

Abstract

To solve a problem in which camera switching for television broadcasts is still done under the direction of a person in charge called a director, such that automation of the switching process has not been considered.SOLUTION: A control device includes a detection unit that is accessible to a plurality of first cameras that capture images of a first subject in a first area within a facility and a plurality of second cameras that capture images of a second subject in a second area within the facility, and detects the gaze direction of a group of spectators within the first subject on the basis of video data of the first subject captured by any of the plurality of first cameras, an identification unit that identifies a gaze area within the second area of the group of spectators on the basis of the gaze direction of the group of spectators and identifies a gaze target of the group of spectators present within the gaze area on the basis of video data of the second subject captured by each of the plurality of second cameras, a determination unit that determines a specific second camera from the plurality of second cameras that is to be the source of the video data on the basis of the gaze target, and a transmission unit that transmits the video data from the specific second camera.SELECTED DRAWING: Figure 1

Description

Incorporation by Reference

[0001] This application is a Japanese patent application filed on October 17, 2017. Priority is claimed from US Pat. No. 6,201,190, the contents of which are incorporated herein by reference. Enter. [Technical Field]

[0002] The present invention relates to a control device, a control system, and a control program. [Background technology]

[0003] Patent Document 1 describes an automatic camera control system used for filming sports events, etc. However, the patent document In Reference 1, the camera for shooting the competition is switched by a switching command from the operator. However, camera switching for television broadcasting is still done by the direction of a person in charge called a director. Therefore, automation of the switchover is not considered. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-5208 Summary of the Invention

[0005] A control device according to one aspect of the technology disclosed in the present application includes: a plurality of first cameras for photographing a body; and a plurality of second cameras for photographing a second subject in a second area within the facility. a control device accessible to the second camera, and Based on the number of spectators in the first subject, a specific one of the plurality of first cameras is selected. a selection unit for selecting a first camera; and a specific first subject photographed by the specific first camera. Detecting the gaze direction of a specific group of spectators within the specific first subject based on the image data of the subject. and a detection unit for detecting the direction of gaze of a specific group of spectators based on the direction of gaze of the specific group of spectators detected by the detection unit. A gaze area of ​​a specific spectator group within the second area is identified, and the gaze area is captured by each of the plurality of second cameras. Based on the video data of the second subject captured by the camera, the specific subject that exists within the gaze area is detected. A specifying unit that specifies a gaze target of a group of spectators, and a and determining a specific second camera from among the plurality of second cameras as a transmission source of the video data. a determination unit for determining whether the second camera is a second camera, and a determination unit for determining whether the second camera is a second camera. and a transmitting unit.

[0006] A control device according to another aspect of the technology disclosed in the present application includes: a first camera for photographing a subject, and a second camera for photographing a second subject in a second area within the facility. and a control device accessible to the first object captured by the first camera. a detection unit that detects the gaze direction of the spectators in the first subject based on video data of the subject; Based on the gaze direction of the spectators detected by the detection unit, a specifying unit for specifying a gaze area within the second area, and a second area based on the result of the specification by the specifying unit. and a control unit that controls the shooting of the camera.

[0007] A control system according to one aspect of the technology disclosed in the present application includes a first area in a facility. a plurality of first cameras for photographing a subject, and a plurality of cameras for photographing a second subject in a second area within the facility; a plurality of second cameras, the plurality of first cameras, and the plurality of second cameras; and a control device configured to control the plurality of first cameras. based on the image data of the first object captured by any one of the first cameras a detection unit for detecting the direction of gaze of the spectators in the first subject; and identifying a gaze area of ​​the spectator group within the second area based on the gaze direction of the spectator group. based on the video data of the second object captured by each of the plurality of second cameras, a specifying unit for specifying a gaze target of the spectators present in the gaze area; Based on the gaze target identified by the above, a source of the video data is identified from among the plurality of second cameras. a determination unit that determines a specific second camera; and a transmitting unit that transmits the video data from the camera.

[0008] A control program according to one aspect of the technology disclosed in the present application is a program for controlling a first area of ​​a facility. a plurality of first cameras for photographing one subject and a plurality of cameras for photographing a second subject in a second area within the facility; a processor accessible to the plurality of second cameras, the processor being configured to detect any of the plurality of first cameras; based on video data of the first subject captured by any one of the first cameras, A detection process for detecting the gaze direction of the spectators in the subject, and a front image detected by the detection process. Identifying a gaze area of ​​the spectator group within the second area based on the gaze direction of the spectator group; The note is based on image data of the second object photographed by each of the plurality of second cameras. A process of identifying a target of gaze of the group of spectators present within a viewing area; Based on the identified gaze target, a source of video data is selected from the plurality of second cameras. a determination process for determining a specific second camera to be used; and A transmission process for transmitting video data from the camera is executed.

[0009] The control program according to another aspect of the technology disclosed in the present application is a program for controlling a first area in a facility. a first camera for photographing a first subject and a second camera for photographing a second subject in a second area within the facility; a processor accessible to the first camera, the first image captured by the first camera; A detection process for detecting the gaze direction of the spectators in the first object based on the image data of the object. and based on the gaze direction of the spectators detected by the detection process, A determination process for determining a gaze area within the second area, and based on the determination result of the determination process, and a control process for controlling the second camera. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of the configuration of a control system. [Figure 2] FIG. 2 is a block diagram illustrating an example of the hardware configuration of the control device. [Figure 3] FIG. 3 is a block diagram illustrating an example of a functional configuration of the control device. [Figure 4] FIG. 4 is an explanatory diagram showing an example of a group of spectators photographed by a spectator camera. [Figure 5] FIG. 5 is an explanatory diagram showing Example 1 of sensitivity to dynamic gaze directions of spectators. [Figure 6] FIG. 6 is an explanatory diagram showing Example 2 of sensitivity to the dynamic gaze direction of an audience member. [Figure 7] FIG. 7 is an explanatory diagram showing Example 3 of sensitivity to the dynamic gaze direction of an audience member. [Figure 8] FIG. 8 is an explanatory diagram showing an example of detecting the gaze direction of a group of spectators. [Figure 9]FIG. 9 is an explanatory diagram showing an example of detecting a motion vector based on a change in the position of the ear. [Figure 10] FIG. 10 is an explanatory diagram showing an example of detecting the gaze direction of a particular spectator. [Figure 11] FIG. 11 is an explanatory diagram showing example 1 of identification of a gaze area and a gaze target by an identification unit. [Figure 12] FIG. 12 is an explanatory diagram showing a second example of specification of a gaze area and a gaze target by the specification unit. [Figure 13] FIG. 13 is an explanatory diagram showing an example of calculation of the area of ​​the fixation region. [Figure 14] FIG. 14 is a flowchart illustrating an example of a control processing procedure performed by the control device. [Figure 15] FIG. 15 is a flowchart showing a detailed example 1 of the processing procedure of the first spectator camera selection process (step S1401) by the selection unit shown in FIG. [Figure 16] FIG. 16 is a flowchart showing a detailed processing procedure example 2 of the first spectator camera selection process (step S1401) by the selection unit shown in FIG. [Figure 17] FIG. 17 is a flowchart showing a detailed example 1 of the processing procedure of the second spectator camera selection processing (step S1504) shown in FIG. [Figure 18] FIG. 18 is an explanatory diagram showing a detailed processing procedure example 2 of the second spectator camera selection process (step S1504) shown in FIG. [Figure 19] FIG. 19 is a flowchart showing a detailed processing procedure example 2 of the second spectator camera selection process (step S1504) shown in FIG. [Figure 20] FIG. 20 is a flowchart illustrating a detailed example of the procedure of the identification process (step S1403) illustrated in FIG. [Figure 21] FIG. 21 is an explanatory diagram of the automatic shutter function of the field camera. [Figure 22] FIG. 22 is an explanatory diagram showing an example of application to marketing. DETAILED DESCRIPTION OF THE INVENTION

[0011] <Control system configuration example> FIG. 1 is an explanatory diagram showing an example of the configuration of a control system. The scenes in the video data broadcast on TV or the Internet are automatically adjusted to match the audience's line of sight. This control system 1 is a system that dynamically switches between the various A facility is a structure or place where spectators can view an object of interest. Specifically, for example, stadiums, arenas, event halls, gymnasiums, sports fields, In this embodiment, the stadium 100 is a soccer stadium. This will be explained using an example.

[0012] The facility has a first area where spectators are present and a second area where a gaze target is present. In the case of the stadium 100, the first area is the spectator seats 101 and the second area is the field 10. 2. The gaze target is the person or thing that the spectators are looking at. For example, in the case of soccer, If we are talking about people, it would be the players, coaches, or referees, and if we are talking about objects, it would be the ball.

[0013] The control system 1 includes a plurality of spectator cameras AC, a plurality of field cameras FC, and a control device. The spectator camera AC captures the first area, spectator seats 10 The field camera FC is a camera that takes video of the stadium 10 as its subject. The field camera FC is a camera that takes moving images of a field 102 of the same type as the camera FC. The camera may automatically track the subject, or may be operated by the photographer.

[0014] The video data from the spectator cameras AC and the field cameras FC is sent to the control device 110. The video data from the field camera FC is transmitted to televisions in ordinary homes via a radio tower. 103, or via the Internet to a personal computer 104 (smartphone) It will be distributed to all devices (including smartphones and tablets).

[0015] The control device 110 controls a plurality of spectator cameras AC, a plurality of field cameras FC, and a data The database 120 is a computer that can communicate with the facility. The design information includes spectator seat information 121, field information 122, spectator camera information 123, The database 120 also stores player information 125 and The database 120 also stores characteristic person information 126. The database 120 is accessible from the control device 110. If possible, it may be provided outside the control system 1.

[0016] The spectator seat information 121 is represented by a local coordinate system 10 consisting of orthogonal X-axis, Y-axis, and Z-axis. This is information indicating the three-dimensional position of the spectator seats 101 in the facility. The plane formed is a plane parallel to the field 102, and the Z axis is the height from the field 102. Indicates the quality.

[0017] The field information 122 is information indicating the three-dimensional position of the field 102 in the facility. The audience camera information 123 indicates the three-dimensional position and shooting conditions of the audience camera AC in the facility. The shooting conditions include, for example, the shooting direction, angle of view, and magnification. The camera information 124 is information indicating the three-dimensional position and shooting conditions of the field camera FC in the facility. is.

[0018] Player information 125 is information that indicates the player's team, uniform color and pattern, and jersey number. The characteristic person information 126 may also include face image data of the player. Specifically, for example, the face image data and clothing image data of the person in question are information indicating the Includes.

[0019] Here, the operation of the control system 1 will be described.

[0020] (1-1) Field Camera FC takes pictures of the players and the ball on the field 102 and creates video data. Get the data. (1-2) Spectator camera AC photographs the crowd of spectators in the spectator seats 101 and acquires video data.

[0021] (2-1) The field camera FC transmits the acquired video data to the control device 110. (2-2) The spectator camera AC transmits the acquired video data to the control device 110.

[0022] (3) The control device 110 uses the acquired video data to determine the field of view that the spectators are watching. The area of ​​the frame 102 (hereinafter referred to as the gaze area) is identified, and the gaze object (a player or a ball) existing in the gaze area is detected. The database 120 is referenced to identify the target object.

[0023] (4) The control device 110 controls the field cameras FC that capture the identified gaze target to be in a plurality of fields. Specifically, the control device 110 determines the gaze area from the field camera FC. and the positional relationship and distance between the field camera FC and the same field camera FC, and the continuous broadcast time The destination is determined based on criteria such as whether the player's face can be recognized or not, The optimal field camera FC is determined.

[0024] This allows the video data from the selected field camera FC to be transmitted to the TV 103 or PC. The image data is broadcast or distributed to the personal computer 104. It is possible to automate the switching of C (real-time control).

[0025] <Example of hardware configuration of control device 110> FIG. 2 is a block diagram showing an example of the hardware configuration of the control device 110. 0 includes a processor 201, a storage device 202, an input device 203, and an output device The processor 201 includes a processor 204 and a communication interface (communication IF) 205. , a storage device 202, an input device 203, an output device 204, and a communication IF 20 5 are connected by a bus 206.

[0026] The processor 201 controls the control device 110. The storage device 202 is The storage device 202 stores various programs and data. The storage device 202 is a non-transitory or temporary recording medium for storing data. For example, ROM (Read Only Memory), RAM (Random Access s Memory), HDD (Hard Disk Drive), and flash memory be.

[0027] The input device 203 inputs data. Examples of the input device 203 include: The output device 204 includes a keyboard, a mouse, a touch panel, a numeric keypad, and a scanner. The output device 204 may be, for example, a display or a printer. There is.

[0028] The communication IF205 receives video data from the spectator cameras ACs and the field cameras FCs. The communication IF 205 is connected to the Internet 200 and a radio tower 210. The video data received from the spectator cameras ACs and the field cameras FCs is then personalised. The image data is transmitted to the central computer 104 and the television 103.

[0029] <Example of functional configuration of control device 110> 3 is a block diagram showing an example of the functional configuration of the control device 110. The control device 110 A detection unit 301, a specification unit 302, a determination unit 303, a transmission unit 304, and a selection unit 305. Specifically, the detection unit 301 to the selection unit 305 are, for example, the storage device shown in FIG. This is realized by having the processor 201 execute a program stored in the device 202. This is a function that can be used.

[0030] The detection unit 301 detects an object captured by one of the spectator cameras AC among the spectator cameras ACs. Based on the captured video data of the first subject, the direction of the gaze of the spectators in the first subject is detected. The gaze direction of a group of spectators is the direction indicated by the resultant vector of the gaze direction of each spectator in the group. be.

[0031] That is, when the scalar value of the composite vector is equal to or greater than a predetermined value, the majority of the spectators are composite vectors. The audience is considered to be looking in the direction of the vector and gazing at the field 102. One method for detecting the gaze direction of an audience member is to detect the eye area. Detects the gaze direction by detecting the position of the iris in the eye area from the video data. The detection unit 301 may also detect the direction of the audience's gaze based on the direction of the audience's face. .

[0032] The identification unit 302 identifies the audience based on the gaze direction of the audience detected by the detection unit 301. Specifically, for example, the identification unit 302 identifies a gaze area within the second area of ​​the audience group. The gaze area on the field 102 that is the direction of the gaze of the person is determined based on the seating information in the database 120. The information 121 and the field information 122 are used to identify the seating of the spectators. The area of ​​the field 102 that is the end of the line of sight from the position as the starting point is defined as the gaze area. The gaze area is an area that includes the point where the extension of the line of sight intersects with the field 102. .

[0033] The specifying unit 302 also determines whether each field camera FC in the field camera group FCs Based on the video data of the captured second subject, the target of attention of the spectators present in the gaze area is determined. Specifically, for example, the specifying unit 302 specifies the image captured by the field camera FC. The gaze target in the gaze area is identified from the video data of the field camera FC.

[0034] In addition, when the spectator camera AC is a so-called spherical camera, the spectator camera AC The image captures the audience seats 101 and the field 102 on the opposite side of the audience seats 101. The control device 110 uses the video data from the audience camera AC to determine the direction of the audience's gaze. This directly identifies the gaze target that exists on the extension of the gaze direction. There is no need to use the image data from the FC camera, and the accuracy of identifying the gaze target can be improved and processed. Speed ​​can be improved.

[0035] The determination unit 303 determines the field color based on the gaze target identified by the identification unit 302. A specific field camera FC is selected from the camera group FCs as the source of video data. Specifically, for example, the determining unit 303 determines the positional relationship between the gaze area and the field camera FC. Whether the player's position, distance, and the continuous broadcast time of the same field camera FC exceeded the threshold, The optimal field camera FC to switch to is selected based on criteria such as whether or not the face can be recognized. Determine.

[0036] This allows the video data from the selected field camera FC to be transmitted to the TV 103 or PC. The image data is broadcast or distributed to the personal computer 104. It is possible to automate the switching of C.

[0037] The transmitting unit 304 receives the signal from the specific field camera FC determined by the determining unit 303. Specifically, for example, the transmitting unit 304 transmits the video data using a communication IF. via a radio tower 210 to a television 103 in a general household, or via the Internet 200 Personal computers 104 (including smartphones and tablets) can be used to Transmits video data from MelaFC.

[0038] Prior to detecting the gaze direction, the selection unit 305 selects a specific spectator from the spectator camera group ACs. Select the camera AC. Specifically, for example, the selection unit 305 selects a plurality of spectator cameras AC. Based on the scores indicating the number of spectators in the first object photographed by each of the spectators, Select a specific spectator camera AC from the audience cameras AC. The score is calculated based on the number of people in the audience. Specifically, for example, the area of ​​the image data of the audience within one frame of the video data is The higher the score, the larger the audience.

[0039] The selection of the audience camera AC by the selection unit 305 is performed by the detection unit 301 after detecting the view of a large number of audience members. This is to detect the line direction. Therefore, a predetermined number or more of the spectator seats 101 are set as the first subject. The shooting conditions of the spectator camera AC are set so that the object fits within the angle of view of the spectator camera AC. The detection unit 301 detects the image captured by the spectator camera AC selected by the selection unit 305. and detecting the gaze direction of the spectators in the first subject based on the acquired video data of the first subject. This will happen.

[0040] The selection unit 305 also selects a specific audience camera based on the positional relationship between the audience group and the gaze area. Specifically, for example, the selection unit 305 selects an AC from among a plurality of spectator cameras AC. , and selects a spectator camera AC that captures a group of spectators close to the gaze area as a specific spectator camera AC. The selection will be described in detail later with reference to FIGS.

[0041] The selection unit 305 also selects spectator cameras AC with scores equal to or greater than a predetermined threshold value as specific observations. Select it as the audience camera AC. The higher the score of the audience camera AC, the more audience members are photographed. Therefore, it is suitable for detecting the gaze direction of a group of spectators. 05 has the highest absolute score among multiple spectator camera ACs, i.e., the score is above the threshold. Select the spectator camera AC that is equal to or greater than the value.

[0042] If there are multiple spectator cameras AC whose scores are equal to or greater than a predetermined threshold, the selection unit 30 5 may select the spectator camera AC with the highest score. Spectator cameras that satisfy the conditions for a specific spectator camera AC in terms of the positional relationship between the camera and the gaze area. May be limited to AC.

[0043] The selection unit 305 also selects a relatively high score from the scores of the multiple spectator cameras AC. Select the spectator camera AC as a specific spectator camera AC. The higher the score, the more likely the spectator camera AC is to be selected. Since it captures a larger number of spectators, it is suitable for detecting the gaze direction of a crowd of spectators. Therefore, the selection unit 305 selects the spectator camera AC with a relatively high score. Just select Customer Camera AC.

[0044] If there are multiple spectator cameras AC with relatively high scores, the selection unit 305 selects one of them. The spectator camera AC with the highest score may be selected. Spectator camera AC that satisfies the conditions as a specific spectator camera AC in terms of its position relative to the area May be restricted.

[0045] The selection unit 305 also selects spectator cameras AC whose scores are less than a predetermined threshold as specific observations. The higher the score of the spectator camera AC, the more spectators there are. Since the system captures the gaze direction of the crowd, it is suitable for detecting the gaze direction of the crowd. If not present, a spectator camera AC with a score below the threshold is selected. If there are multiple spectator cameras AC whose values ​​are less than the threshold value, the selection unit 305 selects the highest value. You may also select the score spectator camera AC.

[0046] In addition, the positional relationship between the spectator seats 101 and the gaze area is such that a specific spectator camera AC The selection unit 305 may limit the number of spectator cameras AC to those that satisfy the conditions. Select spectator cameras AC whose scores are above the threshold, and randomly or periodically It is also possible to select a spectator camera AC that is below the threshold. This allows for consideration of the gaze direction of the minority. For example, the field camera FC can be switched on to capture scenes that the majority of people might miss. It can be replaced.

[0047] The selection unit 305 also selects an observer with a relatively low score among the scores of the multiple spectator cameras AC. Select the customer camera AC as the specific spectator camera AC. The spectator camera with a relatively low score If there are multiple ACs, the selection unit 305 selects the spectator camera AC with the lowest score. In addition, a specific spectator camera may be selected based on the positional relationship between the spectator seats 101 and the gaze area. It may be limited to spectator camera ACs that meet the conditions for a spectator AC.

[0048] Also, the selection unit 305 usually selects the spectator camera AC with the lowest score, Spectator cameras AC with relatively low scores may be selected periodically. It also takes into account the gaze direction of the minority, for example, to highlight scenes that the majority may miss. You can switch between the field cameras.

[0049] In addition, spectator camera AC with a score below the threshold or relatively low is a specific spectator camera A. When the camera AC is selected as the target camera, the detection unit 301 detects the target camera of the audience captured by the camera AC. Instead of the gaze direction, the gaze direction of a specific spectator may be detected. If it is less than or relatively low, the number of spectators is small, so the gaze direction of the spectators is Therefore, in such a case, the detection unit 30 1. Narrow down the audience by removing noise.

[0050] Here, noise refers to noise generated while using a smartphone, eating or drinking (both of which are facing downwards), This is a spectator who is not paying attention, such as a spectator who is looking in the direction of the electronic scoreboard. 301 removes the spectators with such gaze directions and selects the gaze directions of specific spectators from the remaining spectators. Furthermore, the detection unit 301 identifies distinctive people by referring to the database 120. It is also possible to select the person as an audience member and detect their line of sight.

[0051] In addition, spectator camera AC with a score below the threshold or relatively low is a specific spectator camera A. C, the detection unit 301 detects the specific filter determined by the determination unit 303. Based on the shooting direction of the field camera FC and the line of sight direction of each of the specific spectators, The direction of the audience's gaze may also be detected.

[0052] Video data from a specific field camera FC is transmitted to a television 103 for viewing by viewers. Therefore, the shooting direction of a specific field camera FC is the same as the line of sight of the viewer of the television 103. Therefore, the detection unit 301 detects the positions of the individual spectators in the group of spectators located in the opposite direction to the shooting direction. The gaze direction may be detected, and the gaze direction of a particular spectator may be detected from the detected gaze direction.

[0053] In this case, the gaze direction of a particular spectator is the same as the shooting direction or a direction that is close to it within an acceptable range. This allows the television 103 viewer and the audience to watch the video. In addition, the gaze direction of a particular spectator can be measured even if it is outside the allowable range of the shooting direction. This allows viewers of the television 103 to feel a sense of fixation on a target that they cannot fixate on. You can switch between the camera FC.

[0054] In addition, the control device 110 selects the field camera group FCs that has the most past broadcast data. Alternatively, the camera may be switched to a field camera FC that captures similar image data. At this time, the control device 110 determines the vector connecting the audience camera AC and the subject in question in the direction of the audience's line of sight. Based on this gaze direction detection, the detection accuracy can be improved.

[0055] The transmission unit 304 also transmits still image data captured by the field camera FC to the field camera FC. The communication terminal 300 of at least one spectator A among the spectators who are the source of the switching of the digital camera FC. In this case, the control device 110 may transmit the data to a The address information (for example, email address) of the communication terminal 300 of the spectator is stored in the base. The still image data is the data taken as a still image by the field camera FC. The data may be data extracted from video data.

[0056] In addition, the transmission unit 304 transmits a still image data indicating a storage destination of the still image data, rather than the still image data itself. Access information (for example, a URL) may be transmitted to the communication terminal 300. Field Immediately after switching the camera FC, a decisive scene may occur, so the still image data at that time Alternatively, access information can be provided to spectators who may have been watching the scene. In such a scene, the audience is not taking pictures but is instead gazing at the object of their gaze, so the audience is stationary. Even if you are not in a situation where you can take still image data, you can still take still image data of the object you are looking at at that time. It can be obtained.

[0057] In the above-described embodiment, the control device 110 switches between a plurality of field cameras FC. We have explained the above, but we can also calculate the gaze area R using the video data of one or more spectator cameras AC. Alternatively, a target to be watched within the gaze area R is identified and one field camera FC is controlled. The control unit that controls the field camera FC may, for example, Or, control to point the field camera FC in the direction of the gaze target within the specified gaze area R. Execute.

[0058] <Audience members photographed by the audience camera AC> FIG. 4 is an explanatory diagram showing an example of a group of spectators photographed by a spectator camera AC. The spectator cameras AC1 and AC2 are set under the same shooting conditions. The audience camera AC1 photographs the audience 401 in the audience seats 101, and the audience camera AC2 photographs the audience 402 in the audience seats 101. The number of people in the audience groups 401 and 402 is reflected in the score in the selection unit 305 described above. It is determined by calculating

[0059] In the example of FIG. 4, the score of the audience group 401 is equal to or higher than the threshold value or is a relatively high score. The detection unit 301 calculates a composite vector of the gaze direction vectors of each spectator in the spectator group 401 as follows: The score of the audience group 402 is detected as the gaze direction of the audience group 401. Or, the score is relatively low, and the detection unit 301 detects the gaze direction of a particular spectator in the spectator group 401. Detects the direction.

[0060] <Positional relationship between spectators and athletes> Next, the positional relationship between spectators and players will be explained. 2. The spectators can see the objects on the field 102 (players and balls) When tracking the gaze of a person (a person in a room), the control device 110 tracks the dynamic gaze direction and depth of the audience. It is difficult to track in the direction perpendicular to the line of sight, and it is easy to track in the left and right directions or in front of the camera. Therefore, the sensitivity of the dynamic gaze direction and tracking of the depth of the audience is It has poor sensitivity, but good sensitivity for tracking in the left and right directions perpendicular to the line of sight and in the front.

[0061] In the following Figures 5 to 7, we will explain using one audience member as an example, but the same applies to multiple audience members. The same is true for a group of spectators.

[0062] FIG. 5 is an explanatory diagram showing Example 1 of sensitivity to dynamic gaze directions of spectators. Among the spectator seats 101A to 101D, the spectator 500A is located in the spectator seat 101A. 00B is located in spectator seat 101B. Spectator seat 101A faces spectator seat 101B.

[0063] In addition, in the field 102, the area where the player P moves in the X direction is designated as area 102A. The area 102A is closer to the spectator seat 101A than the spectator seat 101B. Area 102B is closer to spectator seat 101B than spectator seat 101A. .

[0064] In FIG. 5, the maximum viewing angle of spectator 500A when player P is in area 102A is is θA, and the maximum viewing angle of spectator 500A when player P is in area 102B is θ Therefore, spectator 500A thinks that player P is in area 102B rather than area 102B. When in A102A, it is easier to see the difference in line of sight due to player P's movement in the X direction. This allows the detection unit 301 to detect the line of sight of the audience member 500A with high sensitivity. .

[0065] Similarly, spectator 500B prefers when player P is in area 102B rather than area 102A. However, it is easy to detect the difference in the line of sight caused by the movement of the player P in the X direction. 1 can detect the gaze direction of the spectator 500A with high sensitivity. In both 500 and 500B, it is difficult to obtain the difference in the line of sight due to the movement of player P in the Y direction. That is, the detection sensitivity in the gaze direction is lower than in the X direction.

[0066] Therefore, when the player P is in the area 102A, the selection unit 305 selects the spectator seats 101A. It is preferable to select the spectator camera AC-A to shoot, and when the player P is in the area 102B, In this case, it is preferable that the selection unit 305 selects the spectator camera AC-B that photographs the spectator seats 101B. It's nice.

[0067] FIG. 6 is an explanatory diagram showing Example 2 of sensitivity to dynamic gaze directions of spectators. Among the spectator seats 101A to 101D, spectator 500C is located in spectator seat 101C. 00D is located in the spectator seat 101D. Also, the spectator seat 101C faces the spectator seat 101D. do.

[0068] In addition, in the field 102, the area where the player P moves in the Y direction is designated as area 102C. Area 102C is closer to spectator seat 101C than spectator seat 101D. Area 102D is closer to spectator seat 101D than spectator seat 101C. .

[0069] In FIG. 6, the maximum viewing angle of the spectator 500C when the player P is in the area 102C is is θC, and the maximum viewing angle of spectator 500C when player P is in area 102D is θ Therefore, the spectator 500C is more likely to see that the player P is in area D than in area 102D. When in A102C, it is easier to see the difference in line of sight due to player P's movement in the Y direction. This allows the detection unit 301 to detect the line of sight of the spectator 500C with high sensitivity. .

[0070] Similarly, spectator 500D prefers when player P is in area 102D rather than area 102C. However, it is easy to detect the difference in the line of sight caused by the movement of the player P in the Y direction. 1 can detect the gaze direction of the spectator 500C with high sensitivity. In both the 500D and 500D, it is difficult to obtain the difference in the line of sight caused by the movement of player P in the X direction. That is, the detection sensitivity of the gaze direction is lower than that of the Y direction.

[0071] Therefore, when the player P is in the area 102C, the selection unit 305 selects the spectator seats 101C. It is preferable to select the spectator camera AC-C to shoot, and when the player P is in the area 102D, In this case, it is preferable that the selection unit 305 selects the spectator camera AC-D that photographs the spectator seats 101D. It's nice.

[0072] FIG. 7 is an explanatory diagram showing Sensitivity Example 3 of the dynamic gaze direction of the audience. Sensitivity Example 3 is a diagram showing Sensitivity Example 1. This is an example of combining area 102A and area 10C. The overlapping area between Area 102A and Area 10D is Area 102AD. The overlapping area between Area 102B and Area 10C is Area 102BC, and Area 10 The overlapping area between 2B and area 102D is area 102BD.

[0073] When the player P is in the area 102AC, the selection unit 305 selects an observer who takes a picture of the spectator seats 101A. It is recommended to select the audience camera AC-A or the audience camera AC-C that captures the audience seat 101C. Preferably, when the player P is in the area 102AD, the selection unit 305 takes a picture of the spectator seats 101A. Select the spectator camera AC-A that will capture the audience seating, or the spectator camera AC-D that will capture the audience seating 101D. It is preferable that:

[0074] <Direction of gaze of the audience> FIG. 8 is an explanatory diagram showing an example of detecting the gaze direction of an audience group 401. The detection unit 301 detects the gaze direction of each individual The motion vector mv of the audience is detected, and the composite motion vector MV of each motion vector mv is calculated as the audience motion vector mv. The detection unit 301 generates the gaze direction of the iris in the eye region of the spectator. The movement vector mv may be detected as the gaze direction due to the change in position, and the head of the audience may be detected as the face pattern. Even if the movement vector mv is detected as the gaze direction due to a change in the position of an object (for example, an ear), good.

[0075] The detection unit 301 may detect the head direction of each spectator and identify the gaze area. In head direction detection, the detection unit 301 detects the head and its direction from the image data. For example, the detection unit 301 uses a deep learning algorithm to detect the head orientation. Head orientation (3 degrees of freedom: Yaw, Roll, Pitch) is learned from facial images. can be detected.

[0076] FIG. 9 is an explanatory diagram showing an example of detecting a motion vector mv due to a change in the position of the ear. If the customer camera AC is a spherical camera, distortion occurs at both ends of each frame of the video data. Therefore, by expanding both ends of the frame, the amount of change in the position of the audience's ears increases. , the motion vector detection unit 301 can more easily detect the motion vector mv.

[0077] FIG. 10 is an explanatory diagram showing an example of detecting the gaze direction of a specific spectator. As shown in the figure, the score indicating the number of people in the audience group 402 is below the threshold or is relatively low, so the selection The selection unit 305 selects a specific audience member. In FIG. 10, audience member A is selected from audience members A1 to A8. is excluded as noise because it is looking at smartphone 1000.

[0078] The direction of the gaze of the audience member A2 is, for example, the direction of the electronic scoreboard, so the noise The location of the electronic signboard is obtained from the facility design information. The selection unit 305 selects an arbitrary spectator from among the spectators A3 to A8 as a specific spectator. If there is a spectator who matches the characteristic person information 126 among the spectators A3 to A8 (for example, A8), spectator A8 may be selected as a particular spectator.

[0079] <Examples of identifying gaze areas and gaze targets> FIG. 11 is an explanatory diagram showing Example 1 of specifying the gaze area and gaze target by the specifying unit 302. The virtual plane VS is a plane parallel to the XZ plane. The spectator camera AC is located on the virtual plane VS. The determination unit 302 calculates the resultant motion vector MV from the virtual plane VS toward the field 102. The image is projected onto the field 102 to identify the gaze area R. 2 is a field camera FC-A to FC-D that are currently capturing an image of the gaze area R. It is identified from the shooting directions PD-A to PD-D of the cameras FC-A to FC-D.

[0080] Here, the specifying unit 302 determines whether the imaging direction PD-B is directed toward the gaze area R. In the case of FIG. 11, the specifying unit 302 excludes the field camera FC-A, From the video data of FC-C and FC-D, the player P who exists within the gaze area R is identified as the gaze target. Determine.

[0081] FIG. 12 is an explanatory diagram showing a second example of specifying the gaze area and gaze target by the specifying unit 302. The specific example of Fig. 12 is an example in which an omnidirectional camera 1200 is used as the spectator camera AC. By using the spherical camera 1200, a synthetic motion vector can be generated for the spherical camera 1200. The field 102 on the opposite side of the spectator group 401 from which the MV was generated is also photographed. The determination unit 302 then determines the composite motion vector MV from the video data of the omnidirectional camera 1200. Simultaneously identify the gaze area R of the field 102 and the gaze target (player P) can be done.

[0082] FIG. 13 is an explanatory diagram showing an example of calculation of the area of ​​the fixation region. In FIG. 13, the fixation region R is When spectator A gazes at the gaze area, the height of spectator A from the field 102 is h , the distance to the gaze area R when the spectator A is projected onto the field 102 is L, and the gaze area R If the lengths of the two perpendicular sides are x and y, then x is given by the following formula (1) and y is given by the following formula (2). However, θ± is expressed by the following formula (3), and the distance D from the spectator A to the gaze area R is expressed by the following formula ( 4) (where the yaw angle θyaw and pitch angle θpitch are as shown in Figure 13 See legend.

[0083]

number

[0084] The area S of the fixation area R is S=x·y. The distance D from A to the gaze area R and the area S of the gaze area R can be determined. The object of attention (for example, player P) in the gaze area R can be identified by the area S. In this example, a certain spectator A is used as a reference, but in the case of the spectator group 401, the composite motion vector M The starting position of V may be the position of spectator A.

[0085] <Example of control processing procedure> FIG. 14 is a flowchart showing an example of a control process procedure performed by the control device 110. The device 110 receives video data from the spectator camera AC and video data from the field camera FC. The control device 110 selects the first spectator camera by the selection unit 305. A selection process is performed to select the audience camera AC for detecting the line of sight (step S14 01). Next, the control device 110 detects the image from the selected audience camera AC by the detection unit 301. The video data is analyzed to detect the line of sight direction (step S1402).

[0086] Next, the control device 110 uses the detected line-of-sight vector by the identification unit 302 to Then, a specification process is executed to specify the gaze area and gaze target (step S1403). The control device 110 then controls the field cameras FC to capture images of the target. Then, the field camera FC is determined (step S1404).

[0087] After that, the control device 110 transmits the image data from the determined field camera FC by the transmitting unit 304. The video data is transmitted via a radio tower 210 or the Internet 200 (step S1405) This allows the system to automatically focus on the target in the direction of the audience's gaze. You can switch between the Ludo Camera FC.

[0088] FIG. 15 shows the first spectator camera selection process (step S1) by the selection unit 305 shown in FIG. 401). The control device 110 is a flowchart showing a detailed processing procedure example 1 of the selection unit 401. 305 calculates a score indicating the number of spectators for each spectator camera AC (step S150 1). Next, the control device 110 selects the calculated score as described above by the selection unit 305. Based on the above, a spectator camera AC is selected (step S1502).

[0089] For example, the control device 110 may detect spectators with scores above a threshold or relatively high scores. In the selection in step S1502, the control device 110 selects the camera AC. If there is a spectator camera AC with a score equal to or higher than the specified value or a relatively high score, The customer camera AC may be selected probabilistically (for example, with a 70% probability).

[0090] Next, the control device 110 selects, by the selection unit 305, the items with a score equal to or greater than a threshold or a relative score. It is determined whether the spectator camera AC with the highest score has been selected (step S1503). When a spectator camera AC with a score above the threshold or a relatively high score is selected (S Step S1503: Yes), the process proceeds to step S1402. 0 is the result of the detection unit 301 detecting the composite motion vector MV of the spectator group 401 as the line of sight direction. This will be done.

[0091] On the other hand, spectator cameras AC with scores above the threshold or relatively high scores are not selected. If so (step S1503: No), the control device 110 selects the first 2. Execute the spectator camera selection process (step S1504) and proceed to step S1402. In the second audience camera selection process (step S1504), a specific audience camera among the audience group 402 is selected. In this case, the control device 110 detects a specific customer A8 by the detection unit 301. The movement vector mv of the spectator A8 is detected as the gaze direction.

[0092] FIG. 16 shows the first spectator camera selection process (step S1) by the selection unit 305 shown in FIG. 15. In FIG. 16, the detailed processing procedure example 2 of the process 401 is The same step numbers are used for the processes, and the explanations thereof will be omitted. The unit 302 identifies the gaze target and its gaze area that were previously identified by the identification unit 302. ,Depending on the positional relationship between the identified gaze area and each spectator camera AC, as shown in Figs. , the spectator camera AC is identified (step S1602).

[0093] There may be multiple spectator cameras AC identified in step S1602. If player P is in area 102AC, the spectator camera AC- After that, as shown in FIG. 15, in step S1602, Steps S1501 to S1504 are executed for the selected spectator camera AC. This makes it easier for the spectator camera AC closest to the gaze target to be selected, and the detection sensitivity of the gaze direction It is possible to improve the

[0094] FIG. 17 shows the detailed process of the second audience camera selection process (step S1504) shown in FIG. 10 is a flowchart showing a first example of a processing procedure. The control device 110 selects each observation Noise is removed from the image data of the customer camera AC (step S1701). The device 110 selects the noise-removed image data from each spectator camera AC by the selection unit 305. It is determined whether image data of a distinctive person exists (step S1702).

[0095] When there is a spectator camera AC that has captured video data containing image data of a distinctive person (Step S1702: Yes), the control device 110 selects the spectator camera by the selection unit 305. The camera AC is selected (step S1703), and the process proceeds to step S1402. If there is no spectator camera AC that has captured video data containing image data of a characteristic person ( Step S1702: No), the control device 110 selects the observation with the highest score by the selection unit 305. A customer camera AC is selected (step S1704), and the process proceeds to step S1704.

[0096] FIG. 18 shows the detailed process of the second spectator camera selection process (step S1504) shown in FIG. 10 is an explanatory diagram showing a processing procedure example 2. Specifically, for example, the control device 110 selects the 5, the viewer TA of TV 103 is watching the same scene. Spectator A11 is selected as a specific spectator.

[0097] The scene that TV103 viewers TA are watching is the scene that a certain field camera FC is watching. The control device 110 controls the currently broadcasted player P (the gaze target) to be photographed in the photographing direction PD. The spectator seats 101 in the opposite direction to the shooting direction PD of the field camera FC in the An audience camera AC that captures the audience seats 101 is selected. Then, the image captured by the selected audience camera AC is A motion vector mv is generated from the group of spectators 402 (including spectators A11 and A12) in the shooting direction P If the angle is the same as D or the deviation in direction is within the allowable range, spectator A11 is considered a specific spectator. Select.

[0098] FIG. 19 shows the detailed process of the second spectator camera selection process (step S1504) shown in FIG. 10 is a flowchart showing a second example of a processing procedure. The shooting direction of the field camera FC that captured the video data is identified (step S1901 ), and then identify the spectator seats 101 in the opposite direction to the identified shooting direction (step S1902). A spectator camera AC capable of photographing the spectator seats 101 is selected (step S1903). Audience members watching the same scene as TV103's TA are watching. A11 can be selected as a specific audience member, improving the accuracy of estimating specific audience members. can be done.

[0099] FIG. 20 shows a detailed example of the procedure of the identification process (step S1403) shown in FIG. 11 to 13. The control device 110 uses the identification unit 302 to As described above, the gaze area on the field 102 is calculated from the gaze direction obtained in step S1402. Next, the control device 110 uses the identifying unit 302 to identify the 11 to 13, the gaze target (or target object) in the gaze area identified in step S2001 For example, a player P is identified (step S2002).

[0100] In the example of FIG. 11, the control device 110 controls the image data of the field camera FC. Using this, a gaze target (for example, player P) within the gaze area identified in step S2001 is identified. This makes it possible to identify the gaze target in the line of sight with high accuracy. In the example of FIG. 12, the control device 110 uses the image data of the omnidirectional camera to Identifying a gaze target (e.g., player P) within the gaze area identified in step S2001 becomes.

[0101] This allows the area of ​​gaze and the target of gaze to be determined using video data from a single spherical camera. (Player P) can be identified at the same time, which can improve the accuracy of the characteristics. This makes it possible to improve the processing speed and increase the detection frequency. The device 110 uses video data from one camera to determine the gaze area or gaze points within the gaze area R. The target may be specified.

[0102] (1) As described above, the control device 110 according to this embodiment is a database 120 that stores design information about a first area ( A plurality of first cameras (for example, audience cameras) for photographing a first object (for example, audience seats 101) and a second subject in a second area (e.g., field 102) within the facility. and multiple secondary cameras (e.g., field cameras FCs) that capture the image. be.

[0103] The control device 110 includes a detection unit 301, an identification unit 302, a determination unit 303, and a transmission unit 304. The detection unit 301 detects the object by any one of the plurality of first cameras. Based on the image data of the first object, the line of sight direction of the spectators 401 in the first object is calculated. Detect.

[0104] The identification unit 302 determines the direction of the gaze of the spectators 401 detected by the detection unit 301. Then, a gaze area R within the second area of ​​the spectator group 401 is identified, and the image is captured by each of the plurality of second cameras. Based on the image data of the second subject, the target of attention of the spectators present in the gaze area R is determined. The determining unit 303 determines multiple targets based on the target of gaze identified by the identifying unit 302. A specific second camera is determined as a source of the video data from among the plurality of second cameras. 304 is a unit for transmitting video data from the specific second camera determined by the determination unit 303. do.

[0105] This allows the video data from the selected second camera (field camera FC) to be The content is broadcast or distributed to a TV 103 or a personal computer 104. It is possible to automate the switching of cameras (field cameras FC).

[0106] (2) Furthermore, the control device 110 of (1) above may also be configured to: Based on a score indicating the number of spectators 401 in the photograph, a first camera is selected from the plurality of first cameras. It has a selection unit 305 for selecting a specific first camera.

[0107] As a result, the detection unit 301 detects the specific first object photographed by the specific first camera. Based on the video data, the gaze direction of a specific group of spectators 401 in a specific first subject is detected. The identification unit 302 detects the direction of the line of sight of the specific audience group 401 detected by the detection unit 301. Based on this, the gaze area R of a specific audience group 401 is identified. This makes it possible to automate switching to the second camera (field camera FC).

[0108] (3) In the control device 110 of (2) above, the selection unit 305 selects the audience group 401 and the attention A specific first camera may be selected based on its positional relationship with the viewing area R. Taking into account the detection sensitivity in the line direction, a specific first camera (field camera FC) can be identified. This can be done.

[0109] (4) In the control device 110 of (2) above, the selection unit 305 selects a game with a predetermined score. A first camera having a value equal to or greater than the threshold may be selected as a specific first camera. Based on the video data from the first camera, which was evaluated as having a large number of spectators, the number of spectators in the group was 40. Therefore, it is possible to detect the gaze direction of the audience. The camera can automatically switch to a second camera that captures the subject being watched, allowing viewers to It is possible to preview the gaze area R and gaze target that many spectators are gazing at.

[0110] (5) In the control device 110 of (2), the selection unit 305 selects a plurality of first cameras. The first camera with a relatively high score among the scores may be selected as the specific first camera. This allows for the use of video data from the first camera, which is considered to have a relatively large number of spectators. Therefore, it is possible to detect the direction of the gaze of the audience 401. It can automatically switch to a second camera that captures the area or object of gaze that the person is looking at. As a result, viewers can preview the gaze area R and gaze target that many spectators are gazing at. Cut.

[0111] (6) In the control device 110 of (2) above, the selection unit 305 selects a game with a predetermined score. A first camera that is less than the threshold may be selected as the specific first camera. Based on the video data from the first camera, which was assessed to have an absolutely small number of spectators, the number of spectators in the group was 4. Therefore, it is possible to detect the gaze direction of the small number of spectators. The camera can automatically switch to a second camera to capture the area or target of attention, allowing viewers to , it is possible to preview the gaze area R or gaze target that a small number of spectators are gazing at. It automatically switches to the video data of the gaze area R or the gaze target that many spectators could not focus on. This can be done.

[0112] (7) In the control device 110 of (2), the selection unit 305 selects a plurality of first cameras. A primary camera with a relatively low score among the scores may be selected as the specific primary camera. This allows for a reduction in the number of spectators in the video data from the first camera, which was deemed to have a relatively small number of spectators. Therefore, it is possible to detect the direction of the gaze of the audience group 402. It can automatically switch to a second camera that captures the area or object of gaze that you are looking at, As a result, viewers can preview the gaze area R and gaze target that a small number of spectators are gazing at. In other words, it is possible to obtain the video data of the gaze area R or gaze target that many spectators could not focus on. It can be switched automatically.

[0113] (8) In the control device 110 of (6) or (7), the detection unit 301 Based on the video data of the specific first object captured by the first camera, The gaze direction of a particular spectator in a particular group of spectators 402 within the subject may be detected. The identification unit 302 determines the direction of the gaze of the specific spectator A8 detected by the detection unit 301. Then, the gaze area R of a particular spectator A8 is identified.

[0114] Therefore, when audience numbers are assessed as small in absolute or relative terms, viewers It is possible to preview the gaze area R and gaze target of a specific audience member. Even if there are no fans, spectators who are watching the game are likely to be loyal fans, so By switching the second camera to the gaze area R where the viewer is gazing and the gaze target, We can provide viewers with video data from an expert's perspective.

[0115] (9) In the control device 110 of (8), the detection unit 301 detects a specific first object. Based on the video data, spectator A1 who is not paying attention to the specific spectator group 402, A specific spectator A8 is selected from the remaining spectators A3 to A8 other than A2, and the line of sight of the specific spectator A8 is This can eliminate noise that is not related to the gaze direction and This makes it possible to improve the detection accuracy.

[0116] (10) In the control device 110 of (8), the database 120 stores characteristic Image data relating to a person (characteristic person information 126) is stored, and the detection unit 301 Based on the video data of the first subject, a distinctive person is identified from a specific audience group 402. Alternatively, the direction of the line of sight of a particular spectator A8 may be detected.

[0117] This will improve the accuracy of detecting specific spectators by matching them with distinctive people. Therefore, the gaze area R and By switching the second camera to focus on the target, video data from an expert's perspective can be provided to viewers. It is possible.

[0118] (11) In the control device 110 of (6) or (7), the detection unit 301 determines The photographing direction of the specific second camera determined by the determination unit 303 and the photographing direction of each of the specific spectators 401 are The gaze direction of a particular spectator A11 may be detected based on the gaze direction of each spectator. The viewer is watching the same gaze area R and gaze target as the video data being watched by the viewer. Since it is possible to identify the customers, it is possible to switch to a scene that is more from the audience's perspective.

[0119] (12) In addition, in the control device 110 of (1), the first camera captures the first area (audience seats) 101) and a camera capable of capturing a second area (field 102) (for example, a spherical camera) The detection unit 301 detects the first image captured by any one of the first cameras. Detect the gaze direction of the spectator group 401 based on video data including the first subject and the second subject. The identification unit 302 determines the direction of the gaze of the audience 401 detected by the detection unit 301. The gaze area R in the second area of ​​the spectator group 401 and the spectator group 401 present in the gaze area R The gaze target may be identified.

[0120] As a result, the gaze area R and the gaze target are identified using the video data from the first camera. Therefore, it does not depend on the image data of the second camera. This improves the accuracy of identification. In addition, there is no need to identify the gaze target using the second camera. This makes it possible to speed up processing.

[0121] (13) In addition, in the control device 110 of (1), the determining unit 303 determines whether the audience group 401 and A specific second camera may be determined based on its positional relationship with the gaze area R. Taking into consideration the positional relationship between the spectator group 401 and the gaze area R, the second camera is automatically set to various shooting directions. can be switched automatically.

[0122] For example, when the above positional relationship is such that the spectator group 401 and the gaze area R are close to each other, It is possible to distribute video data in an environment where an audience 401 is watching, and If you look at it, you can see the same gaze target from a viewpoint close to that of the audience 401. When the relationship is such that the audience group 401 and the gaze area R are far apart, It is possible to deliver video data from angles different from the direction of the viewer's gaze, This allows the audience 401 to view the same object of gaze from a different viewpoint.

[0123] (14) In the control device 110 of (1), the determination unit 303 recognizes the gaze target. Based on the result, a specific second camera may be determined. For example, the second camera is selected so that the face of the player P, who is the target of attention, can be seen. This allows viewers to view Player P from the front in the scene after switching to the second camera. do.

[0124] (15) In the control device 110 of (1), the transmitting unit 304 transmits the signal from a specific second camera. The image data captured by the camera or the information indicating the storage location of the image data is sent to a small number of people in the audience group 401. The second camera may be switched to the communication terminal 300 of at least one spectator A. The scene after the change can be a crucial moment, so in such a scene, the audience is It is thought that 1 was concentrating on gazing at the camera and not taking pictures. The still image data of the scene after switching to such an audience group 401 is transmitted to the communication terminal 30 of the audience group 401. By transmitting the still image data that the audience 401 was unable to capture, The audience 401 can be provided with:

[0125] (16) Furthermore, the control system 1 according to this embodiment can be used to control the facilities (for example, stadium 100) a database 120 storing design information relating to a first area (e.g., observation A plurality of first cameras (for example, a group of audience cameras ACs) for photographing a first subject in the audience seats 101 ) and a plurality of cameras capturing a second subject in a second area (e.g., field 102) within the facility. Secondary cameras (e.g., field cameras FCs) and their accessible control devices and a position 110.

[0126] The control device 110 includes a detection unit 301, an identification unit 302, a determination unit 303, and a transmission unit 304. The detection unit 301 detects the object by any one of the plurality of first cameras. Based on the image data of the first object, the line of sight direction of the spectators 401 in the first object is calculated. Detect.

[0127] The identification unit 302 determines the direction of the gaze of the spectators 401 detected by the detection unit 301. Then, a gaze area R within the second area of ​​the spectator group 401 is identified, and the image is captured by each of the plurality of second cameras. Based on the image data of the second subject, the target of attention of the spectators present in the gaze area R is determined. The determining unit 303 determines multiple targets based on the target of gaze identified by the identifying unit 302. A specific second camera is determined as a source of the video data from among the plurality of second cameras. 304 is a unit for transmitting video data from the specific second camera determined by the determination unit 303. do.

[0128] This allows the video data from the selected second camera (field camera FC) to be The content is broadcast or distributed to a TV 103 or a personal computer 104. It is possible to automate the switching of cameras (field cameras FC).

[0129] (17) The control program according to this embodiment may be used to control the facilities (for example, stadium 100) a database 120 storing design information relating to a first area (e.g., observation A plurality of first cameras (for example, a group of audience cameras ACs) for photographing a first subject in the audience seats 101 ) and a plurality of cameras capturing a second subject in a second area (e.g., field 102) within the facility. a second camera (e.g., a field camera group FCs) and a processor 2 accessible to Let 01 execute it.

[0130] The control program causes the processor 201 to perform a detection process, a specification process, a decision process, and a transmission process. In the detection process, the control program causes the processor 201 to execute a plurality of Based on the video data of the first object captured by any one of the first cameras, Based on this, the direction of the line of sight of the spectator group 401 in the first subject is detected.

[0131] In the specific process, the control program causes the processor 201 to Identifying a gaze area R within the second area of ​​the spectator group 401 based on the gaze direction of the spectator group 401; A gaze area is determined based on the image data of the second object captured by each of the plurality of second cameras. Identify the target of attention of the spectators within area R.

[0132] In the determination process, the control program causes the processor 201 to Based on the gaze target, a specific second camera that is the source of the video data is selected from among the plurality of second cameras. In the transmission process, the control program causes the processor 201 to determine the camera. The video data from the determined specific second camera is then transmitted.

[0133] This allows the video data from the selected second camera (field camera FC) to be The content is broadcast or distributed to a TV 103 or a personal computer 104. Camera (field camera FC) switching can be automated using software. Cut.

[0134] In addition, the second camera is restricted from switching until a certain time has elapsed after switching. In addition, the relationship between the video data before switching and the second camera after switching can be learned. The data is learned and stored in the database 120, and when switching to the second camera, the control device 110 The second camera may be determined by referring to the learned content. The accuracy of the decision can be improved.

[0135] In the above-described embodiment, the control device 110 may store photographed data of the same past competition (for example, For example, information about the director's techniques may be used. The device 110 learns rules regarding optimal cutting positions by machine learning the photographed data. This allows for learning, thereby improving the accuracy of detecting the gaze direction.

[0136] The field camera FC may also have an automatic shutter function. For example, it can be applied to cases where the target in the direction of the gaze of a group of spectators is static.

[0137] FIG. 21 is an explanatory diagram of the automatic shutter function of the field camera FC. The control device 110 refers to the video data of the field camera FC and The line of sight of player P is detected from line of sight d1 to the field camera. When the line direction is changed to d2, the field camera FC releases the shutter and takes a picture of the player P. The still image data 2100 is captured with the player P as the main subject. Still image data of the line of sight can be obtained automatically.

[0138] Also, if the object in the direction of the audience's gaze is a static object that does not move, such as a display This embodiment is applicable to commercial products.

[0139] FIG. 22 is an explanatory diagram showing an example of application to marketing. A crowd 2201 is a group of product exhibitors. The control device 110 captures the crowd 2201 with the audience camera AC, A gaze vector indicating the gaze direction of each person in the crowd 2201 is detected from the photographic data, and a synthetic gaze Generate a vector.

[0140] The control device 110 calculates the composite line-of-sight vector of the group of products displayed on the display shelf 2200. Identify the product indicated by (hereinafter, "specific product"), the number of sales of the specific product, Sales quantity when specified by the composite line of sight vector, Sales units when not specified, selected when specified by the composite line of sight vector In this way, we can manage statistical information such as the number of times a product has received high attention and the number of times it has not. This can be useful for detecting products.

[0141] In the above-described embodiment, the control device 110 controls a plurality of second cameras (field cameras FC). The example of switching between the first camera (audience camera AC) and the second camera (audience camera AC) has been explained. The gaze area R or the gaze target within the gaze area R is identified using the data, and one second camera The second camera (field camera FC) may be controlled. The control unit that controls the gaze area FC may, for example, Executes control to point the second camera (field camera FC) in the direction of the gaze target within area R. .

[0142] In the above-described embodiment, the control device 110 switches between a plurality of field cameras FC. We have explained the above, but we can also calculate the gaze area R using the video data of one or more spectator cameras AC. Or, the gaze target within the gaze area R is identified and generated by taking pictures with one field camera FC. The control device 110 may process the image captured by one field. From the image generated by the camera FC, the specified gaze area R or the specified The control device 110 then identifies the area of ​​the partial image that includes the gaze target within the gaze area R. The specified partial image may be broadcast (transmitted).

[0143] The control device 110 also uses one camera to capture a first area where the audience is present and a second area where the gaze target is present. The control device 110 may process an image of the subject captured in the first area and the second area. For example, from an image generated by one camera, the gaze area R or the gaze area A gaze target within the region R is identified, and a region of the partial image including the identified gaze target is identified. The control device 110 may broadcast (transmit) the identified partial image. [Explanation of symbols]

[0144] AC spectator camera, FC field camera, MV composite motion vector, mv motion vector Tor, P Player, R Gaze Area, 1 Control System, 101 Spectator Seats, 102 Field 110 control device, 120 database, 301 detection unit, 302 identification unit, 30 2 Identification unit, 303 Determination unit, 304 Transmission unit, 305 Selection unit

Claims

[Claim 1] a plurality of first cameras for photographing a first subject in a first area within a facility; a control device accessible to a plurality of second cameras that capture images of the second subject; Based on the number of spectators in the first subject photographed by each of the plurality of first cameras, a selection unit that selects a specific first camera from the plurality of first cameras; Based on the video data of the specific first object captured by the specific first camera, a detection unit for detecting the gaze direction of a specific group of spectators within the specific first subject; Based on the gaze direction of the specific audience group detected by the detection unit, and identifying a gaze area within the second area, and capturing the image captured by each of the plurality of second cameras. Based on the video data of the second subject, the gaze of the specific spectator group present in the gaze area is an identification unit that identifies a target; Based on the gaze target identified by the identification unit, an image captured by one of the plurality of second cameras is a determination unit that determines a specific second camera that is a source of image data; a transmitter that transmits video data from the specific second camera determined by the determination unit; 、 A control device having:

Images