Information processing device, information processing method, and program

Abstract

This information processing device is provided with a control unit that performs processing in which: importance regarding the line of sight of a plurality of persons is determined on the basis of first information that is information pertaining to candidate persons for line-of-sight detection; and a gaze region is determined on the basis of line-of-sight information that reflects the importance.

Description

Information processing device, information processing method, and program

[0001] The present technology relates to an information processing device, an information processing method, and a program, and in particular to a technology for determining a gaze area by detecting a line of sight.

[0002] For example, when filming for broadcasting or distribution at a sports stadium or event venue, it is possible to determine the area of ​​attention of spectators and capture that area. Patent Document 1 below discloses a technology for tracking and capturing an event that is attracting attention during a sporting event in an automatic camera control device used for filming a sporting event. Patent Document 2 below discloses a technology for automatically pointing another camera in the direction of the line of sight of a person being filmed with one camera to capture the person's focus.

[0003] JP 2008-5208 A JP 2011-217202 A

[0004] When determining the gaze area of ​​spectators at sports games and other events, the gazes of many spectators must be detected. However, because not all spectators are looking at the same spot, the accuracy of determining the gaze area decreases. Furthermore, in order to accurately determine the gaze area of ​​spectators, it is important to increase the number of spectators whose gazes are detected. However, increasing the number of spectators increases the processing load, and there is a possibility that the real-time performance that is important in sports filming will be lost. Furthermore, the gaze area is not limited to one location.

[0005] Therefore, an object of the present disclosure is to enable appropriate gaze area determination processing to be realized in terms of determination accuracy and processing load.

[0006] The information processing device according to the present technology includes a control unit that performs a process of determining the importance of the gazes of multiple people based on first information, which is information about gaze detection target candidates, and determining a gaze area based on the gaze information reflecting the importance. For example, rather than using the gazes of multiple people at a sports venue or the like as is, the gaze area is determined by incorporating the viewpoint of the importance of the gazes and reflecting the importance through selection and weighting. The gaze detection target candidates are, for example, spectators, and the gaze target is a sports game, event, or the like that the spectators are watching.

[0007] 1 is an explanatory diagram of an imaging system according to an embodiment of the present technology. FIG. 1 is a block diagram of an imaging system according to an embodiment. FIG. 2 is a block diagram of an information processing device according to an embodiment. FIG. 3 is an explanatory diagram of a functional configuration of a server device according to an embodiment. FIG. 4 is an explanatory diagram of information used in an embodiment. FIG. 5 is a flowchart of a processing example according to an embodiment. FIG. 6 is a flowchart of a calculation process of a gaze point accuracy parameter according to an embodiment. FIG. 7 is a flowchart of a calculation process of an importance parameter according to an embodiment. FIG. 8 is an explanatory diagram of weighting according to an embodiment. FIG. 9 is a flowchart of another processing example according to an embodiment. FIG. 10 is an explanatory diagram of a camera angle facing the intersection of a gaze vector and a horizontal plane. FIG. 11 is an explanatory diagram of rounding at a camera angle according to an embodiment. FIG. 12 is an explanatory diagram of a bar graph of points for each region according to an embodiment. FIG. 13 is an explanatory diagram of clustering according to an embodiment. FIG. 14 is an explanatory diagram of a tentative focus point according to an embodiment. FIG. 15 is an explanatory diagram of rounding in a field according to an embodiment. FIG. 16 is an explanatory diagram of clustering according to an embodiment. FIG. 17 is an explanatory diagram of a monitor display example according to an embodiment. FIG. 18 is an explanatory diagram of a monitor display example according to an embodiment. FIG. 19 is an explanatory diagram of a contour line display example according to an embodiment.

[0008] The following describes the embodiment in the following order: <1. System configuration> <2. Server device configuration> <3. Processing example> <4. Clustering> <5. Monitor display example> <6. Summary and modified examples>

[0009] 1. System Configuration An overview of the imaging system of this embodiment is shown in Fig. 1. It is assumed that multiple cameras 2 are installed at a stadium for soccer, rugby, etc., for example. This imaging system distributes, broadcasts, records, etc., the images captured by the cameras 2, and is designed to automatically capture locations that spectators are particularly watching.

[0010] The figure shows a server device 1, an input device 3, and a monitor device 4. The server device 1 is an information processing device that controls the camera 2. In particular, the server device 1 has a function of detecting the line of sight of the audience and controlling the shooting direction and angle of view of the camera 2.

[0011] The input device 3 is, for example, a keyboard, a mouse, a touch panel, or the like, and is a device that allows the filming staff to input information necessary for control processing of the server device 1. The monitor device 4 is a display device that allows the filming staff to monitor the main line video, the video of the area of ​​interest, and the like.

[0012] 1, systems for broadcasting, distribution, etc. may also include a switcher system that inputs video from camera 2 and selects main line video, a control device for camera settings and staff control, a transmission device for video broadcasting and distribution, and a recording device for recording video captured by camera 2. The following description focuses on camera control in response to line-of-sight detection by server device 1.

[0013] The configuration of the imaging system is shown in Fig. 2. Fig. 2 shows the server device 1, camera 2, input device 3, and monitor device 4 described above.

[0014] The server device 1 is composed of one or more information processing devices. An input device 3 is connected to the server device 1, allowing staff to input information to the server device 1 using the input device 3. A monitor device 4 is connected to the server device 1, allowing the server device 1 to control the display of the monitor device 4. Note that the monitor device 4 receives input of shot video from a switcher system (not shown) or the like, allowing the shot video to be displayed on a monitor. Alternatively, the server device 1 may be configured to include a switcher system.

[0015] The cameras 2 include a PTZ (Pan Tilt Zoom) camera 2A, a fixed camera for players 2B, a fixed camera for spectators 2C, and an overhead camera 2D.

[0016] The PTZ camera refers to a camera controlled by the server device 1. The server device 1 controls the pan, tilt, and zoom of the PTZ camera 2A. The PTZ camera 2A is a camera primarily intended for capturing video for broadcasting or distribution.

[0017] The player fixed cameras 2B are cameras arranged to film the players. For example, they include cameras that track specific players. The spectator fixed cameras 2C are cameras arranged to film the spectators. The overhead camera 2D is a camera arranged to film overhead images.

[0018] There may be multiple PTZ cameras 2A, player fixed cameras 2B, spectator fixed cameras 2C, and overhead camera 2D, or there may be only one of each. The figure shows an example in which multiple PTZ cameras 2A, player fixed cameras 2B, and spectator fixed cameras 2C are installed.

[0019] The fixed camera 2B for players, the fixed camera 2C for spectators, and the overhead camera 2D are also intended to capture video for broadcasting and distribution, but in this embodiment, these captured images are also used to determine the area of ​​interest on the server device 1.

[0020] For the sake of explanation, these cameras 2 (2A, 2B, 2C, 2D) are shown separately, but these types do not necessarily have to be fixed. For example, the PTZ camera 2A may be used in common with the fixed spectator camera 2C, or the fixed player camera 2B may be used as the overhead camera 2D. In this embodiment, it is sufficient to have at least cameras 2 whose shooting direction and shooting angle of view are controlled by the server device 1, and cameras 2 that supply captured images of spectators, players, etc. to the server device 1 so that the server device 1 can determine the spectator's gaze area.

[0021] Figure 2 shows a sensor 5. This is a sensor attached to the ball or a player's uniform, for example, and may be a position sensor, acceleration sensor, angular velocity sensor, altitude sensor, impact sensor, etc. It is a sensor for detecting the position and movement of the ball or player, for example.

[0022] Here, we will discuss EPTS (Electronic Performance and Tracking Systems). Regarding the play of sports such as soccer and basketball, EPTS is a known technology that estimates the posture and position of players and referees on the field, as well as the position and rotation of the ball, from images taken by specially installed cameras and information from sensors (acceleration sensors and GPS sensors) attached to people (players) and objects (balls) involved in the sport. For example, skeletal capture data that represents the posture of people such as players can also be obtained as EPTS data.

[0023] In the filming system of this embodiment, for example, by applying images captured by the fixed spectator cameras 2C and the fixed player cameras 2B to the EPTS, the server device 1 can obtain skeletal capture data of the spectators and players. In this case, by using not only the captured images but also the detection data of the sensor 5, the accuracy of the skeletal capture data can be improved, and the accuracy of determining the situation of the match can also be improved from the trajectories of the ball and player movements.

[0024] 2. Configuration of Server Device

[0025] 3, the configuration of an information processing device 70 used as the server device 1 will be described. The information processing device 70 can be configured, for example, by a dedicated workstation, a general-purpose personal computer, a mobile terminal device, or the like.

[0026] 3 executes various processes in accordance with programs stored in a ROM 72 or a nonvolatile memory unit 74 such as an EEPROM (Electrically Erasable Programmable Read-Only Memory), or programs loaded from a storage unit 79 to a RAM 73. The RAM 73 also stores data necessary for the CPU 71 to execute various processes, as appropriate.

[0027] The CPU 71 functions as a control unit in the server device 1 of this embodiment, and performs processes such as drive control of the camera 2 (particularly the PTZ camera 2A) and display control of the monitor device 4. The CPU 71 is also configured as a processor that performs various types of image processing. For example, it is a processor that can perform EPTS data generation processing.

[0028] Although the figure shows a CPU 71, it is also possible to install a CPU separate from the CPU 71, a GPU (Graphics Processing Unit), a GPGPU (General-purpose computing on graphics processing units), an AI (artificial intelligence) processor, etc., which work together to function as a control unit.

[0029] The CPU 71, ROM 72, RAM 73, and nonvolatile memory unit 74 are interconnected via a bus 83. The bus 83 is also connected to an input / output interface 75.

[0030] An input unit 76 consisting of operators and operation devices is connected to the input / output interface 75. For example, the input unit 76 may be various operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, or a remote controller. The input unit 76 detects a user operation, and a signal corresponding to the input operation is interpreted by the CPU 71. This input unit 76 may function as the input device 3 in FIG. 2.

[0031] The input / output interface 75 is also connected, either integrally or separately, to a display unit 77 such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 78 such as a speaker.

[0032] The display unit 77 displays various types of information as a user interface. The display unit 77 may be, for example, a display device provided on the housing of the information processing device 70 or a separate display device connected to the information processing device 70. The display unit 77 displays various images on the display screen based on instructions from the CPU 71. The display unit 77 also displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the CPU 71. The display unit 77 may function as the monitor device 4 in FIG. 2.

[0033] A storage unit 79 configured with an SSD (Solid State Drive) or an HDD (Hard Disk Drive) or a communication unit 80 configured with a modem or the like may be connected to the input / output interface 75. The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, or the like.

[0034] A drive 82 is also connected to the input / output interface 75 as needed, and a removable recording medium 81 such as a flash memory, a memory card, a magnetic disk, an optical disk, or a magneto-optical disk is appropriately attached. The drive 82 allows data files such as image files and various computer programs to be read from the removable recording medium 81. The read data files are stored in the storage unit 79, and images and sounds contained in the data files are output on the display unit 77 and the audio output unit 78. In addition, the computer programs and the like read from the removable recording medium 81 are installed in the storage unit 79 as needed.

[0035] In this information processing device 70, software can be installed via network communication by the communication unit 80 or via a removable recording medium 81. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, or the like.

[0036] For example, when the information processing device 70 is considered as the server device 1, the functions shown in FIG. 4 are provided in the CPU 71 by software, and the CPU 71 executes the processes according to these functions.

[0037] The CPU 71 is provided with a processing function based on input video information as the image processing unit 10. In this case, the video information is assumed to be images captured by a plurality of cameras 2, such as the fixed camera for players 2B, the fixed camera for spectators 2C, and the overhead camera 2D.

[0038] The image processing unit 10 includes a posture detection unit 11, a gaze detection unit 12, a skeleton estimation unit 13, a face recognition unit 14, a uniform number recognition unit 15, and an event recognition unit 16.

[0039] The posture detection unit 11 detects the postures of spectators, players, etc. For example, the postures are determined based on skeletal capture data. The gaze detection unit 12 detects the gaze of each spectator, for example.

[0040] The skeleton estimation unit 13 generates skeleton capture data as EPTS data from the video information of the multiple cameras 2. The face recognition unit 14 performs face recognition processing on the video information, and performs processes such as person detection and individual identification within the video. The uniform number recognition unit 15 determines the uniform numbers of the players within the video, enabling individual identification of the players within the video.

[0041] The event recognition unit 16 analyzes the video to recognize events in the match. The events recognized here are not limited to planned events, but also include occurrences and scenes that occur during the progress of the match. For example, in the case of soccer, penalty kick scenes, corner kick scenes, shooting scenes, scoring scenes, dribbling scenes, etc. are recognized as events.

[0042] The CPU 71 further includes an importance determination unit 20 , an accuracy determination unit 21 , a gaze area determination unit 22 , a position information management unit 23 , an initial setting storage unit 24 , a camera command generation unit 25 , and a monitor control unit 26 .

[0043] The importance determination unit 20 determines the importance of the spectators' gazes. The accuracy determination unit 21 determines whether real-timeness or accuracy is to be emphasized and the degree of importance. The gaze area determination unit 22 determines the spectators' gaze areas using the gaze detection of each spectator and the determination results of the importance determination unit 20 and the accuracy determination unit 21.

[0044] The position information management unit 23 determines the positions of the ball and players based on the detection information and image information from the sensor 5, and manages the position information. The initial setting storage unit 24 stores initial setting parameters based on the information input by the staff from the input device 3. The camera command generation unit 25 generates control signals for pan, tilt, zoom, etc. for the PTZ camera 2A, and sends them to the PTZ camera 2A. The monitor control unit 26 generates control signals for the monitor device 4 and sends them to the monitor device 4.

[0045] Although the following description will be given using an example in which these functions are realized by software, these functions may also be realized by hardware circuits.

[0046] 3. Processing Example The server device 1 detects the gazes of spectators and determines a gaze area. The gaze area is an area where the gazes of many spectators are focused. The server device 1 also controls the PTZ camera 2A so that the determined gaze area is photographed.

[0047] In order to accurately determine the spectator's gaze area, it is important to increase the number of spectators whose gazes are detected. However, as the number of spectators whose gazes are detected increases, the amount of calculation required to determine the gaze area increases. This can sometimes result in a loss of real-time performance, which is important for sports photography.

[0048] In this embodiment, the server device 1 sets a level of importance for each spectator's line of sight (i.e., for each spectator) based on the state of the match and spectators, and reflects this in determining the gaze area, thereby improving the accuracy of the determination. In addition, the server device 1 changes the amount of calculation required to determine the gaze area based on the state of the match and spectators. This allows the server device 1 to prioritize either the accuracy of the gaze area or real-time performance depending on the situation of the match. Furthermore, the server device 1 determines multiple gaze areas, rather than just one, so as to detect locations that are attracting the second and third most attention of spectators. The server device 1 also achieves highly accurate gaze area determination by using skeletal capture data of spectators when estimating the spectator's gaze.

[0049] Specifically, when shooting sports, etc., as shown in Figure 2, in addition to the overhead camera 2D that captures the entire field, the fixed camera for players 2B that automatically tracks multiple players that are installed surrounding the field, and the PTZ camera 2A, multiple fixed cameras for spectators 2C that can overlook the entire spectators or a portion of the spectators are installed, and the gaze area of ​​the spectators is estimated using facial recognition, gaze detection, skeletal estimation, etc.

[0050] In this case, the server device 1 calculates an importance parameter for the gaze of the spectators based on first information, which is information about gaze detection target candidates. The server device 1 then changes the weight of the gaze of the spectators based on the importance parameter, or selects a gaze to be reflected in determining the gaze area. Gaze detection target candidates are, for example, spectators. They may also include players, staff, referees, etc. In this embodiment, gaze detection target candidates are at least people who can be photographed by the camera 21 and whose gaze can be detected by the system (people who can be subjects for gaze detection). Note that multiple people, such as spectators and staff, may wear gaze detection devices, such as goggles, and gaze detection information may be obtained as a detection result from the gaze detection device. In this case, these people also become gaze detection target candidates.

[0051] As shown in Figure 5, the first information (information used to determine the importance of the line of sight) can be the following information about the situation of the spectators and the situation of the game: - The position of the spectators - The posture and skeletal structure of the spectators - Whether the line of sight is near an event on the field

[0052] Among these, the position, posture, and skeletal structure of spectators are information that can be obtained from the video before gaze detection. This information is referred to as [information iA]. Information on whether the gaze is near an event on the field is information that can be obtained after gaze detection. This information is referred to as [information iB].

[0053] Information about the spectator's position can be used as a factor in determining importance as follows. For example, by determining the attributes of the spectator from their seating position, it is possible to select the gaze of each favored team. It is also possible to select (determine as important) the gaze of people in the front seats. This is because it can be assumed that people in the front are more enthusiastic spectators. The position of a specific person can also be specified by manual operation by staff or by video analysis. For example, people who are overacting, referees, or spectators wearing uniforms. This is because it can be assumed that people who are overacting are watching the game with their eyes fixed on the game and becoming emotional. This is because referees are gazing at important parts of the game. This is because spectators wearing uniforms can be assumed to be enthusiastic spectators supporting their favorite team.

[0054] Information on the audience's posture and skeletal structure is used as a factor in determining importance as follows. By using posture detection based on skeletal structure estimation to determine areas of the audience where emotions are high, it is possible to determine that the gaze of spectators in that area is important. By using posture detection based on skeletal structure estimation, the gaze of people looking down can be determined to be of low importance, and that gaze can be excluded or weighted down. It is also possible to use skeletal structure estimation to detect people who are overacting and assign weights to them.

[0055] Information on whether a gaze is near an event on the field, such as a penalty kick or a shot, determines whether the gaze is focused on that event. By identifying an event as an important point, gazes on that event can be selected or weighted as gazes focused on important points in the game.

[0056] Spectators who are watching an event closely will also look at places related to the event, so their gaze can also be considered important. For example, if the event involves a series of dribbles and passes towards the opponent's goal, enthusiastic spectators will predict the course of the passes and look in that direction. From the perspective of filming important scenes, such gazes will also be of high importance. For example, it may be possible to achieve camerawork that follows the spectators' predictions.

[0057] The server device 1 calculates an importance parameter for each line of sight (each spectator) from the first information described above, and performs processing (selection and weighting) according to the importance of each line of sight to determine the gaze area.

[0058] The server device 1 also sets the gaze point accuracy based on the second information and sets the line of sight used for gaze area determination according to the gaze point accuracy. This gaze point accuracy is a degree required for determining the gaze area, and is a degree that takes into consideration a trade-off with real-time performance in particular.

[0059] In scenes that require a high degree of real-timeness, some decrease in accuracy is tolerated and priority is given to tracking the gaze area. For example, in scenes where players move and move quickly, such as dribbling scenes, the gaze area is determined by tracking the dribble rather than accuracy. On the other hand, in scenes with little movement, the accuracy of determining the gaze area is increased. Therefore, when the required gaze point accuracy is high, priority is given to accuracy, and when it is low, priority is given to tracking. From the perspective of real-timeness, conversely, when real-timeness is high, priority is given to tracking, and when it is low, priority is given to accuracy.

[0060] For the sake of explanation, it is assumed that the server device 1 calculates the gaze point accuracy parameter. A high gaze point accuracy parameter means a low degree of real-timeness, and a low gaze point accuracy parameter means a high degree of real-timeness.

[0061] The server device 1 sets the gaze point accuracy based on the second information and sets the line of sight to be used for gaze area determination according to the gaze point accuracy, which includes determining the number of line of sight to be used for gaze point determination and setting a weighting coefficient of zero for line of sight with low importance so that it is not effectively reflected in the determination.

[0062] As shown in Figure 5, the second information (information used to determine the real-timeness / required accuracy) may include the following information about the situation of the spectators and the situation of the match: Speed ​​of the ball and players' movements Skeleton information of the spectators Events on the field Whether the line of sight is in the direction where the players are concentrating

[0063] Of these, the speed of the ball and player movement, skeletal information of spectators, and events on the field are information that can be obtained from the video before gaze detection. This information is referred to as [information iC]. Information on whether the gaze is directed at a position where players are concentrating is information that can be obtained after gaze detection. This information is referred to as [information iD]. These second pieces of information are elements in determining gaze point accuracy (degree of real-timeness) as follows:

[0064] Information on spectator skeletal estimation is used to determine the variance of posture movements based on skeletal estimation, which becomes a factor in determining accuracy and real-time performance. For example, when each spectator's head movements are fast and diverse, in other words, when the variance is large, it can be assumed that the action on the field is fast and real-time performance is important. On the other hand, when many spectators are facing the same place, it can be determined that an emphasis on accuracy is desirable. Furthermore, by detecting the timing of excitement based on spectator posture detection, and when the game is expected to develop quickly, real-time performance can be emphasized.

[0065] Whether real-timeness is necessary or not can be determined depending on the situation on the field. For special events, the speed of the ball and players' movements is a factor in determining whether real-timeness is important when the speed is fast, and accuracy is important when the speed is slow. In areas where players are concentrated, accuracy is emphasized.

[0066] An example of processing performed by the server device 1 using the first information and second information will now be described. Figure 6 shows an example of processing performed by the server device 1 in which the server device 1 detects the gazes of multiple spectators and selects and weights gaze information using information iA, information iB, information iC, and information iD. This can be considered to be processing by a CPU 71 equipped with the functions of Figure 4 in an information processing device 70 used as the server device 1.

[0067] In step S101, the server device 1 performs an initialization process. For example, an operator of the server device 1 inputs information using the input device 3, and the server device 1 performs initialization in response to the input. For example, the information may include the position of spectators to be emphasized, attributes based on the spectators' seating positions, and the uniform numbers of players. The attributes based on seating positions include attributes such as the team being supported.

[0068] After the initialization, the server device 1 repeats the processes from step S102 to step S112 until it is determined to end in step S113. End here means that the server device 1 has ended control of the PTZ camera 2A.

[0069] In steps S102, S103, and S104, the server device 1 inputs various information, such as information from the overhead camera 2D (images and information accompanying the images), detection information from the sensor 5, and information from the fixed spectator cameras 2C and the fixed player cameras 2B (images and information accompanying the images).

[0070] In step S105, the server device 1 extracts meta information. The meta information here refers to information used in subsequent processes, such as gaze detection, generation of importance parameters, and generation of gaze point accuracy parameters. It may be information attached to the video itself, or information generated by video analysis or other processes. For example, the meta information may include skeletal capture data, face recognition data, uniform number recognition data, event recognition data, posture detection data, ball position data, position data of each player, ball speed data, and speed data of each player, etc., generated by each function of the image processing unit 10.

[0071] In step S106, the server device 1 detects the gazes of the spectators. For example, the server device 1 detects the gazes of all spectators within the range photographed by one or more fixed spectator cameras 2C. For each photographed spectator, the position and gaze angle of that spectator are determined, and this is used as gaze information.

[0072] In step S107, the server device 1 calculates a gaze point accuracy parameter based on the second information. Fig. 7 shows an example of a process for calculating the gaze point accuracy parameter.

[0073] In step S201, the server device 1 acquires meta information for calculating the gaze point accuracy parameter, for example, information relating to the second information from the meta information extracted in step S105.

[0074] In step S202, the server device 1 initializes a gaze point accuracy parameter (x) (x = 0). In step S203, the server device 1 determines whether a specific event has occurred based on the event recognition data. If it is determined that an event has not occurred, the server device 1 increments the gaze point accuracy parameter (x) in step S204.

[0075] In step S204, the server device 1 calculates the variance of posture from the posture detection data of each spectator. It then determines whether the variance of posture is less than a threshold. This can be thought of as determining whether the postures are diverse. If the variance of posture is not less than the threshold, that is, if the spectators are watching with diverse postures, the server device 1 increments the gaze point accuracy parameter (x) in step S206.

[0076] In step S207, the server device 1 determines whether or not there is a bias in the positions of the players based on the position data of each player. If there is no bias, that is, if the players are appropriately dispersed, the server device 1 increments the gaze point accuracy parameter (x) in step S208.

[0077] In step S209, the server device 1 determines whether the speed of the ball or each player exceeds a threshold value. If the speed does not exceed the threshold value, that is, if the movement cannot be said to be fast, the server device 1 increments the gaze point accuracy parameter (x) in step S210.

[0078] Based on the above results, the server device 1 calculates a gaze point accuracy parameter (x) in step S211. For example, x may be set to any value between 0 and 4. When speed is important, such as when an event occurs or when the game progresses quickly in order to catch up with the highlights, the value of the gaze point accuracy parameter (x) decreases, and when the players' positions are not concentrated or when the spectators' postures are highly dispersed, the value of the gaze point accuracy parameter (x) increases. In other words, when real-time performance is not required, the value of the gaze point accuracy parameter (x) increases.

[0079] 6, the server device 1 calculates the importance parameter of each line of sight based on the first information in step S108. An example of the process in step S108 is shown in FIG.

[0080] The server device 1 performs the process shown in Fig. 8 for each spectator whose gaze has been detected. In step S301, the server device 1 acquires the gaze detection result for a certain spectator. In step S302, the server device 1 determines whether to calculate the importance parameter under manually set conditions or automatically. This results in a branching of the process depending on, for example, the mode setting of the operator.

[0081] In the case of manual processing, the server device 1 proceeds to step S310, where it calculates an importance parameter (y) for the current line of sight (spectator) according to the conditions for determining importance set at the time of initialization. For example, if the position of a spectator to be emphasized is specified, the processing increases the importance parameter for the line of sight information of that spectator. In addition, in the case of video distribution of the home team, the processing may increase the importance parameter for the line of sight information of spectators cheering on the home team.

[0082] In the case of automatic calculation, the server device 1 proceeds to step S321 and calculates the importance parameter (y) for the current line of sight (spectator). First, in step S321, it is determined whether the spectator for whom the importance parameter is to be calculated is looking down. For example, it is determined whether the spectator is looking down at a smartphone or the like instead of paying close attention to the game. If the spectator is not looking down, the server device 1 proceeds to step S322 and increments the importance parameter (y).

[0083] In step S323, the server device 1 determines whether the seat position of the target spectator is in the front. If it is in the front, the server device 1 proceeds to step S324 and increments the importance parameter (y).

[0084] In step S325, the server device 1 determines whether or not an emotional surge is observed in the target audience member. For example, the server device 1 determines behavior from skeletal capture data and posture data, and determines whether or not the behavior can be evaluated as an emotional surge. If it is determined that an emotional surge is present, the server device 1 proceeds to step S326 and increments the importance parameter (y).

[0085] In step S327, the server device 1 determines whether the target spectator is looking at the periphery of a specific event. For example, the server device 1 determines whether the target spectator is looking at a location of the event, such as the area around the goal in the case of a penalty kick, based on the event recognition data and the gaze information of the gaze detection result for the spectator. If it is determined that the target spectator is looking at the periphery of the specific event, the server device 1 proceeds to step S328 and increments the importance parameter (y).

[0086] In step S329, the server device 1 determines whether the target spectator is overacting. For example, the server device 1 determines the behavior from skeletal capture data and posture data, and determines whether the behavior can be evaluated as overacting. If the behavior is determined to be overacting, the server device 1 proceeds to step S330 and increments the importance parameter (y).

[0087] In step S331, the server device 1 determines whether the gaze of the target spectator is stable. For example, the server device 1 determines whether the gaze is in the field direction and stable by referring to the gaze detection results for a period of time, such as a predetermined number of past frames of video information. If it is determined that the gaze is stable, the server device 1 proceeds to step S332 and increments the importance parameter (y).

[0088] Based on the above results, in step S333, the server device 1 calculates an importance parameter (y) for the line of sight of the spectator currently being processed. The server device 1 performs the process of Figure 8 for each spectator, thereby calculating an importance parameter (y) for each spectator. The more intently a spectator is watching the game or the more emotionally attached a spectator is to the game, the higher the value of the importance parameter (y).

[0089] After performing the above processing in step S108 of Figure 6, in step S109, the server device 1 weights (or selects) the gaze information detected in step S106 according to the importance parameter and the gaze point accuracy parameter (real-time degree) and plots a provisional gaze point.

[0090] An example of weighting is shown in Figure 9. Here, the gaze point accuracy parameter has a value of four levels, and the importance parameter has a value of five levels. As shown in the figure, the gaze point accuracy and real-timeness have an inverse relationship. When the real-timeness is high, the gaze point accuracy is low.

[0091] For example, when the gaze point accuracy parameter is 1, the weighting factor is set to "5" for gaze information with an importance parameter of 5, "4" for gaze information with an importance parameter of 4, and "0" for gaze information with an importance parameter of 3 or less. This means that when the degree of real-timeness is high, the gaze point based on gaze information with an importance parameter of "4" or "5" is plotted as a tentative gaze point.

[0092] When the gaze point accuracy parameter is 2, the weighting factors for gaze information with importance parameters of 5, 4, and 3 are "5," "4," and "3," respectively, and the weighting factor for gaze information with importance parameters of 2 or less is "0." When the gaze point accuracy parameter is 3, the weighting factors for gaze information with importance parameters of 5, 4, 3, and 2 are "5," "4," "3," and "2," and the weighting factor for gaze information with importance parameter of 1 is "0." When the gaze point accuracy parameter is 4, the weighting factors for gaze information with importance parameters of 5, 4, 3, 2, and 1 are "5," "4," "3," "2," and "1," respectively.

[0093] In other words, the higher the gaze point accuracy parameter, the more gaze points based on gaze information are plotted as provisional gaze points, but the higher the real-timeness, the fewer gaze information items are used as provisional gaze points, thereby reducing the processing load.

[0094] The tentative gaze point is plotted in the angle range (pan angle) of the shooting direction of the PTZ camera 2A, or in an area set on the field. A specific example will be described later.

[0095] In step S110, the server device 1 performs clustering on the tentative gaze points plotted in the angle range of the PTZ camera 2A and in the areas on the field, and calculates one or more areas as gaze areas in descending order of gaze points. As a result, one or more gaze areas are determined.

[0096] In step S111, the server device 1 generates pan / tilt / zoom control signals for one or more PTZ cameras 2A based on the determination result of the gaze area, and transmits the signals to each PTZ camera 2A, thereby controlling each PTZ camera 2A to capture the gaze area.

[0097] In step S112, the server device 1 controls the monitor device 4. For example, the server device 1 controls the monitor device 4 so that the display mode of the plurality of captured images varies depending on the result of the gaze area determination. Display examples on the monitor device 4 will be described later.

[0098] In step S113, the server device 1 determines whether the control has ended, and if not, returns to step S102 to repeat the process.

[0099] According to the processing of FIG. 6 described above, when the degree of real-time is high, the number of pieces of gaze information reflected in determining the gaze area is reduced, reducing the processing load on the server device 1 and enabling control of the PTZ camera 2A with good responsiveness. Even if the number of pieces of gaze information is reduced, the accuracy of determining the gaze area can be maintained by using gaze information with a high importance parameter. However, reducing the number of pieces of gaze information reduces the diversity of locations determined to be gaze areas. While this is disadvantageous from the perspective that there is not just one gaze area, it is suitable for situations where you want to follow the scene in a fast-paced game. On the other hand, when the degree of real-time is low, a variety of gaze areas can be detected by reflecting a larger number of pieces of gaze information when determining the gaze area. Furthermore, reflecting the importance parameter can also improve the accuracy of determining the gaze area.

[0100] Figure 10 shows another example of processing by the server device 1. This is an example of processing in which spectators for gaze detection are selected using information iA and information iC shown in Figure 5, the number of additional gazes is determined using information iD, and the gaze information is weighted from information iA and information iB. In Figure 10, the same steps as in Figure 6 are given the same step numbers to avoid redundant explanation.

[0101] After initialization in step S101, the server device 1 extracts meta-information in step S105 in response to inputs in steps S102, S103, and S104, and then performs processing in step S120. In step S120, the server device 1 calculates the real-time degree and the importance degree using information iA and information iC, i.e., information obtained before gaze detection. For example, the server device 1 may use the gaze point accuracy parameter obtained by performing processing similar to that shown in FIG. 7 as the real-time degree here. A high value (x) of the gaze point accuracy parameter described above indicates a low real-time degree, and a low value (x) of the gaze point accuracy parameter indicates a high real-time degree. The server device 1 may also calculate the importance parameter by performing the processing of FIG. 8 , excluding processing that requires gaze detection results, such as steps S327 and S328.

[0102] In step S122, the server device 1 determines spectators to be the targets of gaze detection based on the calculated importance parameter and real-timeness. For example, the number of spectators is set according to the real-timeness. In other words, the higher the real-timeness, the fewer the number of spectators, and the lower the real-timeness, the more the number of spectators. Then, by selecting the set number of spectators from those with the highest importance parameter, the server device 1 determines spectators to be the targets of gaze detection.

[0103] In step S123, the server device 1 performs gaze detection for the audience members determined as gaze detection targets. That is, for each of the audience members, the server device 1 determines the position and gaze angle of the audience member, and sets this as gaze information.

[0104] In step S124, the server device 1 calculates an accuracy parameter based on the information iD. The accuracy parameter is a parameter that expresses how important the gaze detection result is. In this case, the server device 1 determines whether or not the gaze information of each spectator whose gaze has been detected is concentrated in an area where players are crowded together. A state in which the gazes are concentrated in an area where players are crowded together can be said to be an important scene, so processing is performed to increase the accuracy parameter.

[0105] In step S125, additional gaze detection is performed according to the calculation result of the accuracy parameter. That is, if the accuracy parameter is higher than the threshold, an additional number of people is set as gaze detection targets in order to further improve the determination accuracy. Then, from the spectators excluded in step S122, spectators with high importance parameters are selected for that additional number, and gaze detection is performed. If the accuracy parameter is equal to or lower than the threshold, this additional gaze detection does not need to be performed.

[0106] In step S126, the server device 1 modifies the importance parameter using the information iB, i.e., the information obtained by gaze detection. For example, the server device 1 performs processing such that, for the importance parameter of each spectator whose gaze has been detected, if the gaze is near an event on the field, the server device 1 increases the importance parameter, and if the gaze is not near an event on the field, the server device 1 decreases the importance parameter.

[0107] In step S127, the detected line-of-sight information is weighted according to its importance and plotted as a tentative gaze point. Then, steps S110, S111, S112, and S113 are performed in the same manner as in FIG.

[0108] 10, the number of gaze detection targets can be increased or decreased depending on the situation, and unnecessary gaze detection processing is not performed, thereby reducing the processing load. For example, gaze detection that would ultimately be weighted to "0" and not reflected in the gaze area determination can be avoided.

[0109] 6 and 10. It is difficult to align the gaze vectors of spectators or the intersections of the gaze vectors and horizontal planes at exactly the same positions. Therefore, a certain granularity is set and calculations are performed within that granularity. An example of processing in which the gaze information is rounded within a certain range and weighted for clustering will be described.

[0110] First, an example of rounding the camera angle of the PTZ camera 2A will be described. Figure 11 shows an intersection 100 between the line-of-sight vector 101 of a spectator and the horizontal plane of the field 105. The intersection 100 is plotted by rounding within the range of the angle φ of the PTZ camera 2A.

[0111] 12 shows an example in which the range of 0 to 180 degrees, which is the shooting direction of the PTZ camera 2A, is divided into six. Intersections 100 detected from line-of-sight information (line-of-sight vector 101) are plotted in the six ranges of the shooting direction of the PTZ camera 2A. The numbered circles indicate the positions of the intersections 100, and the numbers indicate the values ​​of the line-of-sight importance parameters. For example, this means that line-of-sight information with an importance level of "1" and line-of-sight information with an importance level of "5" are plotted in the range from 90 to 120 degrees.

[0112] Once each line of sight information has been plotted for each angle range, the points for each angle range are calculated. For example, in the range from 90 degrees to 120 degrees, one line of sight information with an importance level of "1" is calculated as (1 x 1), and one line of sight information with an importance level of "5" is calculated as (5 x 1), and these are added together to get a point of "6." The same calculation is used for the range from 60 degrees to 90 degrees, resulting in a point of "7."

[0113] Next, a one-dimensional bar graph is created for the points for each angle range. An example is shown in FIG. 13. However, FIG. 13 shows an example in which the angle range is divided into 15-degree intervals. Clustering is then performed. For example, as shown in FIG. 14, the peaks on the bar graph are grouped into one group. In the case of FIG. 14, the first and second gaze areas are determined on the field 105 in descending order of the peak points.

[0114] Next, an example of rounding by field area will be described. First, as shown in Figure 15, the point where the spectator's line of sight vector 101 and the field 105 intersect is taken as the tentative gaze point 106. As shown in Figure 16, the field 105 is divided into a certain granularity, and points are calculated from the tentative gaze points 106 (circles or black circles) plotted for each area. That is, points are calculated using the importance parameter and gaze point accuracy parameter of the tentative gaze point 106. For example, in Figure 16, the black circles (black circles) are taken as the tentative gaze points of spectators in the rear, and the open circles (white circles) are taken as the tentative gaze points of spectators in the front, and weights are assigned using this information to calculate the points for each area.

[0115] The points are then graphed two-dimensionally as shown in Fig. 17, and clustering is performed, whereby the first gaze area, the second gaze area, etc. are determined on the field 105 in order from the range of the highest peak of the points.

[0116] 5. Monitor Display Examples Next, a description will be given of display examples on the monitor device 4. The monitor device 4 is a display monitor that is visually recognized by staff such as switcher engineers, and displays images captured by the multiple cameras 2.

[0117] 18 shows an example of a display on the monitor device 4. In response to an operation by a staff member to designate a specific player using the input device 3, the server device 1 can cause the camera 2 to automatically track the specific player. In this case, the server device 1 can display, on the screen of the monitor device 4, an image of the camera 2 automatically tracking the designated player.

[0118] 18, a tracking image 120 is displayed in a portion of the screen. The name 121 of the player being tracked is also displayed. Along with the tracking image 120, the server device 1 also displays the captured image of the gaze area, which it determines as gaze area image 130 (130a, 130b, 130c).

[0119] For example, when the direction of the camera 2 automatically tracking a player designated by a staff member differs from the direction of the spectator's gaze area, the server device 1 performs object recognition on the image of the camera 2 facing the gaze area. As a result, the server device 1 displays, on the monitor device 4, the gaze area image 130 as a scene that the spectator is gazing at, together with the tracking image 120.

[0120] Note that the tracking image 120 may be an image of the gaze area. On the other hand, the tracking image 120 may be an image that is not gazed upon by the audience. For this reason, for example, the server device 1 may perform control so as to automatically switch from the tracking image 120 to the gaze area image 130 without the intervention of a staff member's operation.

[0121] 18, multiple gaze area images 130a, 130b, and 130c are displayed. That is, the area that the audience is most watching is displayed, and also the captured images of the areas that the audience is second and third most watching are displayed. That is, for example, only one tracking image 120 and the most watched gaze area image 130a may be displayed, or captured images of multiple gaze areas may be displayed, such as gaze area images 130a, 130b, and 130c.

[0122] The server device 1 can display the degree of attention 140 as information on the degree of attention each video is receiving from the audience. In the figure, the degree of attention 140 is displayed as a percentage. For example, the server device 1 determines the number of audience members gazing at each area from the plot results of gaze determination, calculates the percentage of the number of audience members gazing at each area, and displays the calculated percentage as the degree of attention 140. This allows staff to understand the degree of attention being paid to the tracking video 120 and the gaze area video 130, which can be used as a reference for switcher operation, etc. The degree of attention 140 may not be displayed as a percentage, but may instead be the number of audience members determined to be gazing at that area by gaze detection. Alternatively, the degree of attention for each screen may be displayed graphically.

[0123] 18, the second and third most watched gaze area images 130b and 130c are displayed in smaller sizes on the screen than the most watched gaze area image 130a. In this case, the tracking image 120 or the main line image is displayed in the largest size, and the gaze area images that are not the main line image are gradually reduced in size according to the degree of attention.

[0124] In the example of Figure 19, gaze area images 130a, 130b, 130c, and 130d are displayed, with the image sizes being set to three levels: large, medium, and small. That is, the image sizes are changed to first gaze area image 130a, second gaze area image 130b, and subsequent gaze area images 130c and 130d. By expressing the degree of attention as a size, as in the examples of Figures 18 and 19, it becomes easier for staff to recognize the degree of attention being paid to each captured image.

[0125] 20 shows an example in which the gaze area images 130a, 130b, and 130c are displayed at the same size, but the degree of attention can be grasped from the position and order. For example, in this example, they are arranged from left to right on the screen in order of gaze level 140. This also makes it easier for staff to recognize the degree of attention for each captured image. The display positions and order may be arranged vertically on the screen to make it possible to grasp the degree of attention.

[0126] The server device 1 may determine, with respect to the gaze area image 130, an image of the gaze area for each location of spectators, such as a location where spectators near the home team's goal are gazing, or a location where spectators near the visitor team's goal are gazing, and display the image of the camera 2 that captures that gaze area. Alternatively, the server device 1 may determine the gaze area image 130 of spectators with a specific attribute, and display the image of the camera 2 that captures that gaze area.

[0127] The server device 1 may also display information such as where the spectators being analyzed are sitting and what their attributes are as additional information for each of the gaze area images 130 .

[0128] The server device 1 may also allow staff to select one of multiple captured images. Furthermore, the server device 1 may also allow staff to select which object to automatically track when multiple objects appear in the gaze area image 130. The server device 1 may also generate a contour map 150 as shown in FIG. 21 from the center of gravity of the spectator's gaze point and display it on the monitor device 4.

[0129] 6. Summary and Modifications According to the above embodiment, the following effects can be obtained.

[0130] The information processing device 70 serving as the server device 1 of the embodiment includes a control unit (CPU 71) that performs processing to determine the importance of the gazes of multiple people based on first information, which is information about gaze detection target candidates, and to determine a gaze area based on the gaze information reflecting the importance. This allows the gaze area to be determined with high accuracy. This is because the gazes of people who are suitable for gaze area determination and those who are not suitable can be selected and weighted based on the importance. Furthermore, by accurately using the gazes of people with high importance to determine the gaze area, it is possible to accurately determine not only one gaze area but also multiple gaze areas.

[0131] In the embodiment, an example was given in which the gaze information for determining the gaze area is information on the gaze of a person selected from the gazes of multiple people based on importance. By using the gaze of a person selected based on importance, it is possible to process the gaze of, for example, a person watching a game. This improves the accuracy of determining a gaze area for an important scene or a scene that should be captured, particularly a gaze area suitable for broadcasting, streaming, recording, etc., and reduces the computational processing load on the server device 1.

[0132] In the embodiment, the gaze information for determining the gaze area is gaze information obtained by weighting the gazes of multiple people according to their importance. By using gaze information in which each gaze is weighted based on its importance, the gaze area can be determined by reflecting the degree of attention each spectator is paying to the game. This also improves the accuracy of determining a gaze area suitable for broadcasting, distribution, recording, etc.

[0133] In the embodiment, the first information includes, for example, position information, posture information, attribute information, etc. obtained from the captured video of the gaze detection target candidate. By capturing a video of a spectator who is a gaze detection target candidate, the first information can be obtained from the captured video, including position information (e.g., the position in front of or behind the seat, the area for each favorite team), posture information (posture itself, behavior, speed of movement, movement of specific parts of the body, etc.), and attribute information (clothing such as uniform, supported team, individual identification using facial recognition, gender, age group, etc.). By using this first information to determine the importance, it is possible to determine who is concentrating on the game, the degree of concentration, and who is not concentrating. Therefore, this information is appropriate for determining the importance of gaze.

[0134] Position information makes it possible to select and weight spectators as candidates for gaze detection. For example, if a seat is located at the front, it can be assumed that the person is highly focused on the game. Also, by using information on which team a fan is based on the seat area, it is possible to determine the importance of each favorite team.

[0135] Regarding posture information, various posture information can be determined by using the results of skeleton estimation based on EPTS data, etc. For example, it is possible to determine "movements corresponding to heightened emotions," "looking at a mobile device with one's head down," "exaggerated actions," and so on, which are desirable information for determining importance.

[0136] By using gender and age group as attribute information and selecting and weighting the gaze, it becomes possible to accurately determine the gaze area according to the attribute, such as the gaze area that children are paying attention to or the gaze area that women are paying attention to.

[0137] Although the position information, posture information, and attribute information have been mentioned, it is desirable that the first information includes at least one of these.

[0138] In the embodiment, the first information includes information obtained from gaze detection results. If the viewing target is a match, situational information about an event occurring in the match, such as a shot or a penalty kick, is used as the first information to determine the importance. A person who is watching the location where an event is occurring at the timing of the event is someone who is paying close attention to the match, and therefore can be determined to have an important gaze for determining the gaze area. Furthermore, depending on the degree of gaze concentration, camerawork can be performed in accordance with spectator predictions, such as filming someone who is about to receive a pass.

[0139] In the embodiment, the control unit (CPU 71) of the server device 1 sets the gaze point accuracy based on second information, which is information about gaze detection target candidates or gaze targets, and sets the gazes used for gaze area determination according to the gaze point accuracy. Increasing the number of gazes used for gaze area determination can improve the gaze area determination accuracy, but increases the processing load. Based on the second information, the number of gazes used for gaze area determination is variably set depending on whether accuracy is required or whether a reduction in processing load is necessary. This enables determination according to the situation. In other words, the amount of calculation required for determining the gaze area can be changed based on the state of the game and the spectators, and the trade-off between gaze area accuracy and real-time performance can be reflected depending on the situation of the game. Note that specific methods for setting the gazes used for gaze point determination may include selecting the gazes to be used in the determination calculation, or setting non-target gazes to a weight of zero so that they are effectively not reflected in the gaze point determination.

[0140] In the embodiment, an example was given in which the control unit (CPU 71) of the server device 1 determines the real-time nature of the viewed object based on the second information and sets the gaze point accuracy. In FIG. 7 , the gaze point accuracy parameter is set based on the determination of real-time nature. For example, in a situation where the gaze area changes rapidly during a game, rapid control of the PTZ camera 2A is required. In such a case, in order to increase the response speed of the control, the gaze point accuracy is lowered and the number of lines of sight used to determine the gaze area is reduced. This enables camerawork that follows the scene in real time even with a slight decrease in accuracy.

[0141] In the embodiment, the second information includes either posture information obtained from a captured video of a candidate for gaze detection, or information on the movement of a person or object being viewed obtained by a sensor. A spectator who is a candidate for gaze detection can be captured, and posture information (posture itself, behavior, speed of movement, movement of specific parts of the body, etc.) can be obtained as the second information from the captured video. The posture information can be used to determine the degree of movement in the game.

[0142] For example, the posture of each spectator can be determined using the skeleton estimation results, and the movement of the game, that is, the movement of the gaze area, can be estimated based on the variance of the posture. When the variance is small, the scene is focused on the spectators and requires high real-time processing, so the processing load can be reduced and tracking performance can be emphasized. When the variance is large, the real-time processing is not required, so accuracy can be emphasized. Therefore, posture information is suitable for setting gaze point accuracy parameters that reflect real-time processing.

[0143] Furthermore, since the movement of the gaze area can be estimated from the movement of the ball and the movement of the players obtained by the sensor 5, information on movement such as the speed of movement is suitable for setting gaze point accuracy parameters in real time.

[0144] It is desirable that the second information includes at least one of the posture information obtained from the captured video of these gaze detection target candidates, or the information on the movement of the person or object being viewed obtained by the sensor.

[0145] In the embodiment, the second information includes information obtained from the gaze detection result. For example, when the gaze is concentrated in an area where players are crowded together, this can be considered an important scene, so processing is performed to increase the accuracy parameter (step S124 in FIG. 10). This allows processing to increase accuracy for important scenes.

[0146] In the embodiment, an example was given in which the control unit (CPU 71) of the server device 1 performs gaze detection, then sets gaze point accuracy based on second information, which is information about gaze detection target candidates or gaze targets, determines the importance of each gaze based on the first information, and determines the gaze area based on the gaze information in which weighting for each importance level is set according to the set gaze point accuracy. For example, as shown in Figure 6, gaze detection is performed in step S106, a gaze accuracy parameter is obtained using the second information in step S107, and an importance parameter is calculated using the first information in step S108. Then, as described in Figure 9, a weighting according to the gaze point accuracy (degree of real time) is set to determine the gaze area. This allows gaze point determination according to the situation.

[0147] In the embodiment, an example has been given in which the control unit (CPU 71) of the server device 1 performs processing to determine the importance of each gaze based on first information before performing gaze detection, set gaze point accuracy based on second information that is information about gaze detection target candidates or gaze targets, determine a gaze detection target based on the set gaze point accuracy, perform gaze detection, and determine a gaze area based on gaze information that reflects the importance. For example, as shown in Figure 10, the gaze point accuracy (real-time degree) and importance are calculated, and then a gaze detection target is determined and gaze detection is performed. The gaze area is then determined by reflecting the importance in the gaze information. This type of processing also allows for gaze point determination according to the situation.

[0148] In this embodiment, the control unit (CPU 71) of the server device 1 controls the shooting direction or shooting angle of view of the imaging device based on the determined gaze area. The server device 1 controls pan, tilt, and zoom so that the gaze area is photographed by the PTZ camera 2A. This allows automatic control so that the PTZ camera 2A photographs the area that the audience is gazing at.

[0149] In the embodiment, an example has been given in which the control unit (CPU 71) of the server device 1 determines multiple gaze areas based on line-of-sight information that reflects importance. Since the gaze area is not limited to one, determining multiple gaze areas allows for capturing images without missing any of the gaze areas. For example, it is possible to detect and capture images of the second and third most popular locations attracting the attention of spectators.

[0150] In the embodiment, it has been described that the control unit (CPU 71) of the server device 1 determines multiple gaze areas based on gaze information reflecting importance and calculates the gaze degree for each of the multiple gaze areas. When determining multiple gaze areas, the server device 1 calculates the gaze degree indicating the degree of gaze for each gaze area. This makes it possible to control the multiple gaze areas according to the degree of gaze.

[0151] In the embodiment, an example has been described in which the control unit (CPU 71) of the server device 1 controls the display of the attention level in accordance with the captured image of the attention area. The server device 1 controls the display on the monitor device 4, for example, as shown in Fig. 18. This allows staff such as switcher engineers to check the image of the actual attention area together with the audience attention level, which is convenient for switching main line images, for example.

[0152] In the embodiment, an example has been given in which the control unit (CPU 71) of the server device 1 sets the display mode of the captured video for each gaze area according to the gaze level. For example, the video is displayed in a display mode that allows staff to intuitively recognize the level of gaze level by viewing each video.

[0153] The display mode of the captured image for each gaze area is set by setting at least one of the display size, display position, and display order of the captured image for each gaze area depending on the level of attention. For example, by using the display size, display position, display order, or a combination of these, staff can intuitively recognize the level of attention by looking at each image.

[0154] In the embodiment, an example has been given in which the control unit (CPU 71) of the server device 1 simultaneously displays one or more gaze area images 130 and images of subjects set under conditions other than the gaze area, and also displays the attention level for the images of subjects set under conditions other than the gaze area. For example, when a tracking image 120 is being captured that tracks a specific player as shown in Figure 18, the gaze area image 130 is also displayed, and the attention level 140 for both the tracking image 120 and the tracking image 120 is also displayed. Such a display serves as a guide for switching from the image of the specified player to an image of another gaze area.

[0155] The program of the embodiment is a program that causes, for example, a CPU, a DSP (digital signal processor), an AI processor, or the like, or an information processing device 70 that includes these, to execute the processing shown in Figures 6, 7, and 8, or the processing shown in Figure 10. That is, the program of the embodiment is a program that causes the information processing device 70 to execute processing that determines the importance of the gazes of multiple people based on first information that is information about gaze detection target candidates, and determines a gaze area based on the gaze information that reflects the importance.

[0156] By using such a program, the information processing device 70 constituting the server device 1 of the embodiment can be realized in, for example, a computer device, a mobile terminal device, or other device capable of executing information processing.

[0157] Such a program can be pre-recorded on a hard disk drive (HDD) or a ROM in a microcomputer having a CPU, which serves as a built-in recording medium in a computer or other device. Alternatively, the program can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disc, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called packaged software. Furthermore, such a program can be installed on a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0158] Furthermore, such a program is suitable for providing a wide range of information processing devices 70 constituting the server device 1 of the embodiment. For example, by downloading the program to a mobile terminal device such as a smartphone or tablet, an imaging device, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), or the like, these devices can function as the information processing device 70 constituting the server device 1 of the present disclosure.

[0159] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0160] The present technology may also be configured as follows. (1) An information processing device including a control unit that determines the importance of the gazes of multiple people based on first information that is information about gaze detection target candidates, and performs processing to determine a gaze area based on the gaze information reflecting the importance. (2) The information processing device described in (1) above, wherein the gaze information is information about the gaze of a person selected from the gazes of multiple people based on the importance. (3) The information processing device described in (1) or (2) above, wherein the gaze information is gaze information obtained by weighting the gazes of multiple people based on the importance. (4) The information processing device described in any of (1) to (3) above, wherein the first information includes any of position information, posture information, and attribute information obtained from a captured video of the gaze detection target candidate. (5) The information processing device described in any of (1) to (4) above, wherein the first information includes information obtained from a gaze detection result. (6) The information processing device described in any of (1) to (5) above, wherein the control unit sets gaze point accuracy based on second information that is information about the gaze detection target candidate or a gaze target, and sets a gaze used for gaze area determination based on the gaze point accuracy. (7) The information processing device according to (6) above, wherein the control unit determines the real-time nature of the gaze target based on the second information and sets gaze point accuracy. (8) The information processing device according to (6) or (7) above, wherein the second information includes either posture information obtained from a captured video of a gaze detection target candidate, or information on the movement of a person or object of the gaze target obtained by a sensor. (9) The information processing device according to any of (6) to (8) above, wherein the second information includes information obtained from a gaze detection result. (10) The information processing device according to any of (1) to (9) above, wherein the control unit, after performing gaze detection, sets gaze point accuracy based on second information that is information on a gaze detection target candidate or a gaze target, determines the importance of each gaze based on the first information, and performs a process of determining a gaze area based on gaze information in which a weight for each importance is set in accordance with the set gaze point accuracy.(11) The information processing device described in any of (1) to (9) above, wherein, before performing gaze detection, the control unit determines the importance of each gaze based on the first information, sets gaze point accuracy based on second information that is information on gaze detection target candidates or gaze targets, determines gaze detection targets based on the set gaze point accuracy, performs gaze detection, and determines a gaze area based on the gaze information reflecting the importance. (12) The information processing device described in any of (1) to (11) above, wherein the control unit controls the shooting direction or shooting angle of view of the imaging device based on the determined gaze area. (13) The information processing device described in any of (1) to (12) above, wherein the control unit determines multiple gaze areas based on gaze information reflecting the importance. (14) The information processing device described in any of (1) to (13) above, wherein the control unit determines multiple gaze areas based on the gaze information reflecting the importance, and performs a process of calculating the gaze degree for each of the multiple gaze areas. (15) The information processing device according to (14) above, wherein the control unit performs control to display the degree of attention in accordance with the captured video of the gaze area. (16) The information processing device according to (14) or (15) above, wherein the control unit sets a display mode of the captured video for each gaze area according to the degree of attention. (17) The information processing device according to (16) above, wherein the control unit sets at least one of the display size, display position, and display order of the captured video of each gaze area according to the level of the degree of attention, as the setting of the display mode of the captured video for each gaze area. (18) The information processing device according to any of (15) to (17) above, wherein the control unit performs control to simultaneously display captured video of one or more gaze areas and captured video of a subject set under conditions other than the gaze area, and also display the degree of attention for the captured video of the subject set under conditions other than the gaze area. (19) An information processing method in which an information processing device executes a process of determining the importance of the gazes of multiple people based on first information that is information about gaze detection target candidates, and determining a gaze area based on the gaze information that reflects the importance.(20) A program that causes an information processing device to execute a process of determining the importance of the gazes of multiple people based on first information that is information about candidates for gaze detection targets, and determining a gaze area based on the gaze information that reflects the importance.

[0161] REFERENCE SIGNS LIST 1 Server device 2 Camera 2A PTZ camera 2B Fixed camera for players 2C Fixed camera for spectators 2D Overhead camera 3 Input device 4 Monitor device 5 Sensor 10 Image processing unit 11 Posture detection unit 12 Gaze detection unit 13 Skeleton estimation unit 14 Face recognition unit 15 Uniform number recognition unit 16 Event recognition unit 20 Importance determination unit 21 Accuracy determination unit 22 Gaze area determination unit 23 Position information management unit 24 Initial setting storage unit 25 Camera command generation unit 26 Monitor control unit 70 Information processing device 71 CPU 130, 130a, 130b, 130c, 130d Gaze area image 140 Gaze degree

Claims

1. An information processing device having a control unit that performs a process of determining the importance of the gaze of multiple people based on first information, which is information regarding candidates for gaze detection targets, and determining the gaze area based on the gaze information that reflects the importance.

2. The information processing device according to claim 1, wherein the gaze information is information on the gaze of a person selected according to importance from among a plurality of people's gazes.

3. The information processing device according to claim 1, wherein the gaze information is gaze information obtained by weighting the gazes of a plurality of people according to their importance.

4. The information processing device according to claim 1, wherein the first information includes any one of position information, posture information, and attribute information obtained from a captured image of the candidate for gaze detection.

5. The information processing device according to claim 1, wherein the first information includes information obtained from a gaze detection result.

6. The information processing device according to claim 1, wherein the control unit sets the gaze point accuracy based on second information which is information relating to a gaze detection target candidate or a visually recognized target, and sets the gaze to be used in gaze area determination according to the gaze point accuracy.

7. The information processing device according to claim 6, wherein the control unit determines the real-time nature of the visually recognized object based on the second information and sets the gaze point precision.

8. An information processing device as described in claim 6, wherein the second information includes either posture information obtained from a captured image of a candidate for gaze detection, or information on the movement of a person or object being viewed obtained by a sensor.

9. The information processing device according to claim 6, wherein the second information includes information obtained from a gaze detection result.

10. The information processing device of claim 1, wherein the control unit, after performing gaze detection, sets gaze point accuracy based on second information which is information relating to a gaze detection target candidate or a visual target, determines the importance of each gaze based on the first information, and performs processing to determine the gaze area based on gaze information in which a weighting for each importance level is set according to the set gaze point accuracy.

11. The information processing device of claim 1, wherein the control unit, before performing gaze detection, determines the importance of each gaze based on the first information, sets the gaze point accuracy based on second information which is information regarding a gaze detection target candidate or a visual target, determines a gaze detection target based on the set gaze point accuracy, performs gaze detection, and determines the gaze area based on the gaze information reflecting the importance.

12. The information processing device according to claim 1, wherein the control unit controls the shooting direction or shooting angle of view of the imaging device based on the determined gaze area.

13. The information processing device according to claim 1, wherein the control unit determines a plurality of gaze areas based on gaze information that reflects importance.

14. The information processing device according to claim 1, wherein the control unit performs processing for determining a plurality of gaze areas based on gaze information reflecting importance, and calculating the gaze degree for each of the plurality of gaze areas.

15. The information processing device according to claim 14, wherein the control unit performs control to display the degree of attention in accordance with the captured image of the gaze area.

16. The information processing device according to claim 14, wherein the control unit sets a display mode of the captured image for each gaze area according to the degree of gaze.

17. The information processing device according to claim 16, wherein the control unit sets the display mode of the captured images for each gaze area by setting at least one of the display size, display position, and display order of the captured images for each gaze area depending on the level of gaze.

18. The information processing device according to claim 15, wherein the control unit performs control to simultaneously display images captured in one or more gaze areas and images captured of a subject set under conditions other than the gaze area, and also to display the degree of attention for the images captured of the subject set under conditions other than the gaze area.

19. An information processing method in which an information processing device executes a process of determining the importance of the gazes of multiple people based on first information, which is information about candidates for gaze detection targets, and determining a gaze area based on the gaze information that reflects the importance.

20. A program that causes an information processing device to execute a process of determining the importance of the gazes of multiple people based on first information, which is information regarding candidates for gaze detection, and determining the gaze area based on the gaze information that reflects the importance.