Region-of-interest detection system, region-of-interest detection method, and program

Abstract

The present invention enables accurate detection of a region of interest being closely observed by a subject such as a player and enables the generation of useful information corresponding to various services relating to the subject.　In this system having a region-of-interest detection device, a processing unit of the region-of-interest detection device acquires, as sensor information and using a sensor mounted on a terminal worn on the head of a subject including a player or a referee, at least one of direction measurement information relating to the direction of the head or the line of sight of the subject and focus measurement information relating to the focal length of the eyes, and estimates, as viewpoint information, at least one of the line-of-sight direction, an observation position, an observation area, and a field-of-view area of the subject on the basis of the sensor information.

Description

Region of interest detection system, region of interest detection method and program

[0001] The present invention relates to a region of interest detection system, a region of interest detection method, and a program.

[0002] In various sports, including ball games, it is considered important to know the areas (areas of interest) that players look at while playing in order to improve and enhance their playing skills. Therefore, there is a need to accurately identify each player's area of ​​interest based on the player's line of sight while playing.

[0003] Similarly, it is important for referees in sports to know their areas of interest in order to improve their refereeing skills, so there is a need to accurately identify the referee's areas of interest based on the referee's line of sight, etc.

[0004] Furthermore, large displays that display various information to spectators, as well as zoomed or replayed footage of players in action, are often installed in stadiums, and it is preferable for these displays to display footage that is of interest to as many spectators as possible. Furthermore, in the distribution or broadcasting of footage, it is preferable to distribute footage that is of interest to users who watch the distributed footage, etc., based on the attributes of the users (e.g., the team they support). Therefore, it is necessary to accurately identify the spectators' areas of interest based on the spectators' line of sight, etc.

[0005] Patent Document 1 discloses a technology for taking aerial photographs of a target person using an unmanned aerial vehicle and analyzing the photographs to identify the direction and position of the target.

[0006] International Publication No. 2019 / 239694

[0007] However, the technology of Patent Document 1 does not take into consideration identifying the areas of interest of athletes and judges. Therefore, it is considered difficult to identify the direction in which athletes and judges are facing or the area they are gazing at (area of ​​interest) in, for example, fast-moving sports or sports involving multiple people. Furthermore, the technology of Patent Document 1 does not take into consideration identifying, displaying, and distributing video that will be of interest to a larger number of spectators.

[0008] The present invention has been made in consideration of the above-mentioned problems, and aims to accurately detect the area of ​​interest that a subject, such as an athlete, is gazing at, and generate useful information corresponding to various services related to the subject.

[0009] The present application includes multiple means for solving at least part of the above-mentioned problems, examples of which are as follows: A region of interest system according to one aspect of the present invention for solving the above-mentioned problems is a region of interest detection system having a region of interest detection device, the region of interest detection device including a processing unit, which acquires, as sensor information, at least one of direction measurement information relating to the direction of the head or gaze of the subject, and focus measurement information relating to the focal length of the eyes, using a sensor mounted on a terminal worn on the head of the subject, including an athlete or a referee, and estimates, as viewpoint information, at least one of the gaze direction, gaze position, gaze area, and field of view area of ​​the subject, based on the sensor information.

[0010] According to the present invention, it is possible to accurately detect the area of ​​interest that an object such as an athlete is gazing at, and to generate useful information corresponding to various services related to the object.

[0011] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments.

[0012] 1 is a diagram showing an example of the overall configuration of a region of interest detection system. FIG. 2 is a diagram showing an example of a user of each device in the region of interest detection system. FIG. 3 is a diagram showing an example of the appearance of an unmanned aerial vehicle. FIG. 4 is a block diagram showing an example of the functional configuration of an unmanned aerial vehicle. FIG. 4 is a diagram showing an example of the appearance of a mobile camera. FIG. 5 is a block diagram showing an example of the functional configuration of a mobile camera. FIG. 5 is a diagram showing an example of the appearance of a control device. FIG. 6 is a block diagram showing an example of the functional configuration of a control device. FIG. 6 is a diagram showing an example of the appearance of a wearable terminal. FIG. 7 is a diagram showing an example of the configuration of a wearable terminal. FIG. 8 is a diagram showing an example of the functional configuration of an information processing terminal. FIG. 9 is a diagram showing an example of the functional configuration of a stadium display control terminal. FIG. 10 is a diagram showing an example of the functional configuration of a distribution / broadcast control terminal 630. FIG. 11 is a block diagram showing an example of the functional configuration of a region of interest detection device. FIG. 12 is a diagram showing an example of the execution order of each process performed by the region of interest detection device for players. FIG. 13 is a flow diagram showing an example of a player identification process. FIG. 14 is a diagram showing an example of an image of the process. FIG. 15 is a diagram showing an example of player identification correspondence information. FIG. 16 is a flow diagram showing an example of a player identification process. 1 is a flow diagram showing an example of a region of interest detection process for a player. FIG. 1 is a diagram showing an example of a region of interest. FIG. 1 is a diagram showing an example of definitions of line of sight, viewpoint, and field of view. FIG. 1 is a flow diagram showing an example of a region of interest detection process. FIG. 1 is a flow diagram showing an example of a region of interest detection process. FIG. 1 is a diagram showing an example of evaluation criterion information. FIG. 1 is a diagram showing an example of a display screen (map format) for analysis result display information. FIG. 1 is a diagram showing an example of a display screen (video synthesis format) for analysis result display information. FIG. 1 is a flow diagram showing an example of a tagging process. FIG. 1 is a diagram showing an example of evaluation criterion information. FIG. 1 is a diagram showing an example of the execution order of each process executed by a region of interest detection device for spectators. FIG. 1 is a diagram showing an example of spectator attribute information. A schematic diagram showing spectators and the direction of a gaze area. FIG. 1 is a diagram showing an example of a display screen (map format) for region of interest visualization information. FIG. 1 is a diagram showing an example of a display screen for region of interest visualization information on which spectator attributes are displayed.It is a figure which shows the example of the display screen of the region of interest visualization information which displays the position of the camera. It is a figure which shows the example of the recommended camera information for distribution / broadcast. It is a figure which shows the example of the hardware configuration of the region of interest detection device.

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0014] <Overall Configuration of Region of Interest Detection System> Figure 1 is a diagram showing an example of the overall configuration of a region of interest detection system (hereinafter, sometimes referred to as "this system") 1000. Region of interest detection system 1000 is a system that uses, for example, an unmanned aerial photography aircraft 200 (e.g., a drone) equipped with a camera or a mobile camera 300 to capture images of a competition (particularly the appearance of subjects including athletes (players) and referees) and spectators, and analyzes the captured images (image information) to generate and provide highly useful information that can be used for various services according to the target or purpose.

[0015] Specifically, the system comprises: an unmanned aerial vehicle 200, a mobile camera 300, and a control device 400 that controls the operation of these devices; wearable terminals 500 worn by athletes (hereinafter sometimes referred to as "players"), referees, and spectators; an information processing terminal 600 used by those involved in the competition, a stadium display control terminal 610 that controls displays 620 installed in the stadium, and a distribution / broadcast control terminal 630 that controls the distribution or broadcasting of video of the competition; and a region of interest detection device 100 that uses images obtained from the unmanned aerial vehicle 200 and the mobile camera 300 and sensor information obtained from the wearable terminal 500 to detect the areas (regions of interest) that athletes, referees, and spectators are gazing at, and generates various information.

[0016] Furthermore, system 1000 centers around region-of-interest detection device 100, and each of unmanned aerial vehicle 200, mobile camera 300, wearable terminal 500, information processing terminal 600, stadium display control terminal 610, and distribution / broadcast control terminal 630 is interconnected for mutual communication via a predetermined communication network N. Here, predetermined network N is, for example, the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network). Note that each device may be interconnected for wireless communication or may be connected by wire.

[0017] FIG. 2 shows an example of the users of each device in the region of interest detection system. As shown in the figure, the control device 400 is used by, for example, the operator of a photographing device such as an unmanned aerial vehicle 200 or a mobile camera 300. The wearable device 500 is worn by athletes, referees, and spectators. The information processing terminal 600 is used by a member of a sports team (e.g., a person who coaches athletes). The stadium display control terminal 610 is used by, for example, a person responsible for displaying the video output on the display 620 (or a person who operates the video displayed on the stadium display 620). The distribution / broadcast control terminal 630 is used by, for example, a person responsible for executing distribution or broadcasting.

[0018] In such a region of interest detection system, the operator of the imaging device photographs the athletes, referees, and spectators by piloting (operating) unmanned aerial vehicle 200 or mobile camera 300. Furthermore, region of interest detection device 100 acquires the photographed images of the athletes, referees, and spectators from unmanned aerial vehicle 200 or mobile camera 300 in real time.

[0019] In addition, the wearable devices 500 worn by the athletes, judges, and spectators detect, for example, the movements of the athletes' heads or eyes, and transmit this information as sensor information to the region of interest detection device 100 in real time.

[0020] Furthermore, region of interest detection device 100 performs analysis processing according to the target and purpose using the acquired video and sensor information. Specifically, region of interest detection device 100 performs analysis processing using the video and sensor information to detect (estimate) regions of interest (gaze areas) of athletes and referees identified by their line of sight, field of view, and viewpoint. Region of interest detection device 100 also evaluates the performance of athletes and referees based on the identified regions of interest.

[0021] Furthermore, the region of interest detection device 100 generates tagging information that links the region of interest of a player or the like with a specific scene during play.

[0022] In addition, based on a display request received from the information processing terminal 600, the area of ​​interest detection device 100 generates analysis result display information for displaying the areas of interest and performance evaluation results of players, etc. during competitive play, and displays it on the corresponding terminal.

[0023] Furthermore, the region of interest detection device 100 detects (estimates) a region of interest identified by the gaze, field of view, and viewpoint of the audience based on analytical processing using sensor information and video.

[0024] In addition, the area of ​​interest detection device 100 generates information that visualizes the area of ​​interest of spectators (hereinafter sometimes referred to as "area of ​​interest visualization information") based on display requests received from the stadium display control terminal 610 and the distribution / broadcast control terminal 630, and displays it on the corresponding terminal.

[0025] Furthermore, people involved with the competition team can, for example, provide coaching to players or analyze tactics based on the displayed analysis result information.

[0026] In addition, the person in charge of displaying the images output to the stadium display 620 will understand the areas of interest of the spectators based on the displayed area of ​​interest visualization information, and will display images of the corresponding areas within the competition field (the area where the athletes play) on the stadium display 620.

[0027] In addition, the person in charge of distribution or broadcasting will understand the areas of interest for each spectator's attributes based on the displayed area of ​​interest visualization information, and will distribute or broadcast video of the competition field that corresponds to the area of ​​interest based on the attributes of the user using the video distribution service.

[0028] Such a region of interest detection system can accurately detect the region of interest that an object such as an athlete is gazing at, and generate useful information for various services related to that object.

[0029] The region of interest detection system will now be described in more detail.

[0030] <Configuration of Unmanned Airplane 200> Figure 3 is a diagram showing an example of the appearance of unmanned airplane 200. As shown in the figure, unmanned airplane 200 is, for example, a drone equipped with a camera for aerial photography. Unmanned airplane 200 flies, for example, above a stadium and photographs players and referees on the field or spectators in the stands.

[0031] It should be noted that a drone is a name for an unmanned aircraft, and may also be called an air vehicle, a multicopter, an RPAS (Remote Piloted Aircraft Systems), a UAS (Unmanned Aircraft System), etc.

[0032] Such an unmanned airplane 200 is equipped with, for example, a propeller, a motor, a battery, cameras (a photography camera and an obstacle detection camera), a GPS (Global Positioning System) receiving device, various sensors, a control device, and a communication device.

[0033] The various sensors include, for example, sensors for detecting the direction, altitude, and speed of the flying unmanned airplane 200, as well as radar sensors such as a ToF (Time Of Flight) sensor and LiDAR.

[0034] The control device is a device that controls the flight, photography, etc. of the unmanned airplane 200, and includes at least a processor (e.g., CPU: Central Processing Unit), non-volatile memory (e.g., ROM: Read Only Memory) that stores programs and various information, and volatile memory (e.g., RAM: Random Access Memory) that temporarily stores information such as calculation results. The control device controls the motor, propellers, camera, etc. based on control signals obtained from the control device 400 of the unmanned airplane 200, thereby performing flight, photography, etc. in accordance with control instructions.

[0035] The communication device is a device that communicates with an external device (for example, the control device 400 or the region-of-interest detection device 100).

[0036] 4 is a block diagram showing an example of the functional configuration of the unmanned airplane 200. As shown in the figure, the unmanned airplane 200 has a state detection unit 210, a flight function unit 220, an obstacle detection unit 230, an image capture unit 240, and a communication unit 250.

[0037] The state detection unit 210 is a functional unit that detects the flight state of the unmanned airplane 200. Specifically, the state detection unit 210 has a position detection unit 211, a direction detection unit 212, an altitude detection unit 213, and a speed detection unit 214 for detecting the position, direction, altitude, and speed of the unmanned airplane 200.

[0038] The position detection unit 211 is realized by, for example, a GPS receiving device. For example, the position detection unit 211 receives a signal from a GPS satellite and detects its own position on the competition field by matching the global coordinate system indicated by the signal with a coordinate system virtually set up on the competition field (hereinafter sometimes referred to as the "field coordinate system").

[0039] The direction detection unit 212, the altitude detection unit 213, and the speed detection unit 214 are realized by corresponding sensors (for example, a direction sensor, an altitude sensor, and a speed sensor), respectively.

[0040] The flight function unit 220 is a function unit that controls the flight of the unmanned airplane 200. Specifically, the flight function unit 220 is composed of a motor 221, a propeller 222, and a flight control unit 223. Here, the flight control unit 223 controls the amount of operation of the motor 221 and the propeller 222 based on control signals acquired from the piloting device 400, thereby controlling flight above the stadium. Note that the flight control unit 223 is realized by a program executed by the control device.

[0041] Obstacle detection unit 230 is a functional unit that detects obstacles around unmanned airplane 200. Specifically, obstacle detection unit 230 is composed of obstacle detection camera 231, ToF sensor 232, and radar sensor 233, and detects obstacles around unmanned airplane 200 based on information detected by these cameras and sensors. Note that obstacle detection unit 230 does not need to be equipped with all of obstacle detection camera 231, ToF sensor 232, and radar sensor 233, and it is sufficient if at least one of them is installed.

[0042] Furthermore, the obstacle detection camera 231 may be provided not only on the front of the unmanned airplane 200 but also on all sides (six sides in total: front / rear, left / right, top / bottom), or on any side (for example, front / rear and top / bottom). The same is true for the ToF sensor 232 and radar sensor 233, and the sides on which they are mounted are not particularly limited.

[0043] The photographing unit 240 is a functional unit that photographs subjects (e.g., players, referees, and spectators). Specifically, the photographing unit 240 is composed of a photographing camera 241, a camera holding unit 242, and a holding control unit 243. Here, the photographing camera 241 is a camera for photographing players and the like, and is, for example, a high-performance camera equipped with a zoom function with a predetermined magnification. The camera holding unit 242 is a connecting mechanism that movably holds and fixes the photographing camera 241 to the unmanned airplane 200. The holding control unit 243 controls changing the orientation of the photographing camera 241 by moving the camera holding unit 242. The holding control unit 243 is realized by a program executed by the control device.

[0044] Communication unit 250 is a functional unit that communicates with external devices (e.g., control device 400 and region-of-interest detection device 100). Specifically, communication unit 250 receives control signals for flight and photography from control device 400, and transmits images captured by photography unit 240 to region-of-interest detection device 100. Note that communication unit 250 is realized by communication device 806.

[0045] <Mobile camera 300> Figure 5 is a diagram showing an example of the appearance of a mobile camera. The upper diagram shows a so-called wire camera 300a. Wire camera 300a moves while suspended from a wire installed above the competition field, and captures images of players and referees on the competition field, as well as spectators in the stands. The lower diagram shows a so-called rail camera 300b. Rail camera 300b moves on rails installed around the perimeter of the competition field, and captures images of players and referees on the competition field, as well as spectators in the stands.

[0046] Such a mobile camera 300 includes, for example, a motor for movement, a photographing camera, a GPS receiving device, various sensors, a control device, and a communication device. Note that the photographing camera, GPS receiving device, various sensors, and communication device are the same as those of the unmanned airplane 200, so detailed description thereof will be omitted.

[0047] The control device is a device that controls the movement and photography of the mobile camera 300, and includes at least a processor such as a CPU, a non-volatile memory such as a ROM, and a volatile memory such as a RAM. The control device controls the motors, the photography camera, etc. based on control signals obtained from the control device 400, thereby performing movement, photography, etc. in accordance with the control instructions.

[0048] 6 is a block diagram showing an example of the functional configuration of a mobile camera 300. As shown in the figure, the mobile camera 300 has a state detection unit 310, a movement function unit 320, an image capture unit 330, and a communication unit 340.

[0049] The state detection unit 310 is a functional unit that detects the movement state of the mobile camera 300. The photographing unit 330 is a functional unit that photographs subjects (e.g., players, referees, and spectators). The communication unit 340 is a functional unit that communicates with external devices (e.g., the control device 400 and the region-of-interest detection device 100). The basic configuration of these functional units is the same as that of the unmanned airplane 200, so a detailed description will be omitted.

[0050] The movement function unit 320 is a function unit that controls the movement of the mobile camera 300. Specifically, the movement function unit 320 is composed of a motor 321 and a movement control unit 322. Here, the movement control unit 322 controls the amount of movement of the motor based on a control signal obtained from the operation device 400, and controls movement around the competition field. The movement control unit 322 is realized by a program executed by the control device.

[0051] It should be noted that the region of interest detection system 1000 is not limited to the mobile camera 300, and may also employ a fixed camera positioned to capture, for example, images that allow the head orientation and uniforms of players, referees, and spectators to be identified.

[0052] <Control Device> Fig. 7 is a diagram showing an example of the appearance of a control device. The control device 400 is used to control the unmanned airplane 200 and the mobile camera 300, and controls the movement and shooting of these devices.

[0053] As shown in the figure, the control device 400 has a controller 410 and a display device 420. Note that the control device 400 may have the controller 410 and the display device 420 integrated into one unit, or the display device 420 may be detachable from the controller 410.

[0054] Display device 420 is a device for displaying images captured by unmanned airplane 200 or mobile camera 300. Note that display device 420 may also be a mobile terminal with a display, such as a smartphone or tablet terminal. In this case, it is sufficient if dedicated application software for displaying images captured by unmanned airplane 200 or mobile camera 300 is pre-installed on the smartphone or the like.

[0055] The controller 410 has a plurality of operation input units to which various functions related to operation are assigned. Specifically, the operation input units have a pair of left and right sticks 411 and input buttons 412.

[0056] When the illustrated control device is a control device for the unmanned airplane 200, the left stick 411 is assigned, for example, functions to turn left and right and move forward and backward of the unmanned airplane 200. Furthermore, the right stick 411 is assigned, for example, functions to move left and right and make the unmanned airplane 200 ascend and descend.

[0057] Such a controller 410 has a control device that generates a control signal according to input information received via an operation input unit such as a stick 411, and a communication device that transmits the generated control signal to an unmanned aircraft 200, etc.

[0058] The control device is a device that generates control signals for the unmanned airplane 200, etc., and includes at least a processor such as a CPU, a non-volatile memory such as a ROM, and a volatile memory such as a RAM. The control device generates a control signal corresponding to input information received from the operation input unit (for example, a control signal for turning the unmanned airplane 200 left when the left stick 411 is tilted left).

[0059] The communication device is a device that communicates with unmanned airplane 200 and mobile camera 300. Communication device 806 transmits control signals generated by the control device to unmanned airplane 200 or mobile camera 300.

[0060] 8 is a block diagram showing an example of the functional configuration of a control device 400. As shown in the figure, the control device 400 has a display unit 401, a control signal generation unit 402, and a communication unit 403.

[0061] The display unit 401 is a functional unit that displays images captured by the unmanned airplane 200 and the mobile camera 300. The display unit 401 is realized by a display device.

[0062] The control signal generating unit 402 is a functional unit that generates a control signal based on input information received via the operation input unit 100. The control signal generating unit 402 is realized by a control device.

[0063] The communication unit 403 is a functional unit that transmits control signals to the unmanned airplane 200 and the mobile camera 300. The communication unit 403 is realized by a communication device 806.

[0064] <Wearable Terminal 500> A diagram showing an example of a wearable terminal. The wearable terminal 500 is a terminal device that is worn on the head of a player, a referee, or a spectator.

[0065] Fig. 9 is a diagram showing an example of the appearance of the wearable terminal 500. Fig. 10 is a diagram showing an example of the configuration of the wearable terminal 500. The first wearable terminals 500a and 500b and the second wearable terminal 500c are realized by, for example, a hair band incorporating a compass sensor 501, eyewear 510, etc.

[0066] As shown in the figure, the first wearable terminal 500a has a pair of compass sensors (orientation sensors) 501 at the front and back of the head where it is worn (front and back of the head), while the first wearable terminal 500b has a pair of compass sensors 501 at the left and right of the head where it is worn.

[0067] The second wearable device 500c also includes a pair of compass sensors 501 mounted on the left and right sides of the wearer's head, and eyewear 510 positioned directly in front of the wearer's eyes. The eyewear 510 is equipped with a sensor capable of measuring a person's eye movements (e.g., eyeball movements and focal length). As shown in the figure, the eyewear 510 detects the gaze direction based on the state of the eyeballs of both eyes and identifies the position where the gaze direction vectors intersect as the focal length. The eyewear may also detect the shape of the lens of one eyeball and identify the focal length based on that shape, for example.

[0068] The wearable terminals 500a, 500b, and 500c are each equipped with a positioning sensor (e.g., a GPS receiver) 502 for acquiring position information of the wearer. The wearable terminals 500a, 500b, and 500c are also equipped with a communication device 503 (not shown) for transmitting information detected or measured by each sensor to an external device (e.g., the region-of-interest detection device 100).

[0069] Such wearable terminals 500a and 500b periodically (e.g., every second) transmit sensor information including information indicating the direction in which the wearer's head is facing and the three-dimensional angle of the head (direction measurement information) and information indicating the wearer's position (e.g., if the wearer is an athlete, the position on the playing field) (position information), as well as identification information of each wearable terminal (hereinafter sometimes referred to as sensor identification information), to region of interest detection device 100. In addition to this information, wearable terminal 500c also transmits to region of interest detection device 100 sensor information including information indicating the gaze direction based on the movement of the wearer's eyeballs detected by eyewear 510 (direction measurement information) and information indicating the focal length (focus measurement information).

[0070] <Information Processing Terminal 600> Fig. 11 is a diagram showing an example of the functional configuration of an information processing terminal. The information processing terminal 600 is a device that displays information acquired from the region of interest detection device 100. Specifically, the information processing terminal 600 is an information processing device such as a smartphone, a tablet terminal, or a personal computer. The information processing terminal 600 has a display unit 601, an input unit 602, and a communication unit 603.

[0071] The display unit 601 is a functional unit that displays various information, such as analysis result display information generated by the region of interest detection device 100, and information such as video captured by the unmanned airplane 200 or the mobile camera 300.

[0072] The input unit 602 is a functional unit that accepts information input via an input device 801 (for example, a touch panel, a keyboard, a mouse, etc.) that is mounted on or connected to the information processing terminal 600. Specifically, the input unit 602 accepts instructions (for example, a request to display analysis result display information or a designation of a specific player for which analysis results are to be displayed) and information input from a person involved with a competition team via the input device.

[0073] The communication unit 603 is a functional unit that communicates information with an external device (e.g., the region-of-interest detection device 100). Specifically, the communication unit 603 transmits input information to the external device. The communication unit 603 also acquires various types of information (e.g., analysis result display information) from the external device.

[0074] <Stadium display control terminal 610> Figure 12 shows an example of the functional configuration of the stadium display control terminal. The stadium display control terminal 610 is a device for controlling the images output to the displays 620 installed in the stadium. Specifically, the stadium display control terminal 610 is an information processing device such as a smartphone, tablet terminal or personal computer. The stadium display control terminal 610 has a display unit 611, an input unit 612 and a communication unit 613.

[0075] The basic functions of display unit 611, input unit 612, and communication unit 613 are the same as those of information processing terminal 600. Display unit 611 displays the region-of-interest visualization information of spectators generated by region-of-interest detection device 100. Input unit 612 accepts instructions and information input from the user of the terminal via the input device (e.g., a request to display region-of-interest visualization information, or a display video specification that specifies a specific video to be displayed from multiple display candidate videos). Communication unit 613 acquires region-of-interest visualization information from region-of-interest detection device 100. Communication unit 613 also transmits instruction information regarding the video to be displayed on stadium display 620 to region-of-interest detection device 100.

[0076] <Distribution / Broadcast Control Terminal 630> Fig. 13 is a diagram showing an example of the functional configuration of the distribution / broadcast control terminal 630. The distribution / broadcast control terminal 630 is a device for controlling video to be distributed or broadcast. Specifically, the distribution / broadcast control terminal 630 is an information processing device such as a smartphone, a tablet terminal, or a personal computer. The information processing terminal 600 has a display unit 631, an input unit 632, and a communication unit 633.

[0077] The basic functions of the display unit 631, the input unit 632, and the communication unit 633 are the same as those of the information processing terminal 600. The display unit 631 displays the region of interest visualization information of the audience generated by the region of interest detection device 100. The input unit 632 accepts instructions and information input from the user of the terminal (for example, a request to display the region of interest visualization information or a video specification that specifies a specific video to be distributed or broadcast) via the input device. The communication unit 633 acquires the region of interest visualization information from the region of interest detection device 100. The communication unit 633 also transmits instruction information regarding the video to be distributed or broadcast to the region of interest detection device 100.

[0078] The functional units of the information processing terminal 600, the stadium display control terminal 610, and the distribution / broadcast control terminal 630 are realized by a computing device such as a CPU executing a program stored in the memory installed in these terminals. Furthermore, the communication units 603, 613, and 633 are realized by a communication device installed in these terminals.

[0079] <Region of interest detection device 100> Figure 14 is a block diagram showing an example of the functional configuration of a region of interest detection device. Region of interest detection device 100 is a device that detects the regions of interest of players, referees, and spectators using video acquired from unmanned aerial vehicle 200 and mobile camera 300 and sensor information acquired from wearable device 500. Furthermore, region of interest detection device 100 generates information to be used for various services according to the target or purpose, based on the detected region of interest.

[0080] As shown in the figure, the region of interest detection device 100 includes a processing unit 110, a storage unit 120, and a communication unit 130.

[0081] The processing unit 110 is a functional unit that performs various processes executed by the region of interest detection device 100. The processing unit 110 also includes an input unit 111, a classification unit 112, a region of interest detection unit 113, a performance evaluation unit 114, an information generation unit 115, an output unit 116, and a camera selection unit 117.

[0082] The input unit 111 is a functional unit that accepts input of instructions and information (e.g., images captured by the unmanned aircraft 200 or the mobile camera 300, sensor information obtained from the wearable terminal 500, and other information stored in the memory unit) from an operator of the area of ​​interest detection device 100 via an input device (e.g., a touch panel, a keyboard, a mouse, etc.) connected to the device.

[0083] The identification unit 112 is a functional unit that uses video or sensor information, or both, to identify players and the like on the playing field.

[0084] The area of ​​interest detection unit 113 uses video or sensor information or both to estimate (detect) the gaze direction, gaze position (viewpoint), field of view (visible area), or gaze area (area of ​​interest) of each identified player, etc. as viewpoint information.

[0085] The performance evaluation unit 114 is a functional unit that evaluates the performance of players and referees using the detected region of interest and information (evaluation criteria information) in the storage unit 120.

[0086] The information generation unit 115 is a functional unit that generates various types of information. Specifically, the information generation unit 115 generates analysis result display information based on the player identification results and the region of interest detection results. The information generation unit 115 also generates region of interest visualization information that visualizes the detected region of interest.

[0087] The output unit 116 is a functional unit that outputs (transmits, distributes, or broadcasts) the generated information. Specifically, the output unit 116 transmits analysis result display information to the information processing terminal 600. The output unit 116 also transmits region of interest visualization information to the stadium display control terminal 610 or the distribution / broadcast control terminal 630.

[0088] Based on the acquired instruction information, the output unit 116 outputs the video to the stadium display 620. Based on the acquired instruction information, the output unit 116 also distributes (or broadcasts) the video of the game play to users viewing the video.

[0089] The camera selection unit 117 is a functional unit that selects a camera that captures an area where the audience's areas of interest are concentrated.

[0090] The storage unit 120 is a functional unit that stores various types of information. Specifically, the storage unit stores player identification correspondence information 121 and evaluation criteria information 122.

[0091] The player identification correspondence information 121 is information that associates predetermined player identification information (e.g., identification number) with identification information of the wearable device 500. The evaluation criterion information 122 is information that serves as a criterion for evaluating the performance of a player. Details of this information will be described later.

[0092] The storage unit 120 also stores images captured by the unmanned aerial vehicle 200 and the mobile camera 300. The storage unit 120 also stores information acquired from the wearable terminal 500, such as sensor information, sensor identification information, eye movements and focal length of the wearer, etc.

[0093] The storage unit 120 also stores the player identification information of each player identified by the identification unit 112. The storage unit 120 also stores the region of interest of each player detected by the region of interest detection unit 113.

[0094] Communication unit 130 is a functional unit that communicates information with external devices. Specifically, communication unit 130 communicates information with unmanned airplane 200, mobile camera 300, wearable device 500, information processing terminal 600, stadium display control terminal 610, and distribution / broadcast control terminal 630.

[0095] 15 is a diagram showing an example of the execution order of each process performed by region of interest detection device 100 on a player. As shown in the figure, region of interest detection device 100 acquires images and sensor information of a player playing a game from unmanned aerial vehicle 200 or mobile camera 300 (steps S10 and S20).

[0096] In addition, the area of ​​interest detection device 100 performs a player identification process (step S30) and an area of ​​interest detection process (step S40) using video, sensor information, etc., in real time during competitive play or at a predetermined timing (for example, any timing after the end of the competition).

[0097] Furthermore, the region-of-interest detection device 100 performs a performance evaluation process in real time during the game or at a predetermined timing (for example, at any timing after the player identification process and region-of-interest detection process are completed) (step S50).

[0098] Furthermore, the region of interest detecting apparatus 100 performs an analysis result display process in response to a display request from the information processing terminal 600 (step S60).

[0099] Furthermore, the area of ​​interest detection device 100 performs tagging processing of a video at a specific point in time (for example, a video at the point in time when a noteworthy scene occurs, hereinafter sometimes referred to as "noteworthy video") based on the performance evaluation results at a predetermined timing (at least at any timing after the performance evaluation processing or when an execution instruction is received from the information processing terminal 600) (step S70).

[0100] <<<<First Form of Player Identification Processing (Step S30)>>> Fig. 16 is a flow chart showing an example of player identification processing. Fig. 17 is a diagram showing an example of the processing.

[0101] When this process starts, the identification unit 112 identifies the position of each player on the competition field based on the sensor information (step S31-1). Specifically, the identification unit 112 identifies the position of each wearable device 500 on the competition field based on the position information included in the acquired sensor information. The identification unit 112 also identifies player identification information corresponding to the sensor device identification information using the player identification correspondence information 121. The identification unit 112 also identifies the position of each player on the competition field by associating the identified player identification information with a corresponding position on the competition field. Note that, hereinafter, positions identified in this way may be referred to as sensor-identified positions.

[0102] 18 is a diagram showing an example of the player identification correspondence information 121. As shown in the figure, the player identification correspondence information 121 is information in which sensor identification information, player identification information, and team identification information are associated with each other.

[0103] By the process of step S31-1, as shown in FIG. 17a, player identification information (for example, A03, B05, etc.) is associated with the position of each player on the playing field (indicated by a star mark).

[0104] Next, the identification unit 112 estimates the position of each player on the competition field based on the video (step S32-1). Specifically, the identification unit 112 performs image analysis (image recognition) using video captured from various directions by the unmanned aerial vehicle 200 and the mobile camera 300, and estimates the position of each player on the competition field based on the analysis results regarding, for example, the positions of the unmanned aerial vehicle 200 and the mobile camera 300, the distance between the camera and the competition field, the angle and length of the white lines drawn on the competition field, the size of the players, etc. Note that any known technology may be used for this image analysis (image recognition). In the following, the positions estimated in this manner may also be referred to as the "video estimated positions."

[0105] By performing the process in step S32-1, the position of each player on the playing field is estimated based on the video, as shown in FIG. 17b.

[0106] Next, the identification unit 112 determines for each player whether a specific sensor position that matches the video estimated position has been identified (step S33-1). Specifically, the identification unit 112 compares the video estimated position with the specific sensor position, and if there is a specific sensor position that is less than a predetermined distance from the video estimated position, it determines that a specific sensor position that matches the video estimated position has been identified. Note that if there are multiple specific sensor positions that are less than the predetermined distance from one video estimated position, the closest specific sensor position can be used.

[0107] The identification unit 112 then proceeds to step S34-1 for players that it determines have been identified. On the other hand, the identification unit 112 returns to step S31-1 for players that it determines cannot be identified. In step S31-1 again, the identification unit 112 performs steps S31-1 to S33-1 based on newly acquired sensor information.

[0108] Next, in step S34-1, the identification unit 112 stores player identification information etc. in association with the players in the video. Specifically, the identification unit 112 stores the player identification information and sensor identification information of the player at the sensor specific position determined to match the video estimated position in association with each player at the corresponding position in the video.

[0109] By the process of step S34-1, player identification information (for example, A03, B02, etc.) is associated with each player, as shown in FIG. 17c.

[0110] In addition, by repeatedly executing steps S31-1 to S34-1, the identification unit 112 associates each player in the video with their player identification information, etc. for the entire duration of the competitive play (match), and then ends this flow.

[0111] <<<<Second Form of Player Identification Processing (Step S30)>>> In this form, the uniforms and jersey numbers of players are identified by image analysis (image recognition) using video, and are stored in association with each player in the video.

[0112] Fig. 19 is a flow chart showing an example of the player identification process according to this embodiment, and Fig. 20 is a diagram showing an example of the process.

[0113] When the process starts, the identification unit 112 identifies the team to which each player belongs based on the video (step S31-2). Specifically, the identification unit 112 performs image analysis using the video images taken from various directions by the unmanned aerial vehicle 200 and the mobile camera 300, and identifies the team to which each player belongs based on the design features of the uniform, such as color, markings, or patterns.

[0114] By the process of step S31-2, the team to which each player on the playing field belongs is identified, as shown in FIG. 20a.

[0115] Next, the identification unit 112 identifies the uniform number of each player based on the video (step S32-2). Specifically, the identification unit 112 identifies the uniform number of each player based on image analysis.

[0116] By the process of step S32-2, the uniform number of each player on the playing field is identified as shown in FIG. 20b.

[0117] Next, the identification unit 112 determines for each player whether or not the team and uniform number of each player can be identified from the video (step S33-2). For players for which it is determined that the player has been identified, the process proceeds to step S34-2. On the other hand, for players for which it is determined that the player cannot be identified, the process returns to step S31-2. Note that in step S31-2 again, the identification unit 112 performs steps S31-2 to S33-2 based on newly acquired video.

[0118] Next, the identification unit 112 uses the player identification correspondence information 121 to identify the player identification information of each player based on the team and uniform number of each identified player (step S34-2).

[0119] 21 is a diagram showing an example of the player identification correspondence information 121 used in this embodiment. As shown in the figure, the player identification correspondence information 121 is information in which player identification information, uniform numbers, and team identification information (uniforms) are associated with each other.

[0120] By the process of step S34-2, as shown in FIG. 20c, the player identification information (for example, A03, B02, etc.) specified based on the player identification correspondence information 121 is associated with each player on the competition field.

[0121] Next, the identification unit 112 stores the identified player identification information in association with each player in the video (step S35-2). Note that if sensor information has also been acquired from the wearable device 500 of each player, the identification unit 112 stores the sensor information in association with each player in addition to the player identification information.

[0122] In addition, by repeatedly executing steps S31-2 to S35-2, the identification unit 112 stores each player in the video in association with their player identification information, etc. for the entire duration of the competitive play (match), and then ends this flow.

[0123] <<<<Third Form of Player Identification Processing (Step S30)>>> This form is a combination of the above-described first and second forms. Specifically, in this form, for players who cannot be identified by the identification processing of the first form, the player is identified based on the identification processing of the second form.

[0124] 22 is a flow diagram showing an example of the player identification process according to this embodiment. Steps S31-3 to S34-3 correspond to steps S31-1 to S34-1 in the first embodiment. Steps S35-3 to S39-3 correspond to steps S31-2 to S35-2 in the second embodiment. Therefore, details of the process in each step will be omitted.

[0125] In this embodiment, in step S34-3, it is determined for each player whether a sensor-specific position that matches the video estimated position has been identified, and for players for whom it is determined that the position cannot be identified, the processes of steps S36-3 to S39-3 are carried out.

[0126] That is, when the identification unit 112 of this embodiment cannot identify each player from the position based on the sensor information and the video, it identifies the corresponding player based on image recognition of the uniform of the relevant player.

[0127] According to the identification processes of the first to third types, each player in the video can be identified with high accuracy using the video and sensor information.

[0128] <<<<First Form of Player's Region of Interest Detection Process (Step S40)>>> Fig. 23 is a flowchart showing an example of a player's region of interest detection process. Note that the process of this form is performed when the wearable devices 500a and 500b shown in Fig. 9 are used.

[0129] When the process starts, the region of interest detection unit 113 identifies the gaze direction in which the player is estimated to be gazing, based on the direction measurement information included in the sensor information acquired from the wearable device 500 (step S41-1).

[0130] Next, the region of interest detection unit 113 identifies the position of each player on the competition field based on the position information included in the sensor information (step S42-1). Note that the region of interest detection unit 113 may estimate the position of the players on the competition field from images captured by the unmanned airplane 200 or the mobile camera 300, for example, in a manner similar to that of step S32-1 described above.

[0131] Next, the region of interest detection unit 113 detects (estimates) the region of interest (gaze area) of each player (step S43-1). For example, the region of interest detection unit 113 estimates a gaze area of ​​a predetermined shape and size based on the direction measurement information. Specifically, the region of interest detection unit 113 estimates a gaze area indicated by a sector (fan shape) or trapezoid whose edge is a predetermined distance (for example, any distance within a range of several meters to several tens of meters) from the player's position in the player's line of sight. The region of interest detection unit 113 then detects such a gaze area as the player's region of interest.

[0132] In addition, the region of interest detection unit 113 may estimate the player's gaze position (viewpoint) on the competition field based on the direction measurement information, and may, for example, estimate a fan-shaped (or trapezoidal) range having a length (distance) from the player's position to the viewpoint and extending at a predetermined angle (for example, any angle in the range of 90 degrees to 120 degrees) as the player's gaze area.

[0133] In addition, the area of ​​interest detection unit 113 may estimate a field of view (field of view) that extends at a predetermined angle (for example, any angle within a range of 60 degrees to 120 degrees) in the player's line of sight based on the direction measurement information, and thereby estimate, for example, a fan-shaped (sector-shaped) or trapezoidal gaze area whose edge is a predetermined distance from the player's position (for example, any distance within a range of several meters to several tens of meters).

[0134] 24 is a diagram showing an example of a region of interest. As shown in the figure, a region of interest 700 is a trapezoidal or fan-shaped region positioned in the player's line of sight (direction in front of the head) 701.

[0135] 25 is a diagram showing an example of a region of interest 700 when the distance to the gaze position (viewpoint) is taken into consideration. As shown in the figure, the size of the region of interest 700 varies depending on the distance from the player's position to the gaze position 702. For example, the shorter the distance from the player's position to the gaze position 702, the smaller the size of the gaze area specified in the player's line of sight, and the longer the distance from the player's position to the gaze position 702, the larger the size of the gaze area.

[0136] Here, the definitions of the line of sight (line of sight direction), viewpoint (gaze position), and field of view (visible area) in this embodiment will be described with reference to FIG.

[0137] 26 is a diagram showing an example of the definition of the line of sight, viewpoint, and field of view. In this embodiment, the line of sight is defined as the direction (vector) of the line of sight. Note that the line of sight includes not only a two-dimensional horizontal vector, but also a three-dimensional vector including an up-down direction.

[0138] The visual field is defined as the range that can be seen without moving the eyes. The same meaning applies to the field of view. As shown in the figure, the visual field is a fan-shaped (or trapezoidal) range (the area inside the boundary line) that includes the central visual field, which is clearly visible, and the peripheral visual field, which is vaguely visible. The visual field includes not only a two-dimensional fan-shaped range in the horizontal direction, but also a three-dimensional cone range that includes the vertical direction.

[0139] The viewpoint is defined as a point, a plane, or a spherical range in three-dimensional space that is being gazed at.

[0140] From this definition, the gaze area (region of interest) is an area included in the field of view, and can be defined as the field of view (field of view) of the distance from the subject's position to the gaze position (viewpoint). The gaze area also includes an area set in a three-dimensional space including not only a two-dimensional plane but also the vertical direction.

[0141] In this way, region of interest detection unit 113 estimates the region of interest of each player for the entire time during competitive play (match) by repeatedly executing steps S41-1 to S43-1. Region of interest detection unit 113 also associates the detected region of interest with the player identification information, etc., of each player in the video identified in step S30, and stores the associated information in the storage unit (step S44-1). Region of interest detection unit 113 then terminates this flow after executing step S44-1.

[0142] <<<<Second Form of Player's Region of Interest Detection Process (Step S40)>>> This form is processing when the wearable terminal 500c (wearable terminal equipped with the compass sensor 501 and eyewear 510) in FIG. 9 is used.

[0143] 27 is a flow diagram showing an example of region-of-interest detection processing according to the second embodiment. When the processing starts, the region-of-interest detection unit 113 identifies the gaze direction in which the player is estimated to be gazing, based on direction measurement information included in the sensor information acquired from the wearable device 500 (step S41-2).

[0144] Next, the region of interest detection unit 113 identifies the position of each player on the competition field based on the position information included in the sensor information (step S42-2). Note that the region of interest detection unit 113 may also estimate the position of the players on the competition field from the video, for example, using a method similar to that of step S42-1 described above.

[0145] Next, the region of interest detection unit 113 detects (estimates) the region of interest (gaze area) of each player (step S43-2). Specifically, the region of interest detection unit 113 estimates a gaze area of ​​a predetermined shape and size based on the direction measurement information (particularly the state of the eyeballs) in the same manner as in step S43-1.

[0146] In addition, the region of interest detection unit 113 may estimate the player's gaze position (viewpoint) on the competition field based on the focus measurement information, and may, for example, estimate a fan-shaped (or trapezoidal) range having a length (distance) from the player's position to the viewpoint and extending at a predetermined angle (for example, any angle in the range of 90 degrees to 120 degrees) as the player's gaze area.

[0147] In addition, the area of ​​interest detection unit 113 may estimate a field of view (field of view) that extends at a predetermined angle (for example, any angle within a range of 60 degrees to 120 degrees) in the player's line of sight based on the focus measurement information, and thereby estimate, for example, a fan-shaped (sector-shaped) or trapezoidal gaze area whose edge is a predetermined distance from the player's position (for example, any distance within a range of several meters to several tens of meters).

[0148] In this way, region of interest detection unit 113 repeatedly executes steps S41-2 to S43-2 to detect the region of interest of each player for the entire duration of competitive play (match). Region of interest detection unit 113 also associates viewpoint information including the detected region of interest with the player identification information, etc., of each player in the video identified in step S30, and stores the associated information in the storage unit (step S44-2). Region of interest detection unit 113 then terminates this flow after processing this step.

[0149] <<<<Third form of player's area of ​​interest detection processing (step S40)>>> In this form, when there is a player whose area of ​​interest cannot be estimated (detected) by the above-mentioned second form of area of ​​interest detection processing, the player's area of ​​interest is estimated from footage captured by an unmanned aircraft 200 or a mobile camera 300.

[0150] 28 is a flowchart showing an example of region of interest detection processing according to the third embodiment. Note that steps S41-3 to S45-3 correspond to steps S41-2 to S45-2 in the second embodiment, and therefore detailed description of the processing will be omitted.

[0151] In step S44-3, the region of interest detection unit 113 determines whether or not the region of interest of each player has been estimated. For players for which the region of interest has been estimated, the process proceeds to step S45-3, and the flow ends. On the other hand, for any player for which the region of interest cannot be estimated, the process proceeds to step S46-3.

[0152] In step S46-3, the region of interest detection unit 113 identifies the player's line of sight (or the direction in which the front of the head is facing) based on image analysis using the video.

[0153] Next, the region of interest detection unit 113 estimates the position of the player on the playing field based on image analysis using the video (step S47-3). Note that this process may be performed using the same method as step S32-1 in the above-mentioned identification process.

[0154] Next, the area of ​​interest detection unit 113 estimates the player's gaze area based on the identified player's gaze direction (or the direction in which the head is facing) and the player's position on the playing field, and detects the gaze area as the player's area of ​​interest (step S48-3).

[0155] In this way, region of interest detection unit 113 estimates (detects) the region of interest of each player for the entire duration of competitive play (match) by repeatedly executing steps S41-3 to S48-3. Furthermore, region of interest detection unit 113 associates viewpoint information including the detected region of interest with the player identification information, etc., of each player in the video identified in step S30, and stores the associated information in the storage unit (step S48-3). Furthermore, region of interest detection unit 113 terminates this flow after processing this step.

[0156] According to the region of interest detection process according to the first to third modes, the region of interest can be detected with high accuracy based on the viewpoint information of each player.

[0157] <<<<Player Performance Evaluation Process (Step S50)>>> The performance evaluation unit 114 evaluates the performance of each player during competitive play based on viewpoint information (gaze direction, gaze position (viewpoint), field of view (visible area), or gaze area (area of ​​interest)) and evaluation criteria information 122.

[0158] 29 is a diagram showing an example of evaluation criteria information used in the player performance evaluation process. As shown in the figure, the evaluation criteria information 122 stores a target player status indicating the playing status of the player to be evaluated, evaluation criteria indicating the criteria (conditions) for evaluation, and gaze abnormality criteria indicating the criteria (conditions) for detecting abnormalities in the gaze, etc. of the player to be evaluated.

[0159] The illustrated evaluation criteria information 122 shows an example of criteria using football (soccer) as an example, but various criteria according to the type of sport and the play (event) to be evaluated may be registered in such evaluation criteria information 122. Furthermore, more detailed events may be registered in the target player status, and more detailed evaluation criteria and gaze abnormality criteria may be set for each event.

[0160] Furthermore, the target player status may also include registered content specific to a particular event depending on the type of sport, such as a corner kick. For example, if the type of sport is football, an example of an evaluation criterion associated with the event could be, "During a corner kick or free kick, a defending player should at least keep their marking player in their peripheral vision before the kick-in, and after the kick-in, change their line of sight so that the ball is in their peripheral vision (because they need to clear the ball)."

[0161] Furthermore, the contents of the evaluation criteria may be derived based on, for example, statistical results of the areas of interest (or gazes, etc.) of top athletes in the sport in question.

[0162] The performance evaluation unit 114 acquires from the memory unit video of the competitive play that is linked to player identification information, etc. and viewpoint information, and evaluates the performance of each player by performing image analysis using the evaluation criteria information 122.

[0163] Specifically, the performance evaluation unit 114 uses video in which player identification information and viewpoint information are associated to determine, for example, every second, whether the line of sight, field of view, viewpoint, or area of ​​interest estimated based on the viewpoint information of each player matches the conditions of the evaluation criteria. Furthermore, if the performance evaluation unit 114 determines that the viewpoint information of a player matches the conditions of the evaluation criteria, it assigns a predetermined evaluation score to the performance of that player.

[0164] Furthermore, the performance evaluation unit 114 determines whether the gaze information of each player matches the conditions of the gaze abnormality criterion, and if it is determined that the information matches, it records the time of play at that time and counts the number of gaze abnormality detections (for example, by adding "1" to the number). For example, if it is determined that the gaze abnormality criterion and the player's gaze information match, such as when the player is not looking at a predetermined noteworthy position registered in the gaze abnormality criterion, or when the player's gaze is fixed in an unbalanced position for more than a predetermined period of time, the performance evaluation unit 114 adds up the number of gaze abnormality detections.

[0165] In this way, the performance evaluation unit 114 evaluates the performance of each player during competitive play, generates performance evaluation result information including an evaluation score, the number of times gaze abnormalities were detected, and the time during competitive play when the gaze abnormality was detected, and stores this information in the memory unit.

[0166] <<<<Analysis Result Display Process (Step S60)>>> When the information generation unit 115 receives a request to display the analysis results from, for example, the information processing terminal 600, it generates analysis result display information. Specifically, the information generation unit 115 generates the analysis result display information based on video of the entire duration of competitive play (match) in which a region of interest is associated with each player. More specifically, the information generation unit 115 uses the video to generate image information in a map format that schematically depicts the competition field and each player, and generates analysis result display information in which the region of interest detected in step S40 is associated with each player.

[0167] In addition, the output unit 116 transmits the analysis result display information of the competition play time input (specified) on the information processing terminal 600 side to the information processing terminal 600 via the communication unit, thereby displaying the analysis result display information on the terminal.

[0168] 30 is a diagram showing an example of a display screen (map format) for analysis result display information. As shown in the figure, the analysis result display information is display information that displays players, player identification information, and each player's region of interest 700 on a schematic competition field 710. The display screen for the analysis result display information also displays a slide bar 711 that accepts instruction inputs for moving the elapsed time forward or backward during competition play.

[0169] The information generating unit 115 may generate analysis result display information (video synthesis format) in which the area of ​​interest of the player is synthesized with the video of the game being played, instead of in the map format.

[0170] 31 is a diagram showing an example of a display screen (video synthesis format) of the analysis result display information. As shown in the figure, the analysis result display information is display information in which the region of interest of each player is synthesized on video 720 of the competition play.

[0171] Here, the person involved with the competition team who is the user of information processing terminal 600 moves the displayed slide bar 711 to input the time designation of the video for which display is requested. For example, a display request for the video at the time indicated by slide bar 711 is transmitted to region-of-interest detection device 100. Based on such operations, the person involved with the competition team can closely check changes in the region of interest of each player during the competition play.

[0172] The analysis result display information can also display only the region of interest of a specified player. Such display is achieved, for example, by hiding regions of interest associated with players other than the specified player. The setting to hide the regions of interest may be executed by the information generating unit 115 that has received an instruction, or may be executed on the information processing terminal 600 that has acquired the analysis result display information.

[0173] The analysis result display information can also display the region of interest in a display format selected from multiple display formats. Here, the multiple display formats for the region of interest include, for example, a fan-shaped display (including a cone display including the height direction) indicating the field of view, a line display (vector display) indicating the line of sight, and a point or circle display indicating the viewpoint. Two or three of these display formats can also be displayed simultaneously based on a specification by the information processing terminal 600. Each display format can be selected by the information processing terminal 600, for example.

[0174] Furthermore, the display screen of the analysis result display information can also display performance evaluation result information. Note that the performance evaluation result information may be included in the analysis result display information in advance, or may be displayed on the display screen of the analysis result display information by the information generating unit 115 based on an instruction input via the information processing terminal 600.

[0175] 32 is a diagram showing an example of a display screen for analysis result display information including performance evaluation result information. As shown in the figure, the display screen displays performance evaluation result information 730 including the identification information, evaluation score, and number of gaze abnormality detections of the specified target player in addition to the player's region of interest.

[0176] <<<<Tagging Process (Step S70)>>> Figure 33 is a flow diagram showing an example of the tagging process. When this process starts, the information generation unit 115 acquires performance evaluation result information of each player (step S71).

[0177] Next, the information generation unit 115 determines for each player whether one or more gaze abnormality detection counts are registered in the performance evaluation result information (step S72). If it is determined that the counts are registered (Yes in step S72), the information generation unit 115 tags a video (e.g., a video used in the performance evaluation process and associated with a region of interest) from a time when a gaze abnormality was detected but that has not yet been tagged (step S73). In this case, the information generation unit 115 generates tag information that registers a tag number and the time of the video, and stores the tag information in the storage unit 120. On the other hand, if it is determined that the number of abnormality detection counts is not registered (No in step S72), the information generation unit 115 proceeds to step S74.

[0178] Next, the information generating unit 115 determines whether or not a manual tagging request has been received from the information processing terminal 600 (step S74). For example, the information generating unit 115 determines whether or not a tagging request has been received based on whether or not input information regarding the time of the video to be tagged has been received from the information processing terminal 600.

[0179] If it is determined that the tagging request has been received (Yes in step S74), the information generation unit 115 tags the video at the acquired time (step S75) and generates tag information in which the tag number and the time are registered. Note that if tag information has already been generated, the information generation unit 115 registers the time of manual tagging and the tag number in the tag information.

[0180] Furthermore, if the information generating unit 115 determines that a tagging request has not been received (No in step S74), the process proceeds to step S76.

[0181] In step S76, the information generation unit 115 determines whether to end the tagging process. Specifically, the information generation unit 115 determines to end the tagging process when tagging has been performed for all detected gaze abnormalities or when an instruction to end the tagging process has been received from the information processing terminal 600.

[0182] If it is determined that the tagging process is to be terminated (Yes in step S76), the information generator 115 terminates this flow. On the other hand, if it is determined that the tagging process is not to be terminated (No in step S76), the information generator 115 returns the process to step S72.

[0183] In addition, the tag information may include not only the time of the tagged video, but also additional information such as the identification information of the player in whose gaze abnormality the tag was detected, and the player's area of ​​interest (gazing area) at that time.

[0184] Furthermore, the tagging may be performed not only on the detection of gaze abnormalities but also on the time of a play that satisfies a predetermined evaluation criterion, that is, on the video of the play time when an evaluation point is awarded.

[0185] By performing such tagging processing, for example, a person involved in the competition who is the user of the information processing terminal 600 can easily display, based on the tag information, a visual abnormality when an eye movement abnormality is detected, a play that deserves evaluation is detected, or a noteworthy video that has been manually tagged.

[0186] <Umpire Performance Evaluation Process> Figure 34 is a diagram showing an example of evaluation criteria information used in the umpire performance evaluation process. As shown in the figure, the evaluation criteria information 122 registers target person / target situation indicating the playing situation of the umpire to be evaluated, evaluation criteria indicating the criteria (conditions) for evaluation, and gaze abnormality criteria indicating the criteria (conditions) for detecting gaze abnormalities in the player to be evaluated. Note that the evaluation criteria and gaze abnormality criteria each register content specific to the umpire of the target sport.

[0187] The performance evaluation unit 114 uses the umpire evaluation criteria information 122 to perform processing similar to the player performance evaluation processing. Note that this processing is premised on the premise that each umpire has been identified in advance (e.g., identification of the chair umpire, linesman, etc.) and that the viewpoint information of each umpire has been detected throughout the entire game. Here, the identification of the umpire and the detection of the viewpoint information can be performed by executing processing equivalent to the player identification processing (step S30) and the region of interest detection processing (step S40) described above.

[0188] In the referee performance evaluation process, the performance evaluation unit 114 acquires from the memory unit video of the competitive play that corresponds to the referee's identification information, etc., and viewpoint information, and evaluates the performance of each referee by performing image analysis using the referee's evaluation criteria information 122.

[0189] Specifically, the performance evaluation unit 114 uses video in which the referee's identification information and viewpoint information are associated to determine, for example, every second, whether the line of sight, field of view, viewpoint, or area of ​​interest of each referee meets the conditions of the evaluation criteria. Furthermore, if the performance evaluation unit 114 determines that the referee's viewpoint information meets the conditions of the evaluation criteria, it assigns a predetermined evaluation score to the performance of that referee.

[0190] In addition, the performance evaluation unit 114 determines whether the viewpoint information of each referee meets the conditions of the gaze abnormality criteria, and if it determines that it does, records the time of play at that time and adds 1 to the number of gaze abnormality detections.

[0191] In this way, the performance evaluation unit 114 evaluates the performance of each referee during the competitive play, generates performance evaluation result information including an evaluation score, the number of times gaze abnormalities were detected, and the time during the competitive play when the gaze abnormality was detected, and stores this information in the memory unit.

[0192] The performance evaluation result information of the umpires may be displayed on the information processing terminal 600 used by the person evaluating the umpires in response to a display request from the terminal.

[0193] 35 is a diagram showing an example of the order in which each process is executed by the region of interest detection device for spectators. As shown in the figure, region of interest detection device 100 acquires images and sensor information of spectators during the game play time (while watching the game) from unmanned aerial vehicle 200 and mobile camera 300 (steps S110 and S120).

[0194] Furthermore, region of interest detection device 100 performs, in real time, a process of identifying attributes of spectators (step S130) and a process of detecting regions of interest (step S140) using video, sensor information, and the like.

[0195] Furthermore, the region of interest detection device 100 executes a display process of region of interest visualization information that visualizes the detected region of interest in real time during the competition in response to a display request from, for example, the stadium display control terminal 610 or the distribution / broadcast control terminal 630 (step S150).

[0196] <<<<Process for Identifying Spectator Attributes (Step S130)>>> In this process, spectator attributes are identified. Specifically, the identification unit 112 performs image analysis using the acquired video to identify each spectator's gender, age, and the team they are supporting (hereinafter sometimes referred to as the "support team") as attributes. Note that these attributes are only examples, and various other types of attributes may also be included. For example, information such as whether the spectators are experienced or inexperienced in the target sport may be included as attributes, based on a prior questionnaire given to the spectators.

[0197] Furthermore, ages may be identified within a predetermined range. Support teams may be identified based on, for example, commonalities between the spectators' clothing and the uniforms of the competing teams (particularly the uniform's color, shape, pattern, and other design features) (for example, when spectators are wearing replica uniforms). The uniform features of the competing teams may be stored in advance in a storage unit.

[0198] Furthermore, the information generating unit 115 generates audience attribute information in which the attributes of the audience identified by the identifying unit 112 are registered.

[0199] FIG. 36 is a diagram showing an example of spectator attribute information. As shown in the figure, the spectator attribute information has records in which an attribute identification number, gender, support team, and age are associated with each other. The attribute identification number is information that identifies attributes classified according to a combination of gender, support team, and age. Age is classified into a predetermined age range, for example, up to 12 years old, 12 to 20 years old, etc. Furthermore, for spectators whose attributes cannot be identified by image analysis, "unknown" is registered.

[0200] <<<<Audience Region of Interest Detection Process (Step S140)>>> In this process, the spectators' regions of interest are detected. Specifically, the region of interest detection unit 113 detects each spectator's region of interest using sensor information (position information and direction measurement information of the spectators) acquired from the wearable devices 500a, 500b worn by the spectators.

[0201] Region of interest detection unit 113 detects the regions of interest of spectators in the same manner as in step S40 described above. Specifically, region of interest detection unit 113 identifies the position of each spectator within the spectator seats based on the direction measurement information.

[0202] The region of interest detection unit 113 also estimates the spectator's gaze position (viewpoint) and field of view (field of view) on the competition field based on the spectator's position and the gaze direction identified by the direction measurement information. The region of interest detection unit 113 also determines (estimates) a gaze area (e.g., an ellipse) of a predetermined area on the competition field that corresponds to the length (distance) from the spectator's position to the gaze position. The region of interest detection unit 113 then detects the estimated gaze area as the spectator's region of interest.

[0203] 37 is a schematic diagram showing spectators and the direction of their gaze areas. As shown in the figure, the region of interest detection unit 113 detects, as a region of interest, a gaze area on the playing field that is estimated to be the spectator's gaze, based on spectator direction measurement information.

[0204] The region of interest detection unit 113 estimates a gaze area of ​​different sizes depending on the distance between the spectator and the gaze area. For example, if the distance from the spectator's position to the gaze area on the competition field is greater than or equal to a predetermined distance (if it is far), the region of interest detection unit 113 determines the gaze area to be an elliptical area of ​​a predetermined size (for example, with a major axis of approximately several tens of meters) corresponding to that distance. On the other hand, if the distance from the spectator's position to the gaze area on the competition field is less than a predetermined distance (if it is short), the region of interest detection unit 113 determines the gaze area to be an elliptical area of ​​a predetermined size (for example, with a major axis of approximately several meters) corresponding to that distance. Spectators far from the competition field are located above the spectator seats, and therefore a wider area of ​​the competition field is included in their field of view. Therefore, it is considered appropriate to estimate a wider gaze area for spectators far from the competition field than for spectators close to the competition field.

[0205] When the wearable device 500c of FIG. 9 is used, the region of interest detection unit 113 estimates the spectator's gaze area based on the eye state and focal length detected by the eyewear 510. Specifically, the region of interest detection unit 113 estimates the player's gaze direction based on the movement of the eyeball. The region of interest detection unit 113 also estimates the gaze position (viewpoint) in the gaze direction and the field of view (field of view) based on the focal measurement distance. The region of interest detection unit 113 also identifies the gaze area on the competition field where the spectator is estimated to be gazing, based on the gaze direction, gaze position, and field of view. This allows the region of interest detection unit 113 to more accurately determine the gaze area on the competition field.

[0206] The spectators' regions of interest can also be detected using video captured by the unmanned aerial vehicle 200 or mobile camera 300 without using sensor information. In this case, the region of interest detection unit 113 performs image analysis using the video of the spectators and estimates the gaze direction, viewpoint, and field of view area on the competition field based on the direction and angle of each spectator's head. The region of interest detection unit 113 also identifies the gaze area on the competition field where the spectators are estimated to be gazing, based on the estimated gaze direction, viewpoint, and field of view area.

[0207] <<<<Display Process of Region of Interest Visualization Information (Step S150)>>> The information generator 115 generates region of interest visualization information when it receives a request to display a region of interest of spectators, for example, from the stadium display control terminal 610 or the distribution / broadcast control terminal 630. Specifically, the information generator 115 generates region of interest visualization information based on the region of interest of spectators detected in real time. More specifically, the information generator 115 generates region of interest visualization information that shows the region of interest of spectators in real time, for example, by overlaying the region of interest of spectators detected in step S140 on a map-format image that schematically depicts the competition field.

[0208] That is, the region of interest visualization information is information that visualizes the region of interest of the audience in real time at each moment, detected based on video and sensor information acquired in real time. Therefore, the region of interest visualization information is updated every time video and sensor information is acquired (for example, every second).

[0209] Furthermore, the output unit 116 displays the generated region of interest visualization information on the terminal that issued the display request via the communication unit 130 .

[0210] 38 is a diagram showing an example display screen (map format) of region of interest visualization information. As shown in the figure, the display screen of the region of interest visualization information displays each spectator's region of interest 760 superimposed on a map-format competition field 750. Furthermore, the range where multiple spectator regions of interest 760 overlap, i.e., the region of interest that is estimated to be watched by more spectators, is highlighted in a darker color as the colors indicating the ranges of each region of interest overlap.

[0211] The region of interest visualization information is not limited to a map-type display, and may be, for example, display information in which a region of interest is superimposed on a video. Furthermore, the region of interest visualization information may display only the regions of interest of spectators corresponding to designated attributes. In this case, regions of interest of spectators corresponding to attributes that are not designated are not displayed. Such display control may be performed, for example, by the information generating unit 115.

[0212] The region of interest visualization information can also display attributes of spectators corresponding to the displayed region of interest. Note that such display control is executed by, for example, the information generating unit 115.

[0213] 39 is a diagram showing an example of a display screen of region of interest visualization information on which spectator attributes are displayed. As shown in the figure, the display screen of the region of interest visualization information displays, for example, a spectator's region of interest 760 corresponding to an attribute specified via the stadium display control terminal 610 or the distribution / broadcast control terminal 630, and the attribute (gender, support team, and age) 770.

[0214] Furthermore, in addition to the region of interest of the spectator, the region of interest visualization information may also display the position information of unmanned airplane 200 and mobile camera 300. Specifically, information generation unit 115 acquires, for example, the identification information and position information of unmanned airplane 200 and mobile camera 300, and displays the camera positions in the region of interest visualization information based on this information.

[0215] 40 is a diagram showing an example of a display screen of region of interest visualization information on which the positions of the cameras are displayed. As shown in the figure, the display screen of region of interest visualization information displays the positional relationship between the rail cameras (cameras A to H) and the camera of unmanned airplane 200 or the wire camera (camera X).

[0216] The display screen also displays the video selected for output to the stadium display 620 or for distribution / broadcast, along with the identification information of the camera capturing the video (camera E in the illustrated example).

[0217] By displaying such area of ​​interest visualization information, the person in charge of displaying the stadium display 620 and the person in charge of distributing or broadcasting can check the trends in the areas of interest of spectators, the areas of attention where spectators' attention is focused, and the identification information of the cameras capturing those areas.

[0218] This enables the person in charge of displaying stadium display 620 to specify the camera capturing an area where the regions of interest of the largest number of spectators overlap, and display the video from that camera on stadium display 620. In this case, when output unit 116 of region of interest detection device 100 receives from stadium display control terminal 610 the designation (selection) of the camera capturing the video to be displayed on stadium display 620, it outputs (displays) the video from the designated camera on stadium display 620.

[0219] Furthermore, the person in charge of distribution or broadcasting can specify the camera capturing the location where the regions of interest of the greatest number of spectators overlap, and distribute or broadcast the video from that camera. In this case, when output unit 116 of region of interest detection device 100 receives from distribution / broadcast control terminal 630 the designation (selection) of the camera capturing the video to be distributed or broadcast, output unit 116 distributes or broadcasts the video from the designated camera.

[0220] Region of interest detection device 100 may automatically select a camera capturing an area that attracts the attention of a larger number of spectators. Specifically, camera selection unit 117 selects a camera capturing an area where the regions of interest overlap more. Output unit 116 outputs, distributes, or broadcasts the video from the camera selected by camera selection unit 117 to stadium display 620.

[0221] With this kind of region of interest detection device 100, it is possible to automatically switch between areas that are being watched by a larger number of spectators, i.e., areas that are estimated to be of high demand to spectators, and display or distribute the images on the stadium display 620, without relying on the person responsible for displaying the stadium display 620 or the person responsible for distributing or broadcasting.

[0222] Furthermore, the region of interest detection device 100 may identify locations where regions of interest are concentrated for each spectator attribute and the cameras capturing those locations, and generate recommended camera information for distribution / broadcasting that associates these locations. Specifically, the information generation unit 115 identifies locations on the competition field where regions of interest are most concentrated for each spectator attribute based on the region of interest visualization information. Furthermore, the information generation unit 115 identifies cameras capturing the identified locations based on location information acquired from the unmanned airplane 200 and the mobile camera 300. At this time, the information generation unit 115 identifies the cameras that best capture the identified locations based on information such as the distance from each camera's position to the identified location and the camera's orientation (direction of capture). Furthermore, the information generation unit 115 extracts corresponding spectator attributes using the spectator attribute information, and generates recommended camera information for distribution / broadcasting that associates the extracted attributes with the identified cameras.

[0223] 41 is a diagram showing an example of recommended camera information for distribution / broadcasting. As shown in the figure, this information has records in which an attribute identification number, a support team, an age, and a recommended camera are associated. For example, a record associated with attribute identification number 0003 indicates that camera E is the camera that best captures the location on the field where the areas of interest of spectators aged 21 to 50 are most concentrated, with support team A.

[0224] Using this recommended camera information for streaming / broadcasting, the person in charge of streaming or broadcasting can identify the appropriate camera that captures footage that is in high demand based on audience attributes. As a result, the person in charge of streaming or broadcasting can identify the camera that captures the region of interest that is being watched by the most audience based on each attribute. As a result, region of interest detection device 100 can contribute to realizing streaming that provides high viewer satisfaction.

[0225] <Hardware Configuration of Region of Interest Detection Apparatus 100> Fig. 42 is a diagram showing an example of the hardware configuration of region of interest detection apparatus 100. Region of interest detection apparatus 100 is, for example, an information processing apparatus such as a server apparatus or a cloud server. As shown in the figure, region of interest detection apparatus 100 has an input device 801, an output device 802, a processing apparatus 803, a main memory device 804, an auxiliary memory device 805, a communication device 806, and a bus 807 that electrically connects these devices together.

[0226] The input device 801 is a device that allows an operator to input information and instructions to the region-of-interest detection device 100. Specifically, the input device 801 is, for example, a touch panel, a keyboard, a mouse, or a voice input device such as a microphone.

[0227] The output device 802 is a device that outputs information generated by the region-of-interest detection device 100. Specifically, the output device 802 is a display, a printer, or a speaker.

[0228] The processing device 803 is, for example, a device that performs arithmetic processing. Specifically, the processing device 803 is, for example, a CPU, a microprocessor, a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), or other semiconductor devices capable of performing calculations.

[0229] The main storage device 804 is a memory device such as a RAM that temporarily stores various types of read information and a ROM that stores programs, application programs, and other various information executed by the processing device 803. The auxiliary storage device 805 is a non-volatile storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory that can store digital information.

[0230] The communication device 806 is a device that performs wireless or wired information communication with external devices.

[0231] The hardware configuration of the region-of-interest detecting apparatus 100 has been described above.

[0232] The processing unit of the region of interest detection device 100 is realized by a program that causes the processing device 803 (e.g., a CPU) to perform processing. These programs are stored, for example, in the main storage device 804 or the auxiliary storage device 805, and are loaded onto the main storage device 804 for execution and executed by the processing device 803. The storage unit 120 may be realized by the main storage device 804 or the auxiliary storage device 805, or by a combination of these. The communication unit 130 is realized by the communication device 806.

[0233] Note that the functional blocks of region of interest detection device 100 are categorized according to the main processing content in order to facilitate understanding of the functions realized in this embodiment. Therefore, the present invention is not limited by the classification method or names of the functions. Furthermore, each configuration of region of interest detection device 100 can be further categorized into more components according to the processing content. Furthermore, a single component can be categorized to perform even more processing.

[0234] Furthermore, all or part of each functional unit may be constructed by hardware (such as an integrated circuit such as an ASIC) implemented in a computer, and the processing of each functional unit may be executed by a single piece of hardware or by multiple pieces of hardware.

[0235] Furthermore, the present invention is not limited to the above-described embodiments and modifications, and includes various other embodiments and modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment or modification, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0236] [Item 1] A region of interest detection system having a region of interest detection device, the region of interest detection device comprising a processing unit, the processing unit acquiring at least one of direction measurement information relating to the direction of the subject's head or gaze and focus measurement information relating to the focal length of the eyes as sensor information using a sensor mounted on a terminal worn on the head of a subject, including an athlete or a referee, and estimating at least one of the subject's gaze direction, gaze position, gaze area, and field of view area as viewpoint information based on the sensor information. [Item 2] The region of interest detection system according to Item 1, wherein the processing unit detects the orientation of the subject's head from a pair of compass sensors mounted on the terminal and positioned to the left and right or in front and behind the subject's head, and estimates at least one of the gaze direction and field of view area. [Item 3] The region of interest detection system according to Item 1, wherein the processing unit estimates at least one of the gaze direction, the gaze position, the gaze area, and the field of view area based on the state of the subject's eyes detected by an eyewear sensor mounted on the terminal. [Item 4] The region of interest detection system according to Item 1, wherein the processing unit acquires video information of the subject and identifies the subject appearing in the captured video based on the video information or the video information and the sensor information. [Item 5] The region of interest detection system according to Item 4, wherein the processing unit acquires detection information from a position detection sensor mounted on the terminal worn by the subject and identification information of the subject associated with the terminal, and identifies the subject appearing in the video. [Item 6] The region of interest detection system according to Item 4, wherein the processing unit identifies the subject appearing in the video based on the uniform design and jersey number information of the subject appearing in the acquired video.[Item 7] The region of interest detection system according to Item 1, wherein the processing unit evaluates the subject based on the estimated gaze point information of the subject. [Item 8] The region of interest detection system according to Item 7, wherein the processing unit determines, based on the gaze point information, that a gaze point abnormality has occurred if the subject is not looking at a position of interest or if the gaze point is fixed at a biased position for a predetermined period of time or longer. [Item 9] The region of interest detection system according to Item 1, wherein the processing unit displays the gaze point information estimated for each subject on a predetermined terminal. [Item 10] The region of interest detection system according to Item 9, wherein the processing unit acquires video information of the subject, identifies the subject appearing in the video, and displays the estimated gaze point information and identification information of the subject by superimposing it on the video of the subject. [Item 11] The region of interest detection system according to Item 9, wherein the processing unit displays the estimated gaze point information for each subject on a map of the stadium identified based on detection information from a position detection sensor mounted on the device worn by the subject. [Item 12] The region of interest detection system according to Item 9, wherein the processing unit evaluates the subject based on the estimated gaze point information of the subject, and displays evaluation information of the subject together with the estimated gaze point information. [Item 13] The region of interest detection system according to Item 9, wherein the processing unit accepts a selection input of the subject for whom the gaze point information is to be displayed. [Item 14] The region of interest detection system according to Item 7 or 8, wherein the processing unit tags video information of the stadium when the subject's evaluation satisfies a predetermined condition or when a line of sight abnormality is determined, and records the video together with the tag information.[Item 15] A region of interest detection system having a region of interest detection device, the region of interest detection device comprising a processing unit, the processing unit acquiring measurement information from sensors mounted on terminals worn on the heads of spectators or video of the spectators, and estimating at least one of gaze direction, gaze position, gaze area, and field of view area as viewpoint information based on the sensor measurement information or the video of the spectators. [Item 16] A region of interest detection system according to Item 15, the processing unit acquiring at least one of direction measurement information relating to the direction of the spectators' heads or gaze direction, and focus measurement information relating to the focal length of the eyes, using the sensors of the terminals worn on the spectators' heads. [Item 17] A region of interest detection system according to Item 15, the processing unit synthesizing the estimated viewpoint information with filmed video of the stadium or a map image of the stadium for display. [Item 18] The region of interest detection system according to Item 17, wherein the processing unit, when combining and displaying the spectator viewpoint information on a captured image or map of a stadium, emphasizes viewpoint positions with a greater number of spectators than viewpoint positions with a smaller number of spectators. [Item 19] The region of interest detection system according to Item 15, wherein the processing unit uses the viewpoint information recorded in association with the spectator attributes to display viewpoint information corresponding to the spectator attributes specified by a user. [Item 20] The region of interest detection system according to Item 15, wherein the processing unit selects video from viewpoint positions at which many spectators are gazing, and proposes the selected video to officials of the competition, or displays the selected video on a display in the stadium, or performs at least one of distributing or broadcasting the selected video.[Item 21] The region of interest detection system according to Item 19, wherein the processing unit selects and distributes or broadcasts video of a viewpoint position corresponding to an attribute selected by a viewer of the distributed or broadcast video, based on the viewpoint information recorded in association with each attribute of the spectators. [Item 22] A region of interest detection method performed by a region of interest detection device, wherein the region of interest detection device performs the steps of: acquiring, using a sensor mounted on a terminal worn on the head of a subject, including an athlete or a referee, at least one of direction measurement information related to the direction of the subject's head or gaze and focus measurement information related to the focal length of the eyes, as sensor information; and estimating, based on the sensor information, at least one of the subject's gaze direction, gaze position, gaze area, and field of view area. [Item 23] A region of interest detection method performed by a region of interest detection device, wherein the region of interest detection device performs the steps of: acquiring measurement information from a sensor mounted on a terminal worn on the head of a spectator, or video of the spectator, and estimating at least one of gaze direction, gaze position, gaze area, and field of view area as viewpoint information based on the sensor measurement information or the video of the spectator. [Item 24] A program that causes a computer to function as a region of interest detection device, the program causing the computer to function as a processing unit that acquires, using a sensor mounted on a terminal worn on the head of a subject, including an athlete or a referee, at least one of direction measurement information related to the subject's head or gaze direction and focus measurement information related to the focal length of the eyes, as sensor information, and estimating at least one of gaze direction, gaze position, gaze area, and field of view area of ​​the subject, as viewpoint information based on the sensor information.[Item 25] A program that causes a computer to function as a region of interest detection device, characterized by causing the computer to function as a processing unit that acquires measurement information from a sensor mounted on a terminal worn on the head of a spectator or video of the spectator, and estimates at least one of the gaze direction, gaze position, gaze area, and field of view area as viewpoint information based on the measurement information from the sensor or the video of the spectator.

[0237] 1000...region of interest detection system, 100...region of interest detection device, 200...unmanned aerial vehicle, 300...mobile camera, 400...control device, 500...wearable terminal, 600...information processing terminal, 610...stadium display control terminal, 620...stadium display, 630...distribution / broadcast control terminal, 801...input device, 802...output device, 803...processing device, 804...main memory device, 805...auxiliary memory device, 806...communication device, 807...bus, N...network

Claims

1. A region of interest detection system having a region of interest detection device, the region of interest detection device comprising a processing unit, the processing unit acquiring at least one of directional measurement information relating to the direction of the subject's head or gaze and focus measurement information relating to the focal length of the eyes as sensor information using a sensor mounted on a terminal worn on the head of a subject, including an athlete or a referee, and estimating at least one of the subject's gaze direction, gaze position, gaze area and field of view area as viewpoint information based on the sensor information.

2. A region of interest detection system as described in claim 1, characterized in that the processing unit is mounted on the terminal and detects the orientation of the subject's head from a pair of compass sensors located to the left and right or in front and behind the subject's head, and estimates at least one of the line of sight direction and the field of view area.

3. A region of interest detection system as described in claim 1, characterized in that the processing unit estimates at least one of the gaze direction, the gaze position, the gaze area, and the field of view area based on the state of the subject's eyeball detected by an eyewear-type sensor mounted on the terminal.

4. A region of interest detection system as described in claim 1, characterized in that the processing unit acquires video information of the subject and identifies the subject appearing in the captured video based on the video information or the video information and the sensor information.

5. A region of interest detection system as described in claim 4, characterized in that the processing unit acquires detection information from a position detection sensor mounted on the terminal worn by the subject and identification information of the subject associated with the terminal, and identifies the subject appearing in the video.

6. A region of interest detection system as claimed in claim 4, characterized in that the processing unit identifies the subject appearing in the acquired image based on the uniform design and jersey number information of the subject appearing in the image.

7. A region of interest detection system as claimed in claim 1, characterized in that the processing unit evaluates the subject based on the estimated viewpoint information of the subject.

8. A region of interest detection system as described in claim 7, characterized in that the processing unit determines, based on the viewpoint information, that a viewpoint abnormality has occurred when the subject is not looking at a position that requires attention, or when the viewpoint is fixed at a biased position for a predetermined period of time or longer.

9. A region of interest detection system as described in claim 1, characterized in that the processing unit displays the viewpoint information estimated for each subject on a specified terminal.

10. A region of interest detection system as described in claim 9, wherein the processing unit acquires video information of the subject, identifies the subject appearing in the video, and displays the estimated viewpoint information and identification information of the subject by superimposing them on the video of the subject.

11. A region of interest detection system as described in claim 9, characterized in that the processing unit displays the estimated viewpoint information for each subject on a map of a stadium identified based on detection information from a position detection sensor mounted on the terminal worn by the subject.

12. A region of interest detection system as described in claim 9, characterized in that the processing unit evaluates the subject based on the estimated viewpoint information of the subject, and displays evaluation information of the subject together with the estimated viewpoint information.

13. A region of interest detection system as claimed in claim 9, characterized in that the processing unit accepts a selection input of the subject for which the viewpoint information is to be displayed.

14. A region of interest detection system as described in claim 7 or 8, characterized in that the processing unit tags video information of the stadium when the subject's evaluation satisfies a predetermined condition or when an abnormality in the line of sight is determined, and records the video together with the tag information.

15. A region of interest detection system having a region of interest detection device, the region of interest detection device comprising a processing unit, the processing unit acquiring measurement information from a sensor mounted on a terminal worn on the head of a spectator or an image of the spectator, and estimating at least one of the gaze direction, gaze position, gaze area and field of view area as viewpoint information based on the measurement information from the sensor or the image of the spectator.

16. A region of interest detection system as described in claim 15, characterized in that the processing unit acquires at least one of directional measurement information relating to the direction of the spectator's head or line of sight, and focus measurement information relating to the focal length of the eyes, using the sensor of the terminal worn on the spectator's head.

17. A region of interest detection system as described in claim 15, characterized in that the processing unit synthesizes the estimated viewpoint information with a photographed image of the stadium or a map image of the stadium and displays it.

18. A region of interest detection system as described in claim 17, characterized in that the processing unit, when synthesizing and displaying the viewpoint information of the spectators on a filmed image or map of the stadium, emphasizes viewpoint positions with a larger number of spectators compared to viewpoint positions with a smaller number of spectators.

19. A region of interest detection system as described in claim 15, characterized in that the processing unit uses the viewpoint information recorded in association with the attributes of the spectators to display viewpoint information corresponding to the attributes of the spectators specified by a user.

20. A region of interest detection system as described in claim 15, wherein the processing unit selects an image of a viewpoint position at which a large number of the spectators are gazing, and at least one of proposing the selected image to people involved in the competition, or displaying the selected image on a display at the competition venue, or distributing or broadcasting the selected image.

21. A region of interest detection system as described in claim 19, characterized in that the processing unit selects and distributes or broadcasts an image of a viewpoint position corresponding to an attribute selected by a viewer of the distributed or broadcast image based on the viewpoint information recorded in association with each attribute of the audience.

22. A region of interest detection method performed by a region of interest detection device, characterized in that the region of interest detection device performs the following steps: acquiring at least one of directional measurement information relating to the direction of the subject's head or gaze and focus measurement information relating to the focal length of the eyes as sensor information using a sensor mounted on a terminal worn on the head of a subject, including an athlete or a referee; and estimating at least one of the subject's gaze direction, gaze position, gaze area and field of view area as viewpoint information based on the sensor information.

23. A region of interest detection method performed by a region of interest detection device, characterized in that the region of interest detection device performs the steps of: acquiring measurement information from a sensor mounted on a terminal worn on the head of a spectator, or an image of the spectator; and estimating at least one of the line of sight direction, gaze position, gaze area, and field of view area as viewpoint information based on the measurement information from the sensor or the image of the spectator.

24. A program that causes a computer to function as a region of interest detection device, characterized in that the program causes the computer to function as a processing unit that acquires at least one of directional measurement information regarding the direction of the subject's head or gaze and focus measurement information regarding the focal length of the eyes as sensor information using a sensor mounted on a terminal worn on the head of a subject, including an athlete or referee, and estimates at least one of the subject's gaze direction, gaze position, gaze area, and field of view area as viewpoint information based on the sensor information.

25. A program that causes a computer to function as a region of interest detection device, comprising the steps of: acquiring measurement information from a sensor mounted on a terminal worn on the head of a spectator, or an image of the spectator, and estimating at least one of the gaze direction, gaze position, gaze area, and field of view area as viewpoint information based on the measurement information from the sensor or the image of the spectator.

Images