Image processing apparatus, method, and program

Abstract

A video processing device according to one embodiment of the present invention comprises: a calculation unit for calculating, on the basis of position information of subjects and position information of a plurality of 360-degree cameras each for capturing a video of a surrounding 360-degree range, imaging directions of videos in which the subjects are imaged from third-person viewpoints; a selection unit for selecting, from among the imaging directions calculated by the calculation unit, an imaging direction which is desired by a user and in which a subject desired by the user who is not any of the subjects is imaged from a third-person viewpoint; and an output unit for outputting a video of a range including the desired subject among videos captured by the 360-degree cameras capable of performing imaging in the imaging direction selected by the selection unit.

Description

Technical Field

[0001] Embodiments of the present invention relate to a video processing apparatus, method, and program.

Background Art

[0002] In video acquisition of competitions held in remote locations or locations with many obstacles, such as windsurfing on the ocean or BMX (Bicycle Motocross) competitions where a course in the forest is run on a mountain bike (MTB), it is difficult to follow the state of the athletes during the competition with a fixed telescopic camera.

[0003] Therefore, in order to shoot a third-person perspective video, which is a video capturing the athlete up close in the above-mentioned competitions, a method using a mobile camera is adopted, such as chasing the athlete with a shooting boat or a shooting drone.

[0004] In this method using a mobile camera, the shooting boat or the shooting drone has limitations on the number of cameras that can run side by side without interfering with the competition and route limitations such as the camera not being able to enter the course. For example, it is a problem to follow professional players moving at high speed.

[0005] As a method that can solve such problems, it is conceivable to apply a technique of obtaining a third-person perspective video by connecting a balloon or a float equipped with a camera to the athlete himself with a long string and pulling the connected balloon or float by the athlete (see, for example, Non-Patent Document 1).

Prior Art Documents

Non-Patent Documents

[0006]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] However, in the above-described method, it is necessary to separate the camera and the athlete by a certain distance. When the athlete moves at high speed or passes through a limited space, in addition to the connected camera getting in the way, the shooting direction of the camera is fixed by the moving direction of the athlete, and there is a problem that the camera work is restricted, for example, only the video from the perspective of the athlete's back can be obtained.

[0008] This invention has been made paying attention to the above circumstances, and an object thereof is to provide a video processing apparatus, method, and program capable of appropriately acquiring a video from a third-person perspective.

Means for Solving the Problems

[0009] A video processing apparatus according to an aspect of the present invention calculates the shooting direction of a video in which a subject is imaged from a third-person perspective by a plurality of 360-degree cameras based on the position information of the subject and the position information of the plurality of 360-degree cameras that shoot videos in a 360-degree range around. and mounted on something other than the subject And from the shooting direction calculated by the calculation unit, as a shooting direction selectable by a user who is not the subject the the calculated shooting direction was presented the uA selection unit that selects a shooting direction in which a desired subject by the user is photographed from a third - person perspective and selects the desired shooting direction by the user, and an output unit that outputs an image of a range including the desired subject among the images photographed by the 360 - degree camera capable of shooting in the shooting direction selected by the selection unit.

[0010] The video processing method according to one aspect of the present invention is a method performed by a video processing apparatus wa and includes: calculating, by the video processing apparatus, a shooting direction of an image in which a subject is photographed from a third - person perspective based on position information of the subject and position information of a plurality of 360 - degree cameras that photograph images of a 360 - degree range around; selecting, by the video processing apparatus, a shooting direction in which a desired subject by the user is photographed from a third - person perspective and selecting the desired shooting direction by the user; and outputting, by the video processing apparatus, an image of a range including the desired subject among the images photographed by the 360 - degree camera capable of shooting in the selected shooting direction. calculation unit by the video processing apparatus, based on position information of the subject and position information of a plurality of 360 - degree cameras that photograph images of the 360 - degree range around, a shooting direction of an image in which the subject is photographed from a third - person perspective by the plurality of 360 - degree cameras and mounted on something other than the subject calculate; by the video processing apparatus as a shooting direction selectable by a user who is not the subject from the calculated shooting direction, selection unit a shooting direction in which a desired subject by the user is photographed from a third - person perspective and selecting the desired shooting direction by the user; and by the video processing apparatus the calculated shooting direction was presented the u a shooting direction in which a desired subject by the user is photographed from a third - person perspective and selecting the desired shooting direction by the user; and by the video processing apparatus output unit outputting an image of a range including the desired subject among the images photographed by the 360 - degree camera capable of shooting in the selected shooting direction.

Advantages of the Invention

[0011] According to the present invention, an image from a third - person perspective can be appropriately acquired.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In this embodiment, each of the boats operated by each of a plurality of athletes as subjects, for example, boat A operated by athlete A, boat B operated by athlete B, and boat C operated by athlete C, is equipped with a 360-degree camera (hereinafter, may also be simply referred to as a camera) capable of recording images in a 360-degree range of the surroundings (hereinafter, may be referred to as 360-degree images or 360-degree camera images) to obtain images during the competition, and based on the position information of each camera and each athlete, an apparatus for extracting and reproducing third-person perspective images of a desired athlete that a user who is not the above athlete, for example, a spectator, wants to focus on will be described.

[0014] FIG. 1 is a diagram showing an application example of an image processing system according to an embodiment of the present invention. In the example shown in FIG. 1, an image processing system having an information acquisition unit (information acquisition device) 10, an information integration / extraction unit (may also be referred to as an information integration unit) 20, a user interface (User Interface: UI) unit 30, and an image output unit 40 is shown. Each element of the above system, namely, the information integration / extraction unit 20, the UI unit 30, and the image output unit 40, can be configured as an integrated image processing device 100. At least two or more of these information integration / extraction unit 20, UI unit 30, and image output unit 40 may be integrated elements. Between each unit, for example, between the information acquisition unit 10 and the information integration / extraction unit 20, they can be communicably connected, for example, by wireless communication.

[0015] In this embodiment, the information acquisition unit 10 is attached to each of the above boats. Each information acquisition unit 10 includes a 360-degree camera that is a camera attached to each of the above boats and can record images in a 360-degree range around the attachment location on the boat, a GPS (Global Posting System sensor), a 9-axis sensor, and a communication device. The GPS sensor is a sensor capable of detecting the latitude and longitude of each 360-degree camera and each player, and the 9-axis sensor is a sensor capable of detecting the acceleration, azimuth, and inclination of each 360-degree camera and each player. The 9-axis sensor may be a combination of an accelerometer, a gyroscope sensor, and an azimuth sensor.

[0016] Each of the information acquisition units 10 acquires the 360-degree camera image by the 360-degree camera, the latitude, longitude, acceleration, azimuth, and inclination of the 360-degree camera and each player, and sets the blind spot information of the camera. Details of these acquisitions or settings will be described later. Also, the 360-degree camera may be directly attached to the back of the above-mentioned player himself, for example, or may be attached to a location somewhat away from the back of the player via an arm connected to a headgear attached to the back of the player.

[0017] The various information acquired or set by each of the information acquisition units 10 is sent to the information integration / extraction unit 20 by the above communication device, for example, by wireless communication (reference symbol a1 in FIG. 1). The information integration / extraction unit 20 detects the positional relationship between each camera and each player based on the latitude and longitude detected by the GPS sensor of each of the information acquisition units 10.

[0018] In addition, based on the detected positional relationships between each camera and each player, the information integration / extraction unit 20 acquires, for each player, a camera angle (which may be referred to as an effective camera angle) that enables the display of a third-person perspective video of each player, that is, the shooting direction of a video in which the player as the subject is photographed from a third-person perspective, as a shooting direction selectable by a user who is not the subject, and sends this acquisition result, together with information indicating the positional relationships between each player, to the UI unit 30 (reference symbol a2 in FIG. 1). This third-person perspective video is a video obtained by cutting out a video of an angle of view in which a specific player is included from the videos shot by each camera.

[0019] The UI unit 30 converts the positional relationships between each player and the camera angles that enable the display of the third-person perspective videos of each player, indicated by the information sent from the information integration / extraction unit 20, into map information, that is, visual information presented in a map format, and presents it to the user by displaying a display screen of this map information on a display device (not shown) of the UI unit 30. FIG. 2 is a diagram showing an example of the positional relationships between each player and the camera angles that enable the display of the third-person perspective videos of each player. In this FIG. 2, map information related to the positional relationships between players A, B, and C and the camera angles (triangle marks in FIG. 2) that enable the display of the third-person perspective videos of players A, B, and C respectively is shown.

[0020] The user can select, by means of a click operation or the like on the display screen of the map information displayed on the UI unit 30, a desired camera angle that enables the display of the third-person perspective video of a desired player among the camera angles that enable the display of the third-person perspective videos of each player indicated by this presented map information, that is, the desired shooting direction of a video in which the desired subject is photographed from a third-person perspective.

[0021] The information integration / extraction unit 20 acquires the information on the selected camera angle (reference symbol a3 in FIG. 1), and sends a 360-degree camera video shot by a 360-degree camera capable of shooting according to this camera angle, together with information related to the positions of the camera and each player, to the video output unit 40 (reference symbol a4 in FIG. 1). The information related to this position corresponds to the azimuth angle and distance described later. That is, the information integration / extraction unit 20 can select a shooting direction in which a desired subject that is not the user is imaged from the above-calculated shooting direction and that is a shooting direction desired by the user from a third-person perspective. The video output unit 40 cuts out a planar video of a range including the desired player from the 360-degree video shot at the camera angle selected by the user from among the 360-degree camera videos sent from the information integration / extraction unit 20, based on the information regarding the positions of the respective cameras and the respective players, and outputs this video to be displayed, for example, on the screen of a display device (not shown). That is, the video output unit 40 outputs a video of a range including a desired subject by the user from among the videos shot by a 360-degree camera capable of shooting in the shooting direction selected by the user.

[0022] FIG. 3 is a top view for explaining an example of the installation of a 360-degree camera in windsurfing. In the example shown in FIG. 3, the 360-degree camera 200 is installed at the boom end 203 behind the boom 202 connected to the mast 201 of the rig part in windsurfing, such that the front of the fisheye lens faces the front direction and the rear direction (reference symbol d in FIG. 3) of the windsurfing. With such an installation, a 360-degree video from the height of the player's arm can be acquired.

[0023] Also, in the example shown in FIG. 3, two 360-degree cameras 200 are paired and mounted on each of the left and right booms 202 connected to the mast 201. Thereby, the dead angles related to each 360-degree camera can cover each other.

[0024] Next, an example of the processing procedure by each of the above units will be described. FIG. 4 is a flowchart showing an example of the processing procedure by each unit of the video processing system. First, each of the information acquisition units 10 sets, based on the mounting position of the 360-degree camera in this information acquisition unit 10, an angular range that becomes a blind spot in the shooting of the player due to an obstacle in the video shot from this position, that is, a blind spot range within the shooting range of the 360-degree camera where the player as the subject is not correctly shot by the obstacle (S11). The information on the set blind spot range is held, for example, in the information storage unit within the information acquisition unit 10.

[0025] Also, each GPS sensor of the information acquisition unit 10 acquires, for each time series, the latitude and longitude information which is the GPS information of each camera and each player. The 9-axis sensor of the information acquisition unit 10 acquires, for each time series, the information on the acceleration, inclination, and azimuth of each camera and each player (S12). The 360-degree camera of the information acquisition unit 10 acquires 360-degree videos for each time series. The set blind spot range, the acquired latitude and longitude information, acceleration, inclination, and azimuth information, and the 360-degree videos are sent to the information integration and extraction unit 20 for each time series together with the information on the blind spot range.

[0026] The information integration and extraction unit 20 designates one of the players as a target candidate (hereinafter, may also be referred to as a target player), and by repeating the following first and second processes for all players for this player, for each player, a list (hereinafter, may be referred to as a valid camera list) of cameras that can be shot at an angle of a valid camera for each player, that is, a camera capable of displaying a third-person perspective video of each player, is created for each time series for each player. The number of valid cameras for each player can be one or more.

[0027] As the first process, the information integration and extraction unit 20 extracts, for each time series, the positions of the 360-degree cameras whose distance from the player who is the target candidate is within a certain threshold based on the latitude and longitude information of each camera and each player from each of the information acquisition units 10 (S21).

[0028] As a second process, based on the latitude and longitude information of each player from each of the information acquisition units 10, the information integration / extraction unit 20 calculates the azimuth angle of the shooting range from the 360-degree camera at the extracted position to the player who is a target candidate (which may be referred to as the azimuth angle from the 360-degree camera to the player in the target direction or the azimuth angle to the player who is a target candidate). The information integration / extraction unit 20 compares this azimuth angle with the blind spot range set for each camera, and thereby extracts the positions of cameras where part or all of the azimuth angle to the target player is not included in part or all of the blind spot range, that is, cameras with no overlapping range between the azimuth angle and the blind spot range. Here, a plurality of camera positions are extracted. The azimuth angle to the player who is a target candidate is the angle between the reference azimuth in the shooting range of the 360-degree camera and the azimuth facing the player who is a target candidate.

[0029] The information integration / extraction unit 20 compares the calculated azimuth angles to the target player for each of the extracted positions of the cameras. When the difference between these azimuth angles is equal to or less than a first threshold, the information integration / extraction unit 20 extracts, as an effective camera that can shoot the player from a third-person perspective in a shooting direction selectable by the user, the camera among the plurality of extracted cameras that is closest to the target player, that is, the camera with a relatively short distance to the player, and adds information of this camera, for example, information including position information and shooting direction, to the effective camera list related to the corresponding target player (S22). That is, the information integration / extraction unit 20 can calculate, as a shooting direction selectable by the user, the shooting direction when the azimuth angle of the shooting range of the 360-degree camera is not included in the blind spot range among the shooting directions of each 360-degree camera.

[0030] In addition, the information integration / extraction unit 20 can also add to the effective camera list a camera among the extracted effective cameras whose difference from the azimuth angle related to another camera is equal to or greater than a second threshold that is greater than the first threshold.

[0031] The information integration and extraction unit 20 outputs the list of valid cameras related to each player to the UI unit 30 as, for example, thumbnails representing camera angles at regular time intervals among the time-series images taken from camera angles from which third-person perspective videos of each player can be displayed for the valid cameras, or as map information in which the positional relationship between the valid cameras and the camera angles from which third-person perspective videos of each player can be displayed are visualized. The above thumbnails are thumbnails of videos taken in the shooting direction of videos in which the subject is photographed from a third-person perspective among the videos taken by the 360-degree camera.

[0032] The UI unit 30 displays a list of the valid camera lists related to each player output from the information integration and extraction unit 20 as the above thumbnails or map information. The UI unit 30 receives an operation by the user related to the selection of a desired valid camera angle related to a desired target player among the valid camera angles related to each player indicated by this information. In response to this operation, the information integration and extraction unit 20 outputs to the video output unit 40 a 360-degree video taken by a 360-degree camera that can be taken at this selected angle, information indicating the azimuth angle from this 360-degree camera to the above-selected target player, and information indicating the distance between the 360-degree camera and the target player (S31).

[0033] Based on the azimuth angle from the camera to the target player and the distance between the camera and the target player, the video output unit 40 cuts out a planar video from the 360-degree video taken at the valid camera angle selected by the user, which is output by the UI unit 30, and outputs and displays this on the screen (S41).

[0034] Next, the calculation of the blind spot range related to the attached 360-degree camera will be described. FIG. 5 is a diagram showing an example of the blind spot range related to the 360-degree camera. In this embodiment, based on the blind spot information caused by the installation position of the 360-degree camera 200 in windsurfing, for example, the blind spot information caused by the mast 101 of windsurfing that does not change with the movement or rotation of the athlete, the blind spot range (reference sign B in FIG. 5) based on the orientation of each 360-degree camera 200 is manually set. In FIG. 5, it is shown that the information acquisition unit 10 can convert the blind spot range B shown in the top view related to the mast into the blind spot range B for each time series with 0 degrees based on the north, based on the detection result by the azimuth sensor mounted on the 360-degree camera 200.

[0035] Next, the calculation of the azimuth angle and distance from the 360-degree camera to the target athlete based on the GPS information from the information acquisition unit 10 by the information integration / extraction unit 20 will be described. The information integration / extraction unit 20 calculates the azimuth angle from each 360-degree camera to the target athlete and the distance between each 360-degree camera and the target athlete, respectively, based on the latitude and longitude of each camera and each athlete indicated by the GPS information from the information acquisition unit 10.

[0036] For example, assume that the latitude and longitude indicated by the GPS information detected by the GPS sensor of each 360-degree camera are longitude L1[n] and latitude P1[n]. Also, assume that the latitude and longitude indicated by the GPS information detected by the GPS sensor of the target athlete are longitude L2 and latitude P2.

[0037] Then, the distance D between the 360-degree camera n and the target athlete and the azimuth angle T from the 360-degree camera n to the target athlete are respectively represented by the following formulas (1) and (2). D = Sqrt(X 2 + Y 2 ) … Formula (1) T = atan2(Y, X) … Formula (2)

[0038] X and Y in Formulas (1) and (2) are respectively represented by the following Formulas (3) and (4). X = R × ΔL × cos(P1) … Equation (3) Y = R × ΔP … Equation (4)

[0039] R in Equations (3) and (4) is expressed as follows. R = 6378137 [m]: Equatorial radius of the earth ΔL in Equation (3) is expressed by the following Equation (5), and ΔP in Equation (4) is expressed by the following Equation (6). ΔL = L2 - L1 [n] … Equation (5) ΔP = P2 - P1 [n] … Equation (6)

[0040] Figure 6 is a diagram showing an example of the azimuth angle and distance from the 360-degree camera to the target player. In the example shown in Figure 6, the distance Dj between the camera j and the target player, and the azimuth angle (which may be referred to as the azimuth angle from the camera j to the target player) Tj from the reference azimuth of 0 degrees north among the angles of the shooting range of the camera j to the azimuth from the camera j to the target player, and the dead angle range B of the camera j with the azimuth of 0 degrees north as the reference azimuth are shown. The azimuth angle Tj shown in Figure 6 is the angle from the azimuth of 0 degrees north, which is the reference azimuth, to the azimuth from the camera j to the target player.

[0041] When the azimuth angle from the camera to the target player is included in the dead angle range of the camera, this camera is not selected as the above-mentioned effective camera by the information integration / extraction unit 20. In the example shown in Figure 6, since a part of the azimuth angle Tj from the camera j to the target player is included in the dead angle range of the camera j, this camera j is not selected as an effective camera.

[0042] Also, when video data for a certain period of time can be stored in the information storage unit in the information acquisition unit 10 during recording or relay by the camera, the user may be notified when the camera angle selected by the user, or the camera angles displayed as a list, enter the dead angle range at a timing close to the future.

[0043] Next, the narrowing down of the effective cameras based on the difference in azimuth angles will be described. FIG. 7 is a diagram showing an example of the setting of the effective cameras based on the difference in azimuth angles. When the difference in the azimuth angle with respect to the target player from each camera is equal to or less than a certain threshold value, the information integration / extraction unit 20 can set the camera with a shorter distance to the target among each camera as the effective camera because the angles of the images obtained by each camera become almost the same.

[0044] In the example shown in FIG. 7, among the angles of the shooting range of the first camera, camera i, from the reference azimuth of 0 degrees north, the azimuth angle from camera i to the target player (which may be referred to as the azimuth angle from camera i to the target player) T_i, and among the angles of the shooting range of the second camera, camera j, from the reference azimuth of 0 degrees north, the azimuth angle from camera j to the target player (which may be referred to as the azimuth angle from camera j to the target player) T_j, when the absolute value of the difference |T_i - T_j| is equal to or less than the threshold value T´, the information integration / extraction unit 20 compares the distance D_i between camera i and the target player with the distance D_j between camera j and the target player. Through this comparison, the information integration / extraction unit 20 sets the camera with the smaller distance to the target player, that is, the camera closer to the target, as the effective camera. This produces the effect of preventing other players from becoming obstacles.

[0045] Next, an example of the display of the effective camera angles of each player will be described. FIG. 8 is a diagram showing an example of the display screen of the effective camera angles of each player by means of thumbnails. The information integration / extraction unit 20 can present by causing the UI unit 30 to display, in time series on the screen, thumbnails representing the camera angles for each player and each angle at regular time intervals. In the example shown in FIG. 8, a thumbnail G1 of a captured video by a camera angle C capable of shooting player A from a third-person perspective is displayed by the UI unit 30 at each time series (time) t of 10 seconds. Thumbnails of captured videos by camera angles A and C capable of shooting player B from a third-person perspective are displayed by the UI unit 30 at each time series of 10 seconds. Thumbnails of captured videos by a camera angle B capable of shooting player C from a third-person perspective are displayed by the UI unit 30 at each time series of 10 seconds. This thumbnail is a thumbnail of a video in which the subject is shot from a third-person perspective according to the shooting direction calculated for the subject among the videos shot by the 360-degree camera.

[0046] The user can view the video of the designated camera angle of the corresponding player by selecting, by a click operation or the like, the thumbnail corresponding to the desired camera angle related to the desired player among the thumbnails displayed by the UI unit 30. Also, the × marks shown in FIG. 8 correspond to the captured videos by the camera angles that are in the blind spots among the respective camera angles.

[0047] Also, when there is no video of the selected camera angle, for example, when the said camera angle is in a blind spot, or when it is likely to enter, the information integration / extraction unit 20 can switch the video that the user can view on the UI unit 30 to the video of another camera angle that captures the corresponding player from a third-person perspective.

[0048] FIG. 9 is a diagram showing an example of a display screen of the effective camera angles of each player based on map information. In addition to the above thumbnail display, based on the latitude and longitude information of each of the above cameras and each player, the information integration / extraction unit 20 visualizes the positional relationship of each player on the map G2 in real time by the UI unit 30. The players on this map can be photographed from a third-party perspective, and the camera angles that the user can view can be displayed by the UI unit 30 as arrow-shaped buttons.

[0049] The user can select, by a click operation or the like, a button corresponding to a desired camera angle related to a desired player among the above arrow-shaped buttons displayed by the UI unit 30, and thereby view the video from the camera angle of the designated destination of the corresponding player. Also, the × mark shown in FIG. 9 corresponds to a camera angle that falls into a blind spot among each camera angle.

[0050] Next, the extraction of a planar video from a 360-degree video by a 360-degree camera will be described. FIGS. 10 to 13 are diagrams for explaining an example of the extraction of a planar video from a 360-degree video. First, as shown in FIG. 10, the video output unit 40 maps an omnidirectional video, for example, an image generated by a DualFishEye image or an equidistant cylindrical projection, onto a spherical surface in a virtual space.

[0051] Next, based on the acceleration, inclination, and azimuth of the 360-degree camera 200, which are the detection results by the 9-axis sensor of the information acquisition unit 10, the video output unit 40 maintains the horizontal level of the 360-degree camera 200 and rotates a virtual sphere, which is a sphere in the virtual space, so that the north direction becomes 0 degrees in the virtual space. For example, as shown in FIG. 11, which is a side view of the virtual sphere, the video output unit 40 rotates the virtual sphere so that the horizontal inclination of the 360-degree camera 200 becomes 0 degrees in the virtual space. Then, as shown in FIG. 12, which is a top view of the virtual sphere, the video output unit 40 rotates the virtual sphere so that the north direction coincides with 0 degrees in the virtual space.

[0052] Next, as shown in FIG. 13 which is a top view of the virtual sphere, the video output unit 40 arranges a virtual camera C for video output at the center of the virtual sphere. The video output unit 40 rotates the virtual camera C in accordance with the azimuth angle T from the virtual camera C to the target player that can be photographed at the specified camera angle. The video output unit 40 changes the field angle F of the virtual camera C in accordance with the distance between the virtual camera C and the target player. This field angle F is set in advance manually as several types of patterns. When the distance between the virtual camera C and the target player is relatively close, the video output unit 40 switches the field angle F of the virtual camera C to a relatively large field angle. On the other hand, when the distance between the virtual camera C and the target player is relatively far, the video output unit 40 switches the field angle F of the virtual camera C to a relatively small field angle. The video output unit 40 cuts out a planar video based on the azimuth angle from the camera to the target player and the distance between the camera and the target player as described above for the 360-degree video photographed at the switched field angle F.

[0053] Next, as a first modification example of the present embodiment, the selection of a camera angle capable of photographing a plurality of players from a third-person perspective will be described. FIG. 14 is a diagram showing an example of the selection of a camera angle capable of photographing a plurality of players from a third-person perspective. Here, an example will be described in which the information integration / extraction unit 20 regards a plurality of players with close distances to each other as a single target player, and displays the camera angle related to this target player on the display device of the UI unit 30 as map information. Among the plurality of players displayed in this map information, the user can select a plurality of players with relatively close distances to each other as a desired target player in a group by a click operation on the screen of the UI unit 30.

[0054] In the example shown in FIG. 14, when a plurality of players A, B, and C on the screen G3 in the UI unit 30 are selected by a click operation by the user (see reference symbol L1 in FIG. 14), the information integration / extraction unit 20 calculates the center of gravity g of the latitudes and longitudes of the selected plurality of players A, B, and C, and sets the coordinates of this center of gravity g as the coordinates of a single target player that can be photographed from cameras D or E.

[0055] Then, in the same manner as in the case where the target player is a single player, the information integration / extraction unit 20 sets the camera angle of the effective camera that can photograph and display the selected plurality of players A, B, and C from the third-person perspective based on the azimuth angle from cameras D or E to the center of gravity g, and causes the information related to this camera angle to be displayed again on the screen in the UI unit 30, and accepts the selection of the desired camera angle related to the desired target player in one group by the user.

[0056] Next, as a second modification example of the present embodiment, setting of a blind spot range using the position information of an obstacle will be described. FIGS. 15 and 16 are diagrams showing an example of setting a blind spot range using the position information of an obstacle. In the example shown in FIG. 15, an obstacle b is arranged between the center of gravity g of the latitudes and longitudes of players A, B, and C, that is, the coordinates of the target player in one group, and camera D, which was also described in FIG. 14. Due to the influence of this obstacle b, a screen G4 in which a blind spot range B due to the obstacle b is generated at the camera angle of camera D is shown to be displayed on the display screen of the UI unit 30.

[0057] The position information and size of an obstacle that becomes a blind spot in the video photographed by the camera, for example, the latitudes and longitudes of the four corners of a bounding box surrounding the obstacle, can be preset by the information integration / extraction unit 20. Then, the information integration / extraction unit 20 grasps the blind spot range due to the obstacle for each time series, and reflects this range on the screen for selecting the desired camera described above.

[0058] In the example shown in FIG. 16, an example is shown where an obstacle b exists in the camera angle of camera j. In this example, from the azimuth based on 0 degrees north, the azimuth angle (which may be referred to as the azimuth angle from camera j to the upper right end b1 of the bounding box of obstacle b in the figure) T_R to the upper right end b1 of the bounding box of obstacle b from camera j, and from the azimuth based on 0 degrees north, the azimuth angle (which may be referred to as the azimuth angle from camera j to the lower left end b2 of the bounding box of obstacle b in the figure) T_L (<T_R) to the lower left end b2 of the bounding box of obstacle b from camera j are respectively shown. Also, in this example, it is shown that the azimuth angle obtained by subtracting azimuth angle T_L from azimuth angle T_R corresponds to the dead angle range B of obstacle b with respect to the camera angle of camera j.

[0059] FIG. 17 is a block diagram showing an example of the hardware configuration of a video processing apparatus according to an embodiment of the present invention. In the example shown in FIG. 17, the video processing apparatus 100 according to the above embodiment is constituted by, for example, a server computer or a personal computer, and has a hardware processor 111A such as a CPU (Central Processing Unit). Then, a program memory 111B, a data memory 112, an input / output interface 113, and a communication interface 114 are connected to the hardware processor 111A via a bus 115. The same applies to each of the information acquisition units 10.

[0060] The communication interface 114 includes, for example, one or more wireless communication interface units, and enables transmission and reception of information to and from a communication network NW. As the wireless interface, for example, an interface adopting a low-power wireless data communication standard such as wireless LAN (Local Area Network) is used.

[0061] Connected to the input / output interface 113 are an input device 400 and an output device 500 that are attached to the video processing apparatus 100 and used by users and the like. The input / output interface 113 captures operation data input by users and the like through an input device 400 such as a keyboard, touch panel, touchpad, or mouse, and performs a process of outputting and displaying the output data to an output device 500 including a display device using liquid crystal or organic EL (Electro Luminescence) or the like. Note that, as the input device 400 and the output device 500, devices built in the video processing apparatus 100 may be used, or input devices and output devices of other information terminals capable of communicating with the video processing apparatus 100 via the network NW may be used.

[0062] The program memory 111B is a non-temporary tangible storage medium, for example, a non-volatile memory that can be written to and read from at any time, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), combined with a non-volatile memory such as a ROM (Read Only Memory), and stores programs necessary for executing various control processes and the like according to one embodiment.

[0063] The data memory 112 is a tangible storage medium, for example, a combination of the above non-volatile memory and a volatile memory such as a RAM (Random Access Memory), and is used to store various data acquired and created in the process of performing various processes.

[0064] The video processing apparatus 100 according to an embodiment of the present invention can be configured as a data processing apparatus having an information integration / extraction unit 20, a UI unit 30, and a video output unit 40 shown in FIG. 1 as processing functional units by software. The same applies to each of the information acquisition units 10.

[0065] Each information storage unit used as a working memory or the like by each unit of the video processing apparatus 100 can be configured by using the data memory 112 shown in FIG. 17. However, these configured storage areas are not essential configurations within the video processing apparatus 100. For example, they may be areas provided in an external storage medium such as a USB (Universal Serial Bus) memory or a storage device such as a database server arranged in the cloud.

[0066] The processing functional units in each of the above information integration / extraction unit 20, UI unit 30, and video output unit 40 can all be realized by causing the hardware processor 111A to read and execute the program stored in the program memory 111B. Note that some or all of these processing functional units may be realized in various other forms including integrated circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). The same applies to each of the information acquisition units 10.

[0067] In addition, the methods described in each embodiment can be implemented as a program (software means) executable on a computer, and stored, for example, in a recording medium such as a magnetic disk (e.g., a floppy disk, a hard disk), an optical disc (e.g., a CD-ROM, a DVD, an MO), or a semiconductor memory (e.g., a ROM, a RAM, a flash memory), and can be distributed by transmission through a communication medium. Note that the program stored on the medium side includes not only software means for causing a computer to execute (including not only an execution program but also a table and a data structure), but also a setting program for configuring software means in a computer. The computer that realizes this apparatus reads the program recorded on the recording medium, and in some cases, constructs software means using the setting program, and executes the above-described processing by controlling the operation by this software means. Note that the recording medium referred to in this specification includes not only a medium for distribution, but also a storage medium such as a magnetic disk or a semiconductor memory provided inside a computer or in a device connected via a network.

[0068] Note that the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist thereof at the implementation stage. Also, the embodiments may be implemented in appropriate combinations, and in such cases, combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combinations selected from a plurality of disclosed constituent elements. For example, even if some constituent elements are deleted from all the constituent elements shown in the embodiments, if the problem can be solved and an effect can be obtained, the configuration from which these constituent elements are deleted can be extracted as an invention.

Explanation of Reference Numerals

[0069] 100... video processing apparatus 10... information acquisition unit 20... information integration / extraction unit 30... user interface (UI) unit 40... video output unit 200…360-degree camera

Claims

1. A calculation unit that captures position information of a subject and images in a 360-degree range around the subject, and calculates, based on the position information of a plurality of 360-degree cameras mounted other than the subject, a shooting direction of an image in which the subject is photographed from a third-party perspective by the plurality of 360-degree cameras as a selectable shooting direction by a user who is not the subject; A selection unit that selects, from the shooting direction calculated by the calculation unit, a desired shooting direction of a subject photographed from a third-party perspective, which is a desired shooting direction by the user, where the calculated shooting direction is presented to the user; An output unit that outputs an image of a range including the desired subject among the images captured by the 360-degree camera capable of shooting in the shooting direction selected by the selection unit; An image processing apparatus comprising the above.

2. The calculation unit: Among the shooting directions by the plurality of 360-degree cameras, calculates, as a shooting direction of an image in which the subject is photographed from a third-party perspective, a shooting direction when the azimuth angle of the shooting range is not included in a dead angle range where the subject is not correctly photographed; The image processing apparatus according to claim 1.

3. The calculation unit: When the difference in the azimuth angle of the shooting range of each of the plurality of 360-degree cameras with respect to the subject satisfies a condition and is small, calculates, as a shooting direction of an image in which the subject is photographed from a third-party perspective, the shooting direction by the 360-degree camera that is relatively close to the subject; The image processing apparatus according to claim 1.

4. The calculation unit: Further calculates visual information in which the positional relationship of a plurality of the subjects and the calculated shooting direction are shown in a map format; The selection unit: Selects, from the shooting direction calculated by the calculation unit, a desired shooting direction of a subject photographed from a third-party perspective, which is a desired shooting direction by the user, with reference to the visual information; The image processing apparatus according to claim 1.

5. The calculation unit: Further calculates a thumbnail of an image in which the subject is photographed from the third-party perspective in the image captured by the 360-degree camera in the shooting direction calculated for the subject; The selection unit: Selects, from the shooting direction calculated by the calculation unit, a desired shooting direction of a subject photographed from a third-party perspective, which is a desired shooting direction by the user, with reference to the thumbnail; The image processing apparatus according to claim 1.

6. A method performed by an image processing apparatus, wherein a calculation unit of the image processing apparatus captures position information of a subject and an image of a 360-degree range around the subject, and based on the position information of a plurality of 360-degree cameras mounted other than the subject, calculates, as a shooting direction selectable by a user who is not the subject, a shooting direction of an image in which the subject is captured from a third-person perspective by the plurality of 360-degree cameras; a selection unit of the image processing apparatus selects, from the calculated shooting directions, a desired shooting direction of the subject captured from a third-person perspective, which is a shooting direction presented to the user and desired by the user; an output unit of the image processing apparatus outputs an image of a range including the desired subject among the images captured by the 360-degree camera capable of shooting in the selected shooting direction; An image processing method comprising the above steps.

7. The calculation by the calculation unit includes calculating, as a shooting direction of an image in which the subject is captured from a third-person perspective, a shooting direction among the shooting directions by the plurality of 360-degree cameras when the azimuth angle of the shooting range is not included in a dead angle range where the subject is not correctly captured. The image processing method according to claim 6.

8. An image processing program that causes a processor to function as each unit of the image processing apparatus according to any one of claims 1 to 5.

Images