Computer, program, information processing method, model generation method and estimation model

Abstract

To provide a computer capable of accurately estimating three-dimensional coordinates of a target in a spherical shape without using a dedicated position-fixed camera, a program, an information processing method, a model generation method and an estimation model.SOLUTION: A reception section 26a receives a still image of a target in a spherical shape, which is picked up by a camera 52, transmitted from a terminal (e.g., a portable communication terminal 50) including the camera 52 and information of an angle of view or a focal distance of the camera 52 imaging the target. An arithmetic section 28a calculates coordinates of four corners of a rectangle 210 enclosing the target from the still image of the target received by the reception section 26a and calculates a direction vector from a focal point of a lens of the camera 52 toward the target from the calculated coordinates of the four corners of the rectangle 210 enclosing the target and the angle of view or the focal distance of the camera 52. An estimation section 30a estimates three-dimensional coordinates of the target on the basis of the direction vector calculated by the arithmetic section 28a.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a computer, a program, an information processing method, a model generation method, and an estimation model. [Background technology]

[0002] Various technologies are known for tracking the trajectory of a ball in ball games such as baseball, soccer, golf, basketball, etc. For example, Patent Documents 1 to 5 disclose technologies for estimating the position and speed of a target sphere moving in a three-dimensional space using a high-speed camera that enables continuous shooting at high speed in a short period of time. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6236600 [Patent Document 2] Patent No. 6763559 [Patent Document 3] Special Publication No. 2016-540185 [Patent Document 4] JP 2019-219373 A [Patent Document 5] Special Publication No. 2005-529339 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional systems for tracking the trajectory of a ball, a dedicated camera is used to capture an image of the ball, and the position of the camera is fixed, and the zoom magnification and focus position of the camera are fixed to capture the image of the ball. However, such conventional systems have the problem that a dedicated camera with a fixed position is required, which increases the introduction cost. Furthermore, if the zoom magnification and focus position of the camera can be changed during shooting, convenience can be further improved.

[0005] The present invention has been made in consideration of these points, and aims to provide a computer, a program, an information processing method, a model generation method, and an estimation model that can accurately estimate the three-dimensional coordinates of a spherical object without using a dedicated camera at a fixed position. [Means for solving the problem]

[0006] The computer of the present invention comprises: a reception unit that receives a still image of a spherical object captured by a camera transmitted from a terminal including the camera, and information on the angle of view or focal length of the camera that captured the object; a calculation unit that calculates coordinates of four corners of a rectangle surrounding the object from the still image of the object received by the reception unit, and calculates a direction vector from the focal point of the lens of the camera toward the object from the calculated coordinates of the four corners of the rectangle surrounding the object and an angle of view or a focal length of the camera; an estimation unit that estimates three-dimensional coordinates of an object based on the direction vector calculated by the calculation unit; an output unit that outputs the three-dimensional coordinates of the object estimated by the estimation unit; The present invention is characterized by comprising:

[0007] In the computer of the present invention, The estimation unit may estimate the three-dimensional coordinates of the object based on the directional vector calculated by the calculation unit using a first estimation model obtained by machine learning using teacher data including a directional vector directed from the focus of the lens of the camera that captured the object to the object and the three-dimensional coordinates of the object.

[0008] The computer of the present invention comprises: The system may further include a model generation unit that uses training data including a direction vector from the focal point of the camera lens that images a spherical object toward the object and the three-dimensional coordinates of the object, to generate a first estimation model through machine learning that estimates the three-dimensional coordinates of the object from the direction vector from the focal point of the camera lens toward the object.

[0009] The computer of the present invention comprises: a reception unit that receives a still image of a spherical object captured by a camera transmitted from a terminal including the camera, and information on the angle of view or focal length of the camera that captured the object; a calculation unit that calculates coordinates of four corners of a rectangle surrounding the object from the still image of the object received by the reception unit; an estimation unit that estimates three-dimensional coordinates of an object based on the coordinates of the four corners of the rectangle calculated by the calculation unit and the angle of view or the focal length of the camera; an output unit that outputs the three-dimensional coordinates of the object estimated by the estimation unit; The present invention is characterized by comprising:

[0010] In the computer of the present invention, The estimation unit may estimate the three-dimensional coordinates of the object based on the coordinates of the four corners of the rectangle calculated by the calculation unit and the angle of view or focal length of the camera, using a second estimation model obtained by machine learning using teacher data including the coordinates of the four corners of a rectangle surrounding a spherical object obtained from a still image of the object, the angle of view or focal length of the camera that captured the object, and the three-dimensional coordinates of the object.

[0011] The computer of the present invention comprises: The system may further include a model generation unit that uses training data including the coordinates of the four corners of a rectangle surrounding a spherical object obtained from a still image of the object, the angle of view or focal length of the camera that captured the object, and the three-dimensional coordinates of the object, to generate the second estimation model by machine learning, which estimates the three-dimensional coordinates of the object from the coordinates of the four corners of a rectangle surrounding the object and the angle of view or focal length of the camera that captured the object.

[0012] In the computer of the present invention, In the terminal, the angle of view or focal length of the camera may be adjustable before capturing an image of the object with the camera, and when the object is captured by the camera, a still image of the captured object and information on the adjusted angle of view or focal length of the camera may be transmitted from the terminal to the computer.

[0013] In addition, an imaging application can be installed on the terminal, and when the imaging application is launched on the terminal, a still image of the object captured by the imaging application when the object is captured by the camera and information on the angle of view or focal length of the camera when the object was captured may be transmitted from the terminal to the computer.

[0014] In addition, information on the angle of view or focal length of the camera can be input in a browser displayed on the monitor of the terminal before the object is captured by the camera, and when the object is captured by the camera, a still image of the captured object and the information on the angle of view or focal length of the camera input in the browser may be transmitted from the terminal to the computer.

[0015] The program of the present invention comprises: A program for causing a computer to function as a receiving means, a calculating means, an estimating means, and an output means, The receiving means receives a still image of a spherical object captured by a camera transmitted from a terminal including the camera, and information on an angle of view or a focal length of the camera that captured the object; the calculation means calculates coordinates of four corners of a rectangle surrounding the object from the still image of the object accepted by the acceptance means, and calculates a direction vector from the focal point of the lens of the camera toward the object from the calculated coordinates of the four corners of the rectangle surrounding the object and an angle of view or a focal length of the camera; The estimation means estimates three-dimensional coordinates of the object based on the direction vector calculated by the calculation means, The output means outputs the three-dimensional coordinates of the object estimated by the estimation means.

[0016] The program of the present invention comprises: A program for causing a computer to function as a receiving means, a calculating means, an estimating means, and an output means, The receiving means receives a still image of a spherical object captured by a camera transmitted from a terminal including the camera, and information on an angle of view or a focal length of the camera that captured the object; The calculation means calculates coordinates of four corners of a rectangle surrounding the object from the still image of the object accepted by the acceptance means; the estimation means estimates three-dimensional coordinates of the object based on the coordinates of the four corners of the rectangle calculated by the calculation means and the angle of view or the focal length of the camera; The output means outputs the three-dimensional coordinates of the object estimated by the estimation means.

[0017] The information processing method of the present invention comprises the steps of: An information processing method performed by a computer having a control unit, a step of receiving, by the control unit, a still image of a spherical object captured by a camera and information on the angle of view or focal length of the camera that captured the object, the still image being transmitted from a terminal including the camera; a step of the control unit calculating coordinates of four corners of a rectangle surrounding the object from the received still image of the object, and calculating a direction vector from the focal point of the lens of the camera toward the object from the calculated coordinates of the four corners of the rectangle surrounding the object and the angle of view or focal length of the camera; The control unit estimates three-dimensional coordinates of the object based on the calculated direction vector; a step of the control unit outputting the estimated three-dimensional coordinates of the object; The present invention is characterized by comprising:

[0018] The information processing method of the present invention comprises the steps of: An information processing method performed by a computer having a control unit, a step of receiving, by the control unit, a still image of a spherical object captured by a camera and information on the angle of view or focal length of the camera that captured the object, the still image being transmitted from a terminal including the camera; The control unit calculates coordinates of four corners of a rectangle surrounding the object from the received still image of the object, The control unit estimates three-dimensional coordinates of the object based on the coordinates of the four corners of the calculated rectangle and the angle of view or the focal length of the camera; a step of the control unit outputting the estimated three-dimensional coordinates of the object; The present invention is characterized by comprising:

[0019] The model generation method of the present invention comprises the steps of: The method is characterized in that a first estimation model is generated by machine learning using training data including a directional vector from the focal point of the camera lens that captured the spherical object toward the object and the three-dimensional coordinates of the object, which estimates the three-dimensional coordinates of the object from the directional vector from the focal point of the camera lens toward the object.

[0020] The model generation method of the present invention comprises the steps of: The method is characterized in that, using training data including the coordinates of the four corners of a rectangle surrounding a spherical object obtained from a still image of the object, the angle of view or focal length of the camera that captured the object, and the three-dimensional coordinates of the object, a second estimation model is generated by machine learning to estimate the three-dimensional coordinates of the object from the coordinates of the four corners of the rectangle surrounding the object and the angle of view or focal length of the camera that captured the object.

[0021] The estimation model of the present invention is The parameter includes parameters obtained by machine learning using training data including a directional vector from the focal point of the camera lens that captured an image of a spherical object toward the object and the three-dimensional coordinates of the object, where the directional vector from the focal point of the camera lens toward the object is input and the three-dimensional coordinates of the object are output.

[0022] The estimation model of the present invention is The parameter includes parameters obtained by machine learning using training data including the coordinates of the four corners of a rectangle surrounding a spherical object obtained from a still image of the object, the angle of view or focal length of the camera that captured the object, and the three-dimensional coordinates of the object, when the coordinates of the four corners of a rectangle surrounding the object and the angle of view or focal length of the camera that captured the object are input, and the three-dimensional coordinates of the object are output. Effect of the Invention

[0023] According to the computer, program, information processing method, model generation method, and estimation model of the present invention, the three-dimensional coordinates of a spherical object can be estimated with high accuracy without using a dedicated camera at a fixed position. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of a computer according to an embodiment of the present invention. [Diagram 2] 2 is a diagram showing a flow of information processing performed by a control unit executing a program in the computer shown in FIG. 1. [Diagram 3] 2 is a flowchart illustrating an outline of the flow of information processing by the computer illustrated in FIG. 1. [Figure 4] FIG. 11 is an explanatory diagram for explaining a method of forming a bounding box surrounding a ball in a still image including the ball. [Diagram 5] FIG. 11 is an explanatory diagram for explaining a method for calculating a direction vector from the focal point of the camera lens toward the ball, based on the coordinates of the four corners of a bounding box surrounding the ball. [Figure 6] 11A and 11B are diagrams showing images displayed on the monitor of the portable communication terminal before and after adjusting the zoom magnification of the camera. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Figures 1 to 6 are diagrams showing a computer 10 and an information processing method according to the present embodiment. The computer 10 according to the present embodiment estimates the three-dimensional coordinates of a spherical object (specifically, a ball 200) of a preset size without using a dedicated camera at a fixed position.

[0026] First, the configuration of a computer 10 according to the present embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram showing the configuration of a computer 10 according to the present embodiment. As shown in Fig. 1, the computer 10 according to the present embodiment is composed of a computer and the like, and includes a control unit 20, a storage unit 40, a communication unit 42, a display unit 44, and an operation unit 46.

[0027] The control unit 20 is composed of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and controls the operation of the computer 10. Specifically, the control unit 20 functions as teacher data receiving means 22, model generating means 24, receiving means 26, calculation means 28, estimation means 30, and output means 32 by executing programs stored in a storage unit 40 described later.

[0028] The teacher data receiving means 22 receives teacher data 100 including a direction vector from the focal point of the lens of the camera 52 of the mobile communication terminal 50 that captured the image of the ball 200 (object) toward the object, and the three-dimensional coordinates of the object. In another aspect, the teacher data receiving means 22 may receive teacher data 100 including the coordinates of the four corners of a rectangle 210 (see FIG. 4, including a square) that is a bounding box surrounding the ball 200 in the still image, the angle of view of the camera 52 of the mobile communication terminal 50 that captured the image of the ball 200, and the three-dimensional coordinates of the ball 200.

[0029] The model generating means 24 uses teacher data 100 including a direction vector from the focal point of the lens of the camera 52 of the mobile communication terminal 50 that captured the ball 200 toward the object and the three-dimensional coordinates of the object to generate an estimation model 110 (first estimation model) that estimates the three-dimensional coordinates of the object from a direction vector from the focal point of the lens of the camera 52 toward the object by machine learning. Note that the model generating means 24 may use teacher data 100 including the coordinates of the four corners of a rectangle 210 surrounding the ball 200 acquired from a still image of the ball 200, the angle of view or focal length of the camera 52 that captured the ball 200, and the three-dimensional coordinates of the ball 200 to generate an estimation model 110 (second estimation model) that estimates the three-dimensional coordinates of the ball 200 from the coordinates of the four corners of the rectangle 210 surrounding the ball 200 and the angle of view or focal length of the camera 52 that captured the ball 200 by machine learning. As such machine learning, various known methods such as deep learning can be used.

[0030] Receiving means 26 receives still image 120 of ball 200 and information on the angle of view or focal length of camera 52 that captured ball 200, transmitted from mobile communication terminal 50. The angle of view or focal length of camera 52 received by receiving means 26 varies depending on the zoom magnification and resolution setting of camera 52.

[0031] Calculation means 28 calculates the coordinates of the four corners of rectangle 210 surrounding ball 200 from still image 120 of ball 200. Furthermore, calculation means 28 calculates a direction vector from the focal point of the lens of camera 52 toward the target object from the coordinates of the four corners of rectangle 210 surrounding ball 200 and the angle of view or focal length of camera 52.

[0032] Estimation means 30 estimates the three-dimensional coordinates of ball 200 based on the coordinates of the four corners of rectangle 210 calculated by calculation means 28 and the angle of view or focal length of camera 52, or based on the direction vector calculated by calculation means 28. At this time, estimation means 30 may estimate the three-dimensional coordinates of ball 200 based on the direction vector calculated by calculation means 28, using estimation model 110 (first estimation model) obtained by machine learning using teacher data 100 including a direction vector from the focus of the lens of camera 52 that captured the image of ball 200 toward the target object and the three-dimensional coordinates of ball 200. Alternatively, the estimation means 30 may estimate the three-dimensional coordinates of the ball 200 based on the coordinates of the four corners of the rectangle 210 calculated by the calculation means 28 and the angle of view or focal length of the camera 52, using an estimation model 110 (second estimation model) obtained by machine learning using teacher data 100 including the coordinates of the four corners of a rectangle 210 surrounding the ball 200 obtained from a still image of the ball 200, the angle of view or focal length of the camera 52 that captured the image of the ball 200, and the three-dimensional coordinates of the ball 200.

[0033] The output means 32 outputs the three-dimensional coordinates of the ball 200 estimated by the estimation means 30. The three-dimensional coordinates of the ball 200 output by the output means 32 are transmitted from the computer 10 to the portable communication terminal 50 and displayed on the monitor 53 of the portable communication terminal 50.

[0034] The functions of these means 22, 24, 26, 28, 30, and 32 will be described in detail later.

[0035] The storage unit 40 is configured with, for example, a hard disk drive (HDD), a random access memory (RAM), a read only memory (ROM), and a solid state drive (SSD). The storage unit 40 is not limited to being built into the computer 10, and may be a storage medium (for example, a USB memory) that is detachably attached to the computer 10. In this embodiment, the storage unit 40 is configured to store the program executed by the control unit 20, the estimation model 110, and the like.

[0036] The communication unit 42 includes a communication interface for transmitting and receiving signals to and from an external device wirelessly or via a wired connection. The communication unit 42 enables the control unit 20 to transmit and receive signals to and from a mobile communication terminal 50, which will be described later.

[0037] The display unit 44 is, for example, a monitor, and is configured to display various screens by receiving a display command signal from the control unit 20. The operation unit 46 is, for example, a keyboard, and is configured to give various commands to the control unit 20. Note that in this embodiment, a display operation unit such as a touch panel in which the display unit 44 and the operation unit 46 are integrated may be used.

[0038] Next, the configuration of the mobile communication terminal 50 that captures the image of the ball 200 will be described with reference to FIG. 1. As shown in FIG. 1, the mobile communication terminal 50 includes a camera 52, a monitor 53, and a communication unit 54. The camera 52 captures a moving image of the ball 200. A still image 120 of each frame of the ball 200 is acquired from the moving image of the ball 200 captured by the camera 52. The monitor 53 displays the moving image and still image 120 of the ball 200 captured by the camera 52. Here, as shown in FIG. 6, the screen displayed on the monitor 53 changes by adjusting the focus of the camera 52. The communication unit 54 includes a communication interface for transmitting and receiving signals to and from an external device wirelessly or by wire. The still image 120 of each frame of the ball 200 captured by the camera 52 is transmitted to the computer 10 by the communication unit 54. For example, a smartphone, a PC tablet, or the like can be used as the mobile communication terminal 50. In addition, a video camera capable of communicating with an external device may be used as the mobile communication terminal 50.

[0039] In this embodiment, an imaging application can be installed in portable communication terminal 50. When the installed imaging application is started in portable communication terminal 50, the imaging application enables ball 200 to be imaged by camera 52. In addition, when ball 200 is imaged by camera 52, a still image of ball 200 captured by the imaging application and information on the angle of view or focal length of camera 52 when ball 200 was imaged are transmitted from portable communication terminal 50 to computer 10.

[0040] Furthermore, information on the angle of view or focal length of camera 52 may be input in the browser displayed on monitor 53 of mobile communication terminal 50 when ball 200 is imaged by camera 52. When ball 200 is imaged by camera 52, a still image of the imaged ball 200 and information on the angle of view or focal length of camera 52 input in the browser are transmitted from mobile communication terminal 50 to computer 10.

[0041] As shown in FIG. 2, in this embodiment, the control unit 20 executes a program stored in the storage unit 40 to function as the teacher data receiving means 22, thereby forming a teacher data receiving unit 22a. The control unit 20 executes a program stored in the storage unit 40 to function as the model generating means 24, thereby forming a model generating unit 24a. The control unit 20 executes a program stored in the storage unit 40 to function as the receiving means 26, thereby forming a receiving unit 26a. The control unit 20 executes a program stored in the storage unit 40 to function as the computing means 28, thereby forming a computing unit 28a. The control unit 20 executes a program stored in the storage unit 40 to function as the estimation means 30, thereby forming an estimation unit 30a. The control unit 20 executes a program stored in the storage unit 40 to function as the output means 32, thereby forming an output unit 32a.

[0042] Next, a method in which the calculation means 28 (calculation unit 28a) calculates the coordinates of the four corners of a rectangle 210 as a bounding box surrounding the ball 200 from the still image 120 of the ball 200 will be described with reference to FIG. 4. FIG. 4(a) shows the still image 120 including the ball 200. The control unit 20 executes a program stored in the storage unit 40, and the calculation means 28 generates a rectangle 210 surrounding the ball 200 in the still image 120 as shown in FIG. 4(b). The y coordinates of the top and bottom sides of such a rectangle 210 are respectively the y coordinate (y1) of the pixel with the smallest y coordinate among all pixels of the color of the ball 200 and the y coordinate (y2) of the pixel with the largest y coordinate. In addition, the x coordinates of the left and right sides of the rectangle 210 are respectively the x coordinate (x1) of the pixel with the smallest x coordinate among all pixels of the color of the ball 200 and the x coordinate (x2) of the pixel with the largest x coordinate. The coordinates of the four corners of rectangle 210 are then expressed as (x1, y1), (x1, y2), (x2, y2), and (x2, y1).

[0043] In addition, in this embodiment, the calculation means 28 may use AI that detects an object in an image to calculate the coordinates of the four corners of the rectangle 210 surrounding the ball 200 from the still image 120 of the ball 200. Specifically, using teacher data including the still image 120 of the ball 200 and the coordinates of the four corners of the rectangle 210 surrounding the ball 200, an estimation model is generated by machine learning, in which the still image 120 of the ball 200 is input and the coordinates of the four corners of the rectangle 210 surrounding the ball 200 are output. As such machine learning, various known methods such as deep learning can be used. Then, the calculation means 28 uses the generated estimation model to calculate the coordinates of the four corners of the rectangle 210 surrounding the ball 200 from the still image 120 of the ball 200.

[0044] Next, a method in which the calculation means 28 (calculation unit 28a) calculates a direction vector from the focal point of the lens of the camera 52 toward the ball 200 (specifically, a direction vector toward the coordinates of the four corners surrounding the target object, for example) from the coordinates of the four corners of the rectangle 210 surrounding the ball 200 and the angle of view of the camera 52 (indicated by reference number 130 in FIG. 5) will be described with reference to FIG. 5. FIG. 5 is a view of the still image 120 shown in FIG. 4 as seen from directly above. A line segment when the still image 120 shown in FIG. 4 is viewed from above is indicated by reference number 120' in FIG. 5. For the sake of explanation, in FIG. 5, the three-dimensional coordinates of the ball 200 are set so that the right direction is the positive x coordinate, the depth direction of the paper is the positive y coordinate, and the upward direction is the positive z coordinate, with the focal point of the lens of the camera 52 as the origin. In addition, the direction vector is scaled so that z=1m. That is, the z coordinate on the three-dimensional coordinates of the direction vector is set to 1 (unit: m). Since the x-coordinates of all the left sides of rectangle 210 shown in Fig. 4 are x1 and the x-coordinates of all the right sides are x2, rectangle 210 shown in Fig. 4 exists between coordinate x1 and coordinate x2 in Fig. 5. The line segment when rectangle 210 shown in Fig. 4 is viewed from above is indicated by reference symbol 210' in Fig. 5.

[0045] When the direction vector is scaled to z=1m, that is, when the z coordinate of the direction vector on the three-dimensional coordinate system is 1 (unit: m), and the horizontal width (number of pixels) of the image is w, the x coordinate of the direction vector toward the right side of the rectangle 210 on the three-dimensional coordinate system is ((x2 / w-0.5)×2×tan(horizontal angle of view / 2)) (unit: m). Similarly, the x coordinate of the direction vector toward the left side of the rectangle 210 on the three-dimensional coordinate system is ((x1 / w-0.5)×2×tan(horizontal angle of view / 2)) (unit: m). This is because the x coordinates of the left and right ends of the still image 120 placed 1 m away from the focal point of the lens of the camera 52 are (±tan(angle of view / 2)) (unit: m). The horizontal angle of view refers to the angle of view of the camera 52 in the horizontal direction. Similarly, by viewing the field of view of camera 52 from the side, the y coordinate on the three-dimensional coordinates of the direction vector pointing to the top side of rectangle 210 and the y coordinate on the three-dimensional coordinates of the direction vector pointing to the bottom side of rectangle 210 can also be calculated. Specifically, when the z coordinate on the three-dimensional coordinates of the direction vector is 1 (unit: m) and the height (number of pixels) of the image is h, the y coordinate on the three-dimensional coordinates of the direction vector pointing to the top side of rectangle 210 is ((y1 / h-0.5)×2×tan(vertical angle of view / 2)) (unit: m). Similarly, the x coordinate on the three-dimensional coordinates of the direction vector pointing to the bottom side of rectangle 210 is ((y2 / h-0.5)×2×tan(vertical angle of view / 2)) (unit: m). The vertical angle of view refers to the angle of view of camera 52 in the vertical direction.

[0046] The directional vectors to the four corners of rectangle 210 have 3 components in the x, y, and z directions × 4 (coordinates of the four corners) = 12, but are scaled to z = 1 (unit: m), and the x component of the upper-left and lower-left vector, the x component of the upper-right and lower-right vector, the y component of the upper-left and upper-right vector, and the y component of the lower-left and lower-right vector are common. Therefore, the following four data are sufficient as elements of the directional vector from the focal point of the lens of camera 52 to ball 200 as training data 100. (a) (x1 / w-0.5)×2×tan(horizontal angle of view / 2) (b) (x2 / w-0.5)×2×tan (horizontal angle of view / 2) (c) (y1 / h-0.5)×2×tan(vertical angle of view / 2) (d) (y2 / h-0.5)×2×tan(vertical angle of view / 2)

[0047] Also, a directional vector from the focal point of the lens of camera 52 toward ball 200 may be calculated using a known method, not from the angle of view of camera 52, but from the focal length of camera 52 and the coordinates of the four corners of rectangle 210 surrounding ball 200. Here, the focal length of camera 52 refers to the distance from the center point of the lens to the image sensor (film surface).

[0048] Next, the operation of such a computer 10 will be described.

[0049] First, a method for generating the estimation model 110 will be described. The control unit 20 of the computer 10 is input with teacher data 100 including a direction vector from the focal point of the lens of the camera 52 of the mobile communication terminal 50 that captured the ball 200 toward the target object and the three-dimensional coordinates of the ball 200. Such teacher data 100 may be input by the operation unit 46, or the teacher data 100 may be transmitted to the control unit 20 from an external device via the communication unit 42. When the teacher data 100 is input to the control unit 20 of the computer 10, the teacher data receiving means 22 receives the teacher data 100. Then, the model generating means 24 performs machine learning using the teacher data 100 to generate an estimation model 110 (first estimation model) that estimates the three-dimensional coordinates of the ball 200 from the direction vector from the focal point of the lens of the camera 52 of the mobile communication terminal 50 that captured the ball 200 toward the target object and the angle of view or focal length of the camera 52 that captured the ball 200.

[0050] The method of generating the estimation model 110 is not limited to this embodiment. Other embodiments are described below. Teacher data 100 including the coordinates of the four corners of a rectangle 210 surrounding the ball 200 obtained from a still image of the ball 200, the angle of view or focal length of the camera 52 capturing the image of the ball 200, and the three-dimensional coordinates of the ball 200 is input to the control unit 20 of the computer 10. Such teacher data 100 may be input by the operation unit 46, or the teacher data 100 may be transmitted to the control unit 20 from an external device via the communication unit 42. When the teacher data 100 is input to the control unit 20 of the computer 10, the teacher data receiving means 22 receives the teacher data 100. Then, the model generation means 24 performs machine learning using the teacher data 100 to generate an estimation model 110 (second estimation model) that estimates the three-dimensional coordinates of the ball 200 from the coordinates of the four corners of a rectangle 210 surrounding the ball 200 obtained from a still image of the ball 200 and the angle of view or focal length of the camera 52 that captured the image of the ball 200.

[0051] Next, the operation when ball 200 is imaged by camera 52 of mobile communication terminal 50 will be described. As described above, in this embodiment, when ball 200 is imaged by camera 52 of mobile communication terminal 50, the zoom magnification and focus position of camera 52 are adjusted, so that moving image and still image 120 of the imaged ball 200 can be made clear. When the zoom magnification of camera 52 is adjusted, the angle of view and focal length of camera 52 also change. Furthermore, as shown in FIG. 6, the screen displayed on monitor 53 also changes when the zoom magnification of camera 52 is adjusted.

[0052] When a moving image of ball 200 is captured by camera 52, still images 120 of each frame of ball 200 are obtained from the captured moving image of ball 200. Still images 120 of each frame of ball 200 are transmitted from mobile communication terminal 50 to computer 10 and accepted by acceptance means 26. Information on the angle of view or focal length of camera 52 is also transmitted from mobile communication terminal 50 to computer 10 and accepted by acceptance means 26 (step S1 in FIG. 3). Note that the angle of view and focal length of camera 52 may be directly input to computer 10 by the operator using operation unit 46, instead of being transmitted from mobile communication terminal 50 to computer 10.

[0053] Next, calculation means 28 calculates the coordinates of the four corners of rectangle 210 surrounding ball 200 from still image 120 of ball 200 (step S2 in FIG. 3). In this manner, information on four coordinates (x1, y1), (x1, y2), (x2, y1), and (x2, y2) as shown in FIG. 4 is acquired. Furthermore, calculation means 28 calculates a direction vector from the focal point of the lens of camera 52 toward ball 200 from the coordinates of the four corners of rectangle 210 surrounding ball 200 and the angle of view or focal length of camera 52 (step S3 in FIG. 3).

[0054] Next, the estimation means 30 estimates the three-dimensional coordinates of the ball 200 based on the coordinates of the four corners of the rectangle 210 calculated by the calculation means 28 and the angle of view or focal length of the camera 52, or based on the direction vector calculated by the calculation means 28 (step S4 in FIG. 3). Then, the output means 32 outputs the estimated three-dimensional coordinates of the ball 200 (step S5 in FIG. 3). Specifically, the three-dimensional coordinates of the ball 200 obtained from the still images 120 of the ball 200 in each frame are joined together to output information on the three-dimensional trajectory of the ball 200. The information output by the output means 32 is displayed on the display unit 44, or is transmitted to an external device by the communication unit 42 and displayed on a monitor of the external device (for example, the monitor 53 of the mobile communication terminal 50).

[0055] According to the computer 10, program, and information processing method of the present embodiment configured as described above, the computer 10 includes a reception unit 26a that receives a still image of the ball 200 (spherical object) captured by the camera 52, transmitted from a terminal (mobile communication terminal 50) including the camera 52, and information on the angle of view or focal length of the camera 52 that captured the image of the ball 200; and a reception unit 26b that calculates the coordinates of the four corners of the rectangle 210 that surrounds the ball 200 from the still image of the ball 200 received by the reception unit 26a, and calculates the coordinates of the four corners of the rectangle 210 that surrounds the ball 200. The computer 10 includes a calculation unit 28a that calculates a direction vector from the focal point of the lens of the camera 52 toward the ball 200 from the coordinates of the four corners of a rectangle 210 that encloses the ball 200 and the angle of view or focal length of the camera 52, an estimation unit 30a that estimates the three-dimensional coordinates of the ball 200 based on the coordinates of the four corners of the rectangle 210 calculated by the calculation unit 28a and the angle of view or focal length of the camera 52, or based on the direction vector calculated by the calculation unit 28a, and an output unit 32a that outputs the three-dimensional coordinates of the ball 200 estimated by the estimation unit 30a. According to such computer 10, program, and information processing method, even if the focus of the camera 52 is changed, information on the angle of view or focal length of the camera 52 according to the adjusted focus of the camera 52 is transmitted from the mobile communication terminal 50 to the computer 10, and the three-dimensional coordinates of the ball 200 are estimated in the computer 10 based on the angle of view or focal length of the camera 52, or based on the direction vector from the focal point of the lens of the camera 52 toward the ball 200. This allows the three-dimensional coordinates of ball 200 to be estimated with high accuracy without using a dedicated camera with a fixed position. More specifically, in conventional systems, a dedicated camera for capturing an image of the ball is used, and the position of this camera is fixed, and then the zoom magnification and focus position are fixed to capture an image of ball 200. However, such conventional systems have a problem in that a dedicated camera with a fixed position is required, resulting in high implementation costs. Furthermore, convenience can be further improved if the zoom magnification and focus position of the camera can be changed during shooting.In contrast, in the present embodiment, a camera 52 attached to a mobile communication terminal 50 such as a smartphone can be used, and the zoom magnification, focus position, resolution, etc. of the camera 52 can be adjusted, thereby solving the problems associated with such conventional systems.

[0056] Furthermore, the estimation unit 30a estimates the three-dimensional coordinates of the ball 200 based on the direction vector calculated by the calculation unit 28a, using an estimation model 110 (first estimation model) obtained by machine learning using teacher data 100 including a direction vector from the focal point of the lens of the camera 52 that captured the ball 200 toward the ball 200 and the three-dimensional coordinates of the ball 200. In this case, the model generation unit 24a uses teacher data 100 including a direction vector from the focal point of the lens of the camera 52 that captured the ball 200 toward the ball 200 and the three-dimensional coordinates of the ball 200 to generate, by machine learning, an estimation model 110 (first estimation model) that estimates the three-dimensional coordinates of the ball 200 from a direction vector from the focal point of the lens of the camera 52 toward the ball 200. In this case, the three-dimensional coordinates of the ball 200 can be estimated more accurately by using the estimation model 110 (first estimation model) obtained by machine learning.

[0057] In addition, the estimation unit 30a may estimate the three-dimensional coordinates of the ball 200 based on the coordinates of the four corners of the rectangle 210 calculated by the calculation unit 28a and the angle of view or focal length of the camera 52, using an estimation model 110 (second estimation model) obtained by machine learning using teacher data 100 including the coordinates of the four corners of the rectangle 210 surrounding the ball 200 obtained from the still image 120 of the ball 200, the angle of view or focal length of the camera 52 that captured the image of the ball 200, and the three-dimensional coordinates of the ball 200. In this case, model generation unit 24a uses teacher data 100 including the coordinates of the four corners of rectangle 210 surrounding ball 200 acquired from still image 120 of ball 200, the angle of view or focal length of camera 52 that captured the image of ball 200, and the three-dimensional coordinates of ball 200, to generate, by machine learning, estimation model 110 (second estimation model) that estimates the three-dimensional coordinates of ball 200 from the coordinates of the four corners of rectangle 210 surrounding ball 200 and the angle of view or focal length of camera 52 that captured the image of ball 200. In this case, by using estimation model 110 (second estimation model) obtained by machine learning, it is possible to estimate the three-dimensional coordinates of ball 200 with higher accuracy.

[0058] Furthermore, in the computer 10, program, and information processing method of this embodiment, as described above, in the mobile communication terminal 50, the angle of view or focal length of the camera 52 can be adjusted before the camera 52 captures an image of the ball 200, and when the camera 52 captures an image of the ball 200, a still image of the captured ball 200 and information on the adjusted angle of view or focal length of the camera 52 are transmitted from the mobile communication terminal 50 to the computer 10.

[0059] Furthermore, in the computer 10, program, and information processing method of this embodiment, as described above, an imaging application can be installed in the mobile communication terminal 50, and when the imaging application is started in the mobile communication terminal 50, a still image of the ball 200 captured by the imaging application when the ball 200 is captured by the camera 52, and information on the angle of view or focal length of the camera 52 when the ball 200 was captured are transmitted from the mobile communication terminal 50 to the computer 10.

[0060] In addition, information on the angle of view or focal length of camera 52 can be input in a browser displayed on monitor 53 of mobile communication terminal 50 before ball 200 is imaged by camera 52, and when ball 200 is imaged by camera 52, a still image of the captured ball 200 and the information on the angle of view or focal length of camera 52 input in the browser can be transmitted from mobile communication terminal 50 to computer 10.

[0061] Moreover, according to the model generation method of the present embodiment, using teacher data 100 including a direction vector from the focal point of the lens of camera 52 capturing an image of ball 200 (spherical object) toward ball 200 and the three-dimensional coordinates of ball 200, an estimation model 110 (first estimation model) that estimates the three-dimensional coordinates of ball 200 from the direction vector from the focal point of the lens of camera 52 toward ball 200 is generated by machine learning. By using estimation model 110 generated by such a model generation method, it becomes possible to accurately estimate the three-dimensional coordinates of ball 200 from still image 120 of ball 200.

[0062] According to another model generation method of the present embodiment, using teacher data 100 including coordinates of the four corners of rectangle 210 (bounding box) surrounding ball 200 (spherical object) acquired from still image 120 of the ball 200, the angle of view or focal length of camera 52 capturing the image of ball 200, and the three-dimensional coordinates of ball 200, an estimation model 110 (second estimation model) is generated by machine learning, which estimates the three-dimensional coordinates of ball 200 from a directional vector pointing from the focal point of the lens of camera 52 toward ball 200. By using estimation model 110 generated by such a model generation method, it becomes possible to accurately estimate the three-dimensional coordinates of ball 200 from still image 120 of ball 200.

[0063] Furthermore, estimation model 110 (first estimation model) of this embodiment includes parameters obtained by machine learning using teacher data 100 including a direction vector from the focal point of the lens of camera 52 that captured the image of ball 200 toward ball 200 and the three-dimensional coordinates of ball 200, with a direction vector from the focal point of the lens of camera 52 toward ball 200 as input and the three-dimensional coordinates of ball 200 as output. By using such estimation model 110, it becomes possible to accurately estimate the three-dimensional coordinates of ball 200 from still image 120 of ball 200.

[0064] Further, another estimation model 110 (first estimation model) of this embodiment includes parameters obtained by machine learning using teacher data 100 including the coordinates of the four corners of a rectangle 210 surrounding ball 200 acquired from still image 120 of ball 200, the angle of view or focal length of camera 52 that captured the image of ball 200, and the three-dimensional coordinates of ball 200, and includes parameters obtained when the coordinates of the four corners of a rectangle 210 surrounding ball 200 and the angle of view or focal length of camera 52 that captured the image of ball 200 are input and the three-dimensional coordinates of ball 200 are output. By using such estimation model 110, it becomes possible to accurately estimate the three-dimensional coordinates of ball 200 from still image 120 of ball 200.

[0065] The computer, program, information processing method, model generation method, estimation model, and the like according to the present invention are not limited to the above-described aspects, and various modifications can be made.

[0066] For example, the method of calculating the coordinates of the four corners of rectangle 210 as a bounding box surrounding ball 200 from still image 120 of ball 200 is not limited to the above-mentioned method. Various methods other than the above-mentioned method may be used to calculate the coordinates of the four corners of rectangle 210 as a bounding box surrounding ball 200 from still image 120 of ball 200.

[0067] Furthermore, the model generating means 24, the receiving means 26, the computing means 28, the estimating means 30, the output means 32, etc. may be provided in separate computers.

[0068] Furthermore, calculation means 28 calculates the coordinates of the four corners of rectangle 210 surrounding ball 200 from the still image of ball 200 accepted by acceptance means 26, but may not calculate a direction vector from the focal point of the lens of camera 52 toward ball 200. Furthermore, a known method other than the above-mentioned method may be used as a method of calculating the direction vector from the focal point of the lens of camera 52 toward ball 200.

[0069] Furthermore, the method of estimating the three-dimensional coordinates of the ball 200 using the estimation model 110 is not limited to the above-mentioned method. Various methods other than the above-mentioned method may be used as the method of estimating the three-dimensional coordinates of the ball 200 using the estimation model 110.

[0070] In addition, instead of the program of the present invention being stored in the memory unit 40 of the computer 10 and the program stored in the memory unit 40 being executed by the control unit 20, the program of the present invention may be stored in the memory unit of a mobile communication terminal 50 such as a smartphone, and the program stored in the memory unit may be executed by the control unit (CPU, etc.) of the mobile communication terminal 50. [Explanation of symbols]

[0071] 10. Computers 20 Control section 22 Teacher data reception method 24 Model Generation Method 26 Reception methods 28 Calculation means 30 Estimation means 32 Output Method 22a Teacher data reception section 24a Model generation section 26a Reception 28a Arithmetic unit 30a Estimation part 32a Output section 40 Storage section 42 Communications Department 44 Display section 46 Control section 50 Mobile communication devices 52 Camera 53 Monitor 54 Communications Department 100 Teacher Data 110 Estimation Model 120, 120´ Still Image 130 Angle of View 200 balls 210 rectangle

Claims

1. A device for receiving, when a still image of a spherical object of a preset size is captured after a field of view or focal length of a camera has been changed, the still image and information on the field of view or focal length of the camera at the time of capturing the image from a terminal including the camera; an estimation unit that estimates three-dimensional coordinates of a center of the object based on coordinates of four corners of a rectangle surrounding the object in the still image received by the reception unit, an angle of view or a focal length of the camera, and a size of the object; an output unit that outputs the three-dimensional coordinates of the center of the object estimated by the estimation unit; A computer equipped with

2. The computer described in Claim 1, wherein the output unit outputs information about the trajectory of the object based on the three-dimensional coordinates of the center of the object obtained from the still image of each frame.

3. The camera further comprises a calculation unit that calculates a direction vector from the focal point of the lens of the camera toward the coordinates of the four corners of the rectangle based on the coordinates of the four corners of the rectangle and the angle of view or the focal length of the camera, The computer according to claim 1 , wherein the estimation unit estimates the three-dimensional coordinates of the center of the object based on the size of the object and the direction vector.

4. The computer described in claim 1, wherein the estimation unit estimates the three-dimensional coordinates of the center of the object using a second estimation model based on the size of the object, the second estimation model being obtained by machine learning using training data including the coordinates of the four corners, the angle of view or focal length of the camera, and the three-dimensional coordinates of the center of the object.

5. A camera with an adjustable angle of view or focal length; a communication unit that, when a still image of a spherical object of a preset size is captured after the angle of view or focal length has been changed, transmits the still image and information on the angle of view or focal length of the camera at the time of capturing the image to an external device, and receives in the external device three-dimensional coordinates of the center of the object that are estimated based on the coordinates of the four corners of a rectangle surrounding the object in the still image, the angle of view or focal length of the camera, and the size of the object; a display unit that displays the three-dimensional coordinates of the center of the object; A mobile terminal equipped with

6. Computer, a receiving unit that, when a still image of a spherical object having a preset size is captured after a field angle or a focal length of the camera is changed, receives the still image and information on the field angle or the focal length of the camera at the time of capturing the image from a terminal including the camera; an estimation unit that estimates three-dimensional coordinates of a center of the object based on coordinates of four corners of a rectangle surrounding the object in the still image received by the reception unit, an angle of view or a focal length of the camera, and a size of the object; and an output unit that outputs the three-dimensional coordinates of the center of the object estimated by the estimation unit; A program to function as a

7. A computer, an imaging control unit that causes a camera with an adjustable angle of view or focal length to capture a still image of a spherical object having a preset size; a transmitter that, when a still image of the object is captured after the angle of view or the focal length is changed, transmits the still image and information about the angle of view or the focal length of the camera at the time of capturing the image to an external device; a receiving unit in the external device that receives three-dimensional coordinates of a center of the object estimated based on coordinates of four corners of a rectangle surrounding the object in the still image, an angle of view or a focal length of the camera, and a size of the object; and a display unit that displays the three-dimensional coordinates of the center of the object; A program to function as a

8. An information processing method performed by a computer having a control unit, a receiving step in which, when a still image of a spherical object having a preset size is captured after a field angle or a focal length of the camera is changed, the control unit receives the still image and information on the field angle or the focal length of the camera at the time of capturing the image from a terminal including the camera; an estimation step in which the control unit estimates three-dimensional coordinates of a center of the object based on coordinates of four corners of a rectangle surrounding the object in the still image received in the receiving step, an angle of view or a focal length of the camera, and a size of the object; an output step in which the control unit outputs the three-dimensional coordinates of the center of the object estimated in the estimation step; An information processing method, including:

9. An information processing method performed by a mobile terminal having a control unit, an imaging step in which the control unit causes a camera having an adjustable angle of view or focal length to capture a still image of a spherical object having a preset size; a transmission step in which, when a still image of the object is captured after the angle of view or the focal length is changed, the control unit transmits the still image and information about the angle of view or the focal length of the camera at the time of capturing the image to an external device; a receiving step in which the control unit receives, in the external device, three-dimensional coordinates of the center of the object estimated based on the coordinates of four corners of a rectangle surrounding the object in the still image, the angle of view or the focal length of the camera, and the size of the object; a display step in which the control unit displays three-dimensional coordinates of the center of the object; An information processing method, including:

Images