Processing apparatus, processing system, processing method, and processing program

Abstract

To provide technique for accurately identifying a player to extract a scene without imposing additional operation.SOLUTION: A processing apparatus which processes golf video includes a memory and a processor. The memory stores video data and player information. The processor is configured to: determine whether an in-frame person detected from the frame of the video data coincides with one of registered players who are players registered in the player information; associate, when the in-frame person coincides with one of the registered players, the registered player with the frame, or newly register, when the in-frame person does not coincide with any of the registered players, the in-frame person on the player information as a registered player; and associate the newly registered player with the frame.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a technique for filming a golf game and creating a digest of the game. [Background technology]

[0002] Patent Documents 1 and 2 disclose a golf digest creation system that films golf play and creates a digest that collects highlight scenes. In order to create a digest of a golf round that incorporates shot scenes of each player from a video of the play during a golf round, it is necessary to identify players in the video and detect the scenes of shots made by the players.

[0003] In order to collect highlight scenes of golf play, image processing, particularly person detection and person determination, is used. In person detection, people appearing in an image are detected by analyzing the image. In person determination, people to be extracted are registered in advance, and a determination is made as to whether or not a person detected from an image is a person registered in advance.

[0004] For example, in a golf digest application, it may be desirable to identify each player who appears in the video and construct the digest so that each player's footage is included somewhat evenly, or evenly, or so that every player appears at least once.

[0005] In this case, if each player wears a GPS receiver at all times during the round to obtain location information, and the camera is also equipped with a GPS receiver to obtain location information, and the direction in which the footage was taken is obtained from the camera, it will be possible to identify the player in the footage based on the directional information of the footage and the location information of the player and camera.

[0006] On the other hand, in the field of image processing, there is a technology that identifies people in an image itself by matching processing such as face recognition. The features obtained from the image of the person to be identified are acquired in advance, and the person is identified by comparing the features of the person in the image with the previously acquired features of the person. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2020-88855 A [Patent Document 2] JP 2022-110177 A Summary of the Invention [Problem to be solved by the invention]

[0008] However, the location information obtained by GPS contains errors, and when multiple players are positioned close to each other from the camera, for example, it is not easy to correctly identify individual players from the GPS information.

[0009] Furthermore, in order to determine the identity of each player through matching processing, it is necessary to acquire image features of each player in advance. In order to register the information of each player in advance, a task of taking an image of each player is required in preparation for creating a digest before starting a round of golf. Such a task is unnecessary for the players, and may reduce convenience and entertainment value.

[0010] One objective of the present disclosure is to provide a technique for accurately identifying a player and extracting a scene without imposing extra work. [Means for solving the problem]

[0011] A processing device according to one aspect included in the present disclosure is a processing device that processes golf video, and has a memory and a processor, the memory stores video data and player information, and the processor determines whether a person in a frame detected from a frame of the video data matches any of the registered players who are players registered in the player information, and if the person in the frame matches any of the registered players, links the registered player to the frame, and if the person in the frame does not match any of the registered players, newly registers the person in the frame as a registered player in the player information, and links the newly registered registered player to the frame. Effect of the Invention

[0012] According to one aspect of the present disclosure, players can be differentiated and extracted without prior registration of the players. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram of a golf digest creating system. [Diagram 2] FIG. 2 is a block diagram of a mobile photography unit. [Diagram 3] FIG. 2 is a block diagram of a processing device. [Figure 4] 13 is a flowchart of a digest creation process. [Diagram 5] 13 is a flowchart of a person tracking process. [Figure 6] FIG. 11 is a diagram showing an example of a player information database. [Figure 7] FIG. 13 is a conceptual diagram for explaining point correction. [Figure 8] FIG. 13 is a conceptual diagram for explaining a point deduction correction. [Figure 9] 13 is a flowchart of a shot scene detection process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] In this embodiment, a golf digest creation system is disclosed that takes videos of multiple players playing multiple holes on a golf course and creates a digest of the highlight scenes from the videos. The video here refers to both video and audio.

[0016] Golf is a sport in which players play an 18-hole golf course in a set order, competing for the total number of strokes they can hit a golf ball from the teeing area to the cup on the green. A round refers to playing an 18-hole golf course. During a round, players travel on and between golf courses in golf carts.

[0017] <System configuration> FIG. 1 is a block diagram of a golf digest creating system according to the present embodiment.

[0018] The golf digest generating system 10 has a mobile camera unit 11 and a processing device 12. The mobile camera unit 11 and the processing device 12 can transmit and receive data to and from each other via wired communication, wireless communication, or both. The mobile camera unit 11 is mounted on a golf cart and captures video including images and audio of players playing a golf round. The processing device 12 creates a digest based on data of the video captured by the mobile camera unit 11. Note that, as an example, the mobile camera unit 11 transmits video data to the processing device 12 via data communication, but as an alternative configuration, data may be handed over from the mobile camera unit 11 to the processing device 12 via a storage medium.

[0019] <Configuration of the mobile shooting unit> FIG. 2 is a block diagram of the mobile photography unit.

[0020] The mobile photography unit 11 is a device that is mounted on a golf cart and automatically records 360-degree video and audio around the golf cart while playing, and has an imaging section 21, a sound collection section 22, a control section 23, a wireless transmission / reception section 24, and an AV memory section 25.

[0021] The imaging unit 21 is installed at a position above the roof of the golf cart where a 360-degree field of view is open, and includes a 360-degree camera (not shown) capable of capturing images in all directions. The imaging unit 21 captures images with the 360-degree camera at a frame rate of 20 fps to 60 fps, and transmits image data with the elapsed time (time code) from the start of capture added to each frame to the AV storage unit 25. At that time, the imaging unit 21 may output images in which distortion caused by the wide-angle lens of the 360-degree camera has been corrected. This distortion correction is realized as real-time processing by an ASIC (Application Specific Integrated Circuit) incorporating a known algorithm. The 360-degree camera may also be configured to cover 360 degrees using multiple cameras, and images captured by each camera may be synthesized. The sound collection unit 22 includes a sound collection microphone (not shown), acquires audio data with the sound collection microphone, and transmits the audio data to the AV storage unit 25 with time information (time code) synchronized with the frames of the video data. The imaging unit 21 and the sound collection unit 22 may be configured as an integrated unit to obtain video data including video data and audio data that are synchronized with each other, and transmit the video data to the AV storage unit 25.

[0022] The AV storage unit 25 is a storage device that stores the video data from the imaging unit 21 and the audio data from the sound collection unit 22. The AV storage unit 25 is a non-volatile storage device that uses a hard disk or a semiconductor memory card as a recording medium, and a semiconductor memory card that has excellent shock resistance is preferable. The wireless transmission / reception unit 24 performs high-speed wireless communication with a fifth-generation wireless base station, for example, and performs medium-range wireless communication (IEEE802.11) up to several hundred meters. A fifth-generation mobile communication system including a fifth-generation wireless base station is connected to the Internet, and enables data communication between the mobile photography unit 11 connected to the fifth-generation wireless base station and the processing device 12 on the Internet. The control unit 23 controls the entire mobile photography unit 11, reads video data including the video data acquired by the imaging unit 21 and the audio data acquired by the sound collection unit 22 from the AV storage unit 25, and transmits the video data to the processing device 12 via the wireless transmission / reception unit 24. The AV storage unit 25, the wireless transmission / reception unit 24, and the control unit 23 may be configured by a computer device equipped with a CPU, a memory, a non-volatile storage device, an input / output interface, and a wireless communication interface. In this case, the control unit 23 realizes its functions by loading a software program stored in a non-volatile storage device into memory and executing the program.

[0023] <Processing device configuration> FIG. 3 is a block diagram of the processing device.

[0024] The processing device 12 includes a network interface unit 31 , an AV storage unit 32 , a control unit 33 , a display unit 34 , an operation unit 35 , an output unit 36 ​​, and a digest storage unit 37 .

[0025] A network interface (NIF) unit 31 is connected to the Internet and enables transmission and reception of data with the outside.

[0026] The AV storage unit 32 is a non-volatile storage device, and is a storage device that stores the video data received from the mobile shooting unit 11. The display unit 34 is a device that displays images and text on a display screen in association with processing by the control unit 33. The operation unit 35 is a device that accepts information input by a user through a keyboard, mouse, or the like, and the input information is used for processing by the control unit 33. The output unit 36 ​​is a device that outputs data such as the processing result of the control unit 33, and outputs the digest video created by the processing device 12 to a storage medium such as a USB memory. The digest storage unit 37 is a non-volatile storage device, and is a storage device that stores data of the digest video created by the processing device 12. The AV storage unit 32 and the digest storage unit 37 may be integrally configured on the same device. The control unit 33 creates a digest video incorporating scenes of shots by each player from the video data received from the mobile shooting unit 11 and stored in the AV storage unit 32, and records the digest video in the digest storage unit 37. A series of processes for creating a digest video will be described in detail later. The network interface unit 31, the AV storage unit 32, the control unit 33, the display unit 34, the operation unit 35, the output unit 36, and the digest storage unit 37 may be configured by a computer device equipped with a CPU, a memory, a non-volatile storage device, an input / output interface, and a wireless communication interface. In this case, the control unit 33 realizes its functions by loading a software program stored in the non-volatile storage device into the memory and executing it.

[0027] <Overall processing> 4 is a flowchart of the digest creation process. The digest creation process is a process for creating a digest moving image from moving image data captured by the mobile shooting unit 11. The digest creation process is started in the processing device 12 by a user instruction input from the operation unit 35, and is executed by the control unit 33.

[0028] First, in step S101, the control unit 33 of the processing device 12 reads video data from the AV storage unit 32 and executes a person tracking process. The person tracking process is a process that simultaneously performs a list of players appearing in a video from the video data and a linking of the players appearing in the video to frames, which are components that make up the video, while improving the accuracy of the list-up and linking. Linking of a player to a frame is, for example, recording a frame and information identifying a player appearing in the frame in a database in association with each other. The person tracking process will be described in detail later.

[0029] Next, in step S102, the control unit 33 executes a shot scene detection process. The shot scene detection process is a process for extracting a shot scene in which a player hits a ball with a golf club from a video. The shot scene detection process makes it possible to extract a shot scene with high accuracy by focusing on a transition of an image that is unique to golf. The shot scene detection process will be described in detail later.

[0030] Next, in step S103, the control unit 33 executes a good shot degree calculation process. The good shot degree calculation process is a process for calculating a good shot degree for each shot scene. The good shot degree is an index for estimating how memorable the shot is in the video. Therefore, the good shot degree does not necessarily coincide with a great shot in a competition. The good shot degree calculation process will be described in detail later.

[0031] Next, in step S104, the control unit 33 executes a digest movie creation process. The digest movie creation process is a process of selecting and combining scenes to create a digest movie. For the digest movie, scenes with a high degree of good shot quality for each player are selected. The digest movie creation process will be described in detail later.

[0032] <People tracking processing> FIG. 5 is a flowchart of the person tracking process.

[0033] First, in step S201, the control unit 33 of the processing device 12 initializes the player information database.

[0034] FIG. 6 is a diagram showing an example of the player information database. As described above, the person tracking process is a process of simultaneously and in parallel listing players appearing in a video from video data and linking the players appearing there to frames constituting the video. The player information database 40 is a database that lists players detected from video data. In the player information database 40, a player identification number (ID) 42 of the player registered in the entry and a feature amount 43 of the image of the player registered in the entry are registered in association with an entry number (No) 41 of each entry. In the example of FIG. 6, the entry with the entry number 1 has a player identification number of 1 and a feature amount of (X, X, X, X, X...), and the entry with the entry number 2 has a player identification number of 1 and a feature amount of (X, Y, Y, X, X...). In this way, data of a plurality of feature amounts may be registered for the same player. Note that the player information database 40 initialized by the process of this step S104 is in a state in which no entry is registered.

[0035] Next, in step S202, the control unit 33 extracts a frame of a video included in the video data. Furthermore, in step S203, the control unit 33 performs human detection on the image of the extracted frame. Although there is no limitation on the method of realizing human detection, it can be realized, for example, by using OpenVINO (registered trademark) and its built-in trained human detection model.

[0036] Next, in step S204, the control unit 33 determines whether or not a person has been detected from the frame in the process of step S203. If a person has not been detected, the control unit 33 proceeds to step S212.

[0037] On the other hand, if a person is detected from the frame, in step S205, the control unit 33 performs a matching process. The matching process is a process for checking whether the person detected from the frame (person in the frame) is the same person as a player (registered player) registered in the player information database 40. If the similarity between the person in the frame and any of the registered players exceeds a predetermined threshold, it is determined that the person in the frame and that registered player are the same person. In the following description, this threshold may be referred to as a first threshold.

[0038] Below are three examples of the similarity between each registered player and the person in the frame.

[0039] (Example 1) The cosine similarity between the feature of each entry in the player information database 40 and the feature of the person in the frame is calculated. Then, the cosine similarity is aggregated for each registered player with each player identification number. The average of the top predetermined number of cosine similarities of each registered player is regarded as the similarity between the registered player and the person in the frame.

[0040] Since people appear in a frame in various ways, the same features do not usually appear even for the same person. Here, various features can be registered for a registered player, and similarities are calculated in a brute-force manner between the features of the person in the frame and each of the player features associated with the registered player, and if the average value of the top predetermined similarities exceeds a threshold, it is determined that the person in the frame is the same person as the registered player. This enables good matching of people in the frame who may appear in various ways.

[0041] (Example 2) In the second example, the similarity in the first example is corrected using information on the person detection results of the previous frame and the current frame. The correction in the second example includes a correction for adding points to the similarity and a correction for subtracting points from the similarity.

[0042] FIG. 7 is a conceptual diagram for explaining the point addition correction.

[0043] When a person is detected in the person detection process, a rectangular area called a bounding box that surrounds the detected person is obtained. The area where the rectangular area surrounding the person in the target frame overlaps with the rectangular area estimated from the rectangular area surrounding the person in the frame detected in the previous frame is taken as a tracking area, and the area of ​​the tracking area is calculated. If the area exceeds a predetermined threshold, a predetermined value is added to the cosine similarity calculated in the first example of that frame or the similarity that is the average value of the cosine similarity.

[0044] In the example of FIG. 7, the nth frame corresponds to the target frame. In the (n-1)th frame, which is the frame immediately before the nth frame, a registered player with an ID of 0 is detected, and a rectangular bounding box BB(n-1) is formed. In the nth frame, a registered player with an ID of 0 is detected, and a rectangular bounding box BBn is formed. In addition, a bounding box BB'n estimated from the bounding box BB(n-1) of the (n-1)th frame is also shown on the nth frame in FIG. 7. The area where the bounding box BB'n and the bounding box BBn overlap is the tracking area TAn. If the area of ​​the tracking area TAn exceeds a threshold, a predetermined value is added to the cosine similarity of the nth frame or the similarity which is the average value of the cosine similarity.

[0045] FIG. 8 is a conceptual diagram for explaining the point deduction correction.

[0046] The movable range is determined to be the estimated movable range from a rectangular area surrounding a person detected in a frame prior to the target frame, and if the rectangular area surrounding the person in the target frame is outside the movable range, a predetermined value is subtracted from the cosine similarity calculated in the first example of that frame or the similarity which is the average value of the cosine similarity.

[0047] In the example of FIG. 8, the nth frame corresponds to the target frame. In the (n-1)th frame, which is the frame immediately before the nth frame, a registered player with an ID of 0 is detected, and a rectangular bounding box BB(n-1) is formed. In the nth frame, a registered player with an ID of 0 is detected, and a rectangular bounding box BBn is formed. In addition, the nth frame in FIG. 8 also shows a movable range MRn estimated from the bounding box BB(n-1) of the (n-1)th frame. Then, since the bounding box BBn is outside the movable range MRn, a predetermined value is subtracted from the cosine similarity of the nth frame or the similarity which is the average value thereof.

[0048] If there is a large overlap between the rectangular area of ​​the target frame and the rectangular area estimated from past frames, it is considered that the player in that frame is moving in a similar manner to the estimation, and if there is a rectangular area outside the possible movement range estimated from past frames, it is considered that the player in that frame is moving in a different manner from the estimation. Points are added to the similarity when the movement is considered to be similar to the estimation from past frames, and points are subtracted when the movement is considered to be different from the estimation from past frames, so that the similarity is well corrected.

[0049] 5, in step S206, the control unit 33 determines whether or not the person in the frame is determined to be the same person as a registered player in the process of step S205. If it is determined that the person in the frame is not the same as any of the registered players, in step S207, the control unit 33 sets the person in the frame as a newly registered player, assigns a new player identification number (ID), and registers the ID and the feature amount of the person extracted from the frame in the player information database 40. Next, in step S207, the control unit 33 links the newly registered player to the frame in the video data, and proceeds to step S212.

[0050] If it is determined in step S206 that the person in the frame is identical to any of the registered players, then in step S209, the control unit 33 links the frame in the video data to the registered player determined to be identical. Next, in step S210, the control unit 33 compares the degree of similarity between the feature amount of the person in the frame extracted from the frame and the feature amount of the registered player registered in the player information database 40 with a second threshold value that is greater than the first threshold value.

[0051] If the similarity is equal to or greater than the second threshold, the control unit 33 proceeds to step S212. On the other hand, if the similarity is smaller than the second threshold, the control unit 33 registers in the player information database 40, in step S211, the feature amount of the person in the frame extracted from the frame as the feature amount of a registered player who is the same person as the person in the frame, and proceeds to step S212.

[0052] In step S212, the control unit 33 determines whether the target frame is the last frame in the video data. If the target frame is not the last frame, the control unit 33 updates the target frame to the next frame in the video data in step S213 and returns to step S202. On the other hand, if the target frame is the last frame, the control unit 33 ends the person tracking process.

[0053] As a result of the person tracking process described above, a player information database in which registered players and their features are registered, information linking registered players to frames of video data, and information on the position and size of the bounding box of the registered player in the frame are obtained.

[0054] <Shot scene detection processing> During a round of golf, there are not only players hitting the ball on the course, but also players just walking around and watching other players play, and these players may appear in the video. Since the scene of the shot in which the player hits the ball is very important for a digest of a golf round, it is desirable to be able to distinguish and detect the shot scene from other scenes. As a method for enabling detection of the shot scene, it is possible to estimate the player's skeleton and the club trajectory from the video to identify the scene in which the player hits the ball. However, unless the direction of the camera relative to the player is suitable and somewhat constant, the player's skeleton cannot be correctly detected from the video, and the shape of the club trajectory in the video differs greatly, so it is not easy to analyze the player's movements and the club trajectory by image processing to identify the shot scene. For example, it is difficult to extract the scene of the player hitting the ball with high accuracy from the video that is continuously shot over a wide area by a 360-degree camera. Therefore, a technology is required to extract the scene of the player hitting the ball with high accuracy from the video of the golf.

[0055] FIG. 9 is a flowchart of the shot scene detection process.

[0056] First, in step S301, the control unit 33 of the processing device 12 reads the result of the person tracking process. Next, in step S302, the control unit 33 focuses on a frame included in the video of the video data, and checks whether or not the sound corresponding to the frame includes a shot sound, which is a sound generated by hitting a ball with a golf club. Furthermore, in step S303, the control unit 33 determines whether or not the shot sound exists in the sound corresponding to the frame, and if the shot sound does not exist, it determines that a shot scene cannot be detected from the frame, and proceeds to step S308.

[0057] On the other hand, if a shot sound is present, in step S304, the control unit 33 extracts the frame from N seconds before to P seconds after the frame as a shot scene candidate, which is a shot scene candidate, and obtains the result of the person detection process of each frame included in the shot scene candidate. The values ​​of N and P are not particularly limited. For example, N and P may be fixed in advance, for example, both N and P may be 1 second. N and P may be external parameters that can be set appropriately. In addition, instead of fixing N and P, a time range to be regarded as one shot may be specified from the video. For example, the time when the person is confirmed to be still may be counted back from the time when the shot sound is detected as the start of the shot, and the time when the person's coordinates change by a certain amount or more may be counted forward from the time when the shot sound is detected as the end of the shot, and the time from the start time to the end time may be extracted as a shot scene candidate.

[0058] Next, in step S305, the control unit 33 first calculates, for each frame included in the shot scene candidate, for each registered player detected in the frame, the total sum of differences between the pixel values ​​of each pixel between the image of the bounding box area in the frame and the image of the area in the frame immediately preceding the frame, calculates a value obtained by normalizing the total sum of differences by the area of ​​the bounding box (normalized total difference value), and calculates an integrated value (integrated total difference value) of the normalized total difference value for each registered player in the shot scene candidate video. Furthermore, the control unit 33 calculates, for each frame included in the shot scene candidate, for each registered player detected in the frame, the distance between the position of the bounding box in the frame and the position of the bounding box in the frame immediately preceding the frame as the amount of movement, and calculates an integrated value (integrated movement amount) of the amount of movement for each registered player in the shot scene candidate video. In this case, for example, the center coordinates of the bounding box are regarded as the position of the bounding box, the difference between the center coordinates of the current frame and the frame immediately preceding it is regarded as the movement vector, and the magnitude of the movement vector is regarded as the amount of movement.

[0059] Next, in step S306, the control unit 33 determines whether or not there is a registered player whose integrated difference value is greater than a predetermined threshold (fourth threshold) and whose integrated movement amount is less than a predetermined threshold (fifth threshold). This determination utilizes the characteristics of a golf shot, that is, when a player is making a shot, the player's body moves while the player's position on the image does not change significantly, for detecting the shot scene.

[0060] If there is no registered player whose accumulated difference value is greater than the fourth threshold and whose accumulated movement amount is less than the fifth threshold, the control unit 33 determines that the shot scene candidate created from the frame was not a shot scene, and proceeds to step S308.

[0061] On the other hand, if there is a registered player whose integrated difference value is greater than the fourth threshold and whose integrated movement amount is less than the fifth threshold, in step S307, the control unit 33 detects the shot scene candidate created from the frame as the shot scene of that registered player, and proceeds to step S308. At this time, if there are multiple registered players whose integrated difference value is greater than the fourth threshold and whose integrated movement amount is less than the fifth threshold, the control unit 33 determines the shot scene to be that of the registered player with the largest integrated difference value.

[0062] In step S308, the control unit 33 determines whether the target frame is the last frame in the video data. If the target frame is not the last frame, the control unit 33 updates the target frame to the next frame in the video data in step S309 and returns to step S302. On the other hand, if the target frame is the last frame, the control unit 33 ends this shot scene detection process.

[0063] <Good shot degree calculation process> The good shot degree calculation process is a process of calculating the good shot degree for each shot scene. The good shot degree is an index that estimates how memorable a shot is in a video, and is calculated by the following formula (1).

[0064] (Good shot degree) = (cumulative voice score) × (shot player area score) ... (1) The sound score in formula (1) is an index indicating the volume of cheers for a predetermined time after a shot, and is calculated as follows.

[0065] First, the presence or absence of human voice is determined at each timing after the shot included in the shot scene, and if there is human voice, the volume at that timing is set as the voice score at that timing, and if there is no human voice, the voice score at that timing is set to zero, and the integrated value of the voice scores at the timing included in the predetermined time after the shot is set as the integrated voice score. The presence or absence of human voice can be determined using a method called Voice Activity Detector (VAD). The above-mentioned predetermined time may be 10 seconds, for example.

[0066] Since cheers do not have a fixed pattern, if you try to detect cheers directly, you cannot detect them with high accuracy. Also, if you decide the audio score based only on the volume regardless of whether it is a cheer or not, only noise will be detected during a golf round, and accuracy will not be ensured. In contrast, the method of this embodiment estimates cheers from the presence or absence of human voices after a shot and the volume, and calculates the audio score, so it is possible to calculate cheers after a shot in a golf round with high accuracy.

[0067] The shot player area score in formula (1) is an index calculated from the area of ​​the area of ​​the player who took the shot in the image of the frame, and is calculated as log10 (shot player area). The area of ​​the area of ​​the player who took the shot is used because a scene in which the player who took the shot is shown large is considered to be suitable for a digest. In addition, by using the logarithm of the area, if the area of ​​the shot player is above a certain level, the audio score is more likely to be reflected in the good shot degree, and conversely, if the area of ​​the shot player is below a certain level, the good shot degree is small even if the audio score is high.

[0068] <Digest video creation process> The digest movie creation process is a process of selecting and combining shot scenes detected in the shot scene detection process to create a digest movie of a predetermined length or less. Hereinafter, the maximum length of the digest movie will be referred to as the digest time. Hereinafter, the length of a shot scene will be set to a maximum of 13 seconds, as an example. The digest time may also be set to 10 minutes, as an example.

[0069] First, if the total time of the shot scenes detected by the above-mentioned shot scene detection process is less than the digest time, all the shot scenes are combined in chronological order to form a digest video. If the total time of the shot scenes exceeds the digest time, shots with a good shot degree less than a predetermined threshold (sixth threshold) are shortened to the first 6 seconds, and if the total time of the shot scenes is less than the digest time in this state, all the shot scenes including the shortened shot scenes are combined in chronological order to form a digest video.

[0070] If the total time of all the shot scenes including the shortened shot scenes exceeds the digest time, the registered player with the largest number of shot scenes is identified, and the shot scene with the lowest good shot degree of the registered player is excluded. Then, in this state, it is determined whether the total time of the shot scenes is equal to or less than the digest time, and if it is equal to or less, the shot scenes are combined in chronological order in this state to create a digest. If the total time of the shot scenes still exceeds the digest time, the registered player with the largest number of shot scenes is identified, and the shot scene with the lowest good shot degree of the registered player is excluded, and this process is repeated until the total time of the shot scenes is equal to or less than the digest time.

[0071] The above-described embodiments are illustrative examples of the present invention, and are not intended to limit the scope of the present invention to these embodiments. Those skilled in the art can implement the present invention in various other forms without departing from the gist of the present invention.

[0072] The above-described embodiment includes the following features, but is not limited to the features included in the embodiment.

[0073] (Item 1) A processing device for processing golf videos has a memory and a processor, the memory stores video data and player information, and the processor determines whether a person in a frame detected from a frame of the video data matches any of the registered players who are players registered in the player information, and if the person in the frame matches any of the registered players, links the registered player to the frame, and if the person in the frame does not match any of the registered players, newly registers the person in the frame as a registered player in the player information and links the newly registered registered player to the frame. This makes it possible to distinguish and extract players without registering them in advance.

[0074] (Item 2) In the processing device described in item 1, the memory stores, as the player information, player features, which are features of an image of a registered player, in association with player identification information that identifies each registered player, and the processor determines whether or not the person in the frame matches the registered player based on the similarity between the features of the person in the frame extracted from the frame and the player features of the registered player. This makes it easy to match the person in the scene with the registered player based on the similarity between the features of the image of the frame included in the scene and the image of the registered player.

[0075] (Item 3) In the processing device described in item 2, the processor calculates a similarity between the feature of the person in the frame and each of the player features associated with the registered player in a brute-force manner, compares the average of a predetermined number of similarities among the calculated similarities with a predetermined first threshold, and if the average exceeds the first threshold, determines that the person in the frame is the registered player. According to this, since people appear in a frame in various ways, the same feature may not appear even if the person is the same. Various feature amounts can be registered for the registered player, and a similarity is calculated between the feature of the person in the frame and each of the player features associated with the registered player in a brute-force manner, and if the average of the predetermined number of similarities exceeds the threshold, determines that the person in the frame is the same person as the registered player. This enables good matching of people in the frame who may appear in various ways.

[0076] (Item 4) In the processing device described in item 3, when the processor determines that the in-frame person is the registered player, if the average value of the similarity between the feature amount of the in-frame person and the feature amount of the registered player is greater than the first threshold and equal to or less than a second threshold greater than the first threshold, the processor additionally registers the feature amount of the in-frame person as the player feature amount of the registered player in the player information. According to this, if the similarity between the feature amount of the in-frame person determined to be a registered player and the player feature amount already registered for the registered player is equal to or less than a certain level, the processor additionally registers the feature amount of the in-frame person as the player feature amount of the registered player, thereby further improving the accuracy of subsequent matching.

[0077] (Item 5) In the processing device described in item 3, if the area of ​​a tracking area, which is an overlapping area of ​​a rectangular area surrounding a person in a first frame and a rectangular area estimated from a rectangular area surrounding a person in a second frame that is a frame prior to the first frame, exceeds a predetermined third threshold, a point is added to the similarity or the average value calculated in the first frame, and if the rectangular area surrounding the person in the first frame is outside the movable range estimated from the rectangular area surrounding the person in the second frame, a point is subtracted from the similarity or the average value calculated in the first frame. According to this, if the rectangular area of ​​the frame overlaps greatly with the rectangular area estimated from a previous frame, it is considered that the player in the frame is moving in the same way as estimated, and if a rectangular area exists outside the movable range estimated from a previous frame, it is considered that the player in the frame is moving differently from estimated. If the movement is deemed to be similar to that estimated from past frames, points are added to the similarity, and if the movement is deemed to be different from that estimated from past frames, points are subtracted, making it possible to link registered players to frames of good golf swings with a high degree of accuracy.

[0078] (Item 6) In the processing device described in item 1, the processor extracts scenes of each registered player from the video data based on the association of the registered players with frames, and creates a digest that collects scenes selected so that the difference between the number of scenes or the total time of each registered player falls within a predetermined value or a predetermined ratio range. This makes it possible to create a digest in which multiple players who have played a round appear roughly evenly.

[0079] (Item 7) In the processing device described in item 1, the processor extracts candidate scenes of a predetermined length from the video data, identifies a scene including a shot action of a player hitting a ball from the candidate scenes based on the degree of change in the image of the area around the person in the scene and the degree of movement of the person, and extracts a scene including a shot action of each registered player from the video data based on the association of the registered players with the frames and the identification of the scene including the shot action. According to this, shot scene candidates are extracted using a characteristic of golf play that the time from when the player addresses the ball to when he finishes hitting the ball falls within a certain range, and further, the shot scene is identified from the candidates using the characteristics related to the body movement and the body movement during the action of the player hitting the ball in golf, so that golf shot scenes can be extracted from the video with high accuracy.

[0080] (Item 8) In the processing device described in item 7, the processor identifies a scene including a shot action of a player hitting a ball from among the scene candidates based on an integrated difference amount obtained by accumulating differences in pixel values ​​in areas around a person between adjacent frames included in the scene candidates and an integrated movement amount obtained by accumulating the movement amount between the frames of a rectangular area indicating human detection. According to this, by calculating the degree of change in an image from the difference in pixel values ​​between frames and calculating the degree of movement of a person from the movement amount of a rectangular area indicating human detection, it is possible to easily extract a golf shot scene from a video with high accuracy.

[0081] (Item 9) In the processing device described in item 7, the processor cuts out the video of the predetermined time including the time when the shot sound occurred in the video data as a candidate for the scene. This cuts out a portion of the predetermined time including the time when the shot sound was observed from the video of the golf play as a scene, so that it is possible to extract with high accuracy the scene of the player hitting the ball from the video of the golf play.

[0082] (Item 10) In the processing device described in item 8, the processor determines that the candidate scene includes a shot motion if the accumulated difference amount in the candidate scene is greater than a predetermined fourth threshold value and the accumulated movement amount is smaller than a predetermined fifth threshold value. This makes it possible to extract shot scenes from videos with high accuracy by identifying shot scenes from the candidates by utilizing the characteristic that the body movement is large while the body movement is small during the action of hitting a ball in golf.

[0083] (Item 11) A processing system comprising: a 360-degree camera installed at a position on the top of a golf cart that provides a 360-degree field of view; and a sound collecting microphone installed on the golf cart; a mobile photographing unit that adds time information to the video data from the 360-degree camera and the audio data from the sound collecting microphone to synchronize the two; and the processing device described in item 6.

[0084] (Item 12) A processing method for a computer having a memory and a processor to process golf videos, wherein the memory stores video data and player information, and the processor determines whether a person in a frame detected from a frame of the video data matches any of the registered players who are players registered in the player information, and if the person in the frame matches any of the registered players, links the registered player to the frame, and if the person in the frame does not match any of the registered players, newly registers the person in the frame as a registered player in the player information, and links the newly registered registered player to the frame.

[0085] (Item 13) A processing program for causing a computer to process golf videos, the processing program causing the computer to execute the following: determine whether a person in a frame detected from a frame of video data matches any of the registered players who are players registered in player information; if the person in the frame matches any of the registered players, link the registered player to the frame; if the person in the frame does not match any of the registered players, newly register the person in the frame as a registered player in the player information, and link the newly registered registered player to the frame. [Explanation of symbols]

[0086] 10...golf digest generation system, 11...mobile photographing unit, 12...processing device, 21...imaging section, 22...sound collection section, 23...control section, 24...wireless transmission / reception section, 25...AV storage section, 31...network interface section, 32...AV storage section, 33...control section, 34...display section, 35...operation section, 36...output section, 37...digest storage section, 40...player information database, 41...entry number, 42...player identification number, 43...feature amount

Claims

1. A processing device for processing golf video, A memory and a processor are included. the memory stores video data and player information; The processor, determining whether a person in a frame detected from a frame of the video data matches any of the registered players who are registered in the player information; If the person in the frame matches any of the registered players, the registered player is associated with the frame; If the person in the frame does not match any of the registered players, the person in the frame is newly registered as a registered player in the player information, and the newly registered registered player is linked to the frame. Processing unit.

2. the memory stores, as the player information, player features that are features of an image of a registered player in association with player identification information that identifies each registered player; The processor, determining whether or not the person in the frame matches the registered player based on a similarity between a feature amount of the person in the frame extracted from the frame and a player feature amount of the registered player; The processing device of claim 1 .

3. The processor, calculating a similarity between the feature of the person in the frame and each of the player features associated with the registered player in a round-robin manner; comparing an average value of a predetermined number of the calculated similarities with a predetermined first threshold value; If the average value exceeds the first threshold, it is determined that the person in the frame is the registered player. The processing device according to claim 2 .

4. The processor, when it is determined that the person in the frame is the registered player, if an average value of the similarity between the feature amount of the person in the frame and the feature amount of the registered player is greater than the first threshold and is equal to or less than a second threshold which is greater than the first threshold, additionally registering the feature amount of the person in the frame as a player feature amount of the registered player in the player information; The processing device according to claim 3 .

5. If an area of ​​a tracking region, which is an overlapping area between a rectangular region surrounding a person in a first frame and a rectangular region estimated from a rectangular region surrounding a person in a second frame that is a frame prior to the first frame, exceeds a predetermined third threshold value, adding points to the similarity or the average value calculated in the first frame; If a rectangular area surrounding the in-frame person detected in the first frame is outside a movable range estimated from a rectangular area surrounding the in-frame person detected in the second frame, a reduction is made in the similarity or the average value calculated in the first frame. The processing device according to claim 3 .

6. The processor, extracting scenes of each registered player from the video data based on the association of the registered player with the frame; creating a digest of scenes selected so that the difference between the number of scenes or the total duration of each registered player among the registered players falls within a predetermined value or a predetermined percentage range; The processing device of claim 1 .

7. The processor, extracting scene candidates of a predetermined length from the video data; Identifying a scene including a shot action of a player hitting a ball from among the scene candidates, based on a degree of change in an image of an area around a person in the scene and a degree of movement of the person; extracting scenes including shot actions of each registered player from the video data based on linking the frames to the registered players and identifying the scenes including the shot actions; The processing device of claim 1 .

8. The processor, Identifying a scene including a shot action of a player hitting a ball from among the scene candidates, based on an integrated difference amount obtained by integrating differences in pixel values ​​in areas around a person between adjacent frames included in the scene candidates, and an integrated movement amount obtained by integrating a movement amount between the frames of a rectangular area indicating human detection; The processing device of claim 7.

9. The processor, extracting a video of the predetermined time period including a time when a shot sound occurred from the video data as a candidate for the scene; The processing device of claim 7.

10. The processor, If the integrated difference amount in the scene candidate is greater than a predetermined fourth threshold value and the integrated movement amount is smaller than a predetermined fifth threshold value, it is determined that the scene candidate includes a shot action. The processing device of claim 8.

11. A mobile photography unit including a 360-degree camera installed at a position on the top of a golf cart where a 360-degree field of view is open, and a sound collecting microphone installed on the golf cart, and which adds time information to the video data from the 360-degree camera and the audio data from the sound collecting microphone in order to synchronize the two; A processing device according to claim 6 ; A processing system having:

12. 1. A processing method for a computer having a memory and a processor to process golf video, comprising: the memory stores video data and player information; The processor, determining whether a person in a frame detected from a frame of the video data matches any of the registered players who are registered in the player information; If the person in the frame matches any of the registered players, the registered player is associated with the frame; If the person in the frame does not match any of the registered players, the person in the frame is newly registered as a registered player in the player information, and the newly registered registered player is linked to the frame. Processing methods.

13. A processing program for causing a computer to process golf video, determining whether a person in a frame detected from a frame of the video data matches any of the registered players who are registered in the player information; If the person in the frame matches any of the registered players, the registered player is associated with the frame; If the person in the frame does not match any of the registered players, the person in the frame is newly registered as a registered player in the player information, and the newly registered registered player is linked to the frame. A processing program that causes a computer to execute a task.

Images