Swing information determination device and program

Abstract

To accurately determine swing information of a batter from a video in a service for generating and distributing an explanatory voice text of a baseball relay program.SOLUTION: A pitching detection means 11 of a swing information determination device 1 inputs a video of a baseball program, detects pitching by using a classifier, and generates pitching detection time T1. The swing detection unit 12 detects the swing of the batter using the swing learning model M1 and generates the swing detection time T2. The screen switching detection means 13 uses the screen switching learning model M2 to detect screen switching indicating that the screen has been switched from the "confrontation scene" to "others", and generates a screen switching detection time T3. The swing information determination unit 15 determines swing information based on the swing detection time T2, the screen change detection time T3, and a predetermined value A.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a service that generates and distributes text for commentary audio of live sports broadcasts, and relates to technology that not only supports television viewing for primarily visually impaired people, but also for many viewers, including able-bodied people who watch television while doing other things. In particular, the present invention relates to a swing information determination device and program that automatically determine swing information of a baseball batter. [Background technology]

[0002] Conventionally, commentary audio services have been known that broadcast live sports programs while providing viewers with audio commentary of the broadcast programs (see, for example, Patent Documents 1 and 2, and Patent Application Publication No. 2023-16933, which was filed by the same applicant as the present patent application and was unpublished at the time of filing the present patent application (hereinafter referred to as Unpublished Patent Document 1)).

[0003] 18 is a diagram illustrating an overview of a system for providing a commentary audio service. This system includes a broadcast transmitting device 101, a broadcast receiving device 102, a commentary audio production and distribution device 103, an application server 104, and a mobile terminal 105.

[0004] The broadcast transmitting device 101, the commentary audio production and distribution device 103, and the application server 104 are installed, for example, in a broadcast station, and the broadcast receiving device 102 is installed, for example, in the home of a viewer 100. The mobile terminal 105 is used by the viewer 100 who watches a broadcast program at home.

[0005] The audio commentary service of this system allows viewers 100 to receive audio commentary along with a broadcast program of audio and video in which an announcer gives commentary on the game situation and a commentator provides commentary. A broadcast transmitting device 101 transmits broadcast program content to a broadcast receiving device 102 via terrestrial digital broadcast waves. The broadcast receiving device 102 is, for example, a television receiver, and receives the broadcast program content transmitted from the broadcast transmitting device 101 via terrestrial digital broadcast waves and plays back the received broadcast program content.

[0006] The commentary audio production and distribution device 103 produces commentary audio for the broadcast program content being transmitted by the broadcast transmission device 101, and transmits the commentary audio to the mobile terminal 105. The application server 104 stores applications that run on the mobile terminal 105, and transmits the applications to the mobile terminal 105 in response to a request from the mobile terminal 105. The application is an abbreviation for application, and in this case, it is a program that receives and plays commentary audio.

[0007] The mobile terminal 105 is, for example, a smartphone or a PDA (Personal Digital Assistant), and plays back audio commentary of the broadcast program content in synchronization with the broadcast program content received by the broadcast receiving device 102. When playing back the audio commentary, the mobile terminal 105 changes the playback speed and the like in accordance with operations by the viewer 100.

[0008] For example, if the broadcast program is a baseball game, the viewer 100 can receive audio commentary that provides detailed information about the current game situation along with the video and audio of the baseball game, allowing them to understand the details of the game. The audio commentary for baseball may include, for example, information about the pitcher, the pitcher's movements, the type of pitch, the ball speed, the course of the ball, information about the batter, the batter's movements, the score, etc., according to the game situation.

[0009] An example of a commentary audio production and distribution device 103 that realizes such a commentary audio service is a system that receives data that conforms to the ODF (Olympic Data Feed) specifications, uses that data to produce and distribute commentary audio (see, for example, non-patent document 1).

[0010] The commentary audio production and distribution device 103 described in Non-Patent Document 1 sequentially receives data on the current state of the game, such as scores and fouls, from one information source that provides Olympic data.

[0011] The commentary audio production and distribution device 103 then generates commentary text according to the game situation by, for example, applying variables to a pre-set template, converts the text into audio using a voice synthesizer, and transmits the audio file of the commentary audio to the mobile terminal 105.

[0012] Fig. 19 is a schematic diagram illustrating an example of the overall configuration of a commentary audio distribution system including the commentary audio production and distribution device 103 (or multiple devices equivalent thereto) shown in Fig. 18. This commentary audio production and distribution system 200 is configured with a commentary audio production device 108, multiple information sources 201, a voice synthesizer 106, a distribution device 107, and a mobile terminal 105.

[0013] The commentary audio production and distribution system 200 corresponds to the commentary audio production and distribution device 103 and the mobile terminal 105 of the system that provides the commentary audio service shown in Fig. 18. The commentary audio production device 108, the voice synthesizer 106, and the distribution device 107 correspond to the commentary audio production and distribution device 103 shown in Fig. 18.

[0014] The commentary audio production device 108 is a device that generates commentary audio text for each utterance when producing commentary audio for a live sports program.

[0015] The commentary audio production device 108 receives real-time data corresponding to the game situation of a live sports program from multiple information sources 201. The commentary audio production device 108 then analyzes the data according to the data format unique to the information source from which the data was input, extracts text elements, assigns labels to the text elements, and stores them in an information management table.

[0016] Here, a text element is one or more elements that make up the commentary voice text to be generated (the text of the content to be spoken). A label is information for identifying the content of a text element.

[0017] In order to generate text for commentary audio of an utterance, the commentary audio production device 108 reads updated text elements from the information management table in accordance with the utterance definition data defined in the template, generates a JSON file including the playback time, and adds the utterance data to generate text for commentary audio.

[0018] The commentary audio production device 108 outputs the utterance data and the commentary audio text to the speech synthesis device 106 for each utterance, and also outputs the utterance data to the distribution device 107.

[0019] As shown in FIG. 19, the multiple baseball information sources 201 include, for example, an information source that distributes Olympic Games-related data in accordance with the ODF specification, an information source that distributes professional baseball-related data in accordance with the BIS specification, and the like.

[0020] Other baseball information sources 201 include information sources that automatically detect the game situation from game footage using image recognition (see, for example, Patent Application No. 2023-70197 (hereinafter referred to as Unpublished Patent Application No. 2) and Patent Application No. 2023-76378 (hereinafter referred to as Unpublished Patent Application No. 3) made by the same applicant as the present patent application, which were unpublished at the time of filing the present patent application).

[0021] The speech synthesizer 106 receives the speech data and the commentary speech text from the commentary speech production device 108, and generates a voice file by generating synthetic speech from the commentary speech text using existing technology. The speech synthesizer 106 then outputs the speech data and the voice file to the distribution device 107.

[0022] The distribution device 107 receives the speech data from the commentary audio production device 108 and also receives the speech data and the audio file from the audio synthesis device 106 , and distributes the audio file of the same speech data to the mobile terminal 105 .

[0023] Furthermore, much research has been conducted on motion recognition and classification technologies. For example, a method has been disclosed in which a machine-learned model is used to detect a person's skeleton from a video and estimate the person's motion (action) (see, for example, Patent Document 3).

[0024] When the method of Patent Document 3 is applied to broadcast content such as baseball, in which people often appear in the frame, it is possible to estimate the actions of each person (sitting, standing, running, raising the right hand, etc.), but it is difficult to determine what kind of scene (action scene) the situation is (taking a stance, throwing, hitting, etc.).

[0025] In contrast to this, the method of the aforementioned unpublished Patent Document 2 makes it possible to determine the action scene in real time by taking into account the correlation and positional relationship between the people in the image. [Prior art documents] [Patent documents]

[0026] [Patent Document 1] Japanese Patent Application Publication No. 2017-203827 [Patent Document 2] Japanese Patent Publication No. 2023-170822 [Patent Document 3] Japanese Patent Publication No. 2022-189456 [Non-patent literature]

[0027] [Non-Patent Document 1] Tadashi Kumano, "Technology for generating audio guides for sports programs," NHK STRL R&D, No. 154, pp. 12-20, 2017 Summary of the Invention [Problem to be solved by the invention]

[0028] The techniques of Patent Document 2 and Unpublished Patent Document 1 are technologies that use data from multiple information sources to provide versatile commentary audio in real time. On the other hand, when data that is not distributed in real time from the information source 201 (for example, when the pitcher takes his stance or throws the ball) is generated as speech data, manual operation by an operator is required.

[0029] Therefore, when there are many manual operations by the operator, the operations may not be completed in time for the generation of speech data, resulting in problems such as a lack of real-timeness, incorrect input of information, and the operator's labor.

[0030] Furthermore, the method of the aforementioned unpublished Patent Document 2 automatically determines action scenes by detecting the skeleton of a person and learning the correlation between the actions of the person in the image and the positional relationship of the skeleton coordinates.

[0031] However, when this method is applied to determining information (swing information) about a baseball batter's swing ("batting" or "swinging and missing"), there is a problem in that the accuracy of the determination is low. This is because the catcher and umpire are located near the batter's area, and there is a high possibility that multiple people will overlap, making it difficult to detect the skeleton of the batter alone, and the accuracy of skeleton detection is low.

[0032] For this reason, there has been a demand for a method to determine swing information without detecting a person's skeleton. By determining swing information with high accuracy and distributing it as audio commentary, the quality of the audio commentary service can be improved.

[0033] Here, swing information refers to information indicating whether the batter "swinged" or "did not swing," and, if the batter "swinged," information indicating whether the swing was a "batting" (hit) or a "swing and a miss."

[0034] Therefore, the present invention has been made to solve the above-mentioned problems, and its purpose is to provide a swing information determination device and program that can accurately determine a batter's swing information from video in a service that generates and distributes commentary text for baseball broadcast programs. [Means for solving the problem]

[0035] In order to solve the above problem, the swing information determination device of claim 1 is a swing information determination device that inputs video of a baseball program and determines swing information including a batter's swing state from the video, the swing information determination device comprising: pitch detection means that uses a predetermined classifier to detect a pitch by the pitcher for each of a plurality of images that make up the video and generates this timing as pitch detection time T1; swing detection means that uses a predetermined swing learning model to detect the batter's swing for each of the plurality of images that make up the video between pitch detection time T1 generated by the pitch detection means and a predetermined threshold Z time, and generates this timing as swing detection time T2; The present invention is characterized by comprising a screen change detection means that, after the swing detection time T2 generated by the swing detection means, uses a predetermined screen change learning model to detect a change of screen from a confrontation scene between the pitcher and the batter to a scene other than the confrontation scene, and generates the timing as a screen change detection time T3, and a swing information determination means that determines the swing information that determines whether the swing state is a batting or a strikeout based on the swing detection time T2 generated by the swing detection means, the screen change detection time T3 generated by the screen change detection means, and the time of a predetermined threshold value A from the swing detection time T2.

[0036] The swing information determination device of claim 2 receives video of a baseball program and generates swing information including a batter's swing state from the video, and includes pitch detection means for detecting a pitch by the pitcher using a predetermined classifier for each of a plurality of images constituting the video and generating the timing as pitch detection time T1, swing detection means for detecting the batter's swing using a predetermined swing learning model for each of a plurality of images constituting the video between pitch detection time T1 generated by the pitch detection means and a predetermined threshold Z, and generating the timing as swing detection time T2, and The apparatus is characterized by comprising: a BSO caption detection means that detects, using a predetermined determiner, that a BSO caption displaying a count of balls (B), strikes (S), and outs (O) has been erased from the screen after the swing detection time T2 generated by the swing detection means, and generates this timing as a BSO caption erasure detection time T4; and a swing information determination means that determines the swing information that determines whether the swing state is a bat or a miss, based on the swing detection time T2 generated by the swing detection means, the BSO caption erasure detection time T4 generated by the BSO caption detection means, and a predetermined threshold value B from the swing detection time T2.

[0037] The swing information determination device of claim 3 receives video of a baseball program and determines swing information including a batter's swing state from the video, and includes pitch detection means for detecting a pitch by the pitcher using a predetermined classifier for each of a plurality of images constituting the video and generating this timing as pitch detection time T1, swing detection means for detecting a swing by the batter using a predetermined swing learning model for each of a plurality of images constituting the video between pitch detection time T1 generated by the pitch detection means and a time of a predetermined threshold Z and generating this timing as swing detection time T2, and swing detection means for detecting a screen change from a confrontation scene between the pitcher and the batter to a scene other than the confrontation scene using a predetermined screen change learning model for each of a plurality of images constituting the video after swing detection time T2 generated by the swing detection means, the timing of which is determined based on the swing detection time T2 generated by the swing detection means, the screen change detection time T3 generated by the swing detection means, the BSO caption counting balls (B), strikes (S), and outs (O) being erased from the screen, and the timing of which is determined based on the swing detection time T2 generated by the swing detection means, the screen change detection time T3 generated by the screen change detection means, the BSO caption counting time T4 generated by the BSO caption detection means, and a predetermined threshold value A and a predetermined threshold value B from the swing detection time T2.

[0038] The swing information determination device of claim 4 is the swing information determination device of claim 1, wherein the swing information determination means counts the elapsed time t from the swing detection time T2, and if the screen change detection time T3 is generated when the elapsed time t is before the predetermined threshold A, generates the swing information that determines the swing state as a batting, and if the screen change detection time T3 has not been generated when the elapsed time t reaches the predetermined threshold A, generates the swing information that determines the swing state as a strikeout.

[0039] The swing information determination device of claim 5 is the swing information determination device of claim 2, wherein the swing information determination means counts the elapsed time t from the swing detection time T2, and if the BSO caption removal detection time T4 is generated when the elapsed time t is before the time of the predetermined threshold B, generates the swing information that determines the swing state as a batting, and if the BSO caption removal detection time T4 has not been generated when the elapsed time t reaches the time of the predetermined threshold B, generates the swing information that determines the swing state as a strikeout.

[0040] Furthermore, a program according to a sixth aspect of the present invention is characterized in that it causes a computer to function as the swing information determination device according to any one of the first to fifth aspects of the present invention. [Effects of the Invention]

[0041] As described above, according to the present invention, in a service that generates and distributes commentary text for live baseball broadcast programs, it is possible to accurately determine batter's swing information from video. [Brief explanation of the drawings]

[0042] [Figure 1] 1 is a block diagram showing an example of the configuration of a swing information determination device according to an embodiment of the present invention. [Figure 2]10 is a flowchart illustrating an example of processing performed by the swing information determining device according to an embodiment of the present invention. [Figure 3] 10A and 10B are diagrams illustrating an example of processing by a pitching detection means. [Figure 4] 10A and 10B are diagrams illustrating an example of processing by a swing detection means. [Figure 5] 10A and 10B are diagrams illustrating an example of processing by a screen change detection unit. [Figure 6] FIG. 10 is a diagram illustrating an example of processing by a BSO telop detection means. [Figure 7] 10A and 10B are diagrams illustrating an example of processing by a swing information determining means. [Figure 8] 10 is a flowchart showing an example of processing (step S701) of a rule-based determiner of the swing information determining means. [Figure 9] FIG. 10 is a diagram illustrating a first setting example of the swing state label “batting.” [Figure 10] FIG. 10 is a diagram illustrating a second setting example of the swing state label “batting.” [Figure 11] FIG. 10 is a diagram illustrating an example of setting a swing state label "miss." [Figure 12] FIG. 1 is a block diagram showing an example of the configuration of a swing information learning device. [Figure 13] 10 is a flowchart showing an example of processing by a swing learning means. [Figure 14] 10A and 10B are diagrams illustrating an example of processing by a swing learning means. [Figure 15] 10 is a flowchart showing an example of processing by a screen switching learning means. [Figure 16] 10A and 10B are diagrams illustrating an example of processing by a screen switching learning means. [Figure 17] FIG. 10 is a block diagram showing an example of the configuration of a swing information determination device according to another embodiment of the present invention. [Figure 18] FIG. 1 is a diagram illustrating an overview of a system that provides a commentary audio service. [Figure 19] 1 is a schematic diagram illustrating an example of the overall configuration of a commentary audio distribution system including a commentary audio production and distribution device. DETAILED DESCRIPTION OF THE INVENTION

[0043] The present invention is characterized in that, when a pitch by a pitcher is detected from video of a live baseball broadcast and a subsequent swing by a batter is detected, the batter's swing information is determined by determining whether or not a screen change from a confrontation scene to another scene has been detected and / or whether or not the BSO caption has been deleted.

[0044] This allows swing information to be determined without detecting a person's skeleton, making it possible to accurately determine a batter's swing information from video in a service that generates and distributes commentary text for baseball broadcast programs.

[0045] [Swing Information Determining Device] First, a swing information determination device according to an embodiment of the present invention will be described. Fig. 1 is a block diagram showing an example of the configuration of a swing information determination device according to an embodiment of the present invention, and Fig. 2 is a flowchart showing an example of the processing performed by the device.

[0046] This swing information determination device 1 includes a control unit 10 and a memory unit 20. Control unit 10 includes pitch detection means 11, swing detection means 12, screen change detection means 13, BSO telop detection means 14, and swing information determination means 15. Memory unit 20 includes pitch storage means 21, swing storage means 22, and screen change storage means 23.

[0047] For example, swing information determination device 1 receives image-by-image video of a live broadcast baseball game (step S201). The video received by swing information determination device 1 is unknown data, consisting of multiple images (frames), and is received in chronological order, image by image.

[0048] (Pitching detection means 11) Pitch detection means 11 receives the video, reads out (the parameters of) a classifier from pitch storage means 21, and uses the classifier (a classifier that functions according to the read-out parameters) to detect a pitch from the video that indicates that the pitcher has thrown the ball, and generates the timing as pitch detection time T1 (step S202). Pitch detection means 11 then outputs pitch detection time T1 to swing detection means 12.

[0049] 3 is a diagram illustrating an example of processing by the pitching detection means 11. The pitching detection means 11 classifies the action scenes for each image in the input video using a classifier, and generates pitching detection output information indicating the action scene "thrown" or "other" (step S301). As a result, pitching detection output information "thrown" or "other" is generated for each image.

[0050] Specifically, the pitching detection means 11 generates skeletal coordinate information for each person from the video, and generates a predicted score for each action from the skeletal coordinate information for each person.The pitching detection means 11 then uses a pre-trained action scene determination model to determine pitching detection output information, which is an action scene, from the skeletal coordinate information for each person and the predicted score for each action.For details, please refer to the above-mentioned unpublished patent document 2.

[0051] When determining whether the pitch detection output information is "thrown" or "other," it is sufficient to determine the pitcher's movements, and the area where the pitcher is located is unlikely to be overlapped by multiple people. Therefore, unlike the batter, where multiple people are likely to overlap, highly accurate skeletal coordinate information can be generated for the pitcher. As a result, highly accurate pitch detection output information is generated, either "thrown" or "other."

[0052] After the process of step S301, the pitching detection means 11 uses the generated pitching detection output information to detect the final pitching detection output information as "thrown" or "other" in accordance with a preset rule.

[0053] When the finally detected pitching detection output information changes from "other" to "thrown," the pitching detection means 11 sets a pitching detection time T1 indicating the timing using an internal clock (step S302).

[0054] As a preset rule, for example, when the pitching detection output information "thrown" is detected for the first time, it is determined that the pitching detection output information "thrown" is the final detection.

[0055] Furthermore, as a preset rule, for example, when the pitching detection output information "thrown" is detected consecutively for n frames (n is an integer equal to or greater than 1) of images, this is the final detection of the pitching detection output information "thrown." In this way, when the pitching detection output information "thrown" is finally detected, the pitching detection time T1 is set.

[0056] (Swing detection means 12) 1 and 2, swing detection means 12 inputs the video and also inputs pitch detection time T1 indicating the timing at which the pitcher's pitch was detected from pitch detection means 11, and reads out (parameters of) swing learning model M1 from swing storage means 22. Swing learning model M1 is a model learned by swing information learning device 2 shown in FIG. 12, which will be described later.

[0057] The swing detection means 12 uses the swing learning model M1 (the swing learning model M1 in which the read parameters are set) to detect a swing of the batter, which indicates that the batter has swung the bat, for the time period from the pitch detection time T1 to the time of the predetermined threshold value Z, and generates that timing as a swing detection time T2 (step S203).The swing detection means 12 then outputs the swing detection time T2 to the screen change detection means 13, the BSO telop detection means 14, and the swing information determination means 15.

[0058] The batter's swing is detected during the time period from pitch detection time T1 to the time of the predetermined threshold value Z because it is predicted in advance that a swing will occur immediately after pitch detection time T1.

[0059] It should be noted that for any time period other than the period from pitch detection time T1 to the time of predetermined threshold value Z, the batter's swing is not detected and swing detection time T2 is not generated.

[0060] 4 is a diagram illustrating an example of processing by the swing detection means 12. The swing detection means 12 determines the batter's scene for each image of the input video using the swing learning model M1 for the time period from the pitch detection time T1 to the time of a predetermined threshold Z, and generates swing detection output information indicating "swing (right)," "swing (left)," or "other" (step S401). As a result, the swing detection output information "swing (right)," "swing (left)," or "other" is generated for each image.

[0061] "Swing (Right)" is a scene in which a batter in the right batter's box takes a swing, and "Swing (Left)" is a scene in which a batter in the left batter's box takes a swing. "Other" is all scenes of batters other than "Swing (Right)" and "Swing (Left)."

[0062] The swing detection means 12 uses the generated swing detection output information to detect the final swing detection output information "swing (right)", "swing (left)" or "other" in accordance with a preset rule.

[0063] When the finally detected swing detection output information changes from "other" to "swing (right)" or "swing (left)", the swing detection means 12 sets a swing detection time T2 indicating the timing using an internal clock (step S402).

[0064] As a preset rule, for example, if the swing detection output information "swing (right)" or "swing (left)" is detected for the first time during the time period from the pitch detection time T1 to the time of a predetermined threshold value Z, it is determined that the final swing detection output information is "swing (right)" or "swing (left)".

[0065] Furthermore, as a preset rule, for example, when the swing detection output information "swing (right)" or "swing (left)" is detected consecutively for n frames (n is an integer equal to or greater than 1) of images, the final swing detection output information is determined to be "swing (right)" or "swing (left)." In this way, when the swing detection output information "swing (right)" or "swing (left)" is finally detected, the swing detection time T2 is set.

[0066] (Screen change detection means 13) 1 and 2, the screen change detection means 13 inputs the video and also inputs the swing detection time T2 indicating the timing at which the batter's swing is detected from the swing detection means 12, and reads out the screen change learning model M2 (parameters thereof) from the screen change storage means 23. The screen change learning model M2 is a model learned by the swing information learning device 2 shown in FIG. 12, which will be described later.

[0067] Using the screen change learning model M2 (the screen change learning model M2 to which the read parameters are set), the screen change detection means 13 detects a screen change indicating that the screen has changed from "contest scene" to "other" after the timing of the swing detection time T2, and generates this timing as the screen change detection time T3 (step S204). Then, the screen change detection means 13 outputs the screen change detection time T3 to the swing information determination means 15.

[0068] A "confrontation scene" is a scene where a pitcher and a batter fight, and when the pitcher throws the ball and the batter swings the bat, the batter either hits the ball or swings and misses. "Other" is all scenes other than "confrontation scenes."

[0069] The reason why a screen change indicating that the screen has changed from "confrontation scene" to "other" is detected after the timing of swing detection time T2 is that if the batter's swing is "batting," the screen is predicted to change immediately (for example, to a screen of the infield or a scene of the outfield), and if the batter's swing is "striking and missing," the screen is predicted to change after a short time has passed. In other words, whether the batter's swing is "batting" or "striking and missing" is determined by swing information determination means 15 at a subsequent stage according to the timing of screen change detection time T3 after swing detection time T2.

[0070] It is assumed that the screen has already displayed the "confrontation scene" at the timing of swing detection time T2.

[0071] 5 is a diagram illustrating an example of processing by the screen change detection means 13. The screen change detection means 13 determines the scene of the screen for each image of the input video using the screen change learning model M2, and generates confrontation scene detection output information indicating a "confrontation scene" or "other" (step S501). As a result, confrontation scene detection output information "confrontation scene" or "other" is generated for each image.

[0072] The screen change detection means 13 uses the generated confrontation scene detection output information to detect the final confrontation scene detection output information "confrontation scene" or "other" in accordance with a preset rule.

[0073] When the finally detected confrontation scene detection output information changes from "confrontation scene" to "other", the screen change detection means 13 sets a screen change detection time T3 indicating the timing using an internal clock (step S502).

[0074] As a preset rule, for example, if the confrontation scene detection output information "Other" is detected for the first time (when it changes from "Confrontation Scene" to "Other") at a timing after the swing detection time T2, it is determined that the final confrontation scene detection output information "Other" has been detected.

[0075] Furthermore, as a preset rule, for example, when the confrontation scene detection output information "other" is detected consecutively for n frames (n is an integer equal to or greater than 1) of images, the final confrontation scene detection output information is "other." In this way, when the confrontation scene detection output information "other" is finally detected, the screen change detection time T3 is set.

[0076] (BSO telop detection means 14) Returning to FIGS. 1 and 2, the BSO telop detection means 14 receives the video as an input, and also receives from the swing detection means 12 the swing detection time T2 indicating the timing at which the batter's swing is detected.

[0077] The BSO telop detection means 14 uses a rule-based decision device to detect that the BSO telop has been erased from the screen after the timing of the swing detection time T2 based on the brightness information of the image, and generates this timing as a BSO telop erasure detection time T4 (step S205).The BSO telop detection means 14 then outputs the BSO telop erasure detection time T4 to the swing information determination means 15. The BSO telop is a telop that displays the count of ball (B), strike (S), and out (O).

[0078] The reason why the removal of the BSO telop from the screen is detected after swing detection time T2 is that if the batter's swing is a "batting," the BSO telop is predicted to be removed immediately, and if the batter's swing is a "swing-and-miss," the BSO telop is predicted to be removed after a short time has passed. In other words, whether the batter's swing is a "batting" or a "swing-and-miss" is determined by swing information determination means 15 at a subsequent stage according to the timing of BSO telop removal detection time T4 after swing detection time T2.

[0079] It is assumed that the BSO telop is already displayed on the screen at the timing of the swing detection time T2.

[0080] 6 is a diagram illustrating an example of processing by the BSO telop detection means 14. The BSO telop detection means 14 focuses on a plurality of specific areas (areas with high and low brightness) within a predetermined region within the image (a pre-set region where BSO telops are displayed) for each image of the input video, and calculates the average brightness value for each pixel in each area as brightness information.

[0081] The BSO telop detection means 14 uses a rule-based determiner to compare the brightness information with a preset threshold, and based on the comparison result, determines whether or not a BSO telop is being displayed on the screen, and generates BSO telop detection output information of "BSO telop present" or "BSO telop not present" (step S601). For example, if the brightness information in a high-brightness area is equal to or greater than the threshold, and if the brightness information in a low-brightness area is equal to or less than the threshold, the BSO telop detection output information of "BSO telop present" is generated. In this way, the BSO telop detection output information of "BSO telop present" or "BSO telop not present" is generated for each image.

[0082] "With BSO caption" indicates that a BSO caption is displayed on the screen, and "without BSO caption" indicates that a BSO caption is not displayed on the screen.

[0083] The BSO telop detection means 14 uses the generated BSO telop detection output information to detect the final BSO telop detection output information "BSO telop present" or "BSO telop absent" in accordance with a preset rule.

[0084] When the finally detected BSO telop detection output information changes from "BSO telop present" to "BSO telop absent," the BSO telop detection means 14 sets a BSO telop deletion detection time T4 indicating the timing using an internal clock (step S602).

[0085] As a pre-set rule, for example, if the BSO caption detection output information "No BSO caption" is detected for the first time at a timing after the swing detection time T2 (when it changes from "BSO caption present" to "No BSO caption"), the final BSO caption detection output information "No BSO caption" is detected.

[0086] Furthermore, as a preset rule, for example, when the BSO telop detection output information "no BSO telop" is detected consecutively for n frames (n is an integer equal to or greater than 1) of images, the final BSO telop detection output information is determined to be "no BSO telop." In this way, when the BSO telop detection output information "no BSO telop" is finally detected, a BSO telop erasure detection time T4 is set.

[0087] (Swing information determination means 15) 1 and 2, swing information determination means 15 receives swing detection time T2, which indicates the timing at which the batter's swing is detected, from swing detection means 12. Swing information determination means 15 also receives screen change detection time T3, which indicates the timing at which the screen changed from "confrontation scene" to "other" after swing detection time T2, from screen change detection means 13. Swing information determination means 15 also receives BSO telop removal detection time T4, which indicates the timing at which the BSO telop was removed from the screen after swing detection time T2.

[0088] The swing information determination means 15 uses a rule-based determiner to determine swing information based on the swing detection time T2, the screen change detection time T3, the BSO caption erasure detection time T4, and preset thresholds A and B (step S206), and outputs the determined swing information (step S207).

[0089] Specifically, the swing information determination means 15 counts the elapsed time t from the timing of the swing detection time T2, and determines the swing information depending on whether the elapsed time t reaches the screen change detection time T3 before the time of threshold A, or whether the elapsed time t reaches the BSO caption erasure detection time T4 before the time of threshold B.

[0090] The timing at which the swing detection time T2 is input indicates the timing at which the batter takes a swing, and the swing information determining means 15 determines whether the swing is a "batting" or a "swing and miss."

[0091] In a normal state where the batter is not swinging, swing information determination means 15 outputs swing information indicating "not swinging." Then, swing information determination means 15 inputs swing detection time T2 and determines swing information, thereby outputting swing information indicating "swinging" at the timing of that determination and indicating whether the swing was a "batting" or a "swing-and-miss."

[0092] The control unit 10 determines whether the processing of the swing information determination device 1 has been completed (step S208), and if it determines that the processing has not been completed (step S208: N), the control unit 10 proceeds to step S201 and performs the processing of steps S201 to S207.

[0093] On the other hand, if the control unit 10 determines in step S208 that the processing is completed (step S208: Y), the control unit 10 ends the processing.

[0094] 7 is a diagram illustrating an example of processing by swing information determination means 15. Using a rule-based determiner, swing information determination means 15 determines a swing state based on swing detection time T2, screen change detection time T3, BSO caption erasure detection time T4, and preset thresholds A and B, and generates a swing state label indicating "batting" or "swinging and missing" (step S701).

[0095] The swing information determination means 15 generates swing information indicating that "a swing was made" and including a swing state label (a label indicating whether the swing was a "batting" or a "swing and miss") (step S702), and outputs this.

[0096] 8 is a flowchart showing an example of processing by the rule-based determiner of swing information determining means 15, and shows an example of processing in step S701 of FIG. 7 when threshold value A is equal to or less than threshold value B (A≦B). When the rule-based determiner receives swing detection time T2 (step S801), it counts the elapsed time t from that timing (step S802). In other words, the elapsed time t from when the batter's swing was detected is counted.

[0097] The rule-based determinator determines whether the elapsed time t is before the time of threshold value A (whether the time of threshold value A has not elapsed since the batter's swing was detected) (step S803). If the rule-based determinator determines in step S803 that the elapsed time t is before the time of threshold value A (step S803: Y), it proceeds to step S804, and if it determines that the elapsed time t is not before the time of threshold value A (step S803: N), it proceeds to step S805.

[0098] The rule-based determinator proceeds from step S803 (Y) and determines whether or not the screen change detection time T3 has been input at the elapsed time t (whether or not the screen has changed from "confrontation scene" to "other" at the elapsed time t) (step S804). If the rule-based determinator determines in step S804 that the screen change detection time T3 has been input at the timing of the elapsed time t (step S804: Y), the rule-based determinator proceeds to step S807. On the other hand, if the rule-based determinator determines in step S804 that the screen change detection time T3 has not been input at the timing of the elapsed time t (step S804: N), the rule-based determinator proceeds to step S803.

[0099] The rule-based determinator proceeds from step S803(N) to determine whether the elapsed time t is before the time of threshold B (whether the time of threshold B has not elapsed since the batter's swing was detected) (step S805). If the rule-based determinator determines in step S805 that the elapsed time t is before the time of threshold B (step S805: Y), it proceeds to step S806, and if it determines that the elapsed time t is not before the time of threshold B (step S805: N), it proceeds to step S808.

[0100] The rule-based determinator proceeds from step S805 (Y) to determine whether or not the BSO telop removal detection time T4 has been input at the elapsed time t (whether or not the BSO telop has been removed from the screen at the elapsed time t) (step S806). If the rule-based determinator determines in step S806 that the BSO telop removal detection time T4 has been input at the timing of the elapsed time t (step S806: Y), the rule-based determinator proceeds to step S807. On the other hand, if the rule-based determinator determines in step S806 that the BSO telop removal detection time T4 has not been input at the timing of the elapsed time t (step S806: N), the rule-based determinator proceeds to step S803.

[0101] The rule-based determiner proceeds from step S804(Y) or step S806(Y) and sets the swing state label "batting" (step S807).

[0102] FIG. 9 is a diagram for explaining a first setting example of the swing state label "batting," and shows the case where the process moves from step S804(Y) to step S807 in FIG.

[0103] Pitch detection time T1 is the timing when pitch detection output information "thrown" is generated and the pitcher's pitch is detected, and swing detection time T2 is the timing when swing detection output information "swing (right)" is generated and the batter's swing is detected. The same applies to Figures 10 and 11 described below.

[0104] The screen change detection time T3 is the timing when the competition scene detection output information "other" is generated and the screen changes from "competition scene" to "other." The swing information determination means 15 inputs the screen change detection time T3 when the elapsed time t is reached. The elapsed time t is the time T obtained by subtracting the swing detection time T2 from the screen change detection time T3. a and is assumed to be before the time of threshold A (the time of threshold A has not passed).

[0105] In this case, the process moves from step S803(Y) and step S804(Y) shown in FIG. 8, and in step S807, the swing state label "batting" is set.

[0106] FIG. 10 is a diagram for explaining a second setting example of the swing state label "batting," and shows the case where the process moves from step S806(Y) to step S807 in FIG.

[0107] The BSO telop erasure detection time T4 is the timing when the BSO telop detection output information "No BSO telop" is generated and the BSO telop is erased from the screen. The swing information determination means 15 receives the BSO telop erasure detection time T4 at the time of the elapsed time t. The elapsed time t is the time T obtained by subtracting the swing detection time T2 from the BSO telop erasure detection time T4. b and is before the time of threshold B (the time of threshold B has not passed).

[0108] In this case, the process moves from step S805(Y) and step S806(Y) shown in FIG. 8, and in step S807, the swing state label "batting" is set.

[0109] 8, the rule-based determinator proceeds from step S805(N) and sets the swing state label "miss" (step S808). Then, the rule-based determinator proceeds from step S807 or step S808 and outputs the swing state label (step S809).

[0110] FIG. 11 is a diagram for explaining an example of setting the swing state label "whisk", and shows the case where the process moves from step S805(N) to step S808 in FIG.

[0111] The elapsed time t corresponds to the time of threshold B (>threshold A), and at this timing, it is assumed that there is no input of the screen change detection time T3 and the BSO telop removal detection time T4.

[0112] In this case, the process moves from step S803(N) and step S805(N) shown in FIG. 8, and in step S808, the swing state label "miss" is set.

[0113] In FIG. 8, if threshold A is greater than threshold B (A>B), the rule-based decision unit performs processing in the case where step S803 and step S805 are interchanged and step S804 and step S806 are interchanged.

[0114] (Storage unit 20) Returning to Fig. 1, pitch storage means 21 stores a classifier (or classifier parameters), swing storage means 22 stores a swing learning model M1 (or parameters of swing learning model M1), and screen switching storage means 23 stores a screen switching learning model M2 (or parameters of screen switching learning model M2).

[0115] The swing learning model M1 and the screen switching learning model M2 are models used in the image recognition field, and are configured using, for example, a CNN (convolutional neural network) and a Vision Transformer.

[0116] Through the learning process of the swing information learning device 2 shown in Figure 12 described below, a swing learning model M1 and a screen switching learning model M2 are generated, and the swing learning model M1 is stored in the swing memory means 22, and the screen switching learning model M2 is stored in the screen switching memory means 23.

[0117] As described above, according to the swing information determination device 1 of the embodiment of the present invention, pitch detection means 11 receives video of a baseball program, detects the pitcher's pitch from the video using a classifier, and generates pitch detection time T1. Swing detection means 12 uses swing learning model M1 to detect the batter's swing from pitch detection time T1 to the time of threshold value Z, and generates swing detection time T2.

[0118] The screen change detection means 13 uses the screen change learning model M2 to detect a screen change indicating that the screen has changed from "confrontation scene" to "others" after the timing of the swing detection time T2, and generates a screen change detection time T3.

[0119] The BSO telop detection means 14 uses a rule-based decision device to detect that the BSO telop has been erased after the timing of the swing detection time T2, and generates a BSO telop erase detection time T4.

[0120] The swing information determination means 15 uses a rule-based determiner to determine the swing information based on the swing detection time T2, the screen change detection time T3, the BSO telop erasure detection time T4, and the threshold values ​​A and B set in advance.

[0121] Specifically, the swing information determination means 15 counts the elapsed time t from the timing of the swing detection time T2, and if the screen change detection time T3 is input when the elapsed time t is before the time of threshold A, or if the BSO caption erasure detection time T4 is input when the elapsed time t is before the time of threshold B, it determines that the swing information includes "batting."

[0122] Furthermore, if the elapsed time t reaches the larger of the threshold values ​​A and B and both the screen change detection time T3 and the BSO caption erasure detection time T4 have not been input, the swing information determination means 15 determines that the swing information includes a "miss."

[0123] In this way, pitches and swings are detected from footage of baseball programs using AI, etc., and whether the swing is a "batting strike" or a "swing and miss" is determined by detecting whether the screen changes after the swing is detected or whether the BSO caption disappears.

[0124] This allows swing information to be determined without detecting a person's skeleton, making it possible to accurately determine a batter's swing information from video in a service that generates and distributes commentary text for baseball broadcast programs.

[0125] This reduces the workload of operators creating commentary text and prevents input errors, improving the quality of service and enabling many viewers, including not only the visually impaired but also able-bodied viewers who watch baseball broadcasts while doing other things, to enjoy baseball broadcasts comfortably.

[0126] [Swing Information Learning Device] Next, a swing information learning device will be described with reference to Fig. 12. Fig. 12 is a block diagram showing an example of the configuration of the swing information learning device.

[0127] This swing information learning device 2 includes a control unit 30 and a storage unit 40. The control unit 30 includes a swing learning means 31 and a screen change learning means 32, and the storage unit 40 includes a swing storage means 41 and a screen change storage means 42.

[0128] The swing information learning device 2 inputs known footage from a baseball broadcast program on an image-by-image basis as training data, and also inputs a label indicating whether the action scene corresponding to the image is a "swing (right)," "swing (left)," or "other," and a label indicating whether the action scene corresponding to the image is a "confrontation scene" or "other."

[0129] The video input to swing information learning device 2 is known data, consisting of multiple images, and is input in chronological order, image by image. Each image is assigned two types of labels. Specifically, it is labeled with information such as "swing (right)," "swing (left)," or "other," and is also labeled with information such as "confrontation scene" or "other."

[0130] The swing learning means 31 uses images and labels from a baseball broadcast program to learn a swing learning model M1 for generating swing detection output information indicating "swing (right)", "swing (left)", or "other".

[0131] 13 is a flowchart showing an example of processing by swing learning means 31. Swing learning means 31 inputs known footage of a baseball broadcast program on an image-by-image basis as training data, and also inputs a label indicating whether the action scene corresponding to the image is "swing (right)," "swing (left)," or "other" (step S1301).

[0132] The swing learning means 31 uses this teacher data to learn the swing learning model M1 (step S1302). Then, the swing learning means 31 determines whether or not the learning process of the swing learning means 31 is complete (step S1303). Whether or not the learning process is complete is determined, for example, by the number of times the learning process has been performed and whether or not the errors of the parameters used in the swing learning model M1 have converged.

[0133] If it is determined in step S1303 that the learning process is not complete (step S1303: N), the swing learning means 31 proceeds to step S1301 and performs the processes of steps S1301 and S1302.

[0134] On the other hand, if it is determined in step S1303 that the learning process is completed (step S1303: Y), the swing learning means 31 stores the swing learning model M1 (parameters thereof) in the swing storage means 41 (step S1304), and ends the learning process.

[0135] 14 is a diagram illustrating an example of processing by swing learning means 31. Swing learning means 31 uses swing learning model M1 to generate swing detection output information indicating "swing (right)," "swing (left)," or "other" for each image of the input video (step S1401).

[0136] The swing learning means 31 updates the parameters of the swing learning model M1 so that the input label indicating "swing (right)", "swing (left)" or "other" matches the generated swing detection output information indicating "swing (right)", "swing (left)" or "other" other than these (step S1402).

[0137] For example, back propagation is used as the learning process by the swing learning means 31. The learned swing learning model M1 (parameters thereof) is stored in the swing storage means 41.

[0138] Returning to FIG. 12, the image transition learning means 32 uses the video and labels of a baseball broadcast program to learn an image transition learning model M2 for generating confrontation scene detection output information indicating "confrontation scene" or "other."

[0139] 15 is a flowchart showing an example of processing by the screen transition learning means 32. The screen transition learning means 32 inputs, as training data, known images of a baseball broadcast program on an image-by-image basis, and also inputs a label indicating that the action scene corresponding to the image is a "confrontation scene" or "other" (step S1501).

[0140] The screen transition learning means 32 uses this teacher data to learn the screen transition learning model M2 (step S1502). Then, the screen transition learning means 32 determines whether the learning process of the screen transition learning means 32 is complete (step S1503). Whether the learning process is complete is determined, similarly to the swing learning means 31, by, for example, the number of times the learning process has been performed and whether the errors of the parameters used in the screen transition learning model M2 have converged.

[0141] If it is determined in step S1503 that the learning process is not complete (step S1503: N), the screen switching learning means 32 proceeds to step S1501 and performs the processes of steps S1501 and S1502.

[0142] On the other hand, if the screen switching learning means 32 determines in step S1503 that the learning process is completed (step S1503: Y), it stores the screen switching learning model M2 (parameters thereof) in the screen switching storage means 42 (step S1504) and terminates the learning process.

[0143] 16 is a diagram illustrating an example of processing by the screen transition learning means 32. The screen transition learning means 32 generates confrontation scene detection output information indicating "confrontation scene" or "other" for each image of the input video using the screen transition learning model M2 (step S1601).

[0144] The screen transition learning means 32 updates the parameters of the screen transition learning model M2 so that the input label indicating "confrontation scene" or "other" matches the generated confrontation scene detection output information indicating "confrontation scene" or "other" (step S1602).

[0145] The learning process by the screen switching learning means 32 uses, for example, back propagation, similarly to the swing learning means 31. The learned screen switching learning model M2 (parameters thereof) is stored in the screen switching storage means 42.

[0146] As described above, according to the swing information learning device 2, the swing learning means 31 inputs known footage and labels of a baseball broadcast program, which are training data, and learns a swing learning model M1 for generating swing detection output information indicating "swing (right)," "swing (left)," or "other."

[0147] In addition, the screen transition learning means 32 inputs known images and labels of baseball broadcast programs, which are training data, and learns a screen transition learning model M2 for generating confrontation scene detection output information indicating "confrontation scene" or "other."

[0148] The swing learning model M1 and screen switching learning model M2 learned in this manner are then used by the swing information determination device 1 shown in FIG. 1, which determines swing information from footage of a baseball broadcast program.

[0149] This allows swing information to be determined without detecting a person's skeleton, making it possible to accurately determine a batter's swing information from video in a service that generates and distributes commentary text for baseball broadcast programs.

[0150] [Other swing information determination devices] Next, a swing information determination device according to another embodiment of the present invention will be described below. Fig. 17 is a block diagram showing an example of the configuration of a swing information determination device according to another embodiment of the present invention.

[0151] This swing information determination device 3 is a device that adds control unit 30 of swing information learning device 2 shown in Fig. 12 to swing information determination device 1 shown in Fig. 1. Swing information determination device 3 inputs known video and teacher data that is a label to learn swing learning model M1 and screen switching learning model M2, and determines swing information for unknown video using the learned swing learning model M1 and screen switching learning model M2.

[0152] This swing information determination device 3 includes a control unit 50 and a memory unit 20. Control unit 50 includes pitch detection means 11, swing detection means 12, screen change detection means 13, BSO telop detection means 14, swing information determination means 15, swing learning means 31, and screen change learning means 32. Memory unit 20 includes pitch storage means 21, swing storage means 22, and screen change storage means 23.

[0153] The pitch detection means 11, swing detection means 12, screen change detection means 13, BSO caption detection means 14, and swing information determination means 15 provided in the control unit 50 correspond to the pitch detection means 11, swing detection means 12, screen change detection means 13, BSO caption detection means 14, and swing information determination means 15 provided in the control unit 10 of the swing information determination device 1 shown in Figure 1.

[0154] 12. The swing learning means 31 and the screen switching learning means 32 provided in the control unit 50 correspond to the swing learning means 31 and the screen switching learning means 32 provided in the control unit 30 of the swing information learning device 2 shown in FIG.

[0155] The swing learning means 31 stores the learned swing learning model M1 (parameters thereof) in the swing storage means 22, and the screen switching learning means 32 stores the learned screen switching learning model M2 (parameters thereof) in the screen switching storage means 23.

[0156] Furthermore, pitch storage means 21 provided in storage unit 20 corresponds to pitch storage means 21 provided in storage unit 20 of swing information determination device 1 shown in Fig. 1. Swing storage means 22 and screen switching storage means 23 provided in storage unit 20 correspond to swing storage means 22 and screen switching storage means 23 provided in storage unit 20 of swing information determination device 1 shown in Fig. 1, and also correspond to swing storage means 41 and screen switching storage means 42 provided in storage unit 40 of swing information learning device 2 shown in Fig. 12.

[0157] The operation modes of the swing information determination device 3 include a learning mode and an evaluation mode (determination mode).

[0158] When the operating mode is the learning mode, the swing learning means 31 provided in the control unit 50 inputs a known image and label (a label indicating "swing (right)", "swing (left)" or "other"), and learns the swing learning model M1 and stores it in the swing storage means 22.

[0159] Furthermore, the screen transition learning means 32 inputs known video and labels (labels indicating "confrontation scene" or "other"), learns a screen transition learning model M2, and stores it in the screen transition storage means .

[0160] On the other hand, when the operating mode is the evaluation mode, pitch detection means 11 provided in control unit 50, upon input of unknown video, generates pitch detection time T1, swing detection means 12 generates swing detection time T2, screen change detection means 13 generates screen change detection time T3, BSO caption detection means 14 generates BSO caption deletion detection time T4, and swing information determination means 15 determines and outputs swing information.

[0161] As described above, swing information determination device 3 according to an embodiment of the present invention achieves the same effects as the previously described swing information determination device 1 and swing information learning device 2. In other words, because swing information is determined without detecting a person's skeleton, it is possible to accurately determine a batter's swing information from video in a service that generates and distributes commentary text for live baseball broadcasts.

[0162] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical concept thereof.

[0163] For example, in the swing information determination device 1 shown in FIG. 1, the control unit 10 includes a pitch detection means 11, a swing detection means 12, a screen change detection means 13, a BSO caption detection means 14, and a swing information determination means 15, and the swing information determination means 15 determines the swing information based on a swing detection time T2, a screen change detection time T3, a BSO caption erasure detection time T4, and preset thresholds A and B.

[0164] Alternatively, control unit 10 may include pitch detection means 11, swing detection means 12, screen change detection means 13, and swing information determination means 15. In this case, swing information determination means 15 determines swing information based on swing detection time T2, screen change detection time T3, and preset threshold value A. In the processing example of the rule-based determiner of swing information determination means 15 shown in FIG. 8, processing is performed excluding steps S805 and S806, and the process moves from step S803(N) to step S808.

[0165] Control unit 10 may also include pitch detection means 11, swing detection means 12, BSO telop detection means 14, and swing information determination means 15. In this case, swing information determination means 15 determines swing information based on swing detection time T2, BSO telop removal detection time T4, and preset threshold B. In the processing example of the rule-based determiner of swing information determination means 15 shown in FIG. 8, processing is performed excluding steps S803 and S804, and the process moves from step S802 to step S805. The same applies to control unit 50 of swing information determination device 3 shown in FIG. 17.

[0166] Furthermore, when comparing the swing information determination devices 1 and 3 shown in Figures 1 and 17 with the aforementioned device that does not have the screen change detection means 13 or the BSO caption detection means 14, the former can determine swing information with higher accuracy.

[0167] Furthermore, for example, the swing information determination device 1 shown in FIG. 1 and the swing information determination device 3 shown in FIG. 17 are provided with pitch storage means 21, swing storage means 22, and screen change storage means 23, respectively, but they may be provided together in a single storage device. In other words, pitch storage means 21, swing storage means 22, and screen change storage means 23 may be provided in each of the areas when the storage area of ​​a single storage device is divided into multiple sections. Furthermore, the swing information learning device 2 shown in FIG. 12 is provided with swing storage means 41 and screen change storage means 42, respectively, but they may be provided together in a single storage device. In other words, the swing storage means 41 and screen change storage means 42 may be provided in each of the areas when the storage area of ​​a single storage device is divided into multiple sections.

[0168] Furthermore, although swing information determination device 1 shown in FIG. 1 and swing information determination device 3 shown in FIG. 17 are provided with storage unit 20, an external storage device may be provided with storage unit 20. In this case, swing information determination device 1 is provided with only control unit 10, and swing information determination device 3 is provided with only control unit 50, but not with storage unit 20. Furthermore, swing information learning device 2 shown in FIG. 12 is provided with storage unit 40, but an external storage device may be provided with storage unit 40. In this case, swing information learning device 2 is provided with only control unit 30, but not with storage unit 40.

[0169] Furthermore, for example, the swing learning model M1 and the screen switching learning model M2 are assumed to be models (CNN (Convolutional Neural Network), Vision Transformer) that are effective in the field of image recognition where learning is performed by supervised learning. However, in supervised learning, machine learning that is effective in other general image recognition fields may also be used.

[0170] Furthermore, for example, the BSO telop detection means 14 of the swing information determination device 1 shown in Fig. 1 and the BSO telop detection means 14 of the swing information determination device 3 shown in Fig. 17 use a rule-based determiner to generate the BSO telop deletion detection time T4. However, instead of the rule-based determiner, the BSO telop detection means 14 may use a determiner trained by machine learning that is effective in the general image recognition field. As the determiner, for example, a CNN (convolutional neural network) or a Vision Transformer is used.

[0171] Also, for example, swing information determination means 15 of swing information determination device 1 shown in Fig. 1 and swing information determination means 15 of swing information determination device 3 shown in Fig. 17 use a rule-based determiner to determine swing information. In contrast, swing information determination means 15 may use a determiner trained by general machine learning instead of the rule-based determiner. As the determiner, for example, a neural network, a support vector machine (SVM), or a conditional random field (CRF) is used.

[0172] A normal computer can be used as the hardware configuration of swing information determination device 1 according to the embodiment of the present invention. Swing information determination device 1 is configured by a computer equipped with a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, an interface, etc. The same applies to swing information learning device 2 and swing information determination device 3.

[0173] The functions of the control unit 10 (pitching detection means 11, swing detection means 12, screen change detection means 13, BSO caption detection means 14, and swing information determination means 15) and memory unit 20 (pitching memory means 21, swing memory means 22, and screen change memory means 23) provided in the swing information determination device 1 are each realized by having the CPU execute a program that describes these functions.

[0174] In addition, the functions of the control unit 30 (swing learning means 31 and screen switching learning means 32) and memory unit 40 (swing memory means 41 and screen switching memory means 42) provided in the swing information learning device 2 are also realized by having the CPU execute a program that describes these functions.

[0175] In addition, each function of the control unit 50 (pitching detection means 11, swing detection means 12, screen change detection means 13, BSO caption detection means 14, swing information determination means 15, swing learning means 31, and screen change learning means 32) and memory unit 20 (pitching memory means 21, swing memory means 22, and screen change memory means 23) provided in the swing information determination device 3 is realized by having the CPU execute a program that describes these functions.

[0176] These programs are stored in the storage medium and are read and executed by the CPU. These programs can also be stored in a storage medium such as a magnetic disk (HDD, etc.), an optical disk (CD-ROM, DVD, etc.), or a semiconductor memory (SSD, etc.) and distributed, or can be transmitted and received via a network. [Explanation of symbols]

[0177] 1,3 Swing information determination device 2. Swing information learning device 10,30,50 Control section 11. Pitching detection means 12 Swing detection means 13 Screen change detection method 14 BSO caption detection method 15 Swing information determination means 20,40 storage section 21 Pitching memory means 22,41 Swing memory means 23,42 Screen switching storage means 31 Swing Learning Tools 32 Screen switching learning method 100 viewers 101 Broadcast transmitting equipment 102 Broadcast receiving equipment 103 Audio commentary production and distribution device 104 Application Server 105 Mobile Devices 106 Voice Synthesizer 107 Distribution Device 108 Commentary audio production device 200 Commentary audio production and distribution system 201 Source of information M1 Swing Learning Model M2 Screen Switching Learning Model T1 pitch detection time T2 swing detection time T3 Screen change detection time T4BSO caption deletion detection time Z, A, B thresholds t elapsed time T a ,T b time

Claims

1. A swing information determination device inputs a video of a baseball program and determines swing information including a batter's swing state from the video, For each of the multiple images that make up the video, a predetermined classifier is used to detect the pitcher's pitch, and the timing is defined as a pitch detection time T 1 a pitching detection means for generating the pitch; For each of the plurality of images constituting the video, the pitching detection time T 1 Between the time of the swing detection time T and the time of the predetermined threshold Z, the swing of the batter is detected using a predetermined swing learning model, and the timing is defined as the swing detection time T 2 a swing detection means for generating the swing time; The swing detection time T generated by the swing detection means for each of the plurality of images constituting the video. 2 After that, a predetermined screen transition learning model is used to detect a screen transition from a confrontation scene in which the pitcher and the batter face off to a scene other than the confrontation scene, and the timing is defined as a screen transition detection time T 3 a screen change detection means for generating the following; The swing detection time T 2 , the screen change detection time T generated by the screen change detection means 3 , and the swing detection time T 2 a swing information determining means for determining the swing information that determines whether the swing state is a batting or a miss based on a time period of a predetermined threshold A from the time when the swing state is reached; A swing information determination device comprising:

2. A swing information determination device receives a video of a baseball game program and generates swing information including a batter's swing state from the video, For each of the multiple images that make up the video, a predetermined classifier is used to detect the pitcher's pitch, and the timing is defined as a pitch detection time T 1 a pitching detection means for generating the pitch; For each of the plurality of images constituting the video, the pitching detection time T 1 Between the time of the swing detection time T and the time of the predetermined threshold Z, the swing of the batter is detected using a predetermined swing learning model, and the timing is defined as the swing detection time T 2 a swing detection means for generating the swing time; The swing detection time T generated by the swing detection means for each of the plurality of images constituting the video. 2 After that, a predetermined judgement device is used to detect that the BSO telop, which displays the counts of ball (B), strike (S) and out (O), has been erased from the screen, and the timing is defined as a BSO telop erasure detection time T 4 a BSO telop detection means for generating the telop as follows: The swing detection time T 2 , the BSO telop erasure detection time T 4 , and the swing detection time T 2 a swing information determining means for determining the swing information that determines whether the swing state is a batting or a miss based on a predetermined threshold B from the A swing information determination device comprising:

3. A swing information determination device inputs a video of a baseball program and determines swing information including a batter's swing state from the video, For each of the multiple images that make up the video, a predetermined classifier is used to detect the pitcher's pitch, and the timing is defined as a pitch detection time T 1 a pitching detection means for generating the pitch; For each of the plurality of images constituting the video, the pitching detection time T 1 Between the time of the swing detection time T and the time of the predetermined threshold Z, the swing of the batter is detected using a predetermined swing learning model, and the timing is defined as the swing detection time T 2 a swing detection means for generating the swing time; The swing detection time T generated by the swing detection means for each of the plurality of images constituting the video. 2 After that, a predetermined screen transition learning model is used to detect a screen transition from a confrontation scene in which the pitcher and the batter face off to a scene other than the confrontation scene, and the timing is defined as a screen transition detection time T 3 a screen change detection means for generating the following; The swing detection time T generated by the swing detection means for each of the plurality of images constituting the video. 2 After that, a predetermined judgement device is used to detect that the BSO telop, which displays the counts of ball (B), strike (S) and out (O), has been erased from the screen, and the timing is defined as a BSO telop erasure detection time T 4 a BSO telop detection means for generating the telop as follows: The swing detection time T 2 , the screen change detection time T generated by the screen change detection means 3 , the BSO telop erasure detection time T 4 , and the swing detection time T 2 a swing information determination means for determining the swing information that determines whether the swing state is a batting or a miss based on a predetermined threshold A and a predetermined threshold B from the A swing information determination device comprising:

4. 2. The swing information determination device according to claim 1, The swing information determination means The swing detection time T 2 Count the elapsed time t from When the elapsed time t is before the time of the predetermined threshold A, the screen change detection time T 3 If the swing state is batting, the swing information is generated. When the elapsed time t reaches the predetermined threshold A, the screen change detection time T 3 If the swing information determination device does not generate the swing information indicating that the swing state is a miss, the swing information determination device generates the swing information indicating that the swing state is a miss.

5. 3. The swing information determining device according to claim 2, The swing information determination means The swing detection time T 2 Count the elapsed time t from When the elapsed time t is before the time of the predetermined threshold value B, the BSO telop erasure detection time T 4 If the swing state is batting, the swing information is generated. When the elapsed time t reaches the predetermined threshold value B, the BSO telop erasure detection time T 4 If the swing information determination device does not generate the swing information indicating that the swing state is a miss, the swing information determination device generates the swing information indicating that the swing state is a miss.

6. A program for causing a computer to function as the swing information determination device according to any one of claims 1 to 5.

Images