Commentary text generation device and program for use in sports broadcasts

Abstract

In a commentary audio service that provides commentary audio synchronized with sports broadcasts, the system generates commentary text from video footage of broadcast programs focusing on racket sports. [Solution] The object detection unit 20 of the person / ball position processing unit 10 provided in the explanatory text generation device 1 detects the image coordinates of each player and the ball from the image, and the court detection unit 22 detects the image coordinates of the court from the image. The projection transformation unit 23 projects the image to generate a true overhead image and uses the projection transformation matrix to obtain the actual coordinates of each player, the actual coordinates of the ball, and the actual coordinates of the court. The motion information generation unit 24 generates motion information for each player based on the actual coordinates of each player and the actual coordinates of the court, and generates motion information for the ball based on the actual coordinates of the ball and the actual coordinates of the court. The explanatory text generation unit 25 generates explanatory text about the players' play based on the motion information for each player, and generates explanatory text about the players' play based on the ball's motion information.

Description

Technical Field

[0001] The present invention relates to an explanatory text generation device and a program for generating an explanatory text for commentary audio of sports broadcasts.

Background Art

[0002] Broadcasting is a medium that provides information to viewers' vision and hearing. However, it is difficult for visually impaired people to fully receive video information, and in particular, information displayed on the screen cannot be transmitted to completely blind people. Thus, the information that can be transmitted to visually impaired people is limited.

[0003] Therefore, an explanatory broadcast service is provided in television broadcasts. The explanatory broadcast service is a service that superimposes explanatory audio that complements visual information on the secondary audio channel of a television. Viewers can listen to the explanatory audio by operating the audio switching button on the remote control to switch the main audio to the secondary audio.

[0004] The explanatory broadcast service is generally produced with time and manpower mainly for recorded programs such as dramas and documentaries, and thus is rarely implemented for live broadcast programs such as news and sports broadcasts.

[0005] It is desirable that the explanatory audio be uttered clearly during the intervals of the main audio. The fact that it is difficult to instantaneously determine the utterance timing and content during a live broadcast is also a reason for the lack of implementation of the explanatory broadcast service in live broadcast programs.

[0006] In contrast, in order to apply the explanatory broadcast service to live broadcast programs, a system has been developed that provides explanatory audio for sports broadcasts by delivering the explanatory audio to viewers' mobile terminals separately from the television audio (see, for example, Patent Documents 1 and 2).

[0007] This system primarily broadcasts baseball sports programs and provides commentary audio to viewers. It analyzes baseball sports broadcast footage to determine ball counts, player actions, etc., and automatically generates commentary. Furthermore, for commentary that is difficult to generate automatically, a simple commentary creation tool allows for manual generation. This system enables the provision of diverse data from broadcast programs as commentary audio to viewers.

[0008] Figure 18 is a diagram illustrating the configuration of a system that provides commentary audio services. This system consists of a broadcast transmission device 101, a broadcast reception device 102, a commentary audio production and distribution device 103, an application server 104, and a mobile terminal 105.

[0009] The broadcast transmission device 101, the commentary audio production and distribution device 103, and the application server 104 are installed, for example, at a broadcasting station, while the broadcast receiving device 102 is installed, for example, in the home of a viewer 100 who watches broadcast program content. The mobile terminal 105 is used by the viewer 100.

[0010] This system's commentary audio service allows viewers 100 to receive commentary audio along with the broadcast program, which includes audio and video of the match being explained by the announcer and commentators.

[0011] The broadcast transmission device 101 transmits broadcast program content to the broadcast reception device 102 via terrestrial digital broadcast waves. The broadcast reception device 102 is, for example, a television receiver, which receives the broadcast program content transmitted from the broadcast transmission device 101 via terrestrial digital broadcast waves and plays the broadcast program content.

[0012] The commentary audio production and distribution device 103 produces commentary audio for the broadcast program content transmitted by the broadcast transmission device 101 and sends the commentary audio to the mobile terminal 105. The application server 104 stores applications that run on the mobile terminal 105 and sends applications to the mobile terminal 105 in response to requests from the mobile terminal 105. "Application" is an abbreviation for application, and in this case, it is a program that receives and plays the commentary audio.

[0013] The mobile device 105 is, for example, a smartphone, and plays commentary audio for the broadcast program content in synchronization with the broadcast receiving device 102. When playing the commentary audio, the mobile device 105 changes the playback speed and other settings according to the viewer's (100) input.

[0014] For example, if the broadcast program content is a live baseball sports broadcast, viewers 100 can receive not only video and audio of the baseball game, but also commentary that provides a detailed explanation of the game situation, allowing them to understand the details of the game. Baseball commentary may include information such as the pitcher's actions, pitch type, speed, and location, as well as information about the batter, the batter's actions, and the score, depending on the game situation.

[0015] Figure 19 is a conceptual diagram illustrating the image of a system that provides commentary audio services. It is assumed that in the home, the viewer 100, operating the mobile terminal 105, has pre-set options such as speaker gender selection ("male") and playback speed (four levels, "slow").

[0016] In the commentary audio production and distribution device 103 installed at the broadcasting station, commentary text generated from manually entered data, and commentary text generated from automatically entered data by character recognition, are sent to the speech synthesis server 106 in the cloud. The speech synthesis server 106 then synthesizes commentary audio from the commentary text, and the commentary audio is distributed from the distribution server 107 in the cloud to the mobile terminal 105.

[0017] For example, when viewer 100 hears the television audio "He missed the fast ball" from the broadcast receiving device 102, the mobile terminal 105 plays an audio commentary of "145 km / h fastball" based on pre-set speaker gender selection "male" and playback speed of 4 levels "slow". In other words, the mobile terminal 105 plays visual information (on-screen superimposed, player movements, etc.) that the announcer deliberately chooses not to mention.

[0018] In this way, the generated commentary text is converted into commentary audio by the speech synthesis server 106 in the cloud and distributed from the distribution server 107 to the viewer's mobile device 105. This allows the viewer 100 to listen to the commentary audio on their mobile device 105 in sync with the audio of the sports broadcast on television. [Prior art documents] [Patent Documents]

[0019] [Patent Document 1] Japanese Patent Publication No. 2023-170822 [Patent Document 2] Japanese Patent Publication No. 2024-112086 [Overview of the Initiative] [Problems that the invention aims to solve]

[0020] The commentary audio service shown in Figures 18 and 19 provides real-time commentary synchronized with sports broadcasts, utilizing manually and automatically generated commentary text.

[0021] However, conventional audio commentary services had problems such as a high proportion of manual generation of commentary text, which could lead to delays in speaking timing and errors in the content of the speech.

[0022] In addition, the target of the commentary voice service is mainly the broadcast program of the live relay of baseball, and baseball has unique rules compared to other sports. Therefore, there has been a problem that it is difficult to directly apply the function of generating commentary sentences in the commentary voice service of baseball to other sports (such as racket sports like tennis, table tennis, badminton, etc.).

[0023] Therefore, the present invention has been made to solve the above problems, and its object is to provide an explanatory sentence generation device and a program that generate explanatory sentences from the video of a broadcast program targeting racket sports in a commentary voice service that provides commentary voices synchronized with the live relay broadcast program of sports.

Means for Solving the Problems

[0024] In order to solve the above problems, the explanatory sentence generation device according to claim 1 is an explanatory sentence generation device that generates an explanatory sentence corresponding to the commentary voice when providing the commentary voice synchronized with the video of the live relay of racket sports to the viewer. The device detects the image coordinates of the players and the image coordinates of the ball included in the image from the images constituting the video, and uses the image coordinates of the court included in the image to projectively transform the image into a true overhead view image, thereby obtaining the positions of the players corresponding to the image coordinates of the players and the positions of the balls corresponding to the image coordinates of the balls. Based on the transition of the positions of the players, a first explanatory sentence regarding the playing content of the players is generated, and based on the transition of the positions of the balls, a second explanatory sentence regarding the playing content of the players is generated. It is characterized by comprising a person and ball position processing unit.

[0025] Further, the explanatory text generation device according to claim 2 is the explanatory text generation device according to claim 1, wherein the person / ball position processing unit projects and transforms the image into a true bird's-eye view image for each image constituting the video using the image coordinates of the court, and obtains the actual coordinates of the player, the actual coordinates of the ball, and the actual coordinates of the court included in the true bird's-eye view image corresponding to the image coordinates of the player, the image coordinates of the ball, and the image coordinates of the court; a motion information generation unit that generates player motion information indicating the transition of the player's position on the court based on the actual coordinates of the player and the actual coordinates of the court obtained by the projection conversion unit, and generates ball motion information indicating the transition of the ball's position on the court based on the actual coordinates of the ball and the actual coordinates of the court; and an explanatory text generation unit that generates the first explanatory text based on the player motion information generated by the motion information generation unit and generates the second explanatory text based on the ball motion information.

[0026] Further, the explanatory text generation device according to claim 3 is an explanatory text generation device that generates an explanatory text corresponding to the explanatory audio when providing the explanatory audio synchronized with the video of the racket sports relay to the viewer. The device includes a human motion processing unit that generates player skeleton information by detecting the skeleton of the player included in the image from the image constituting the video, detects the player's motion based on the player's skeleton information, and generates an explanatory text regarding the player's motion. The human motion processing unit includes a motion detection unit that detects the player's motion for each image constituting the video based on the player's skeleton information in a predetermined number of consecutive images constituting the video using a predetermined learning model; a threshold processing unit that determines the motion as a confirmed motion of the player when the number of images in which the player's motion detected by the motion detection unit is the same exceeds a predetermined threshold within a time window of a predetermined number of preset images; and an explanatory text generation unit that generates an explanatory text regarding the player's motion based on the confirmed motion of the player determined by the threshold processing unit.

[0027] Furthermore, the explanatory text generation device of claim 4 is an explanatory text generation device that generates explanatory text corresponding to the explanatory audio when providing explanatory audio synchronized with video footage of a racket sports broadcast to viewers, and is characterized by comprising a scene description unit that generates text representing the content of images constituting the video using a predetermined learning model, determines the class of the image from a pre-set explanatory text indicating the class of each player using a predetermined multimodal learning model, replaces abstract nouns of people included in the text with proper nouns of players corresponding to the class, and generates explanatory text relating to the scene description.

[0028] Furthermore, the explanatory text generation device of claim 5 is characterized in that, in the explanatory text generation device of claim 4, the scene description unit comprises: a text generation unit that generates text representing the content of each image constituting the video using the learning model; a classification unit that uses the multimodal learning model to determine the class with the highest similarity as a classification result for each image constituting the video, based on the similarity between the respective feature vectors in the explanatory text indicating the class of each player and the feature vector of the image; and an explanatory text generation unit that identifies pre-set abstract nouns of people from the text generated by the text generation unit, replaces the abstract nouns of people in the text with proper nouns of players corresponding to the classification result determined by the classification unit, and generates the replaced text as an explanatory text relating to the scene description.

[0029] Furthermore, the program of claim 6 is characterized by causing the computer to function as the explanatory text generation device described in claim 1. [Effects of the Invention]

[0030] As described above, according to the present invention, in a commentary audio service that provides commentary audio synchronized with a sports broadcast program, commentary text can be generated from video footage of a broadcast program focusing on racket sports. [Brief explanation of the drawing]

[0031] [Figure 1] This diagram illustrates the overview of a system that provides explanatory audio services using an explanatory text generation device according to an embodiment of the present invention. [Figure 2] This is a block diagram showing an example configuration of an explanatory text generation device. [Figure 3] This is a block diagram showing an example configuration for the person / ball position processing unit. [Figure 4] This flowchart shows an example of the processing performed by the person / ball position processing unit. [Figure 5] This diagram illustrates an example of processing in the person / ball position processing unit. [Figure 6] This flowchart shows an example of the processing in the projection transformation unit (details of the processing in step S404). [Figure 7] This is a block diagram showing an example configuration of the human motion processing unit. [Figure 8] This flowchart shows an example of the processing performed by the human motion processing unit. [Figure 9] This diagram illustrates an example of processing in the human motion processing unit. [Figure 10] This figure shows an example of a player's skeletal information in a 10-frame (1-sequence) image. [Figure 11] This is a block diagram showing an example of the composition of a scene description section. [Figure 12] This figure shows an example of a table data structure. [Figure 13] This flowchart shows an example of how the scene description section is processed. [Figure 14] This diagram illustrates an example of how to process the descriptive section of a scene (when the text contains abstract nouns about people). [Figure 15] This diagram illustrates an example of how to process the scene description section (when the text does not contain abstract nouns about people). [Figure 16] This figure shows an example of explanatory text generated by the explanatory text generation device according to an embodiment of the present invention. [Figure 17] This figure shows the experimental results. [Figure 18] This is a diagram illustrating the system's configuration for providing commentary audio services. [Figure 19] This is a conceptual diagram illustrating the image of a system that provides explanatory audio services. [Modes for carrying out the invention]

[0032] The embodiments for carrying out the present invention will be described in detail below with reference to the drawings. [Overall System] First, the overall system including the explanatory text generation device according to an embodiment of the present invention will be described. Figure 1 is a configuration diagram illustrating the outline of a system that provides explanatory audio services using the explanatory text generation device according to an embodiment of the present invention.

[0033] The system providing this commentary audio service corresponds to the system shown in Figure 18 and consists of a broadcast transmission device 101, a broadcast reception device 102, a commentary text generation device 1, a mobile terminal 105, a speech synthesis server 106, and a distribution server 107.

[0034] The broadcast transmission device 101 and the explanatory text generation device 1 are installed, for example, at a broadcasting station; the broadcast receiving device 102 is installed, for example, in the home of a viewer 100 who watches broadcast program content; and the mobile terminal 105 is used by the viewer 100.

[0035] The broadcast transmission device 101, broadcast reception device 102, mobile terminal 105, voice synthesis server 106, and distribution server 107 are the same as the components shown in Figures 18 and 19, and therefore their explanation is omitted here.

[0036] The explanatory text generation device 1 is a device that generates explanatory text for broadcast program content transmitted by the broadcast transmission device 101. Details of the explanatory text generation device 1 will be described later.

[0037] The explanatory text generated by the explanatory text generation device 1 is sent to the speech synthesis server 106 on the cloud, and the distribution server 107 on the cloud synthesizes explanatory audio from the explanatory text. The explanatory audio is then distributed from the distribution server 107 to the mobile terminal 105.

[0038] As a result, the mobile device 105 plays commentary audio synchronized with the broadcast program content, and viewers 100 can receive commentary audio along with the broadcast program, which includes audio and video of the announcer's play-by-play and the commentator's explanation of the match situation.

[0039] Embodiments of the present invention expand the scope of target sports to racket sports by introducing a new video analysis function to the functions of the commentary audio production and distribution device 103 shown in Figures 18 and 19. In the following description, a tennis sports broadcast program will be used as an example, and the commentary text generation device 1 will generate commentary text for the tennis sports broadcast.

[0040] [Explanatory text generation device 1] Next, we will describe in detail the explanatory text generation device 1 shown in Figure 1. Figure 2 is a block diagram showing an example of the configuration of the explanatory text generation device 1.

[0041] This commentary text generation device 1 comprises a person / ball position processing unit 10, a person motion processing unit 11, a scene description unit 12, and a manual input unit 13. When providing commentary audio synchronized with the video of a tennis sports broadcast to the viewer 100, the commentary text generation device 1 inputs the video, generates commentary text about the players' play and actions as well as the scene description shown in the video, and transmits the commentary text to the speech synthesis server 106 shown in Figure 1.

[0042] The person / ball position processing unit 10 detects the positions of players and the ball from the video and generates explanatory text about the players' actions based on the changes in their positions.

[0043] Specifically, the person / ball position processing unit 10 receives video input and uses the learning model 21 shown in Figure 3 (described later) to detect the image coordinates of each player (two players in this example) and the ball included in the images that make up the video. The person / ball position processing unit 10 also detects the image coordinates of the court included in the image. Then, using the image coordinates of the court, the person / ball position processing unit 10 projects the image into a true overhead view image and obtains the actual coordinates (positions) of the players corresponding to the players' image coordinates and the actual coordinates (positions) of the ball corresponding to the ball's image coordinates.

[0044] The player / ball position processing unit 10 generates explanatory text about the player's play (for example, "Player A moves forward") based on the player's position changes and the player's movement information on the court. The player / ball position processing unit 10 also generates explanatory text about the player's play (for example, "Player A serves") based on the ball's position changes and the ball's movement information on the court. Finally, the player / ball position processing unit 10 transmits the explanatory text. Details of the player / ball position processing unit 10 will be described later.

[0045] Furthermore, the person / ball position processing unit 10 may, instead of detecting the image coordinates of the court included in the image, accept pre-set image coordinates of the court as input. Real coordinates are coordinates in real space.

[0046] The human motion processing unit 11 estimates the athlete's posture from the video and generates explanatory text about the athlete's movements using motion recognition technology based on the athlete's posture. Generally, in television sports broadcasts, information about the athlete's movements is conveyed visually through the screen of the broadcast receiving device 102, so announcers rarely provide commentary. For this reason, the human motion processing unit 11 generates explanatory text about the athlete's movements, and the mobile terminal 105 plays the audio of that explanation.

[0047] Specifically, the human motion processing unit 11 receives video input, uses the learning model 31 shown in Figure 7 (described later) to detect skeletal information for each player from the images that make up the video, and uses the learning model 33 shown in Figure 7 (described later) to detect the player's actions (for example, "running") from the player's skeletal information.

[0048] The human motion processing unit 11 generates an explanatory text about the player's actions (for example, "Player B, dashing") based on the player's actions. The human motion processing unit 11 then transmits the explanatory text. Details of the human motion processing unit 11 will be described later.

[0049] The scene description unit 12 generates explanatory text about the scene depicting the person included in the image, such as by generating text that represents the content of the image from the image.

[0050] Specifically, the scene description unit 12 receives video as input and uses the learning model 41 shown in Figure 11 (described later) to generate text that represents the content of the image from the images that make up the video (for example, "woman with a white sun visor").

[0051] The scene description unit 12 uses the learning model 43 shown in Figure 11, which will be described later, to determine the class of the image in question (for example, "Player A's class a") from the descriptive text indicating the pre-set class for each player (for example, "Player A's class a", "Player B's class b").

[0052] The scene description unit 12 replaces abstract nouns of people in the generated text (e.g., "woman") with proper nouns of players corresponding to the determined class (e.g., "Player A"), and generates an explanatory text about the scene (e.g., "Player A wearing a white visor"). The scene description unit 12 then transmits the explanatory text. Details of the scene description unit 12 will be described later.

[0053] The manual input unit 13 allows a user to manually operate the explanatory text generation device 1 to input the explanatory text and transmit the explanatory text.

[0054] <Person / Ball Position Processing Unit 10> Next, we will explain in detail the person / ball position processing unit 10 shown in Figure 2. Figure 3 is a block diagram showing an example configuration of the person / ball position processing unit 10, and Figure 4 is a flowchart showing an example of processing by the person / ball position processing unit 10. Figure 5 is a diagram illustrating an example of processing by the person / ball position processing unit 10.

[0055] This person / ball position processing unit 10 includes an object detection unit 20, a learning model 21, a court detection unit 22, a projection transformation unit 23, a motion information generation unit 24, and an explanatory text generation unit 25.

[0056] The object detection unit 20, court detection unit 22, and projection transformation unit 23 of the person / ball position processing unit 10 receive video (image by image) of a tennis sports broadcast (step S401).

[0057] The object detection unit 20 uses the learning model 21 to detect objects in each image that makes up the input video, thereby inferring the image coordinates of each player and the ball (step S402). The object detection unit 20 then outputs the image coordinates of each player and the ball to the projection transformation unit 23.

[0058] As shown in Figure 5, the process in step S402 infers, for example, the image coordinates of player A, player B, and the image coordinates of the ball.

[0059] For example, YOLOv8, a general-purpose object detection model, is used as the learning model 21. Generally, when a standard model is used as the learning model 21, many false positives and missed detections occur, making it difficult to infer stable image coordinates. Therefore, additional learning is performed on the learning model 21 by annotating the positions of people and balls included in the image to be detected.

[0060] In this embodiment of the present invention, annotation is performed by specifying the positions of players and the ball using bounding boxes from video footage of a tennis rally. For example, LabelMe can be used as a tool for annotation. For instance, approximately 15,000 overhead images of a tennis match are prepared, and the positions of players and the ball included in these images are annotated and further training is performed. This improves the accuracy of detecting people and the ball.

[0061] The court detection unit 22 detects the court for each image constituting the input video, for example using a known line detection method, and obtains the image coordinates of the court (for example, the image coordinates of the four corners of the court) (step S403). The image coordinates of the four corners of the court are obtained, for example, by detecting the four outer lines (lines) of the court using a line detection method and finding the intersection of two lines. The court detection unit 22 then outputs the image coordinates of the court to the projection transformation unit 23.

[0062] Furthermore, the court detection unit 22 may detect the image coordinates of the four corners of the court using a method other than the linear detection method. Alternatively, as described above, the court detection unit 22 may manually set the image coordinates of the four corners of the court. In this case, assuming that the position of the camera that photographs the court is fixed and the position of the court in the image is unchanging, the projection transformation unit 23 inputs the image coordinates of the four corners of the court that have been set manually in advance.

[0063] However, even if we use the image coordinates of each player, the ball, and the court detected by the object detection unit 20 and the court detection unit 22, we cannot determine their respective positions on the actual court in real space. Therefore, we use a projection transformation performed by the projection transformation unit 23, described later, which utilizes the image coordinates of the court (image coordinates of the four corners) to obtain the actual coordinates, which are the coordinates in real space as seen from directly above the court.

[0064] The projection transformation unit 23 receives video input, as well as image coordinates for each player and the ball from the object detection unit 20, and further inputs image coordinates for the court from the court detection unit 22. The projection transformation unit 23 then generates a true overhead image by projecting each image that makes up the video, and uses the resulting projection transformation matrix to determine the actual coordinates of each player, the ball, and the court (step S404).

[0065] Figure 6 is a flowchart showing an example of the processing of the projection transformation unit 23 (details of the processing in step S404). In the processing of step S404, the projection transformation unit 23 uses the image coordinates of the coat to generate a true overhead image from the image and calculates the projection transformation matrix (step S601).

[0066] The projection transformation unit 23 uses a projection transformation matrix to convert the image coordinates of each player into real coordinates, as well as the image coordinates of the ball into real coordinates, and further converts the image coordinates of the court into real coordinates (step S602). As a result, the real coordinates of each player, the real coordinates of the ball, and the real coordinates of the court are obtained for each of the multiple images that make up the video. The projection transformation unit 23 then outputs the real coordinates of each player, the real coordinates of the ball, and the real coordinates of the court to the motion information generation unit 24.

[0067] Furthermore, if the camera position for capturing the court is fixed, the actual coordinates of the court remain unchanged. Therefore, the actual coordinates of the court obtained in the initial transformation may be used for subsequent images.

[0068] As shown in Figure 5, the process in step S404 yields, along with the overhead view image, the actual coordinates corresponding to the image coordinates of player A, player B, the ball, and the court.

[0069] In the overhead view, players A and B are almost always touching the ground and can be considered to have a height of zero. Therefore, the actual coordinates of player A and player B on the actual court obtained by the projection transformation in step S404 are highly accurate.

[0070] In contrast, for the ball, an error occurs between the actual coordinates and the ball's position depending on its height from the ground. However, since the actual coordinates are those viewed from directly above the court, the error due to the height from the ground is not a significant problem, and the explanatory text about the players' actions, based on the changes in the ball's position generated by step S406 (described later), will be accurate.

[0071] Returning to Figures 3 to 5, the motion information generation unit 24 receives the actual coordinates of each player, the actual coordinates of the ball, and the actual coordinates of the court from the projection transformation unit 23.

[0072] The motion information generation unit 24 generates motion information showing the change in the position of each player on the court based on the change in the actual coordinates of each player and the actual coordinates of the court (fixed actual coordinates), and also generates motion information showing the change in the position of the ball on the court based on the change in the actual coordinates of the ball and the actual coordinates of the court (fixed actual coordinates) (step S405).

[0073] The motion information generation unit 24 outputs motion information showing the change in each player's position on the court as player-specific motion information, and motion information showing the change in the ball's position on the court as ball motion information, to the explanatory text generation unit 25.

[0074] For example, the motion information generation unit 24, based on the change in player A's actual coordinates and the actual coordinates of the court, determines that player A is approaching the net located in the center of the court, which is obtained from the actual coordinates of the court, and generates player motion information indicating that "player A is approaching the net." As shown in Figure 5, the processing in step S405 generates, for example, player motion information indicating that "player A is approaching the net."

[0075] Furthermore, for example, if the motion information generation unit 24 determines, based on the transition of the ball's actual coordinates and the actual coordinates of the court, that "the ball has moved for the first time from player A's court to player B's court," that is, if it determines that the ball has moved across the court between players for the first time since play began, and that the ball has moved from player A's court to player B's court, it generates motion information of the ball indicating that fact.

[0076] Furthermore, for example, if the motion information generation unit 24 determines, based on the transition of the ball's actual coordinates and the actual coordinates of the court, that "after the first movement of the ball (after the serve), the ball moved back to player A's side of the court," that is, if it determines that the ball has moved across the court between players for the second time since the start of play, and the ball has moved from player B's side of the court to player A's side of the court, it generates ball motion information indicating this.

[0077] Furthermore, for example, if the motion information generation unit 24 determines, based on the transition of the ball's actual coordinates and the actual coordinates of the court, that "after the second movement of the ball (after the receive), the ball moved back to player B's court," that is, if it determines that the ball moved across the court between players for the third time since the start of play, and the ball moved from player A's court to player B's court, it generates ball motion information indicating this.

[0078] Furthermore, when generating motion information, if the motion information generation unit 24 determines, based on the actual coordinates of the ball, that the ball has moved from one player's court to another player's court, it will make another determination based on the actual coordinates of the ball after a predetermined time (number of frames) has elapsed. Then, if the motion information generation unit 24 makes the same determination again (for example, if it initially determines that "the ball has moved for the first time from player A's court to player B's court," and then makes the same determination again after a predetermined time has elapsed), it will generate motion information.

[0079] This eliminates the incorrect real coordinates of the ball (the wrong real coordinates of the ball) that were misconverted by the projection transformation unit 23, enabling the generation of highly accurate ball movement information, and as a result, highly accurate explanatory text can be generated.

[0080] The explanatory text generation unit 25 receives movement information for each player and movement information for the ball from the movement information generation unit 24. The explanatory text generation unit 25 then generates explanatory text about the players' play based on the movement information for each player, derived from the changes in the players' positions on the court, and also generates explanatory text about the players' play based on the movement information for the ball, derived from the changes in the ball's position on the court (step S406). The explanatory text generation unit 25 then outputs the explanatory text (step S407).

[0081] For example, if the explanatory text generation unit 25 receives player movement information indicating "Player A is approaching the net," it generates an explanatory text indicating "Player A moves forward" (based on the change in the player's position). As shown in Figure 5, the process in step S406 generates an explanatory text indicating "Player A moves forward."

[0082] Furthermore, for example, if the explanatory text generation unit 25 receives information indicating the movement of the ball, such as "the ball has moved for the first time from player A's court to player B's court," it generates an explanatory text indicating "player A serves" (based on the change in the ball's position).

[0083] Furthermore, for example, if the explanatory text generation unit 25 receives ball movement information indicating that "after the initial movement of the ball (after the serve), the ball moved back to player A's side of the court," it generates an explanatory text indicating "player B receives" (based on the change in the ball's position).

[0084] Furthermore, for example, if the explanatory text generation unit 25 receives information indicating that "after the second movement of the ball (after the receive), the ball moved back to player B's side of the court," it generates an explanatory text indicating "player A, shot" (based on the change in the ball's position).

[0085] Examples of explanatory text based on the changes in players' positions include, in addition to those mentioned above, "Player A, second serve," and "Player B, shot to the opposite side." Similarly, examples of explanatory text based on the changes in the ball's position include, in addition to those mentioned above, "Player A, moving backward," and "Player B, moving to the left."

[0086] As described above, according to the explanatory text generation device 1 of the embodiment of the present invention, the object detection unit 20 of the person / ball position processing unit 10 uses a learning model 21 to detect the image coordinates of each player and the ball from the image. In addition, the court detection unit 22 detects the image coordinates of the court from the image.

[0087] The projection transformation unit 23 generates a true overhead image by projecting the image, and uses the resulting projection transformation matrix to determine the actual coordinates of each player, the ball, and the court. Then, the motion information generation unit 24 generates motion information for each player based on the actual coordinates of each player and the court, and generates motion information for the ball based on the actual coordinates of the ball and the court.

[0088] The explanatory text generation unit 25 generates explanatory text about the player's play based on the movement information of each player, and generates explanatory text about the player's play based on the movement information of the ball.

[0089] This enables the generation of commentary text from video footage of sports broadcasts, specifically for tennis, in a commentary audio service that provides commentary audio synchronized with the broadcast program. In particular, the person / ball position processing unit 10 can generate commentary text about the players' play based on the changes in the players' positions, using the actual coordinates of each player on the court. It can also generate commentary text about the players' play based on the changes in the ball's position, using the actual coordinates of the ball on the court.

[0090] The explanatory text generated by the person / ball position processing unit 10 is then delivered as explanatory audio to the mobile device 105 used by the viewer 100. The viewer 100 can listen to the explanatory audio about the players' actions in sync with the audio of the tennis sports broadcast.

[0091] <Human motion processing unit 11> Next, the human motion processing unit 11 shown in Figure 2 will be described in detail. Figure 7 is a block diagram showing an example of the configuration of the human motion processing unit 11, and Figure 8 is a flowchart showing an example of processing by the human motion processing unit 11. Figure 9 is a diagram illustrating an example of processing by the human motion processing unit 11.

[0092] This human motion processing unit 11 includes a skeleton detection unit 30, learning models 31 and 33, a motion detection unit 32, a threshold processing unit 34, and an explanatory text generation unit 35.

[0093] The skeleton detection unit 30 of the human motion processing unit 11 receives video footage (image by image) of a tennis sports broadcast (step S801).

[0094] The skeleton detection unit 30 uses the learning model 31 to detect the skeleton of each player included in each image that makes up the input video, thereby inferring skeletal information for each player (step S802). The skeleton detection unit 30 then outputs the skeletal information for each player to the motion detection unit 32.

[0095] Skeletal information is represented as a graph using nodes (joints) and edges (connections between joints) in skeletal coordinates. By connecting nodes between adjacent images (frames), it is possible to analyze the temporal changes of each joint.

[0096] As shown in Figure 9, the process in step S802 infers, for example, the skeletal information of player A and the skeletal information of player B.

[0097] As the learning model 31, for example, the aforementioned YOLOv8, which is an image recognition model generated by deep learning, is used.

[0098] The motion detection unit 32 receives skeletal information for each player from the skeletal detection unit 30. Then, using the learning model 33, the motion detection unit 32 detects (infers) the motion for each player from the skeletal information for each player (a predetermined number of consecutive images, for example, in units of 10 images (10 frames)) (step S803). As a result, for example, the motion for each player in relation to the first image (the first image) is detected from 10 consecutive frames.

[0099] Then, using 10 frames starting with the second image, the actions of each player in relation to the second image are detected, and then, using 10 frames starting with the third image, the actions of each player in relation to the third image are detected. This process is carried out in units of 10 images, shifting one image at a time, so that the actions of each player can be obtained for each image that makes up the video. The action detection unit 32 outputs the actions of each player to the threshold processing unit 34.

[0100] As shown in Figure 9, the process in step S803 detects, for example, that player A's action is "Run" and player B's action is "Stand".

[0101] As a learning model 33, for example, an ST-GCN (Spatial-Temporal Graph Convolutional Network) is used, which represents a spatial graph of the positional relationships of each joint and a time graph between adjacent frames (representing skeletal information as a spatiotemporal graph). This learning model 33 is generated by learning the weights of the skeleton in each movement using skeletal movement, which is the time-series change of skeletal information, as training data, and can recognize various movements from video. Furthermore, compared to CNNs that use images as input data, it can learn with a small amount of data and enables highly accurate recognition that takes the time axis into consideration.

[0102] Figure 10 shows an example of a player's skeletal information in a 10-frame (1-sequence) image. When training the learning model 33, as shown in Figure 10, for example, a player's skeletal information in units of 10 frames (1-sequence) is used.

[0103] The learning model 33 uses the skeletal information of a player in one sequence and the corresponding player's movements as training data. For example, it learns the player's movements using 500 sequences of skeletal information. In this example, the player's movements are of five types: "Stand," "Walk," "Run," "Stroke," and "Other."

[0104] Returning to Figures 7 to 9, the threshold processing unit 34 receives input for each player's actions from the motion detection unit 32 for each image that makes up the video. The threshold processing unit 34 then sets a predetermined number of images (frames) in a time window for each player, starting with the image in question. If the number of frames within this time window that contain the same action (the actions detected by the motion detection unit 32 are the same) exceeds a predetermined threshold, the threshold processing unit 34 determines that the occurrence of that action is the final action (confirmed action) in that image (step S804). The threshold processing unit 34 then outputs the determined final action for each player to the explanatory text generation unit 35.

[0105] In other words, the threshold processing unit 34 determines that, for each of the input player actions (player actions), if the number of frames in which the same action occurs within the time window exceeds a predetermined threshold, that identical action is the final action.

[0106] For example, the thresholding unit 34 determines that the final action is "Run" if the number of frames of the same action "RUN" within a 10-frame time window exceeds a threshold (for example, 5 frames). On the other hand, the thresholding unit 34 does not determine the final action if the number of frames of the same action within the time window does not exceed the threshold. In this case, no explanatory text is generated.

[0107] As shown in Figure 9, the process in step S804 determines, for example, that player A's action is "Run" and player B's action is "Stand".

[0108] The explanatory text generation unit 35 receives input for each player's actions from the threshold processing unit 34 and generates explanatory text about the players' actions based on those actions (step S805). Then, the explanatory text generation unit 35 outputs the explanatory text (step S806).

[0109] For example, if the explanatory text generation unit 35 receives input indicating the action of player A, "Run," it generates an explanatory text about player A's action, indicating "Player A, dashing." As shown in Figure 9, the process in step S805 generates an explanatory text, for example, indicating "Player A, dashing."

[0110] Furthermore, for example, if the explanatory text generation unit 35 receives input indicating the action of player B, which is "Stand," it generates an explanatory text about the action of player B, which is "Player B, in stance." As shown in Figure 9, the process in step S805 generates, for example, an explanatory text indicating "Player B, in stance."

[0111] Furthermore, for example, if the explanatory text generation unit 35 receives input indicating the action of player A, "Stroke," it will generate explanatory text about player A's action, indicating "Player A, shot."

[0112] As described above, according to the explanatory text generation device 1 of the embodiment of the present invention, the skeleton detection unit 30 of the human motion processing unit 11 infers skeletal information for each player from the image using the learning model 31. Then, the motion detection unit 32 infers the movements of each player from the skeletal information for each player using the learning model 33.

[0113] The threshold processing unit 34 determines, for each player, that if the number of frames of the same action within a predetermined time window of frames exceeds a preset threshold, the occurrence of that action is the final action in that image. Then, the explanatory text generation unit 35 generates explanatory text about the player's action based on the player's action.

[0114] This enables the generation of commentary text from video footage of sports broadcasts in an audio commentary service that provides commentary synchronized with the broadcast program, specifically for tennis. In particular, the human motion processing unit 11 detects movements from the skeletal information of each player and generates commentary text about the player's movements.

[0115] The explanatory text generated by the human motion processing unit 11 is then delivered to the mobile terminal 105 as explanatory audio, allowing the viewer 100 to listen to the explanatory audio about the players' movements in sync with the audio of the tennis sports broadcast.

[0116] <Scene description part 12> Next, the scene description unit 12 shown in Figure 2 will be explained in detail. Figure 11 is a block diagram showing an example of the configuration of the scene description unit 12, Figure 12 is a diagram showing an example of the data structure of a table, and Figure 13 is a flowchart showing an example of processing by the scene description unit 12. Figure 14 is a diagram illustrating an example of processing by the scene description unit 12 (when the text contains abstract nouns of people), and Figure 15 is a diagram illustrating an example of processing by the scene description unit 12 (when the text does not contain abstract nouns of people).

[0117] This scene description unit 12 includes a text generation unit 40, learning models 41 and 43, a classification unit 42, an explanatory text generation unit 44, and tables 45 and 46.

[0118] The text generation unit 40 and classification unit 42 of the scene description unit 12 receive video footage (image by image) of a tennis sports broadcast (step S1301).

[0119] The text generation unit 40 generates text representing the content of each image that makes up the input video, using the learning model 41 (step S1302). The text generation unit 40 then outputs the text to the explanatory text generation unit 44.

[0120] As shown in Figure 14, the process in step S1302 generates the text "Woman with a white sun visor" from the image input in the process in step S1301. Also, as shown in Figure 15, the text "Group of people sitting in the stands at a sporting event" is generated.

[0121] As a learning model 41, for example, CoCa, a text generation model that generates text from images, can be used.

[0122] Here, the text generation unit 40 generates text that describes the content of the image, but it does not generate text that includes the player's name. For example, if the text "Woman in a white visor" is generated, this text includes the abstract noun "woman." Such a descriptive text containing an abstract noun about a person is insufficient as an explanation if "woman" is a player, and it may raise questions for the listener 100 who are listening to the audio explanation of this text.

[0123] In this case, by adding a face recognition function to the text generation unit 40, it becomes possible to identify players in the image. However, adding a face recognition function requires prior training, and the accuracy of player recognition cannot be guaranteed.

[0124] Therefore, in the classification unit 42 described later, a learning model 43, which is a multimodal model of images and text, is used to determine the class of the player included in the image as the classification result (a / b). Then, in the explanatory text generation unit 44 described later, the abstract nouns of people included in the text are replaced with the proper nouns of the players indicated by the classification result (a / b), and an explanatory text is generated.

[0125] The classification unit 42 inputs video footage and also inputs pre-set descriptive text indicating the class of each player (class description text (descriptive text indicating player A's class a, and descriptive text indicating player B's class b)).

[0126] The classification unit 42 uses the learning model 43 to determine the class of the player (Player A or Player B (Player A / Player B)) included in each image that makes up the video, based on the explanatory text, as the classification result (a / b) (step S1303). The classification unit 42 then outputs the classification result (a / b) to the explanatory text generation unit 44.

[0127] Specifically, the classification unit 42 uses the learning model 43 to obtain feature vectors for the descriptive text indicating class a for player A and the descriptive text indicating class b for player B, as well as the feature vector of the image. The classification unit 42 then calculates the similarity between the feature vectors of the descriptive texts for players A and B and the feature vector of the image, and determines the class of the player (player A / player B) in the descriptive text with the highest similarity as the classification result (a / b).

[0128] Classification result (a) indicates that the image is more likely to contain player A than player B, and classification result (b) indicates that the image is more likely to contain player B than player A.

[0129] When using the English version of the learning model 43, pre-set descriptions indicating each player's class are entered, such as "a tennis player wearing a navy shirt" for player A's class and "a tennis player wearing a white shirt" for player B's class.

[0130] As shown in Figure 14, the process in step S1303 determines, for example, a classification result (a) indicating the class of player A included in the image, based on the image input in the process in step S1301.

[0131] The learning model 43 used is CLIP (Contrastive Language-Image Pre-training), a multimodal model of images and text. Because CLIP handles images and text in the same feature space, it can classify images without training by specifying class features in the description, thus enabling zero-shot classification.

[0132] The explanatory text generation unit 44 receives text from the text generation unit 40 and classification results (a / b) from the classification unit 42. The explanatory text generation unit 44 then searches for abstract nouns of people contained in the text by referring to table 45 (step S1304). The explanatory text generation unit 44 then determines whether or not there are abstract nouns of people in the text (whether or not they are included) (step S1305).

[0133] Furthermore, the explanatory text generation unit 44 may search for pre-defined abstract nouns of people contained in the text without using table 45.

[0134] If the explanatory text generation unit 44 determines in step S1305 that the text contains an abstract noun referring to a person (step S1305:Y), it proceeds to step S1306. On the other hand, if the explanatory text generation unit 44 determines that the text does not contain an abstract noun referring to a person (step S1305:N), it proceeds to step S1309.

[0135] As shown in Figure 12(1), Table 45 stores predefined abstract nouns of people, such as "athlete" and "woman".

[0136] As shown in Figure 14, if the text is "woman in a white sun visor," then "woman" is stored in Table 45, and in steps S1304 and S1305(Y), it is determined that the text contains an abstract noun referring to a person.

[0137] Furthermore, as shown in Figure 15, if the text is "a group of people sitting in the stands at a sporting event," referring to Table 45 will determine in steps S1304 and S1305(N) that the text does not contain any abstract nouns about people.

[0138] The explanatory text generation unit 44, moving from step S1305(Y), reads the proper noun of the player corresponding to the classification result (a / b) from table 46 if the text contains an abstract noun of a person (step S1306).

[0139] As shown in Figure 12(2), Table 46 consists of multiple sets of data (for each classification result), with each set containing a classification result and the player's proper name. In this example, Table 46 stores classification result (a) and the corresponding player's proper name, "Player A," and classification result (b) and the corresponding player's proper name, "Player B."

[0140] In the example in Figure 14, in step S1306, "Player A" is read from table 46 as the proper noun of the player corresponding to classification result (a).

[0141] The explanatory text generation unit 44 replaces the abstract nouns of people in the text searched in step S1304 with the proper nouns of players read in step S1306 (step S1307), and sets the replaced text as an explanatory text about the scene (step S1308).

[0142] Furthermore, if Table 46 is not used in step S1306, the explanatory text generation unit 44 may replace the abstract nouns of people in the text with the pre-set proper nouns of players corresponding to the classification result (a / b).

[0143] In the example in Figure 14, during step S1308, the abstract noun "woman" of a person contained in the text "woman in a white visor" is replaced with the proper noun "Player A" of the player corresponding to the classification result (a), and the replacement explanatory text "Player A in a white visor" is set.

[0144] On the other hand, the explanatory text generation unit 44, moving from step S1305(N), sets the text as is in the explanatory text if the text does not contain any abstract nouns of people (step S1309).

[0145] In the example in Figure 15, during step S1309, the text "A group of people sitting in the stands at a sporting event" is set as the explanatory text.

[0146] Then, the explanatory text generation unit 44 outputs the explanatory text set in step S1308 or step S1309 (step S1310).

[0147] As described above, according to the explanatory text generation device 1 of the embodiment of the present invention, the text generation unit 40 of the scene description unit 12 generates text from an image using a learning model 41. The classification unit 42 uses a learning model 43 to determine the class of the player included in the image as a classification result (a / b) based on a pre-set explanatory text indicating the class of each player.

[0148] The explanatory text generation unit 44, if the text contains abstract nouns of people, replaces the abstract nouns of people in the text with the proper nouns of the players corresponding to the classification result (a / b), and sets the replaced text as the explanatory text. On the other hand, if the text does not contain abstract nouns of people, the explanatory text generation unit 44 sets the text as is as the explanatory text.

[0149] This enables the generation of commentary text from video footage of sports broadcasts in an audio commentary service that provides commentary synchronized with the broadcast program, specifically for tennis. In particular, the scene description unit 12 can automatically generate commentary text about the players' scenes from the broadcast program's video footage, and since this commentary text will include the players' proper names, the quality of the commentary text can be improved. Furthermore, the scene description unit 12 can also automatically generate commentary text about scenes other than those involving players.

[0150] The explanatory text generated by the explanatory text generation device 1 is then delivered as explanatory audio to the mobile terminal 105, allowing the viewer 100 to listen to the explanatory audio about the scene in sync with the audio of the tennis sports broadcast.

[0151] <Example of explanatory text> Next, we will explain an example of explanatory text. Figure 16 is a diagram showing an example of explanatory text generated by the explanatory text generation device 1, and it shows explanatory text for the period from the start to the end of one point of tennis play. In the multiple explanatory texts shown in Figure 16, time progresses from top to bottom.

[0152] In the example shown in Figure 16, first, the scene description unit 12 generates a descriptive text about the scene, "Player A wearing a white visor," from the top right image. Then, the person / ball position processing unit 10 generates descriptive texts about the players' actions, "Player A serving," "Player B receiving," and "Player A shooting to the opposite side," from the overhead image obtained by transforming the second image from the top on the right.

[0153] Next, the person motion processing unit 11 generates a descriptive text about the player's actions, "Player B, dashing," from the third image from the top on the right side (skeletal information). Then, the person / ball position processing unit 10 generates descriptive texts about the players' actions, such as "Player B, shooting," "Player A, shooting to the opposite side," and "Player A moving forward."

[0154] Next, the manual input unit 13 generates explanatory text: "Player A's hit was a goal" and "Player B is unable to move."

[0155] In this way, explanatory text is generated for the time-series images that make up the video input to the explanatory text generation device 1 by the person / ball position processing unit 10, the person motion processing unit 11, the scene description unit 12, and the manual input unit 13, respectively.

[0156] In other words, the explanatory text generation device 1 generates explanatory texts about the players' actions, the players' movements, and the scenes, so that the match situation and scenes that cannot be understood from the video and audio of the television broadcast can be conveyed to a large number of viewers 100, including the visually impaired. This improves the accessibility of video content for viewers 100.

[0157] <Experimental Results> Next, I will explain the experimental results. Figure 17 shows the experimental results.

[0158] Figure 17(1) shows the detection accuracy of the person / ball position processing unit 10. 500 images were extracted from untrained match footage, and the accuracy was calculated using these images, with the detection of a player or ball considered correct.

[0159] As a result, the detection accuracy was 97.5% precision, 100% recall, and 98.7% F-score for players, and 91.6% precision, 71.4% recall, and 80.3% F-score for balls.

[0160] These experimental results show that while there were cases where the ball was lost in images at the moment of high-speed movement, overall, false detections were relatively few.

[0161] Furthermore, the average processing time for the person / ball position processing unit 10 was 40.5 msec per frame. This corresponds to 24.7 frames / sec, indicating that processing is possible for the majority of frames of broadcast camera footage shot at 30 frames / sec.

[0162] Figure 17(2) shows the detection accuracy of the human motion processing unit 11. 200 sequences of images different from those used for training were prepared, and the detection accuracy for the five types of motions mentioned above was calculated using these images.

[0163] As a result, the detection accuracy for the five types of operations is shown in Figure 17(2), with the average being a precision of 69.9%, a recall of 74.9%, and an F-score of 72.3%.

[0164] These experimental results show that the learning model 33 (ST-GCN in this experiment) can detect athlete movements with relatively high accuracy by learning the time-series changes in skeletal information for each athlete, thereby suppressing the influence of noise in the images.

[0165] Figure 17(3) shows the detection accuracy of the scene description unit 12. Using approximately 300 images (images containing players) from three matches, the detection accuracy was calculated when classifying two players in each match using the learning model 43 (CLIP in this experiment). The class descriptions were related to the color of the players' clothing, such as "a tennis player wearing a white shirt."

[0166] As a result, the accuracy rates for each match are shown in Figure 17(3), with the average accuracy rate being 85.9%.

[0167] These experimental results show that the learning model 43 can be used to classify players with high accuracy. However, the precision for the third match was 70.8%, which is slightly lower than the other matches. This is likely because the players' clothing was similar in the third match.

[0168] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments and can be modified in various ways without departing from the technical concept.

[0169] In the above embodiment, the explanatory text generation device 1 was configured to generate explanatory text for a tennis sports broadcast. However, the present invention may also be configured to generate explanatory text for sports broadcasts other than tennis, such as table tennis, badminton, and other racket sports.

[0170] Furthermore, in the above embodiment, the explanatory text generation device 1 is applicable not only to generating explanatory text for singles matches but also for doubles matches in tennis sports broadcasts, and is also applicable to generating explanatory text for wheelchair tennis.

[0171] Furthermore, in the above embodiment, the scene description unit 12 of the explanatory text generation device 1 is configured to output the generated explanatory text (send it to the speech synthesis server 106). Alternatively, the operator of the explanatory text generation device 1 may check the explanatory text generated by the scene description unit 12, and if it is determined that there are no problems, the scene description unit 12 may output the explanatory text.

[0172] Furthermore, in the above embodiment, the person / ball position processing unit 10 of the explanatory text generation device 1 generates and outputs explanatory text about the players' actions, such as "Player A serves," "Player B receives," etc., "Player A is approaching the net." In addition, the person motion processing unit 11 generates and outputs explanatory text about the players' actions, such as "Player A dashes," "Player B gets into position," "Player A takes a shot," etc.

[0173] In contrast, the person / ball position processing unit 10 and the person motion processing unit 11 may stop processing (generating or outputting) the specified explanatory text in accordance with the operator's operation to specify a particular explanatory text (or according to a pre-set specification). The same applies to the scene depiction unit 12.

[0174] For example, if the operator specifies the explanatory texts "Player A, shot" and "Player B, shot" to the character motion processing unit 11, the character motion processing unit 11 stops the process of generating (or outputting) these explanatory texts. In this case, the explanatory audio corresponding to the explanatory texts "Player A, shot" and "Player B, shot" generated by the character motion processing unit 11 will not be delivered to the mobile terminal 105, and instead, explanatory audio corresponding to the explanatory texts generated by the other character / ball position processing unit 10, character motion processing unit 11, scene depiction unit 12, and manual input unit 13 will be delivered.

[0175] As a result, if the accuracy of the generation process of explanatory texts "Player A, shot" and "Player B, shot" by the person motion processing unit 11 is low, viewers 100 will not have to listen to inaccurate match information. The person / ball position processing unit 10 also generates explanatory texts "Player A, shot" and "Player B, shot," and these explanatory audios are delivered to the mobile terminal 105, so viewers 100 can listen to the match information obtained from the players' continuous play and actions.

[0176] Furthermore, in the above embodiment, the person / ball position processing unit 10, the person motion processing unit 11, and the scene depiction unit 12 of the explanatory text generation device 1 each generate and output explanatory text, and explanatory audio corresponding to the explanatory text is delivered to the mobile terminal 105. In contrast, the person / ball position processing unit 10, the person motion processing unit 11, and the scene depiction unit 12 may each stop the aforementioned processing (generation processing or output processing) in accordance with an operator's operation to stop processing of one, two, or three of the components of the person / ball position processing unit 10, the person motion processing unit 11, and the scene depiction unit 12 (or according to a pre-set specification).

[0177] For example, if the operator stops the processing of the character motion processing unit 11, the character motion processing unit 11 stops the process of generating (or outputting) all explanatory text. In this case, the explanatory audio corresponding to the explanatory text generated by the character / ball position processing unit 10, the scene description unit 12, and the manual input unit 13, excluding the character motion processing unit 11, will be delivered to the mobile terminal 105. As a result, if, for example, the accuracy of the explanatory text generation process by the character motion processing unit 11 is low, the viewer 100 will not have to listen to the inaccurate match situation.

[0178] Furthermore, in the above embodiment, the system providing the audio description service shown in Figure 1 can be applied to video media services such as broadcasting and the internet. It can also be applied to various other uses, such as security systems represented by emergency alerts via mobile apps.

[0179] Furthermore, a standard computer can be used as the hardware configuration for the explanatory text generation device 1 according to the embodiment of the present invention. The explanatory text generation device 1 is composed of a computer equipped with a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, and an interface.

[0180] The functions of the person / ball position processing unit 10, the person motion processing unit 11, the scene description unit 12, and the manual input unit 13, all of which are provided in the explanatory text generation device 1, are realized by having the CPU execute a program that describes these functions.

[0181] These programs are stored in the aforementioned storage medium and are read and executed by the CPU. These programs can also be stored and distributed on storage media such as magnetic disks (HDDs, etc.), optical disks (CD-ROMs, DVDs, etc.), and semiconductor memory (SSDs, etc.), and can be transmitted and received over a network. [Explanation of Symbols]

[0182] 1. Explanatory text generation device 10. Person / Ball Position Processing Unit 11. Human Motion Processing Unit 12 Scene description section 13 Manual input section 20 Object detection unit 21, 31, 33, 41, 43 Learning Models 22 Coat detection unit 23 Projection Transformation Unit 24 Motion Information Generation Unit 25,35,44 Explanatory text generation unit 30 Skeleton detection unit 32 Motion detection unit 34. Threshold Processing Unit 40 Text generation unit 42 Classification Department Tables 45, 46 100 viewers 101 Broadcasting Transmitter 102 Broadcast receiving equipment 103 Commentary Audio Production and Distribution Device 104 Application Server 105 Mobile devices 106 Speech Synthesis Server 107 Distribution Server

Claims

1. In a commentary text generation device that generates commentary text corresponding to the commentary audio when providing viewers with commentary audio synchronized with video footage of a racket sports broadcast, An explanatory text generation device characterized by comprising a person and ball position processing unit that detects the image coordinates of a player and the image coordinates of a ball contained in an image that constitutes the aforementioned video, and uses the image coordinates of the court contained in the image to project the image into a true overhead view image to determine the position of the player corresponding to the player's image coordinates and the position of the ball corresponding to the ball's image coordinates, generates a first explanatory text about the player's play based on the change in the player's position, and generates a second explanatory text about the player's play based on the change in the ball's position.

2. In the explanatory text generation device according to claim 1, The aforementioned person / ball position processing unit, For each image constituting the aforementioned video, a projection transformation unit uses the image coordinates of the court to project the image into a true overhead view image, and determines the actual coordinates of the player, the ball, and the court included in the true overhead view image, corresponding to the image coordinates of the player, the ball, and the court. A motion information generation unit generates player movement information showing the change in the player's position on the court based on the actual coordinates of the player and the actual coordinates of the court obtained by the projection transformation unit, and generates ball movement information showing the change in the ball's position on the court based on the actual coordinates of the ball and the actual coordinates of the court. An explanatory text generation unit generates a first explanatory text based on the player's movement information generated by the movement information generation unit, and generates a second explanatory text based on the ball's movement information, A device for generating explanatory text, characterized by having the following features.

3. In a commentary text generation device that generates commentary text corresponding to the commentary audio when providing viewers with commentary audio synchronized with video footage of a racket sports broadcast, The system includes a human motion processing unit that detects the skeleton of a player from the images constituting the video, generates skeletal information of the player, detects the player's movements based on the skeletal information of the player, and generates explanatory text about the player's movements. The aforementioned human motion processing unit is A motion detection unit detects the movement of the player for each image constituting the video, based on the player's skeletal information in a predetermined number of consecutive images constituting the video, using a predetermined learning model. A threshold processing unit determines that if the number of images in which the player's actions detected by the action detection unit are identical within a predetermined time window of a set number of images exceeds a predetermined threshold, the action is determined to be a confirmed action of the player. Based on the determined actions of the player as determined by the threshold processing unit, an explanatory text generation unit generates an explanatory text about the player's actions. A device for generating explanatory text, characterized by having the following features.

4. In a commentary text generation device that generates commentary text corresponding to the commentary audio when providing viewers with commentary audio synchronized with video footage of a racket sports broadcast, An explanatory text generation device characterized by comprising a scene description unit that generates text representing the content of images constituting the video using a predetermined learning model, determines the class of the image from a pre-set explanatory text indicating the class of each player using a predetermined multimodal learning model, replaces abstract nouns of people included in the text with proper nouns of players corresponding to the class, and generates an explanatory text relating to the scene description.

5. In the explanatory text generation device according to claim 4, The aforementioned scene depiction section is, A text generation unit generates text representing the content of each image that makes up the video, using the learning model. A classification unit that, using the multimodal learning model, determines the class with the highest similarity as the classification result for each image constituting the video, based on the similarity between the respective feature vectors in the descriptive text indicating the class of each player and the feature vector of the image; An explanatory text generation unit identifies pre-defined abstract nouns of people from the text generated by the text generation unit, replaces the abstract nouns of people in the text with proper nouns of players corresponding to the classification results determined by the classification unit, and generates the resulting text as an explanatory text relating to the scene description. A device for generating explanatory text, characterized by having the following features.

6. A program for causing a computer to function as the explanatory text generation device described in claim 1.

Images