Learning device for generating model for determining content of character superimposition of video, character superimposition determination device and program
The learning device and determination device automatically identify and generate explanatory text for superimposed text in live broadcasts, addressing operational challenges and ensuring timely delivery of commentary audio.
Patent Information
- Application Number
- JP2023076378
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-03
- Publication Date
- 2024-11-15
AI Technical Summary
Existing audio commentary systems for live broadcasts, particularly sports events, face challenges in automatically generating explanatory text for superimposed text due to high operational burden on operators and the inability to handle multiple events simultaneously, leading to delays and inaccuracies in providing real-time explanations.
A learning device and determination device that generate and utilize a model to automatically identify superimposed text in videos by binarizing pixel regions, rotating images, and using consistency checks to ensure accurate and timely generation of explanatory text.
Stable and quick determination of superimposed text content in commentary audio services, reducing operator burden and enabling real-time explanation delivery to users, including visually impaired individuals.
Smart Images

Figure 2024160884000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a learning device that generates a model for determining the content of subtitles in video, a subtitle determination device that determines the content of subtitles in video, and the programs for these devices, and in particular to technology used in audio distribution services aimed at many users, including able-bodied people who watch while doing other things, in technology to assist visually impaired people in viewing. [Background technology]
[0002] Broadcasting is a medium that provides information to the viewer's visual and auditory senses. However, the video information provided by broadcasting is hardly understood by the visually impaired. Therefore, a commentary audio service has been proposed as a service to complement video information (see, for example, Patent Document 1).
[0003] The audio commentary service is, for example, a service that broadcasts a program and provides audio commentary of the broadcast program to viewers. However, for live broadcasts of sports broadcasts and the like, the audio commentary service has hardly been realized due to cost issues, technical difficulties, and the like.
[0004] In light of this situation, in order to realize an audio commentary service for live broadcast programs, a system for producing and distributing audio commentary has been proposed, as described in Patent Application Publication No. 2022-082878, which was unpublished at the time of filing this patent application and was made by the same applicant of this patent application.
[0005] This commentary audio production and distribution system uses image recognition technology to automatically add commentary text and expand the sources of information that provide commentary audio, thereby making the system itself scalable and versatile, and aiming to improve the quality of commentary audio services.
[0006] In addition, a technology for use in the commentary audio service has also been proposed in Patent Application Publication No. 2023-016933, which was unpublished at the time of filing this patent application and was made by the same applicant of this patent application. This technology improves the quality of the commentary audio service by individually setting the amount of information of the commentary audio according to user operations.
[0007] On the other hand, as a general-purpose image recognition technology that recognizes characters from an image, an optical character recognition engine called Tesseract that uses optical character recognition (OCR) is known (see, for example, Non-Patent Document 1). This optical character recognition engine has been updated since 2007 and continues to be used to this day (see, for example, Non-Patent Document 2). [Prior art documents] [Patent documents]
[0008] [Patent Document 1] JP 2017-203827 A [Non-patent literature]
[0009] [Non-Patent Document 1] Ray Smith, “An Overview of the Tesseract OCR Engine”, ICDAR, pp.629-633, 2007 [Non-Patent Document 2] “tesseract OCR / tesseract”, [online], GitHub,Inc., [Retrieved April 6, 2023], Internet<URL:https: / / github.com / tesseract-ocr / tesseract> Summary of the Invention [Problem to be solved by the invention]
[0010] The above-mentioned commentary audio production and distribution system provides a commentary audio service while reducing costs and technical difficulties, but there is a problem in that the commentary text is often generated manually by an operator, which places limitations on operation.
[0011] For example, when a large number of events to which explanations should be added occur at once, the manual process of generating explanation text by an operator cannot adequately handle the situation, and places a heavy burden on the operator.
[0012] In addition, operators are likely to miss information that is displayed for a short time, such as superimposed text showing the ball speed in a baseball broadcast, and if they miss it, they will not be able to provide commentary to the user.
[0013] Thus, there has been a demand for a mechanism that reduces the burden on the operator of manually generating commentary text and provides users with commentary on video scenes in broadcast programs in real time in accordance with the video.
[0014] Therefore, the present invention has been made to solve the above-mentioned problems, and its object is to provide a learning device, a character superimposition determination device, and a program that can stably and quickly determine the content of character superimpositions when providing commentary audio for sports broadcasts, etc. to users through a commentary audio service. [Means for solving the problem]
[0015] In order to solve the above problem, the learning device of claim 1 is a learning device that generates a model for determining the content of a subtitle, which is used when generating explanatory text for a video including subtitles, and is characterized in that it comprises: a binarization unit that inputs an image of the subtitle as a subtitle image, binarizes each pixel that constitutes the subtitle image so as to distinguish between an area of characters contained in the subtitle image and an area excluding the characters, and generates a binary image; and a learning unit that uses the binary image and the correct label generated by the binarization unit to learn the relationship between the input data and the output data in a model in which the binary image is used as input data and the label for identifying characters contained in the binary image is used as output data, thereby generating a super content determination model, the learning unit being characterized in that the label assigned to the subtitle image, the label for identifying characters contained in the subtitle image, is used as a correct label, and the learning unit uses the binary image generated by the binarization unit to learn the relationship between the input data and the output data in a model in which the binary image is used as input data and the label for identifying characters contained in the binary image is used as output data.
[0016] In addition, the learning device of claim 2 is characterized in that, in the learning device described in claim 1, it further includes a bulking unit that bulks up the superimposed character image by rotating the superimposed character image at a random angle within a predetermined maximum angle, and the binarization unit generates the binary image for each of the multiple superimposed character images bulked by the bulking unit.
[0017] Furthermore, the character superimposition determination device of claim 3 is a character superimposition determination device that determines the content of a video including a character superimposition in order to generate explanatory text for the video, and is characterized in that it comprises: an image extraction unit that inputs a video and extracts an image of the character superimposition as a character superimposition image from a predetermined area of the video; a binarization unit that binarizes each pixel that constitutes the character superimposition image to generate a binary image so as to distinguish between an area of characters contained in the character superimposition image extracted by the image extraction unit and an area excluding the characters; and a determination unit that uses a superimposition content determination model generated by the learning device described in claim 1 or 2 to determine a label for identifying the characters contained in the binary image generated by the binarization unit from the binary image generated by the binarization unit.
[0018] In addition, the character superimposition determination device of claim 4 is characterized in that, in the character superimposition determination device described in claim 3, the image extraction unit calculates an average brightness value of a predetermined area of the image, and if the average brightness value is equal to or less than a predetermined threshold, determines that the image contains the character superimposition, extracts the character superimposition image from the area, and if the average brightness value is greater than the threshold, determines that the image does not contain the character superimposition.
[0019] In addition, the character superimposition determination device of claim 5 is characterized in that, in the character superimposition determination device described in claim 3, it further includes an explanatory text generation unit that inserts characters corresponding to the label determined by the determination unit into a preset template and generates the explanatory text to explain the scene in the video.
[0020] The subtitle determination device of claim 6 is the subtitle determination device of claim 3, further comprising: a matching unit that inputs a first label, with the label determined by the determination unit as a first label and a label input immediately before the first label as a second label, and if it is determined that the first label and the second label are different, determines a consistency check OK and outputs the first label, and if it is determined that the first label and the second label are the same, determines a consistency check NG; and a commentary text generation unit that inserts characters corresponding to the first label output by the matching unit into a preset template and generates the commentary text to explain a scene in the video.
[0021] The subtitle determination device of claim 7 is the subtitle determination device of claim 3, further comprising: a matching unit that determines an elapsed time from a predetermined time point to a time point when the third label is input, using the label determined by the determination unit as a third label when the label is input, and determines a consistency check OK and outputs the third label if it is determined that the elapsed time has exceeded a preset threshold time, and determines a consistency check NG if it is determined that the elapsed time has not exceeded the threshold time; and a commentary text generation unit that inserts characters corresponding to the third label output by the matching unit into a preset template and generates the commentary text for explaining a scene in the video, wherein the predetermined time point for determining the elapsed time by the matching unit is a time point when the commentary text generation unit generates the commentary text for characters corresponding to the label output by the matching unit immediately before the third label.
[0022] Furthermore, the program of claim 8 is characterized in that the computer constituting a learning device that generates a model for determining the content of the subtitle used when generating explanatory text for a video including the subtitle is made to function as a binarization unit that inputs an image of the subtitle as a subtitle image, binarizes each pixel constituting the subtitle image so as to distinguish between areas of characters contained in the subtitle image and areas excluding the characters, and generates a binary image, and a learning unit that uses the binary image generated by the binarization unit and the correct label, which is a label assigned to the subtitle image and is used to identify characters contained in the subtitle image, as a correct label, to learn the relationship between the input data and the output data in a model in which the binary image is used as input data and the label for identifying characters contained in the binary image is used as output data, and generates a super content determination model.
[0023] In addition, the program of claim 9 is characterized in that the computer constituting a subtitle determination device that determines the content of a video including subtitles in order to generate explanatory text for the video includes an image extraction unit that inputs a video and extracts an image of the subtitle as a subtitle image from a predetermined area of the video, a binarization unit that binarizes each pixel that constitutes the subtitle image so as to distinguish between an area of characters contained in the subtitle image extracted by the image extraction unit and an area excluding the characters, and a determination unit that uses a superimposition content determination model generated by the learning device of claim 1 or 2 to determine a label for identifying the characters contained in the binary image generated by the binarization unit from the binary image generated by the binarization unit. Effect of the Invention
[0024] As described above, according to the present invention, when providing a commentary audio of a sports broadcast or the like to a user through a commentary audio service, the contents of the superimposed text can be determined stably and quickly. [Brief description of the drawings]
[0025] [Figure 1] 1 is a schematic diagram illustrating an example of the overall configuration of a commentary audio production and distribution system including a learning device and a superimposition determination device according to an embodiment of the present invention. [Diagram 2] 1 is a block diagram showing an example of the configuration of a learning device according to an embodiment of the present invention. [Diagram 3] 13 is a flowchart showing an example of a ball speed learning process of the learning device. [Figure 4] FIG. 11 is a diagram for explaining an example of processing (step S304) of a learning unit. [Diagram 5] FIG. 13 is a schematic diagram showing an example of the structure of a super content judgment model for ball speed. [Figure 6] 1 is a block diagram showing an example of the configuration of a superimposition determination device according to an embodiment of the present invention; [Figure 7] 13 is a flowchart showing an example of processing performed by the superimposition determination device; [Figure 8] 13 is a flowchart showing an example of processing by the offense / defense determination unit (step S702). [Figure 9] 11A and 11B are diagrams illustrating an example of processing by an image cutout unit (steps S703 and S704). [Figure 10] 1 is a diagram showing an example of a video, and FIG. 2 is a diagram showing an example of a subtitle. [Figure 11] FIG. 13 is a diagram for explaining an example of processing (step S706-1) of a determination unit. [Figure 12] 13 is a flowchart showing an example of a ball speed consistency check by a consistency section (step S707-1). [Figure 13] 13 is a flowchart showing an example of a ball count (strike) consistency check by a consistency section (step S707-2). [Figure 14] 13 is a flowchart showing an example of a ball count (out) consistency check by a consistency unit (step S707-2). [Figure 15] 13 is a flowchart showing an example of a batting order consistency check by a consistency section (step S707-3). [Figure 16] 11A and 11B are diagrams illustrating an example of processing by an explanatory text generating unit. [Figure 17] FIG. 13 is a diagram showing experimental results. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0026] Hereinafter, the embodiment of the present invention will be described in detail with reference to the drawings. The present invention focuses on real-time display information added to the video of a program. For example, in a baseball broadcast of a sports broadcast program, a trained model is generated using the superimposed text (ball speed, ball count, and batting order) displayed on the video as training data, and the trained model is used to perform image recognition of the video to determine the content of the superimposed text. Then, from the determined content of the superimposed text, a commentary text for explaining the scene in the video is generated, and the commentary audio is provided to the user.
[0027] This makes it possible to stably and quickly determine the content of superimposed text when providing users with commentary audio for a sports broadcast or the like through a commentary audio service.
[0028] [Audio commentary production and distribution system] First, a commentary audio production and distribution system for realizing a commentary audio service will be described. Fig. 1 is a schematic diagram illustrating an example of the overall configuration of a commentary audio production and distribution system including a learning device and a superimposition determination device according to an embodiment of the present invention.
[0029] This commentary audio production and distribution system 6 is configured to include a commentary audio production device 1, a plurality of information sources 2, a voice synthesis device 3, a distribution device 4, and a mobile terminal 5.
[0030] The commentary audio production device 1 is a device that generates commentary text for each utterance when producing commentary audio for a live-streamed sports program, etc. The commentary audio production device 1 inputs real-time data corresponding to the game situation, etc. of the live-streamed sports program from each of a plurality of information sources 2.
[0031] Then, the commentary audio production device 1 analyzes the data according to a data format unique to the information source 2 from which the data is input, extracts text elements from the data, assigns labels to the text elements, and stores the labeled text elements in an information management table.
[0032] Here, a text element is one or more elements that make up the commentary text to be generated (text to be spoken). A label is attribute information for identifying the content of a text element.
[0033] In order to generate commentary text for an utterance, the commentary audio production device 1 reads out labeled text elements from the information management table according to the utterance definition data defined in the template. Then, the commentary audio production device 1 generates utterance data, which is JSON data including the labeled text elements and playback times, and generates commentary text.
[0034] The commentary audio production device 1 outputs the utterance data and commentary text to the voice synthesis device 3 for each utterance, and also outputs the utterance data to the distribution device 4.
[0035] The information source 2 may consist of multiple information sources for each sport. As shown in Fig. 1, the multiple information sources for baseball include information source 2-1 that distributes Olympic-related data according to the ODF specification, information source 2-2 that distributes professional baseball-related data according to the BIS specification, and information source 2-3 that distributes baseball-related data according to a predetermined specification, which is generated by the operation of a manual input tool by an operator watching the broadcast program.
[0036] Furthermore, there is information source 2-4 that automatically and in real time determines the content of the subtitles by image recognition from the video using an automatic input tool that uses a pre-generated superimposition content determination model without manual operation by an operator, generates explanatory text reflecting the content of the subtitles, generates speech data consisting of JSON data, and distributes the speech data as real time data. This information source 2-4 corresponds to the learning device 10 and the subtitle determination device 20 according to an embodiment of the present invention.
[0037] Furthermore, when the commentary audio production device 1 inputs speech data, etc. from this information source 2-4, it outputs the speech data and commentary text to the voice synthesis device 3 and outputs the speech data to the distribution device 4 without performing the process of generating the utterance data and commentary text.
[0038] The voice synthesizer 3 receives the speech data and commentary text from the commentary audio production device 1, and generates a voice file by generating synthetic voice from the commentary text using existing technology. The voice synthesizer 3 then outputs the speech data and the voice file to the distribution device 4.
[0039] The distribution device 4 inputs the speech data from the commentary audio production device 1, and also inputs the speech data and the audio file from the audio synthesis device 3, and distributes the audio file of the same speech data to the mobile terminal 5 together with the speech data.
[0040] The mobile terminal 5 receives the speech data and the voice file distributed from the distribution device 4, and plays back the voice of the target speech using the speech data and the voice file.
[0041] The commentary audio production and distribution system 6 shown in Figure 1 can be applied to video media services such as broadcasting and the Internet, and can also be applied to a variety of uses, such as security systems, represented by emergency news on mobile apps.
[0042] Hereinafter, an example of generating a trained model for determining the content of the superimposed text included in a video of a baseball broadcast program among sports broadcast programs will be described. Also, an example of determining the content of the superimposed text included in the video of a baseball broadcast program using the trained model and generating a commentary text reflecting the content will be described. Note that the characters include numbers, and for the video of the following baseball broadcast program, the numbers in the superimposed text will be determined.
[0043] [Learning device 10] First, a learning device 10 according to an embodiment of the present invention will be described below. Fig. 2 is a block diagram showing an example of the configuration of the learning device 10 according to the embodiment of the present invention.
[0044] The learning device 10 is a device that generates a superimposed content determination model, which is a trained model for determining the content of superimposed text included in a video. The learning device 10 includes bulk padding units 11-1, 11-2, and 11-3, binarization units 12-1, 12-2, and 12-3, learning units 13-1, 13-2, and 13-3, and a storage unit 14.
[0045] The bulking unit 11-1, the binarization unit 12-1, and the learning unit 13-1 generate a superimposed content determination model for ball speed using the superimposed text image for ball speed and the label (correct label) assigned to the superimposed text image for ball speed. The correct label for the superimposed text image for ball speed is one of the labels [0] to [9] and [None].
[0046] A text superimposition image for ball speed is an image that corresponds to a single digit in the text superimposition used to display the ball speed in the video of a live baseball game. For example, the text superimposition in the video shown in FIG. 10(1) below shows the ball speed as "142km / h." A text superimposition image for ball speed corresponds to an image that contains a single digit corresponding to each of these numbers [1][4][2].
[0047] In addition, the bulking unit 11-2, the binarization unit 12-2, and the learning unit 13-2 generate a ball count superimposed content determination model using the ball count superimposed image and the label (correct label) assigned to the ball count superimposed image. The correct label of the ball count superimposed image is one of the labels [0] to [3] and [None].
[0048] There are three ball counts: ball (B), strike (S) and out (O).
[0049] A ball count text superimposition image is an image that corresponds to a single digit in the ball count display used in the text superimposition included in the video of a baseball broadcast. For example, the text superimposition included in the video shown in FIG. 10(1) described later displays the ball count as [2] for ball (B), [2] for strike (S), and [0] for out (O). A ball count text superimposition image corresponds to an image that includes a single digit corresponding to each of these numbers [2][2][0].
[0050] In addition, the bulking unit 11-3, the binarization unit 12-3, and the learning unit 13-3 generate a batting order superimposition content determination model using the batting order superimposition image and the label (correct label) assigned to the batting order superimposition image. The correct label of the batting order superimposition image is one of the labels [1] to [9] and [None].
[0051] A batting order text superimposition image is an image that corresponds to a single digit in the batting order display used in the text superimposition included in the video of a baseball broadcast. For example, the text superimposition included in the video shown in FIG. 10(1) described later displays the batting order [4] [5] [6]. A batting order text superimposition image corresponds to an image that includes a single digit corresponding to each of these numbers [4] [5] [6].
[0052] The text superimposition image for ball speed used to generate a super content judgment model for ball speed, the text superimposition image for ball count used to generate a super content judgment model for ball count, and the text superimposition image for batting order used to generate a super content judgment model for batting order are cut out from text superimposition images included in, for example, baseball broadcast footage.
[0053] In this way, the model generated by the learning device 10 is not one model corresponding to all of the ball speed, ball count, and batting order, but a total of three models corresponding to each of them. In other words, a distinction is made so that a super content judgment model for ball speed is generated for the ball speed, a super content judgment model for ball count is generated for the ball count, and a super content judgment model for batting order is generated for the batting order.
[0054] This is because the fonts and sizes of ball speed, ball count, and batting order are different, and the ranges are also different. For example, the range of each digit of ball speed is 0 to 9, the range of ball count is 0 to 3, and the range of batting order is 1 to 9.
[0055] Therefore, by using separate models, it is possible to improve the accuracy of judging ball speed, ball count, and batting order compared to using a single model.
[0056] (Generation of a super content judgment model for ball speed) First, a process of generating a super content judgment model for ball speed by the bulking unit 11-1, the binarization unit 12-1, and the learning unit 13-1 will be described. FIG.
[0057] With reference to Figures 2 and 3, the learning device 10 inputs a ball speed text caption image containing a single digit (0 to 9) used for each of the multiple digits of the ball speed displayed in the text caption included in the video, and inputs a label (correct label, a label indicating any one of [0] to [9]) assigned to the ball speed text caption image (step S301).
[0058] In this case, the learning device 10 inputs the above-mentioned data as positive example data to handle the case where the display of superimpositions is set to on (super on). On the other hand, the learning device 10 also inputs negative example data to handle the case where the display of superimpositions is set to off (super off). In other words, the learning device 10 inputs an image in which the ball speed is not reflected and a label assigned to this image (correct label, label indicating [None]) as a superimposition image for ball speed and a correct label.
[0059] The bulking unit 11-1 inputs a ball speed superimposing image including a single digit (0 to 9) or a ball speed superimposing image not including a digit. The bulking unit 11-1 then sets a plurality of random angles within a preset maximum angle, and performs bulking by rotating the ball speed superimposing image at each of the plurality of random angles (step S302).
[0060] The bulking unit 11-1 generates a plurality of bulked post-rotation ball speed superimposition images, and outputs each of these images to the binarization unit 12-1 as a bulked post-rotation ball speed superimposition image.
[0061] For example, 5 degrees is used as the preset maximum angle. This is because in a baseball broadcast program, the numbers representing the ball speed do not invert or fall to the side, and there is relatively little change. Also, the bulking unit 11-1 generates, for example, 1,000 bulked ball speed superimposed images from one ball speed superimposed image.
[0062] This allows efficient learning by adding rotation to the input ball speed superimposed image to increase its volume, and prevents overlearning. Also, in the superimposed image determination device 20 described later, the recognition rate for superimposed images included in unknown video can be improved.
[0063] The binarization unit 12-1 inputs the padded ball speed text image from the padding unit 11-1, and binarizes each pixel constituting the ball speed text image so as to distinguish between areas of characters (numbers in this example) included in the ball speed text image and areas other than characters (step S303).Then, the binarization unit 12-1 generates a ball speed binary image by binarization, and outputs the ball speed binary image to the learning unit 13-1.
[0064] In this way, the binarization unit 12-1 generates a binary image for ball speed, and the learning unit 13-1 performs learning processing using the binary image for ball speed, so that the learning processing can be simplified. The same applies to the binarization units 12-2 and 12-3 and the learning units 13-2 and 13-3 described later.
[0065] The learning unit 13-1 inputs the binary image for ball speed from the binarization unit 12-1, and also inputs the correct label that has been assigned to the superimposed text image for ball speed input by the learning device 10.
[0066] The learning unit 13-1 uses the ball speed binary image and the ground truth label to learn the relationship between the input data and the output data in a model in which the ball speed binary image is input data and the label of the ball speed binary image (ball speed label) is output data, and generates a ball speed super content judgment model as a learned model (step S304).Then, the learning unit 13-1 stores the ball speed super content judgment model (parameters of the ball speed super content judgment model) in the storage unit 14 (step S305).
[0067] Fig. 4 is a diagram for explaining a processing example (step S304) of the learning unit 13-1. As shown in Fig. 4, the binary image for ball speed generated by the binarization unit 12-1 is input to a model (super content judgment model for ball speed to be generated) in which a binary image for ball speed is input data and a label for ball speed is output data, and the label for ball speed is judged and output. The label for ball speed is a label indicating any one of 0 to 9 and None.
[0068] Then, the update unit updates the parameters of the model so that the correct label input by the learning unit 13-1 and the ball speed label output by the model become the same.
[0069] For example, back propagation is used as a learning process for updating the parameters of such a model. As the model, for example, a neural network shown in Fig. 5 described later is used. The model may be any neural network as long as it can learn the relationship between input data and output data by supervised learning.
[0070] Fig. 5 is a schematic diagram showing an example of the structure of a super content judgment model for ball speed. This super content judgment model for ball speed is mainly composed of a Convolutional Neural Network (CNN) and a Feed Forward Neural Network (FFN). In addition, a dropout layer is provided to suppress overlearning.
[0071] Specifically, this super content judgment model for ball speed has three layers: CNN, rectified linear unit (ReLU), and maximum pooling layer, and is further composed of a dropout layer, a flatten layer, FFN and ReLU, and FFN and softmax function layers.
[0072] A binary image of ball speed is input as input data to the input layer. From the output layer, the ball speed label with the maximum score among the ball speed labels corresponding to [0] to [9] and [None] is output as output data. The example in Figure 5 shows that the ball speed label [2] is output.
[0073] The example structure in Figure 5 can be applied not only to the super content judgment model for ball speed, but also to the super content judgment model for ball count and the super content judgment model for batting order. These models have the same structure except for the number of dimensions of the final output layer.
[0074] In this way, the text superimposition image for ball speed is augmented to generate a binary image for ball speed, and a superimposition content determination model for ball speed is generated using the binary image for ball speed and the correct label.
[0075] (Generation of a super content judgment model for ball counting) Next, a process in which the bulking unit 11-2, the binarization unit 12-2, and the learning unit 13-2 generate a super content determination model for ball counting will be described.
[0076] The learning device 10 inputs a ball count text superimposition image including a ball count consisting of single-digit numbers indicating the number of balls, strikes, and outs displayed in the text superimposition included in the video, and also inputs a label assigned to the ball count text superimposition image (a correct answer label, a label indicating any one of [0] to [3]).
[0077] In this case, the learning device 10 inputs the above-mentioned data as positive example data to cope with the case where Super ON is set. On the other hand, the learning device 10 also inputs negative example data to cope with the case where Super OFF is set. In other words, the learning device 10 inputs an image in which the ball count is not reflected and a label assigned to this image (correct label, label indicating [None]) as a superimposed text image for ball count and a correct label.
[0078] The bulking unit 11-2 inputs a ball counting text superimposition image including a single digit (0 to 3) or a ball counting text superimposition image not including a digit. The bulking unit 11-2 then performs the same process as the bulking unit 11-1, and outputs the bulked ball counting text superimposition image to the binarization unit 12-2.
[0079] The binarization unit 12-2 inputs the padded ball count character superimposition image from the padding unit 11-2, performs the same processing as the binarization unit 12-1, and outputs the ball count binary image to the learning unit 13-2.
[0080] The learning unit 13-2 inputs the ball count binary image from the binarization unit 12-2, and also inputs the correct label assigned to the ball count superimposed image input by the learning device 10. Then, using the ball count binary image and the correct label, the learning unit 13-2 learns the relationship between the input data and the output data in a model in which the ball count binary image is input data and the label of the ball count binary image (ball count label) is output data, in the same manner as the processing of the learning unit 13-1, and generates a ball count super content determination model as a learned model. Then, the learning unit 13-2 stores (the parameters of) the ball count super content determination model in the storage unit 14.
[0081] In this way, the ball count text super image is padded to generate a ball count binary image, and a ball count super content determination model is generated using the ball count binary image and the correct answer label.
[0082] (Generation of a super content judgment model for batting order) Next, a process of generating a super content determination model for batting order by the bulk padding unit 11-3, the binarization unit 12-3, and the learning unit 13-3 will be described.
[0083] The learning device 10 inputs a batting order text caption image containing a single digit number (1 to 9) for a batting order consisting of a single digit number displayed in the text caption included in the video, as well as a label assigned to the batting order text caption image (a correct label, a label indicating one of [1] to [9]).
[0084] In this case, the learning device 10 inputs the above-mentioned data as positive example data to cope with the case where Super ON is set. On the other hand, the learning device 10 also inputs negative example data to cope with the case where Super OFF is set. In other words, the learning device 10 inputs an image in which the batting order is not reflected and a label assigned to this image (correct label, label indicating [None]) as a batting order text superimposition image and a correct label.
[0085] The bulk padding unit 11-3 inputs a batting order text image including a single digit (1 to 9) or a batting order text image not including a digit. The bulk padding unit 11-3 then performs the same process as the bulk padding unit 11-1, and outputs the bulk padded batting order text image to the binarization unit 12-3.
[0086] The binarization unit 12-3 inputs the padded batting order character superimposition image from the padding unit 11-3, performs the same processing as the binarization unit 12-1, and outputs the batting order binary image to the learning unit 13-3.
[0087] The learning unit 13-3 inputs the batting order binary image from the binarization unit 12-3, and also inputs the correct label assigned to the batting order superimposed text image input by the learning device 10. Then, using the batting order binary image and the correct label, the learning unit 13-3 learns the relationship between the input data and the output data in a model in which the batting order binary image is input data and the label of the batting order binary image (batting order label) is output data, in the same manner as the processing of the learning unit 13-1, and generates a batting order super content determination model as a learned model. Then, the learning unit 13-3 stores the batting order super content determination model (parameters of the batting order super content determination model) in the storage unit 14.
[0088] In this way, the batting order text superimposition image is augmented to generate a batting order binary image, and a batting order superimposition content determination model is generated using the batting order binary image and the correct answer label.
[0089] As described above, according to the learning device 10 of the embodiment of the present invention, the bulking unit 11-1 bulks the ball speed superimposed text image by rotating it at a random angle within a preset maximum angle. The binarization unit 12-1 binarizes the bulked ball speed superimposed text image to generate a binary image for ball speed.
[0090] The learning unit 13-1 uses the binary image for ball speed and the correct label to learn the relationship between the input data and output data in a model in which the binary image for ball speed is input data and the label for ball speed is output data, and generates a super content judgment model for ball speed as a learned model and stores it in the memory unit 14.
[0091] The bulking unit 11-2, the binarization unit 12-2 and the learning unit 13-2 perform processing similar to that of the bulking unit 11-1, the binarization unit 12-1 and the learning unit 13-1, and use the ball count character super image and the correct label to generate a super content judgment model for ball counting, and store it in the memory unit 14.
[0092] The bulking unit 11-3, the binarization unit 12-3 and the learning unit 13-3 perform the same processing as the bulking unit 11-1, the binarization unit 12-1 and the learning unit 13-1, and use the batting order character superimposition image and the correct label to generate a batting order superimposition content judgment model and store it in the memory unit 14.
[0093] The superimposition content determination model for ball speed, the superimposition content determination model for ball count, and the superimposition content determination model for batting order thus generated are used by the character superimposition determination device 20.
[0094] The subtitle determination device 20 inputs video of a baseball broadcast and uses these models to determine labels for ball speed, ball count, and batting order from the subtitles contained in the video of the baseball broadcast, and can generate commentary text that reflects each of these labels.
[0095] Therefore, when generating commentary text for a video that includes subtitles, the content of the subtitles can be determined stably and quickly.
[0096] In other words, as shown in Fig. 1, by transmitting the speech data and audio files of the commentary text from the commentary audio production and distribution system 6 to the mobile terminal 5, a commentary of the contents of the superimposed text included in the video of the baseball broadcast can be provided to the user in real time, reducing the burden on the operator of manually generating the commentary text. This makes it possible to convey information about the baseball broadcast in an easy-to-understand manner to many users, including the visually impaired and able-bodied people who watch while doing other things.
[0097] [Subtitle determination device 20] Next, a description will be given of the superimposed character determination device 20 according to an embodiment of the present invention. Fig. 6 is a block diagram showing an example of the configuration of the superimposed character determination device 20 according to an embodiment of the present invention, and Fig. 7 is a flowchart showing an example of the processing of the superimposed character determination device 20.
[0098] The subtitle determination device 20 is a device that determines the content of subtitles included in a video by using a subtitle content determination model, which is a trained model generated by the learning device 10, and generates commentary text.
[0099] The superimposition determination device 20 includes an offense / defense determination unit 21, an image clipping unit 22, binarization units 23-1, 23-2, and 23-3, a memory unit 24, determination units 25-1, 25-2, and 25-3, matching units 26-1, 26-2, and 26-3, and commentary text generation units 27-1, 27-2, and 27-3.
[0100] The storage unit 24 stores super content judgment models (a super content judgment model for ball speed, a super content judgment model for ball count, and a super content judgment model for batting order) which are learned models generated by the learning device 10. In other words, the super content judgment models stored in the storage unit 14 shown in FIG. 2 are the same as the super content judgment models stored in the storage unit 24.
[0101] The binarization unit 23-1, the judgment unit 25-1, the matching unit 26-1 and the commentary text generation unit 27-1 use a ball speed superimposition content judgment model to judge a ball speed label from a ball speed superimposition image extracted from a video, and generate commentary text reflecting the ball speed corresponding to the ball speed label.
[0102] In addition, the binarization unit 23-2, the judgment unit 25-2, the matching unit 26-2 and the explanatory text generation unit 27-2 use a ball count super content judgment model to judge a ball count label from a ball count text super image cut out from the video, and generate explanatory text reflecting the ball count corresponding to the ball count label.
[0103] In addition, the binarization unit 23-3, the judgment unit 25-3, the matching unit 26-3 and the commentary text generation unit 27-3 use a batting order superimposition content judgment model to judge a batting order label from a batting order text superimposition image cut out from the video, and generate commentary text reflecting the batting order corresponding to the batting order label.
[0104] The superimposition determination device 20 receives video of a live baseball game (step S701). The offense / defense determination unit 21 periodically receives baseball game data from the commentary audio production device 1 shown in Fig. 1 and determines whether the offense and defense have switched based on the game data (step S702).
[0105] In this case, the commentary audio production device 1 acquires baseball game data from, for example, a BIS information source 2-2, and the game data is input from the information source 2-2 to the offense / defense determination unit 21 of the superimposition determination device 20 via the commentary audio production device 1.
[0106] The match data includes offensive and defensive data indicating which of the first and second batting teams is currently on offense or defense. The match data makes it possible to determine which team is on offense and which team is on defense, and changes in the match data make it possible to determine whether the offense and defense are switching.
[0107] When the offense / defense determination unit 21 determines that a switch of offense and defense has occurred, it generates offense / defense switch information including information indicating that a switch of offense and defense has occurred and information indicating which team is on offense and defense, and outputs the offense / defense switch information to matching units 26-1, 26-2, and 26-3.
[0108] This enables the matching units 26-1, 26-2, and 26-3 to determine that a switch between offense and defense has occurred and which team is on offense and which team is on defense.
[0109] 8 is a flowchart showing an example of the process (step S702) of the offense and defense determination unit 21. The offense and defense determination unit 21 inputs baseball game data (step S801), extracts offense and defense data from the game data, and refers to this as offense and defense data p new The offensive and defensive data newly inputted and extracted by the offensive and defensive determination unit 21 and stored in the memory as p new And this offensive and defensive data p new The offensive and defensive data is input and extracted at a time before p and stored in the memory in step S805 described later. pre Let us assume that.
[0110] The offense and defense determination unit 21 reads the offense and defense data p pre ,p new Read the past offensive and defensive data p pre And new (current) offensive and defensive data p new In step S803, the offense / defense determination unit 21 determines whether the offense / defense data p pre And offensive and defensive data p newIf it is determined that they are different (step S803: Y), it is determined that a change in offense and defense has occurred, and offense and defense change information is generated and output to matching units 26-1, 26-2, and 26-3 (step S804), and the process proceeds to step S805.
[0111] As a result, the data held in the matching units 26-1, 26-2, and 26-3 is cleared, as will be described in detail later.
[0112] On the other hand, in step S803, the offense and defense determination unit 21 pre And offensive and defensive data p new If it is determined that they are the same (step S803: N), the process proceeds to step S805.
[0113] The offense and defense determination unit 21 proceeds from step S804 or step S803(N) to obtain the offense and defense data p pre Offensive and defensive data new Updated the offensive and defensive data to p pre is stored in memory as past data (step S805).
[0114] In this way, when the offense / defense determination unit 21 determines that a switch of offense and defense has occurred, the offense / defense switch information is output to the matching units 26-1, 26-2, and 26-3, and the stored data is cleared. This prevents the commentary text generation units 27-1, 27-2, and 27-3, which will be described later, from failing to generate commentary text. Details will be described later.
[0115] Returning to Figures 6 and 7, the image cropping unit 22 inputs video of a baseball broadcast and detects subtitles from a preset area of the video (area where subtitles are displayed) to determine whether or not there are subtitles (step S703).
[0116] If the image cropping unit 22 determines in step S703 that text superimpositions have been detected (step S703: Y), it crops out an image for ball speed (text superimposition image for ball speed), an image for ball count (text superimposition image for ball count), and an image for batting order (text superimposition image for batting order) from the video (steps S704-1, S704-2, S704-3).
[0117] The image clipping unit 22 outputs a ball speed text superimposition image to a binarization unit 23-1, outputs a ball count text superimposition image to a binarization unit 23-2, and outputs a batting order text superimposition image to a binarization unit 23-3.
[0118] On the other hand, if it is determined in step S703 that superimposed text has not been detected (step S703: N), the image clipping unit 22 proceeds to step S702.
[0119] FIG. 9 is a diagram for explaining an example of processing (steps S703, S704) of the image cutout unit 22, FIG. 10(1) is a diagram showing an example of an image, and FIG. 10(2) is a diagram showing an example of superimposed text.
[0120] For example, the image cropping unit 22 inputs the image shown in Fig. 10(1). This image includes a text caption in the lower right area, and the text caption includes the ball speed (142km / h), ball count (ball count 2, strike count 2, out count 0), and batting order (4,5,6) as shown in Fig. 10(2).
[0121] 9, the image clipping unit 22 specifies a preset area in which the ball count in the ball count subtitle is displayed, from among areas in which the subtitle is displayed (step S901).
[0122] When the Super On state is selected, the ball count in the ball count superimposition is displayed in the preset area, whereas when the Super Off state is selected, the ball count in the ball count superimposition is not displayed in the preset area.
[0123] The image cutout unit 22 calculates the average luminance value L mean (step S902), and the average luminance value L mean It is determined whether or not the value is equal to or smaller than a preset threshold value θ (step S903).
[0124] In step S903, the image cropping unit 22 calculates the average luminance value L mean is equal to or smaller than the threshold value θ (step S903: Y), it is determined that the ball is on, and a superimposed text image for the ball speed, a superimposed text image for the ball count, and a superimposed text image for the batting order are extracted from the video (step S904). The processes in steps S901 to S904 correspond to steps S703, S704-1, S704-2, and S704-3 shown in FIG.
[0125] Here, the average luminance value L mean is compared with the threshold value θ, and the average brightness value L mean If the threshold value θ is less than or equal to the threshold value θ, the superimposed image is judged to be superimposed because it contains many black pixels and the average brightness value L mean This is because becomes smaller.
[0126] On the other hand, in step S903, the image cutout unit 22 calculates the average luminance value L mean is greater than the threshold value θ (step S903: N), it is determined that the state is super-off.
[0127] The image cropping unit 22 may specify a region other than the region in which the ball count in the ball count subtitle is displayed as a predefined region among the regions in which the subtitles are displayed, and may identify the region.
[0128] For example, the image cropping unit 22 may set the area where the ball speed subtitles are displayed as the preset area, or may set the area where the batting order subtitles are displayed as the preset area. Also, all areas including the ball speed subtitles, ball count subtitles, and batting order subtitles (subtitle areas) may be set as the preset area, or all areas including these numbers may be set as the preset area.
[0129] Here, when the image cutout unit 22 determines that the region including the numbers of the ball count superimposition is the preset region, in step S902, the image cutout unit 22 inverts the color of the region and then calculates the average luminance value L mean Calculate.
[0130] (Process to determine ball speed and generate explanatory text) 6 and 7, the binarization unit 23-1, the determination unit 25-1, the matching unit 26-1, and the commentary text generation unit 27-1 will now be described with respect to the process of determining a ball speed label from a ball speed superimposition image using a ball speed superimposition content determination model, and generating a commentary text for the ball speed. This process corresponds to steps S705-1, S706-1, S707-1, S708-1, and S709 in FIG. 7.
[0131] The binarization unit 23-1 inputs the ball speed text superimposition image from the image clipping unit 22, and binarizes each pixel constituting the ball speed text superimposition image so as to distinguish between a text area and a non-text area included in the ball speed text superimposition image (step S705-1).Then, the binarization unit 23-1 generates a ball speed binary image by binarization, and outputs the ball speed binary image to the determination unit 25-1.
[0132] The determination unit 25-1 reads out the super content determination model for ball speed from the storage unit 24, and inputs the binary image for ball speed from the binarization unit 23-1. Then, the determination unit 25-1 determines a label for ball speed from the binary image for ball speed using the super content determination model for ball speed (step S706-1).
[0133] For example, when the ball speed is three digits, the determination unit 25-1 determines a ball speed (first digit) label, a ball speed (second digit) label, and a ball speed (third digit) label from the ball speed (first digit) binary image, the ball speed (second digit) binary image, and the ball speed (third digit) binary image for each digit. The determination unit 25-1 outputs the ball speed labels to the matching unit 26-1.
[0134] Fig. 11 is a diagram for explaining a processing example (step S706-1) of the determination unit 25-1. As shown in Fig. 11, the determination unit 25-1 determines a ball speed label from the ball speed binary image input from the binarization unit 23-1 using a super content determination model for ball speed.
[0135] Returning to FIG. 6 and FIG. 7, the matching unit 26-1 receives the ball speed label from the determination unit 25-1, and performs a consistency check (ball speed consistency check) on the ball speed labels for the number of digits of the ball speed (step S707-1).
[0136] The ball speed consistency check checks the consistency of ball speed by retaining or deleting the most recent ball speed label in order to provide commentary that is in line with the actual baseball broadcast.
[0137] Even if the ball speed label is accurately determined by the determination unit 25-1, this determination process is performed for each frame image, for example, so as long as the video contains superimposed text (superimposed text is set to ON), the same determination result may be output in some cases. Also, if the same ball speed label is determined for two consecutive pitches, the same determination result will continue to be output.
[0138] The ball speed consistency check is performed by the matching unit 26-1 to prevent the comment text generating unit 27-1 in the subsequent stage from generating the same ball speed comment text even if the matching unit 26-1 inputs a ball speed label with the same judgment result within a preset time period. Also, when [None] is input as the ball speed label, the output process is not performed so that the comment text generating unit 27-1 in the subsequent stage does not generate a ball speed comment text.
[0139] Furthermore, the ball speed consistency check involves outputting a ball speed label of the same ball speed from the matching unit 26-1 to the commentary text generation unit 27-1 after a preset time period has elapsed so that when a ball speed label of the same ball speed is input for two consecutive balls, the commentary text generation unit 27-1 in the subsequent stage will generate the same commentary text for the ball speed.
[0140] If the ball speed consistency check is OK in step S707-1 (step S707-1: OK), the matching unit 26-1 outputs the ball speed label input from the determination unit 25-1 to the commentary text generation unit 27-1. On the other hand, if the ball speed consistency check is NG in step S707-1 (step S707-1: NG), the matching unit 26-1 does not perform output processing to the commentary text generation unit 27-1 and proceeds to step S702.
[0141] In addition, the matching unit 26-1 receives information on a change in offense and defense from the offense and defense determination unit 21, and when it determines that a change in offense and defense has occurred, creates a ball speed label k new is deleted from the memory (not shown) of the matching unit 26-1.
[0142] 12 is a flowchart showing an example of ball speed consistency check (step S707-1) by the matching unit 26-1. The matching unit 26-1 inputs the ball speed label from the determination unit 25-1 and checks whether the ball speed label k corresponds to the number of digits of the ball speed. new is stored in memory (step S1201).
[0143] The matching unit 26-1 inputs a new ball speed label in step S1201 from the point in time when the commentary text is generated by the commentary text generation unit 27-1 in the subsequent stage using the past ball speed label (output by the matching unit 26-1 immediately before). new The time T that has elapsed between the time when the signal is stored in memory and the time when the signal is stored in memory is calculated (step S1202).
[0144] The matching unit 26-1 judges whether the elapsed time T has passed a preset threshold value δ (time) (step S1203). When the matching unit 26-1 judges in step S1203 that the elapsed time T has passed the threshold value δ (step S1203: Y), the matching unit 26-1 reads the ball speed label k new Read out the ball speed label k new It is determined whether or not the label is not [None] (step S1204).
[0145] In step S1204, the matching unit 26-1 new If it is determined that the label is not [None] (step S1204: Y), it is determined that the consistency check is OK (step S1205). Then, the consistency unit 26-1 checks the input ball speed label (ball speed label k for the number of digits of the ball speed). new ) to the explanatory text generator 27-1. (Step S1206).
[0146] On the other hand, when the matching unit 26-1 determines in step S1203 that the elapsed time T does not exceed the threshold value δ (step S1203: N), the process proceeds to step S1207. new If it is determined that the label is [None] (step S1204: N), the process proceeds to step S1207.
[0147] The matching unit 26-1 proceeds from step S1203(N) or step S1204(N), determines that the consistency check is NG (step S1207), and does not perform output processing to the commentary text generation unit 27-1.
[0148] In this way, the matching unit 26-1 inputs a new ball speed label after the commentary text is generated using the previous ball speed label, and then matches the ball speed label k new The time T between storing the ball in memory exceeds the threshold δ, and the ball speed label k new If it is determined that the label is not [None], the consistency check is OK, and the input ball speed label is output to the commentary text generation unit 27-1, causing the commentary text generation unit 27-1 to generate new commentary text for the ball speed.
[0149] On the other hand, if the elapsed time T does not exceed the threshold value δ, or if the ball speed label k new If it is [None], the consistency check is determined to be NG, and output processing to the commentary text generating unit 27-1 is not performed, and the commentary text generating unit 27-1 is not caused to generate new commentary text for the ball speed.
[0150] As a result, when ball speed labels with the same ball speed are input to the matching unit 26-1 for two consecutive balls, the commentary text generation unit 27-1 can generate commentary text for the same ball speed, assuming that the elapsed time T has exceeded the threshold value δ.
[0151] Returning to FIG. 6 and FIG. 7, the commentary text generating unit 27-1 receives ball speed labels (ball speed labels k for the number of digits of the ball speed) from the matching unit 26-1. new ) is input. The commentary text generating unit 27-1 then generates commentary text by using the ball speed label and a preset template, for example by inserting the ball speed corresponding to the ball speed label into the text of the template (step S708-1).
[0152] The commentary text generation unit 27-1 generates speech data consisting of JSON data by, for example, adding attribute labels to the commentary text, and outputs the speech data, etc. to the commentary audio production device 1 as real-time data (step S709).
[0153] Fig. 16 is a diagram for explaining an example of processing by commentary text generating units 27-1, 27-2, and 27-3. As shown in Fig. 16, for example, when [1][4][7] are input as ball speed labels in commentary text generating unit 27-1, commentary text "ball speed
[0147] km" is generated based on a preset template, and speech data, etc. are output.
[0154] (Processing to determine ball count and generate explanatory text) 6 and 7, the binarization unit 23-2, the determination unit 25-2, the matching unit 26-2, and the commentary text generation unit 27-2 will now be described with reference to the process of determining the ball count label from the ball count superimposition image using the ball count superimposition content determination model, and generating the commentary text for the ball count. This process corresponds to steps S705-2, S706-2, S707-2, S708-2, and S709 in FIG. 7.
[0155] The binarization unit 23-2 inputs the ball count superimposed image from the image clipping unit 22, and binarizes each pixel constituting the ball count superimposed image in the same manner as the binarization unit 23-1 (step S705-2). The binarization unit 23-2 then generates a ball count binary image by binarization, and outputs the ball count binary image to the determination unit 25-2.
[0156] The determination unit 25-2 reads out the ball count super content determination model from the storage unit 24, and inputs the ball count binary image from the binarization unit 23-2. Then, the determination unit 25-2 determines the ball count label from the ball count binary image using the ball count super content determination model (step S706-2).
[0157] For example, the determination unit 25-2 determines a label for the ball count (ball), a label for the ball count (strike), and a label for the ball count (out) from a binary image for the ball count (ball), a binary image for the ball count (strike), and a binary image for the ball count (out) in the order of ball, strike, and out. The determination unit 25-2 outputs the label for the ball count to the matching unit 26-2.
[0158] The matching unit 26-2 receives the ball count labels from the determination unit 25-2, and performs a consistency check (ball count consistency check) on each of the ball count (ball) label, the ball count (strike) label, and the ball count (out) label (step S707-2).
[0159] The ball count consistency check checks the consistency of the ball count by keeping or removing the most recent ball count label in order to provide commentary that is in line with the reality of the baseball broadcast.
[0160] Even if the ball count label is accurately determined by the determination unit 25-2, this determination process is performed, for example, for each frame image, and therefore as long as the image contains text superimpositions (as long as superimpositions are set to on), the same determination result may be output in some cases.
[0161] The ball count consistency check is performed by the matching unit 26-2 to prevent the comment text generating unit 27-2 from generating the same ball count commentary text even if the matching unit 26-2 inputs a ball count label with the same judgment result for each of the ball count (ball) label, the ball count (strike) label, and the ball count (out) label. Also, when [None] is input as the ball count label, the output process is not performed so that the comment text generating unit 27-2 does not generate the ball count commentary text.
[0162] If the ball count consistency check is OK in step S707-2 (step S707-2: OK), the matching unit 26-2 outputs the ball count label input from the determination unit 25-2 to the commentary text generation unit 27-2. On the other hand, if the ball count consistency check is NG in step S707-2 (step S707-2: NG), the matching unit 26-2 does not perform output processing to the commentary text generation unit 27-2 and proceeds to step S702. The ball count consistency check by the matching unit 26-2 is performed individually for each of the ball count (ball) label, the ball count (strike) label, and the ball count (out) label.
[0163] In addition, the matching unit 26-2 receives information on a change in offense and defense from the offense and defense determination unit 21, and when it determines that the offense and defense have changed, it generates a label s for ball count (strike) shown in FIG. 13 and FIG. 14, which will be described later. new ,s pre and ball count (out) labels new ,o pre The label b for ball count (ball) is deleted from the memory (not shown) of the matching unit 26-2. new ,b pre The same applies to.
[0164] 13 is a flowchart showing an example of ball count (strike) consistency check (step S707-2) by the consistency unit 26-2. The consistency unit 26-2 receives the ball count (strike) label from the determination unit 25-2 and converts it into the ball count (strike) label s new and stores it in memory (step S1301).
[0165] Here, the matching unit 26-2 newly inputs and stores the ball count (strike) label in the memory as s new Then, the ball count (strike) label s new The ball count (strike) label s newThe label for the ball count (strike) closest to the time of pre Let us assume that.
[0166] The matching unit 26-2 reads the ball count (strike) label s from the memory. new and the ball count (strike) label s new It is determined whether or not the label is not [None] (step S1302).
[0167] In step S1302, the matching unit 26-2 performs a ball count (strike) label s new If it is determined that the label s is not [None] (step S1302: Y), the label s for ball count (strike) is retrieved from the memory. pre Then, the matching unit 26-2 reads out the past ball count (strike) label s pre and new (current) ball count (strike) labels s new It is determined whether or not the values are different (step S1303).
[0168] In step S1303, the matching unit 26-2 performs the ball count (strike) label s pre and ball count (strike) labels new If it is determined that the counts are different (step S1303: Y), it is determined that the ball count (strike) has changed.
[0169] The matching section 26-2 is a label for ball count (strike) s pre The ball count (strike) for labels new and the updated ball count (strike) label s pre The matching unit 26-2 then determines that the matching check is OK (step S1305) and stores the ball count (ball) label b new The ball count (strike) label s entered is read out. new And the label b for the ball count (ball) readnew to the commentary text generating unit 27-2 (step S1306).
[0170] On the other hand, in step S1302, the matching unit 26-2 performs the ball count (strike) label s new If it is determined that the label is [None] (step S1302: N), the process proceeds to step S1307.
[0171] In step S1303, the matching unit 26-2 performs the ball count (strike) label s pre and ball count (strike) labels new If it is determined that the number of balls is the same as the number of strikes (step S1303: N), it is determined that the ball count (strike) has not changed, and the process proceeds to step S1307.
[0172] The matching unit 26-2 proceeds from step S1302(N) or step S1303(N), determines that the consistency check is NG (step S1307), and does not perform output processing to the commentary text generation unit 27-2.
[0173] In this way, the matching unit 26-2 matches the ball count (strike) label s new is not [None] and the past ball count (strike) label s pre If it is determined that the consistency check is OK, the input ball count (strike) label s new and the label b for ball count (ball) read from memory new to the commentary text generating unit 27-2, thereby causing the commentary text generating unit 27-2 to generate a new commentary text for the ball count (strike, ball).
[0174] On the other hand, the matching unit 26-2 is a label for ball count (strike) s new If it is determined that the value is [None], or if the past ball count (strike) label s preIf it is determined that the commentary text is the same as the above, the consistency check is deemed NG, and output processing to the commentary text generation unit 27-2 is not performed, and the commentary text generation unit 27-2 is not prompted to generate a new commentary text for the ball count (strike, ball).
[0175] As a result, when the same ball count (strike) label is input to the matching unit 26-2 consecutively, the commentary text generating unit 27-2 does not generate commentary text for the same ball count (strike, ball). Also, when [None] is input to the matching unit 26-2 as the ball count (strike) label, the commentary text generating unit 27-2 does not generate a new ball count (strike, ball) commentary text.
[0176] The example of the ball count (strike) consistency check shown in FIG. 13 can also be applied to the ball count (ball) consistency check. new Let the label for this ball count (ball) be b new The label for the ball count (ball) entered at a time before and stored in memory is b pre Let us assume that.
[0177] In this way, the matching unit 26-2 uses the ball count (ball) label b new is not [None] and the past ball count (ball) label b pre If it is determined that the label is different from the label, the consistency check is OK and the entered ball count (ball) label b new and the ball count (strike) label s read from the memory new to the commentary text generating unit 27-2, thereby causing the commentary text generating unit 27-2 to generate a new commentary text for the ball count (strike, ball).
[0178] On the other hand, the matching unit 26-2 uses the ball count (ball) label b newIf it is determined that the value is [None], or the label for past ball count (ball) b pre If it is determined that the commentary text is the same as the above, the consistency check is deemed NG, and output processing to the commentary text generation unit 27-2 is not performed, and the commentary text generation unit 27-2 is not prompted to generate a new commentary text for the ball count (strike, ball).
[0179] As a result, when the same ball count (ball) label is input to the matching unit 26-2 consecutively, the commentary text generating unit 27-2 does not generate commentary text for the same ball count (strike, ball). Also, when [None] is input to the matching unit 26-2 as the ball count (ball) label, the commentary text generating unit 27-2 does not generate a new ball count (strike, ball) commentary text.
[0180] 14 is a flowchart showing an example of ball count (out) consistency check (step S707-2) by the consistency unit 26-2. The consistency unit 26-2 receives the ball count (out) label from the determination unit 25-2, and converts it into the ball count (out) label o new and stores it in memory (step S1401).
[0181] Here, the matching unit 26-2 newly inputs and stores the ball count (out) label in the memory. new And then, label this ball count (out) new The ball count (out) label o was input at a time earlier than the time of the ball count (out) and stored in the memory in step S1404 described later. new Label for ball count (closest to the time of pre Let us assume that.
[0182] The matching unit 26-2 reads the ball count (out) label from the memory. new Read out the label for ball count (out) new It is determined whether or not the label is not [None] (step S1402).
[0183] In step S1402, the matching unit 26-2 new If it is determined that the label o for ball count (out) is not [None] (step S1402: Y), pre Then, the matching unit 26-2 reads out the past ball count (out) label o pre and a new (current) label for the ball count (out) new It is determined whether or not the values are different (step S1403).
[0184] In step S1403, the matching unit 26-2 pre and ball count (out) labels new If it is determined that they are different (step S1403: Y), it is determined that the ball count (out) has changed.
[0185] The matching section 26-2 is a label for ball count (out). pre The ball count(out) for label o new and update the label for the updated ball count (out) pre The matching unit 26-2 stores the ball count (out) label o pre It is determined whether or not is not a label of [0] (step S1405).
[0186] In step S1405, the matching unit 26-2 pre If it is determined that the label o for the ball count (out) is not [0] (step S1405: Y), it is determined that the consistency check is OK (step S1406), and the input label o for the ball count (out) is new to the commentary text generating unit 27-2 (step S1407).
[0187] On the other hand, in step S1402, the matching unit 26-2new If it is determined that the label is [None] (step S1402: N), the process proceeds to step S1408.
[0188] In step S1403, the matching unit 26-2 pre and ball count (out) labels new If it is determined that the counts are the same (step S1403: N), it is determined that the ball count (out) has not changed, and the process proceeds to step S1408.
[0189] In step S1405, the matching unit 26-2 pre is determined to be a label of [0] (step S1405: N), the process proceeds to step S1408.
[0190] The matching unit 26-2 proceeds from step S1402(N), step S1403(N) or step S1405(N), determines that the consistency check is NG (step S1408), and does not perform output processing to the commentary text generation unit 27-2.
[0191] In this way, the matching unit 26-2 is a label for ball count (out). new is not [None] and the past ball count (out) label o pre and the ball count (out) label o pre If it is determined that the label is not [0], the consistency check is OK and the input ball count (out) label o new to the commentary text generating unit 27-2, thereby causing the commentary text generating unit 27-2 to generate a new commentary text for the ball count (out).
[0192] On the other hand, the matching unit 26-2 is a label for ball count (out) new If is [None], the label for past ball count (out) o preIf it is determined to be the same as the ball count (out), the label o pre is determined to be a label of [0], the consistency check is deemed to be NG, and output processing to the commentary text generation unit 27-2 is not performed, and the commentary text generation unit 27-2 is not prompted to generate new commentary text for the ball count (out).
[0193] As a result, when the same ball count (out) label is input to the matching unit 26-2 consecutively, the commentary text generating unit 27-2 does not generate the same commentary text for the ball count (out). Also, when [None] or [0] is input to the matching unit 26-2 as the ball count (out) label, the commentary text generating unit 27-2 does not generate a new commentary text for the ball count (out).
[0194] Returning to FIG. 6 and FIG. 7, the commentary text generating unit 27-2 receives the ball count label (ball count (strike) label s new and ball count (ball) label b new , or ball count (out) label new ) is input. The commentary text generating unit 27-2 then uses the ball count label and a preset template to generate commentary text by, for example, inserting the count corresponding to the ball count label into the text of the template (step S708-2).
[0195] The commentary text generation unit 27-2 generates speech data consisting of Json data, similar to the above-mentioned step S709, and outputs the speech data, etc. to the commentary audio production device 1 as real-time data.
[0196] 16, for example, in the commentary text generating unit 27-2, when [2] and [2] are input as the label for the ball count (strike) and the label for the ball count (ball), the commentary text "[Two] balls, [Two] strikes" is generated based on a preset template, and speech data, etc. are output. Also, when [1] is input as the label for the ball count (out), the commentary text "[One] out" is generated based on a preset template, and speech data, etc. are output.
[0197] (Process to determine batting order and generate explanatory text) 6 and 7, the binarization unit 23-3, the determination unit 25-3, the matching unit 26-3, and the commentary text generation unit 27-3 will now be described with reference to the process of determining the batting order label from the batting order text superimposition image using the batting order superimposition content determination model, and generating the commentary text for the batting order. This process corresponds to steps S705-3, S706-3, S707-3, S708-3, and S709 in FIG. 7.
[0198] The binarization unit 23-3 inputs the batting order text superimposed image from the image clipping unit 22, and binarizes each pixel constituting the batting order text superimposed image in the same manner as the binarization unit 23-1 (step S705-3). The binarization unit 23-3 then generates a batting order binary image by binarization, and outputs the batting order binary image to the determination unit 25-3.
[0199] The determination unit 25-3 reads out the batting order super content determination model from the storage unit 24, and inputs the batting order binary image from the binarization unit 23-3. The determination unit 25-3 then uses the batting order super content determination model to determine the batting order label from the batting order binary image (step S706-3). The determination unit 25-3 outputs the batting order label to the matching unit 26-3.
[0200] The matching unit 26-3 receives the batting order labels from the determination unit 25-3 and performs a consistency check on the batting order labels (batting order consistency check) (step S707-3).
[0201] The batting order consistency check checks the consistency of the batting order by keeping or deleting the most recent batting order label in order to provide commentary that is in line with the reality of the baseball broadcast.
[0202] Even if the batting order label is accurately determined by the determination unit 25-3, this determination process is performed, for example, for each frame image, and therefore as long as the image contains superimpositions (as long as superimpositions are set to on), the same determination result may be output in some cases.
[0203] The batting order consistency check is a process in which, even if the matching unit 26-3 inputs a batting order label with the same determination result, no output process is performed so that the commentary text generating unit 27-3 in the subsequent stage does not generate the same batting order commentary text. Also, even if [None] is input as the batting order label, no output process is performed so that the commentary text generating unit 27-3 in the subsequent stage does not generate the batting order commentary text.
[0204] If the batting order consistency check is OK in step S707-3 (step S707-3: OK), the matching unit 26-3 outputs the batting order label input from the determination unit 25-3 to the commentary text generation unit 27-3. On the other hand, if the batting order consistency check is NG in step S707-3 (step S707-3: NG), the matching unit 26-3 does not perform output processing to the commentary text generation unit 27-3 and proceeds to step S702.
[0205] In addition, the matching unit 26-3 receives the offense / defense change information from the offense / defense determination unit 21, and when it determines the offense / defense change, it creates a batting order label d new ,d pre is deleted from the memory (not shown) of the matching unit 26-3.
[0206] This makes it possible to prevent the commentary text generation unit 27-3 from failing to generate commentary text. new ,d preIf we assume that the process of deleting is not performed, for example, if the offense and defense switch when the fourth batter is out and the next batting team starts with the fourth batter, "N" will be determined in step S1503 of Figure 15 described below, the consistency check will be judged as NG, and output processing to the explanatory text generation unit 27-3 will not be performed, and explanatory text will not be generated.
[0207] 15 is a flowchart showing an example of a batting order consistency check by the consistency unit 26-3 (step S707-3). The consistency unit 26-3 receives the batting order label from the determination unit 25-3 and converts it into the batting order label d new and stores it in memory (step S1501).
[0208] Here, the matching unit 26-3 newly inputs and stores the batting order label in the memory as d new Let the batting order label d new and stored in memory in step S1504 (later described) (batting order label d new (the closest time to the batting order label) pre Let us assume that.
[0209] The matching unit 26-3 reads the batting order label d from the memory. new Read out the batting order label d new It is determined whether or not the label is not [None] (step S1502).
[0210] In step S1502, the matching unit 26-3 new If it is determined that the label is not [None] (step S1502: Y), the batting order label d is read from the memory. pre Then, the matching unit 26-3 reads out the past batting order label d pre and new (current) batting order label d new It is determined whether or not the values are different (step S1503).
[0211] In step S1503, the matching unit 26-3 pre and batting order label dnew If it is determined that the batting order is different (step S1503: Y), it is determined that the batting order has changed.
[0212] The matching part 26-3 is the batting order label d pre The batting order label d new and update the batting order label to d pre is stored in memory as past data (step S1504). Then, the matching unit 26-3 determines that the matching check is OK (step S1505) and stores the input batting order label d new to the commentary text generating unit 27-3 (step S1506).
[0213] On the other hand, in step S1502, the matching unit 26-3 new If it is determined that the label is [None] (step S1502: N), the process proceeds to step S1507.
[0214] In step S1503, the matching unit 26-3 pre and batting order label d new If it is determined that the batting order is the same (step S1503: N), it is determined that the batting order has not changed, and the process proceeds to step S1507.
[0215] The matching unit 26-3 proceeds from step S1502(N) or step S1503(N), determines that the consistency check is NG (step S1507), and does not perform output processing to the commentary text generating unit 27-3.
[0216] In this way, the matching unit 26-3 uses the batting order label d new is not [None] and the past batting order label d pre If it is determined that the consistency check is OK, the batting order label d new to the commentary text generating unit 27-3, thereby causing the commentary text generating unit 27-3 to generate a new commentary text for the batting order.
[0217] On the other hand, the matching unit 26-3 uses the batting order label d new If it is determined that is [None], or if the past batting order label d pre If it is determined that the batting order commentary text is the same as the batting order commentary text, the consistency check is deemed NG, output processing to the commentary text generating unit 27-3 is not performed, and the commentary text generating unit 27-3 is not caused to generate a new batting order commentary text.
[0218] As a result, when the same batting order label is input to the matching unit 26-3 consecutively, the commentary text generating unit 27-3 does not generate the same batting order commentary text. Also, when [None] is input as the batting order label to the matching unit 26-3, the commentary text generating unit 27-3 does not generate a new batting order commentary text.
[0219] Returning to FIGS. 6 and 7, the commentary text generating unit 27-3 inputs the player data of the baseball game from the commentary audio producing device 1 shown in FIG.
[0220] In this case, the commentary audio production device 1 acquires player data for a baseball game from, for example, the BIS information source 2-2, and the player data is input from the information source 2-2 to the commentary text generation unit 27-3 of the superimposition determination device 20 via the commentary audio production device 1. The player data includes the name of the team to which the player belongs, his uniform number, his name, turn-at-bat information such as whether he is a right-handed or left-handed batter, etc.
[0221] The commentary text generating unit 27-3 inputs the batting order label from the matching unit 26-3. Then, the commentary text generating unit 27-3 uses the batting order label, the player data, and a preset template to generate commentary text for introducing the batter by inserting the batting order label and the player data into the text of the template (step S708-3).
[0222] The commentary text generation unit 27-3 generates speech data consisting of Json data, similar to the above-mentioned step S709, and outputs the speech data, etc. to the commentary audio production device 1 as real-time data.
[0223] As shown in Figure 16, for example, when [4] is input as a batting order label in the commentary text generation unit 27-3, the commentary text “Number "[4] [Giken] Uniform number
[99] [Right-handed]” is generated based on the player data and a pre-set template, and speech data, etc. are output.
[0224] As described above, according to the superimposition determination device 20 of the embodiment of the present invention, the offense / defense determination unit 21 determines a change of offense / defense based on the match data input from the commentary audio production device 1, and generates offense / defense change information.
[0225] The image cutout unit 22 detects the presence or absence of superimposed text from the video of the baseball broadcast, and cuts out a superimposed text image for ball speed, a superimposed text image for ball count, and a superimposed text image for batting order from the video.
[0226] The binarization unit 23-1 binarizes the ball speed text superimposition image to generate a ball speed binary image. The judgment unit 25-1 judges the ball speed label from the ball speed binary image using a trained ball speed superimposition content judgment model. The matching unit 26-1 checks the consistency of the ball speed label, and the commentary text generation unit 27-1 generates commentary text for the ball speed corresponding to the ball speed label if the consistency check is OK.
[0227] Binarization unit 23-2, judgment unit 25-2, matching unit 26-2 and explanatory text generation unit 27-2 perform processing similar to that of binarization unit 23-1, judgment unit 25-1, matching unit 26-1 and explanatory text generation unit 27-1, binarize the ball count text superimposition image to generate a ball count binary image, judge a ball count label from the ball count binary image using a trained ball count superimposition content judgment model, perform a consistency check of the ball count label, and if the consistency check is OK, generate a ball count explanatory text corresponding to the ball count label.
[0228] The binarization unit 23-3, the judgment unit 25-3, the matching unit 26-3 and the commentary text generation unit 27-3 perform the same processing as the binarization unit 23-1, the judgment unit 25-1, the matching unit 26-1 and the commentary text generation unit 27-1, binarize the batting order text superimposition image to generate a batting order binary image, judge a batting order label from the batting order binary image using a trained batting order superimposition content judgment model, perform a consistency check of the batting order label, and if the consistency check is OK, generate commentary text for the batting order corresponding to the batting order label.
[0229] Therefore, when generating commentary text for a video that includes subtitles, the content of the subtitles can be determined stably and quickly.
[0230] In other words, as shown in Fig. 1, by transmitting the speech data and audio files of the commentary text from the commentary audio production and distribution system 6 to the mobile terminal 5, a commentary of the contents of the superimposed text included in the video of the baseball broadcast can be provided to the user in real time, reducing the burden on the operator of manually generating the commentary text. This makes it possible to convey information about the baseball broadcast in an easy-to-understand manner to many users, including the visually impaired and able-bodied people who watch while doing other things.
[0231] [Experimental Results] Next, a description will be given of the results of an experiment using the learning device 10 and the superimposed character determination device 20 according to the embodiment of the present invention. Fig. 17 is a diagram showing the results of the experiment.
[0232] Figure 17(1) is a diagram comparing the experimental results of the recognition rate of each super of ball speed, ball count, and batting order between the conventional technology (Non-Patent Documents 1 and 2) and the embodiment of the present invention. The conventional technology (Non-Patent Documents 1 and 2) uses an optical character recognition engine called Tesseract, which is commonly used to recognize characters on an image. Still images (2,235 images) extracted from a baseball broadcast of about one hour were used as verification data.
[0233] From FIG. 17(1), it can be seen that the embodiment of the present invention has a higher recognition rate for each super, such as ball speed, ball count, and batting order, than the conventional techniques (Non-Patent Documents 1 and 2).
[0234] Fig. 17(2) is a diagram comparing the processing time per image between the conventional techniques (Non-Patent Documents 1 and 2) and the embodiment of the present invention. The method of the conventional techniques (Non-Patent Documents 1 and 2) is similar to that shown in Fig. 17(1).
[0235] From FIG. 17(2), it can be seen that the processing time per wafer is 1.34 seconds in the conventional technology (Non-Patent Documents 1 and 2) and 0.26 seconds in the embodiment of the present invention, and thus the processing time is shorter in the embodiment of the present invention than in the conventional technology (Non-Patent Documents 1 and 2).
[0236] Moreover, from the experimental results of FIGS. 17(1) and (2), it can be said that the embodiment of the present invention can determine the contents of superimposed characters more stably and quickly than the conventional techniques (Non-Patent Documents 1 and 2).
[0237] FIG. 17(3) is a diagram comparing the time taken from displaying the ball speed to speaking it, between manual input (when the information source 2-3 of the manual input tool in FIG. 1 is used) and an embodiment of the present invention.
[0238] Measurements were performed for 35 balls using manual input, and 26 balls using the embodiment of the present invention. With manual input, speech was successful for 13 balls, but with the embodiment of the present invention, speech was successful for all balls. In addition, in cases where the ball speed display was less than 1 second, speech was successful for 1 / 3 of the total balls using manual input, but with the embodiment of the present invention, speech was successful for all balls.
[0239] As shown in Figure 17 (3), the average time it took from when the ball speed was displayed on the television screen until it was spoken was 9.0 seconds for manual input and 2.8 seconds for the embodiment of the present invention, and the minimum time was 5.0 seconds for manual input and 1.0 second for the embodiment of the present invention.
[0240] From the experimental results in FIGS. 17(1) to (3), it is clear that the embodiment of the present invention enables commentary audio to be created and distributed more accurately and quickly than the conventional techniques (Non-Patent Documents 1 and 2) and manual input.
[0241] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical concept thereof.
[0242] For example, in the above embodiment, the learning device 10 and the superimposed text determination device 20 were described with respect to video footage of a baseball broadcast program among sports broadcast programs. The present invention is not limited to target video footage of baseball, but is also applicable to video footage of ball games other than baseball, such as volleyball, rugby, and table tennis. Furthermore, the present invention is applicable not only to sports broadcast programs, but also to programs other than sports broadcast programs.
[0243] In short, the present invention can be applied to any video as long as it generates a trained model for determining the content of subtitles contained in a video and can use the trained model to determine the content of subtitles contained in unknown video.
[0244] In the above embodiment, for example, a case has been described in which video of a baseball game program is distributed by broadcast, but the present invention is applicable not only to broadcast but also to various video media services such as the Internet.
[0245] In addition, for example, in the above embodiment, a trained model for determining the numbers in the subtitles included in a video is generated, and the trained model is used to determine the numbers in the subtitles included in an unknown video. The present invention is not limited to the determination target being the numbers in the subtitles, and various characters such as hiragana, kanji, katakana, and alphabets may be determined as the determination target.
[0246] Also, for example, in the above embodiment, the bulking unit 11-1 of the learning device 10 inputs the ball speed superimposition image and performs bulking, and the binarization unit 12-1 inputs the ball speed superimposition image after bulking and performs binarization. In contrast, the binarization unit 12-1 may directly input the ball speed superimposition image before bulking. The same applies to the binarization unit 12-2 that processes the ball count superimposition image and the binarization unit 12-3 that processes the batting order superimposition image.
[0247] Also, for example, in the above embodiment, the judgment unit 25-1 of the superimposition judgment device 20 outputs the ball speed label to the matching unit 26-1, but it may be output directly to the commentary text generation unit 27-1. In this case, the commentary text generation unit 27-1 inputs the ball speed label from the judgment unit 25-1 and generates commentary text reflecting the ball speed corresponding to the ball speed label. The same applies to the judgment unit 25-2 and commentary text generation unit 27-2 that process the ball count label, and the judgment unit 25-3 and commentary text generation unit 27-3 that process the batting order label.
[0248] A normal computer can be used as the hardware configuration of the learning device 10 and the subtitle determination device 20 according to the embodiment of the present invention. The learning device 10 and the subtitle determination device 20 are configured by a computer equipped with a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, an interface, etc.
[0249] The functions of the bulking units 11-1, 11-2, and 11-3, the binarization units 12-1, 12-2, and 12-3, the learning units 13-1, 13-2, and 13-3, and the memory unit 14 provided in the learning device 10 are each realized by causing a CPU to execute a program in which these functions are written.
[0250] In addition, each of the functions of the offense / defense determination unit 21, image extraction unit 22, binarization units 23-1, 23-2, 23-3, memory unit 24, determination units 25-1, 25-2, 25-3, matching units 26-1, 26-2, 26-3 and explanatory text generation units 27-1, 27-2, 27-3 provided in the character superimposition determination device 20 is realized by having the CPU execute a program that describes these functions.
[0251] These programs are stored in the storage medium and are read and executed by the CPU. These programs can also be distributed by storing them in a storage medium such as a magnetic disk (floppy disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), or a semiconductor memory, and can also be transmitted and received via a network. [Explanation of symbols]
[0252] 1. Audio commentary production equipment 2 Sources of information 3. Voice synthesizer 4. Distribution Device 5. Mobile devices 6. Audio commentary production and distribution system 10 Learning Device 11-1, 11-2, 11-3 Bulking section 12-1, 12-2, 12-3, 23-1, 23-2, 23-3 Binarization section 13-1, 13-2, 13-3 Learning Section 14,24 Storage section 20. Superimposed Text Judgment Device 21 Offense and Defense Judgment Department 22 Image Extraction Section 25-1,25-2,25-3 Judgment section 26-1,26-2,26-3 Matching part 27-1, 27-2, 27-3 Explanation text generation section p pre ,p new Offensive and defensive data L mean Average Luminance Value θ,δ threshold k newBall speed label T Elapsed time s pre ,s new Ball count (strike) labels b pre ,b new Ball count (ball) labels o pre ,o new Ball count (out) label d pre ,d new Batting order labels
Claims
1. A learning device for generating a model for determining the content of a transcribed image, the model being used when generating a commentary text for a video including the transcribed image, comprising: a binarization unit that inputs the image of the superimposed character as a superimposed character image, binarizes each pixel constituting the superimposed character image so as to distinguish a character area included in the superimposed character image from an area other than the character, and generates a binary image; A label is assigned to the superimposed text image, and the label is used to identify a character included in the superimposed text image. The label is used as a correct label. a learning unit that uses the binary image generated by the binarization unit and the correct label to learn a relationship between the input data and the output data in a model in which the binary image is input data and a label for identifying a character included in the binary image is output data, thereby generating a super content determination model; A learning device comprising:
2. 2. The learning device according to claim 1, Further, a volume increasing unit is provided for increasing the volume of the superimposed character image by rotating the superimposed character image at a random angle within a preset maximum angle, The binarization unit includes: A learning device characterized in that the binary image is generated for each of the multiple character superimposition images that have been padded by the padding unit.
3. A subtitle determination device that determines the content of a subtitle in order to generate a commentary text for a video including the subtitle, an image extraction unit that receives a video and extracts an image of the subtitles as a subtitle image from a preset area of the video; a binarization unit that binarizes each pixel constituting the superimposed character image and generates a binary image so as to distinguish a character area included in the superimposed character image cut out by the image cutout unit from an area other than the character; a determination unit that determines a label for identifying the character included in the binary image generated by the binarization unit, from the binary image generated by the binarization unit, using a super content determination model generated by the learning device according to claim 1 or 2; A character superimposition determination device comprising:
4. 4. The character superimposition determination device according to claim 3, The image cropping unit is A character superimposition determination device comprising: a first region of the image that is a predetermined region; a second region that is a predetermined threshold region; a third region that is a predetermined region; a fourth region that is a predetermined region; a fourth region that is a predetermined region; a fifth region that is a predetermined region; a fifth region that is a predetermined region;
5. 4. The character superimposition determination device according to claim 3, The character superimposition determination device further comprises an explanatory text generation unit that inserts characters corresponding to the label determined by the determination unit into a preset template and generates the explanatory text for explaining the scene in the video.
6. 4. The character superimposition determination device according to claim 3, a matching unit which inputs the first label, sets the label determined by the determination unit when the label is input as a first label, sets the label input immediately before the first label as a second label, and if it is determined that the first label and the second label are different, determines that a consistency check is OK and outputs the first label, and if it is determined that the first label and the second label are the same, determines that the consistency check is NG; a commentary text generating unit that generates the commentary text for explaining a scene of the video by inserting characters corresponding to the first label output by the matching unit into a preset template; A character superimposition determination device comprising:
7. 4. The character superimposition determination device according to claim 3, a matching unit which determines an elapsed time from a predetermined time point to a time point when the third label is input, using the label determined by the determining unit as a third label, and outputs the third label when it is determined that the elapsed time has exceeded a preset threshold time, and determines an OK consistency check and outputs the third label when it is determined that the elapsed time has not exceeded the threshold time; and a commentary text generating unit that generates the commentary text for explaining a scene of the video by inserting characters corresponding to the third label output by the matching unit into a preset template, A character superimposition determination device characterized in that the specified point in time for the matching unit to calculate the elapsed time is the point in time when the commentary text generation unit generates the commentary text for a character corresponding to a label output by the matching unit immediately before the third label.
8. A computer constituting a learning device that generates a model for determining the content of a character superimposition, which is used when generating a commentary text for a video including the character superimposition, a binarization unit that inputs the image of the superimposed character as a superimposed character image, binarizes each pixel constituting the superimposed character image so as to distinguish a character area included in the superimposed character image from an area other than the character, and generates a binary image; and A label is assigned to the superimposed text image, and the label is used to identify a character included in the superimposed text image. The label is used as a correct label. A program for functioning as a learning unit that uses the binary image generated by the binarization unit and the correct label to learn the relationship between the input data and the output data in a model in which the binary image is input data and labels for identifying characters contained in the binary image are output data, and to generate a super content judgment model.
9. A computer constituting a subtitle determination device that determines the content of a subtitle in order to generate a commentary text for a video including the subtitle, an image extraction unit that inputs a video and extracts an image of the subtitles as a subtitle image from a preset area of the video; a binarization unit that binarizes each pixel constituting the superimposed text image and generates a binary image so as to distinguish a text area included in the superimposed text image cut out by the image cutout unit from an area other than the text; and A program for causing a super content judgment model generated by the learning device according to claim 1 or 2 to function as a judgment unit that judges a label for identifying the character contained in the binary image generated by the binarization unit from the binary image generated by the binarization unit.
Citation Information
Patent Citations
Explanation voice reproduction device and program thereof
JP2017203827A
