Subtitle display position learning device, subtitle display position inference device and program

Abstract

To automatically adjust a subtitle display position to a proper position corresponding to a video.SOLUTION: A subtitle area extraction part 11 of a subtitle display position learning device 1-1 extracts a subtitle area by using image data (with no subtitle) and image data (with a subtitles), and finds subtitle left upper coordinates (x, y) from the subtitle area. A coordinate conversion part 13 converts the subtitle left upper coordinates (x, y) into a subtitle display position (X, Y) by using a table 12. A storage part 14 is stored with the image data (with no subtitle) and the subtitle display position (X, Y) as teacher data. A learning part 15 reads the teacher data consisting of the image data (with no subtitle) and the subtitle display position (X, Y) out of the storage part 14, and learns using the teacher data to generate a learning-completed model.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a subtitle display position learning device that learns the display position of subtitles added to video, a subtitle display position inference device that infers the display position of subtitles added to video, and a program. [Background technology]

[0002] Conventionally, television broadcasting provides closed captioning services for the hearing impaired, in which the audio of a broadcast program is displayed on the screen as text. The subtitles sent out for live broadcast programs are manually transcribed from the audio of the live broadcast program.

[0003] Methods for producing subtitles for live broadcast programs include a method in which a person inputs the audio while listening to it, and a semi-automatic method in which the results of speech recognition are manually corrected (see, for example, non-patent document 1).

[0004] In such subtitled broadcasting, when a viewer turns on the display of the subtitled broadcasting, the receiver displays the decoded characters superimposed on the main video image of the live broadcast program.

[0005] In this case, if the subtitle text overlaps with information-rich parts of the main video (e.g., captions, maps, etc.), the viewer will have difficulty seeing the CG superimpositions, captions, etc. For this reason, when producing subtitles, the position of the subtitles is manually adjusted to avoid information-rich parts of the main video.

[0006] Figure 39 shows examples of subtitle display positions before and after adjustment in a conventional subtitle display position adjustment method. In the broadcast screen before adjustment shown in Figure 39(1), the subtitle "This city is said to be the entrance to Europe" is covered by the CG superimposition. This makes it difficult for viewers to see the CG superimposition of the main video, and they are unable to correctly obtain the information in the CG superimposition.

[0007] So, while watching the unadjusted broadcast screen shown in Fig. 39(1), the person in charge in the subtitle production room uses the operation box to change the subtitle display position to an appropriate position so that the subtitles displayed on the broadcast screen do not overlap the CG superimposition. In other words, the person in charge moves the subtitle display position shown in Fig. 39(1) to the position shown in Fig. 39(2).

[0008] This moves the subtitles to a position where they do not overlap with the CG superimpositions, allowing the viewer to accurately obtain not only the subtitle information but also the CG superimposition information.

[0009] In particular, in live subtitling rooms where subtitles are added in real time to live or partially recorded broadcast programs, staff create subtitles using voice recognition or high-speed typing, and manually adjust the position of the subtitles while watching the main program footage. [Prior art documents] [Non-patent literature]

[0010] [Non-Patent Document 1] Tomoyasu Komori, "Latest Trends in Automatic Subtitling for Live Broadcast Programs," [online], NHK Science & Technology Research Laboratories, Summer 2020 Issue Commentary 02, NHK Broadcasting Technology Research Laboratories, [Retrieved February 3, 2023], Internet<URL:https: / / www.nhk.or.jp / strl / publica / rd / 182 / 3.html> Summary of the Invention [Problem to be solved by the invention]

[0011] As mentioned above, when producing subtitles for broadcast programs, it is necessary to appropriately determine the display position of the subtitles on the screen so that they do not overlap with CG superimpositions, etc., and the person in charge adjusts the display position of the subtitles (height, right-justified, left-justified) by operating buttons on an operation box while closely watching the broadcast screen. However, there was a problem in that manual adjustment work was a heavy burden for the person in charge.

[0012] For this reason, a new method that places less of a burden on the person in charge of adjusting the position of subtitles was desired. Automating the operation of adjusting the position of subtitles is expected to improve the efficiency of subtitle production. Also, for programs that are currently being sent without adjusting the position of subtitles due to the need for personnel and the efficiency of subtitle production, automation would make it possible to adjust the position without additional personnel, which would also lead to improved viewer service.

[0013] To solve this problem, a method has been proposed in Japanese Patent Application No. 2022-028169, which was unpublished at the time of filing the present patent application and was made by the same applicant of the present patent application. This method detects specific objects (human faces, subtitles, maps, etc.) contained in a video, maps a score determined for each object to coordinates, and determines the subtitle display area with the smallest score as the subtitle display position.

[0014] However, in this method, since mapping is performed based on a score determined for each object, it is not always possible to obtain a subtitle display position equivalent to that obtained by mapping performed by manual adjustment. For this reason, a method that can obtain a subtitle display position equivalent to that obtained by manual adjustment has been desired.

[0015] Therefore, the present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide a subtitle display position learning device, a subtitle display position inference device, and a program that are capable of automatically adjusting the subtitle display position to an appropriate position according to the video. [Means for solving the problem]

[0016] In order to solve the above problem, the subtitle display position learning device of claim 1 is a subtitle display position learning device that learns the display position of subtitles added to a video, and is characterized by comprising: a subtitle area extraction unit that extracts a position of a subtitle area where the subtitles are displayed based on data including the subtitles as a subtitle top left coordinate indicating the top left coordinate of the subtitle area in an image coordinate system in which an image is displayed; a coordinate conversion unit that converts the top left coordinate extracted by the subtitle area extraction unit into a subtitle display position in the subtitle coordinate system in which the subtitles are displayed using a preset conversion rule; and a learning unit that uses teacher data consisting of image data to which the subtitles are not added as image data (without subtitles) and the image data (without subtitles) and the subtitle display position converted by the coordinate conversion unit to learn a relationship between the input data and the output data for a learning model in which the image data (without subtitles) is used as input data and the subtitle display position in which the subtitles are displayed in the image data (without subtitles) as output data, and generates a learned model.

[0017] Moreover, the subtitle display position inference device of claim 2 is a subtitle display position inference device that infers a display position of subtitles added to video using a trained model, characterized in that it comprises: a memory unit in which the trained model generated by the subtitle display position learning device of claim 1 is stored; and an inference unit that inputs image data to which no subtitles are added as image data (without subtitles) and infers a subtitle display position at which the subtitles will be displayed using the trained model stored in the memory unit.

[0018] The subtitle display position inference device of claim 3 is the subtitle display position inference device of claim 2, further comprising: inputting the subtitle display position inferred by the inference unit, and continuously outputting the subtitle display position for a preset time period; inputting the subtitle display position, and outputting the subtitle display position if the subtitle display position is the same a preset number of times in succession; inputting the subtitle display position, setting the currently inputted subtitle display position as a first subtitle display position and the immediately preceding inputted subtitle display position as a second subtitle display position, and outputting the first subtitle display position if a difference between the first subtitle display position and the second subtitle display position is equal to or greater than a preset threshold value; and a control output unit that controls whether to continue outputting the subtitle display position that was being output when the difference between the first image data and the second image data is equal to or greater than a preset threshold value, or to input the subtitle display position and input the image data (without subtitles) corresponding to the subtitle display position, set the currently input image data as first image data and the image data input immediately before as second image data, determine that it is time for a cut change when a difference between the first image data and the second image data is equal to or greater than a preset threshold value, and output the subtitle display position that was input at the time of the cut change, or determine that it is not time for a cut change when the difference is less than the threshold value, and continue outputting the subtitle display position that was being output.

[0019] The program of claim 4 is characterized in that it causes a computer constituting a subtitle display position learning device that learns the display position of subtitles added to video to function as: a subtitle area extraction unit that extracts a position of a subtitle area where the subtitles are displayed based on data including the subtitles as a subtitle top left coordinate indicating the top left coordinate of the subtitle area in an image coordinate system in which an image is displayed; a coordinate conversion unit that converts the top left coordinate extracted by the subtitle area extraction unit into a subtitle display position in the subtitle coordinate system in which the subtitles are displayed using a preset conversion rule; and a learning unit that uses teacher data consisting of image data to which the subtitles are not added as image data (without subtitles) and the image data (without subtitles) and the subtitle display position converted by the coordinate conversion unit to learn a relationship between the input data and the output data for a learning model in which the image data (without subtitles) is displayed as output data and the subtitle display position in which the subtitles are displayed in the image data (without subtitles) to generate a learned model.

[0020] The program of claim 5 is characterized in that the program includes a memory unit in which the trained model generated by the subtitle display position learning device of claim 1 is stored, and the computer constituting the subtitle display position inference device infers the display position of subtitles added to video using the trained model stored in the memory unit, and functions as an inference unit that inputs image data to which no subtitles are added as image data (without subtitles) and infers the subtitle display position at which the subtitles will be displayed using the trained model stored in the memory unit. Effect of the Invention

[0021] As described above, according to the present invention, the subtitle display position can be automatically adjusted to an appropriate position according to the video. [Brief description of the drawings]

[0022] [Figure 1]Fig. 1(1) is a diagram illustrating the upper left coordinates (x, y) of the subtitle and the subtitle display area (cx, cy, w, h) in the image coordinate system, and Fig. 1(2) is a diagram illustrating the subtitle display position (X, Y) in the subtitle coordinate system. [Diagram 2] 1 is a block diagram showing an example of the configuration of a subtitle display position learning device according to a first embodiment. [Diagram 3] 1 is a schematic diagram illustrating an overall process flow in a subtitle display position learning device according to a first embodiment. [Figure 4] 5 is a flowchart showing an example of processing performed by the subtitle display position learning device according to the first embodiment. [Diagram 5] FIG. 4 is a diagram illustrating an example of a data configuration of a table. [Figure 6] FIG. 2 is a diagram illustrating an example of teacher data stored in a storage unit. [Figure 7] 1 is a block diagram showing an example of the configuration of a subtitle display position inference device according to a first embodiment. [Figure 8] FIG. 2 is a schematic diagram illustrating an overall process flow in the subtitle display position inference device of the first embodiment. [Figure 9] 4 is a flowchart showing an example of processing performed by the subtitle display position inference device according to the first embodiment. [Figure 10] FIG. 11 is a block diagram showing an example of the configuration of a subtitle display position learning device according to a second embodiment. [Figure 11] FIG. 11 is a schematic diagram for explaining the overall process flow in the subtitle display position learning device of the second embodiment. [Figure 12] 13 is a flowchart showing an example of processing performed by the subtitle display position learning device according to the second embodiment. [Figure 13] FIG. 2 is a diagram illustrating an example of teacher data stored in a storage unit. [Figure 14] FIG. 11 is a block diagram showing an example of the configuration of a subtitle display position inference device according to a second embodiment. [Figure 15] FIG. 11 is a schematic diagram illustrating an overall process flow in the subtitle display position inference device according to the second embodiment. [Figure 16] 13 is a flowchart showing an example of processing performed by the subtitle display position inference device according to the second embodiment. [Figure 17]FIG. 11 is a block diagram showing an example of the configuration of a subtitle display position learning device according to a third embodiment. [Figure 18] FIG. 11 is a schematic diagram for explaining the overall process flow in the subtitle display position learning device of the third embodiment. [Figure 19] 13 is a flowchart showing an example of processing performed by the subtitle display position learning device according to the third embodiment. [Figure 20] FIG. 2 is a diagram illustrating an example of teacher data stored in a storage unit. [Figure 21] FIG. 11 is a block diagram showing an example of the configuration of a subtitle display position inference device according to a third embodiment. [Figure 22] FIG. 11 is a schematic diagram illustrating an overall process flow in the subtitle display position inference device according to the third embodiment. [Figure 23] 13 is a flowchart showing an example of processing performed by the subtitle display position inference device according to the third embodiment. [Figure 24] FIG. 4 is a diagram illustrating an example of a data configuration of a table. [Diagram 25] FIG. 13 is a diagram for explaining a subtitle display position (X, Y, Z) in a subtitle coordinate system according to a fourth embodiment. [Figure 26] FIG. 13 is a block diagram showing an example of the configuration of a subtitle display position learning device according to a fourth embodiment. [Figure 27] FIG. 13 is a schematic diagram for explaining the overall process flow in the subtitle display position learning device of the fourth embodiment. [Figure 28] 13 is a flowchart showing an example of processing performed by the subtitle display position learning device according to the fourth embodiment. [Figure 29] FIG. 2 is a diagram illustrating an example of teacher data stored in a storage unit. [Diagram 30] FIG. 13 is a block diagram showing an example of the configuration of a subtitle display position inference device according to a fourth embodiment. [Diagram 31] FIG. 13 is a schematic diagram illustrating an overall process flow in the subtitle display position inference device according to the fourth embodiment. [Diagram 32] 13 is a flowchart showing an example of processing performed by the subtitle display position inference device according to the fourth embodiment. [Diagram 33] FIG. 4 is a diagram illustrating an example of a data configuration of a table. [Diagram 34]FIG. 13 is a diagram for explaining the subtitle display position (X, Y, Z, XW) in the subtitle coordinate system of the fifth embodiment. [Diagram 35] FIG. 13 is a block diagram showing an example of the configuration of a subtitle display position inference device according to a fifth embodiment. [Diagram 36] FIG. 13 is a schematic diagram illustrating an overall process flow in the subtitle display position inference device according to the fifth embodiment. [Figure 37] 13 is a flowchart showing an example of processing performed by the subtitle display position inference device according to the fifth embodiment. [Figure 38] FIG. 4 is a diagram illustrating an example of a data configuration of a table. [Figure 39] 1A and 1B are diagrams illustrating examples of subtitle display positions before and after adjustment in a conventional subtitle display position adjustment method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. The subtitle display position learning device of the present invention is characterized in that it performs machine learning of the relationship between the video of a program and the position of the subtitles to generate a trained model. Also, the subtitle display position inference device of the present invention is characterized in that it infers the position of the subtitles to be added to the video of the program using the trained model generated by the subtitle display position learning device.

[0024] This makes it possible to automatically adjust the subtitle display position to an appropriate position according to the video.

[0025] [Definition of terms] First, the terms used in the following Examples 1, 2, 3, 4, 5, etc. are defined. Fig. 1(1) is a diagram for explaining the upper left subtitle coordinates (x, y) and the subtitle display area (cx, cy, w, h) in the image coordinate system. The image coordinate system indicates the coordinate system of an image when adding subtitles to a video of a program, and the coordinates are represented by x and y on the x-axis and y-axis.

[0026] As shown in Fig. 1 (1), it is assumed that subtitles are displayed in area α in an image. Then, the upper left subtitle coordinates (x, y) indicating the position (subtitle position) of area α where the subtitles are displayed are the upper left coordinates of area α in the image coordinate system. Also, the subtitle display area (cx, cy, w, h) indicating the position of area α is an area represented by the center coordinates (cx, cy), width w, and height h of area α in the image coordinate system.

[0027] Fig. 1(2) is a diagram for explaining the subtitle display position (X, Y) in the subtitle coordinate system. The subtitle coordinate system indicates the coordinate system of the subtitle when the subtitle is added to the video, and the coordinates are represented by X and Y on the X-axis and the Y-axis. The subtitle display position (X, Y, Z) in the fourth embodiment and the subtitle display position (X, Y, Z, XW) in the fifth embodiment will be described later.

[0028] The overall rectangular frame β shown in Fig. 1(2) corresponds to the area of ​​the image shown in Fig. 1(1). The subtitle display position (X, Y) indicates the position where the subtitle is displayed within the frame β corresponding to the image in the subtitle coordinate system, and in this example, is represented by one of 24 patterns described later.

[0029] If we divide the X and Y axes into 21 and 8 regions, respectively, X is either left-justified (1st column), center (4th column), or right-justified (7th column) (X=left-justified, center, right-justified), and Y is an integer value between 1 (1st row) and 8 (8th row) (Y=1,...,8).

[0030] As shown in FIG. 1(2), for example, if subtitles are displayed in the upper left corner of the image, the subtitle display position (X, Y) = (left-justified, 1st row), and in this case the subtitles will be displayed in area β1 from the 1st column and the 1st and 2nd rows to the 15th column and the 1st and 2nd rows.

[0031] For example, if the subtitles are displayed in the center of the image, the subtitle display position (X, Y) = (center, 4th row), and the subtitles in this case will be displayed in area β2 from column 4 and rows 4 and 5 to column 18 and rows 4 and 5. For example, if the subtitles are displayed in the bottom right of the image, the subtitle display position (X, Y) = (right-justified, 7th row), and the subtitles in this case will be displayed in area β3 from column 7 and rows 7 and 8 to column 21 and rows 7 and 8.

[0032] Thus, in the example of Figure 1(2), the subtitle placement is a combination of three horizontal patterns represented by X: "left-justified," "center," and "right-justified," and eight vertical patterns represented by Y: "1st line" through "8th line."

[0033] In other words, when the screen is divided into a mesh-like display frame of 21 columns x 8 columns, the subtitle display position (X, Y) is determined by dividing the screen into 21 columns x 8 columns of consecutive display frames of 15 columns x 2 rows into one subtitle display frame, and the subtitle display position (X, Y) can be any of a total of 8 x 3 = 24 patterns, including combinations of "right-justified," "center," or "left-justified" horizontally and "first row" to "eighth row" vertically.

[0034] In the following first, second, third, fourth, and fifth embodiments, as well as the first and second embodiments, the subtitle display position learning device generates a trained model by performing training using image data to which no subtitles are added and subtitle position information as teacher data. The subtitle display position inference device infers subtitle position information from image data to which no subtitles are added, using the trained model generated by the subtitle display position learning device.

[0035] Example 1 Next, a description will be given of Example 1. Example 1 is an example in which image data is extracted from video data to which no subtitles are added, a trained model is generated using the image data to which no subtitles are added and the subtitle display position (X, Y) as training data, and the trained model is used to infer the subtitle display position (X, Y).

[0036] Hereinafter, video data without subtitles will be referred to as "video data (without subtitles)," and video data with subtitles added, which will be described later, will be referred to as "video data (with subtitles)." Furthermore, image data without subtitles will be referred to as "image data (without subtitles)," and image data with subtitles added, which will be described later, will be referred to as "image data (with subtitles)."

[0037] For example, the video data (without subtitles) is video of a program produced in a studio and broadcast, and the video data (with subtitles) is video with actual subtitles displayed by a receiver.

[0038] (Example 1 / Subtitle display position learning device) First, a description will be given of a subtitle display position learning device according to the first embodiment. Fig. 2 is a block diagram showing a configuration example of the subtitle display position learning device according to the first embodiment, and Fig. 3 is a schematic diagram showing an overall process flow in the subtitle display position learning device according to the first embodiment.

[0039] The subtitle display position learning device 1-1 includes an image clipping unit 10, a subtitle area extraction unit 11, a table 12, a coordinate conversion unit 13, a storage unit 14, a learning unit 15, and a memory unit 16.

[0040] The subtitle display position learning device 1-1 inputs video data (without subtitles) and video data (with subtitles) (the video data (with subtitles) in FIG. 3 shows an area γ to which subtitles have been added. In this area γ, subtitles with a gray background, for example, are displayed). Then, the subtitle display position learning device 1-1 generates a subtitle display position (X, Y) using the image cropping unit 10, the subtitle area extraction unit 11, the table 12, and the coordinate conversion unit 13, thereby generating teacher data consisting of the image data (without subtitles) and the subtitle display position (X, Y) (for example, (X, Y) = (right-justified, seventh line) with reference to FIG. 3) (step S301), and stores this in the storage unit 14.

[0041] The subtitle display position learning device 1-1 reads the teacher data from the storage unit 14 in the learning unit 15, learns the relationship between the image data (without subtitles) and the subtitle display position (X, Y) using the teacher data, and generates a learned model (step S302). Then, the subtitle display position learning device 1-1 stores the learned model in the storage unit 16.

[0042] 4 is a flowchart showing an example of the process of the subtitle display position learning device 1-1 of Example 1. Hereinafter, the example of the process of the subtitle display position learning device 1-1 will be described with reference to FIGS.

[0043] The image cropping unit 10 receives as input video data (without subtitles) and video data (with subtitles) synchronized with (corresponding to) the video data (without subtitles) (step S401).

[0044] The image cutout unit 10 cuts out still images from the video data (without subtitles) and the video data (with subtitles), respectively, to generate image data (without subtitles) and image data (with subtitles) (step S402). For example, the image cutout unit 10 cuts out still images at preset time intervals (for example, every 10 to 60 seconds).

[0045] The image cropping unit 10 outputs the image data (without subtitles) and the image data (with subtitles) to a subtitle region extraction unit 11, and also stores the image data (without subtitles) in a storage unit .

[0046] The subtitle region extraction unit 11 inputs image data (without subtitles) and corresponding image data (with subtitles) from the image cropping unit 10. Then, the subtitle region extraction unit 11 uses the image data (without subtitles) and the image data (with subtitles) to extract a subtitle region (region where subtitles exist) by, for example, a background subtraction method, and obtains the upper left subtitle coordinates (x, y) from the subtitle region (step S403). The subtitle region extraction unit 11 outputs the upper left subtitle coordinates (x, y) to the coordinate conversion unit 13.

[0047] The coordinate conversion unit 13 inputs the upper left subtitle coordinates (x, y) from the subtitle area extraction unit 11, and reads out the subtitle display position (X, Y) corresponding to the upper left subtitle coordinates (x, y) from the table 12, thereby converting the upper left subtitle coordinates (x, y) to a subtitle display position (X, Y) (step S404).

[0048] 5 is a diagram showing an example of the data configuration of table 12. In this table 12, a plurality of sets of data are stored, each set being a top left subtitle coordinate (x, y) and a corresponding subtitle display position (X, Y).

[0049] The relationship between the upper left subtitle coordinates (x, y) indicating the position of the subtitle added to the video data (with subtitles) and the corresponding subtitle display position (X, Y) can be set in advance, and the correspondence data indicating this relationship is stored in table 12.

[0050] The coordinate conversion unit 13 may perform the conversion process using an arithmetic expression in which the conversion rule is defined, instead of the table 12 in which the conversion rule is stored. The same applies to the coordinate conversion unit 21 shown in Fig. 17 and the coordinate conversion unit 26 shown in Fig. 26, which will be described later.

[0051] 2 and 4, the coordinate conversion unit 13 stores the subtitle display position (X, Y) in the storage unit 14. That is, the image cropping unit 10 and the coordinate conversion unit 13 store the image data (without subtitles) and the corresponding subtitle display position (X, Y) in the storage unit 14 as teacher data (step S405).

[0052] Fig. 6 is a diagram showing an example of teacher data stored in the storage unit 14. As shown in Fig. 6, the storage unit 14 stores a plurality of sets of data (a plurality of teacher data) with each set of image data (without subtitles) and the corresponding subtitle display position (X, Y) as a unit of teacher data.

[0053] 2 and 4, the learning unit 15 reads out teacher data consisting of image data (without subtitles) and the corresponding subtitle display position (X, Y) from the storage unit 14. Then, using the teacher data, the learning unit 15 learns the relationship between the input data and the output data by deep learning for a learning model in which the image data (without subtitles) is used as input data and the subtitle display position (X, Y) is used as output data (step S406).

[0054] The learning unit 15 generates a trained model by performing learning using a predetermined number of teacher data, and stores the trained model in the storage unit 16 (step S407).

[0055] As described above, according to the subtitle display position learning device 1-1 of the first embodiment, the subtitle area extraction unit 11 extracts a subtitle area using image data (without subtitles) and image data (with subtitles) and obtains the upper left subtitle coordinate (x, y) from the subtitle area, and the coordinate conversion unit 13 converts the upper left subtitle coordinate (x, y) into a subtitle display position (X, Y) using the table 12.

[0056] In the storage unit 14, image data (without subtitles) and the subtitle display position (X, Y) are stored as teacher data.

[0057] The learning unit 15 learns these relationships using training data consisting of image data (without subtitles) and subtitle display positions (X, Y), and generates a learned model.

[0058] This makes it possible to obtain a trained model that follows the conventional method of manually adjusting the subtitle display position. Then, by using this trained model, a subtitle display position inference device 2-1 of a first embodiment described later can automatically adjust the subtitle display position (X, Y) to an appropriate position according to the video.

[0059] (Example 1 / subtitle display position inference device) Next, a description will be given of a subtitle display position inference device according to the first embodiment. Fig. 7 is a block diagram showing a configuration example of the subtitle display position inference device according to the first embodiment, and Fig. 8 is a schematic diagram showing an overall process flow in the subtitle display position inference device according to the first embodiment.

[0060] This subtitle display position inference device 2-1 includes an image cropping unit 30, a storage unit 31, an inference unit 32, and a control output unit 33. The storage unit 31 stores a trained model generated by the subtitle display position learning device 1-1 shown in FIG.

[0061] The subtitle display position inference device 2-1 inputs video data (without subtitles) and cuts out image data (without subtitles) in the image cutout unit 30 (step S801). Then, the subtitle display position inference device 2-1 infers the subtitle display position (X, Y) (for example, (X, Y)=(center, fifth line) with reference to FIG. 8) from the image data (without subtitles) using the trained model stored in the storage unit 31 in the inference unit 32 (step S802). Then, the subtitle display position inference device 2-1 controls the subtitle display position (X, Y) in the control output unit 33.

[0062] 9 is a flowchart showing an example of the process of the subtitle display position inference device 2-1 of Example 1. Hereinafter, an example of the process of the subtitle display position inference device 2-1 will be described with reference to FIGS.

[0063] The image cropping unit 30 inputs video data (without subtitles) (step S901), crops out still images from the video data (without subtitles), and generates image data (without subtitles) (step S902). For example, the image cropping unit 30 inputs video data (without subtitles) for broadcasting in HD-SDI format, and crops out cropped images of still images for broadcasting at preset intervals (for example, 0.3 to 5 seconds). The image cropping unit 30 outputs the image data (without subtitles) to the inference unit 32.

[0064] The inference unit 32 inputs the image data (without subtitles) from the image cropping unit 30, reads the trained model from the storage unit 31, and infers the subtitle display position (X, Y) from the image data (without subtitles) using the trained model (step S903). Then, the inference unit 32 outputs the subtitle display position (X, Y) to the control output unit 33.

[0065] The control output unit 33 receives the subtitle display position (X, Y) from the inference unit 32, performs predetermined control using the subtitle display position (X, Y) (step S904), and outputs the controlled subtitle display position (X, Y) (step S905).

[0066] The subtitle display position (X, Y) output by the subtitle display position inference device 2-1 needs to be an optimal position selected within the image data (without subtitles). However, the current subtitle display position (X, Y) inferred using the trained model does not take into account the subtitle display position (X, Y) inferred in the previous and previous scans. For this reason, for example, even if the background image has not changed significantly, the inferred subtitle display position (X, Y) may change frequently. Frequent changes to the subtitle display position (X, Y) are not appropriate because they cause discomfort to the viewer.

[0067] Therefore, the subtitle display position inference device 2-1 is provided with a control output unit 33, and the control output unit 33 performs control so that the subtitle display position (X, Y) is not frequently changed. The same applies to the second and third embodiments described later. The same applies to the control output unit 46 provided in the subtitle display position inference device 2-4 of the fourth embodiment described later and the control output unit 51 provided in the subtitle display position inference device 2-5 of the fifth embodiment described later.

[0068] (First control) Specifically, the control output unit 33 continuously outputs the subtitle display position (X, Y) input from the inference unit 32 for a preset time period, and after the time period has elapsed, continues to output the subtitle display position (X, Y) input from the inference unit 32 at that time (by scanning) for a similarly preset time period.

[0069] As a result, the subtitle display position (X, Y) output from the control output unit 33 is not changed for a preset time period, making it possible to suppress frequent changes and preventing discomfort to the viewer.

[0070] (Second control) Furthermore, the control output unit 33 judges whether or not the subtitle display position (X, Y) input from the inference unit 32 is the same for a preset number of times (number of scans). If the control output unit 33 judges that the subtitle display position (X, Y) is the same for a preset number of times, it outputs the subtitle display position (X, Y), and if it judges that the subtitle display position (X, Y) is not the same for a preset number of times, it continues to output the subtitle display position (X, Y) that has been output so far.

[0071] As a result, the subtitle display position (X, Y) output from the control output unit 33 is not changed for a preset number of times, making it possible to suppress frequent changes and preventing discomfort to the viewer.

[0072] (Third control) Furthermore, the control output unit 33 obtains the difference between the subtitle display position (X, Y) of the current scan (first subtitle display position (X, Y)) input from the inference unit 32 and the subtitle display position (X, Y) of the immediately preceding (one before) scan (second subtitle display position (X, Y)). Then, the control output unit 33 compares the difference with a preset threshold, and when it is determined that the difference is equal to or greater than the threshold (when it is determined that the difference in X is equal to or greater than the threshold, or the difference in Y is equal to or greater than the threshold), it determines that the two positions are significantly different, and outputs the subtitle display position (X, Y) of the current scan. On the other hand, when it is determined that the difference is less than the threshold (when it is determined that the difference in X is less than the threshold and the difference in Y is less than the threshold), it determines that the two positions are not significantly different, and continues to output the subtitle display position (X, Y) that has been output up until now (output in the previous scan).

[0073] For example, assume that the thresholds are preset to an X difference (column difference) of 1 or more and a Y difference (row difference) of 3 or more. In this case, if the subtitle display position of the current scan is (X, Y) = (left justified, sixth row) and the subtitle display position of the previous scan is (X, Y) = (left justified, second row), the difference in X is 0 and the difference in Y is 4. Then, it is determined that the difference in Y is greater than or equal to the threshold (the two positions are determined to be significantly different), and the subtitle display position of the current scan is output as (X, Y) = (left justified, sixth row).

[0074] Furthermore, if the subtitle display position of the current scan is (X, Y) = (left justified, sixth line) and the subtitle display position of the previous scan is (X, Y) = (left justified, fifth line), then the difference in X is 0 and the difference in Y is 1. Then, it is determined that the difference in X is less than the threshold value and the difference in Y is less than the threshold value (it is determined that the two positions are not significantly different), and the subtitle display position (X, Y) = (left justified, fifth line) that has been output up until now will continue to be output.

[0075] Furthermore, if the subtitle display position of the current scan is (X, Y) = (left justified, 6th line) and the subtitle display position of the previous scan is (X, Y) = (center, 6th line), the difference in X is 3 (see Figure 1 (2)) and the difference in Y is 0. Then, it is determined that the difference in X is equal to or greater than the threshold (the two positions are determined to be significantly different), and the subtitle display position of the current scan is output as (X, Y) = (left justified, 6th line).

[0076] As a result, the subtitle display position (X, Y) output from the control output unit 33 is not changed when the difference between the subtitle display position (X, Y) of the current scan and the subtitle display position (X, Y) of the immediately previous scan is less than the threshold, that is, when it is determined that the two positions will not differ significantly, so frequent changes can be suppressed and the viewer will not feel uncomfortable.

[0077] (Fourth control) Furthermore, the control output unit 33 inputs image data (without subtitles) from the image cropping unit 30, and finds the difference between the input image data (without subtitles) of the current scan (first image data (without subtitles)) and the image data (without subtitles) of the immediately preceding (previous) scan (second image data (without subtitles)). The control output unit 33 then compares the difference with a preset threshold, and if it determines that the difference is equal to or greater than the threshold, it determines that it is time to switch cuts, and outputs the subtitle display position (X, Y) of the current scan, which is the timing for switching cuts. On the other hand, if it determines that the difference is less than the threshold, it determines that it is not time to switch cuts, and continues to output the subtitle display position (X, Y) that has been output up until now (output in the previous scan).

[0078] For example, the control output unit 33 outputs image data of the nth frame of the current scan (without subtitles) as I n (x,y), and the image data of the n-1th frame of the previous scan (without subtitles) is I n-1 Let (x,y) be the difference image I d Calculate (x, y) using the following formula. [Number 1] I d (x,y)=|I n (x,y)-I n-1 (x,y)| (1)

[0079] The control output unit 33 outputs the difference image I d Average value of (x,y) I' d is calculated by the following formula: N is the difference image I d (x, y) is the number of pixels in the x-axis direction, and M is the number of pixels in the y-axis direction. [Number 2] TIFF2024176222000002.tif20170

[0080] The control output unit 33 outputs the difference image I d Average value of (x,y) I' dBy comparing this with a preset threshold, as described above, it is determined whether it is time to switch cuts, and either outputs the subtitle display position (X,Y) for this scan, or continues to output the subtitle display position (X,Y) that has been output up to now.

[0081] As a result, the subtitle display position (X, Y) output from the control output unit 33 is not changed when the difference between the image data (without subtitles) of the current scan and the image data (without subtitles) of the immediately previous scan is less than the threshold, that is, when it is determined that it is not the time to switch cuts, so that frequent changes can be suppressed and the viewer will not feel uncomfortable.

[0082] Here, the control output unit 33 may perform any one of the above-mentioned first to fourth controls, or may perform two or three of the above-mentioned controls, or may perform all of the above-mentioned controls. For example, when performing all of the controls, the control output unit 33 outputs the subtitle display position (X, Y) if the above-mentioned first control satisfies a condition that the time period has elapsed since the subtitle display position (X, Y) was input, and the above-mentioned second control satisfies a condition that the input subtitle display position (X, Y) is determined to be the same for a preset number of consecutive times, and the above-mentioned third control satisfies a condition that the difference is determined to be equal to or greater than a threshold value and that the two positions are significantly different, and the above-mentioned fourth control satisfies a condition that the difference is determined to be equal to or greater than a threshold value and that it is the timing for cutting, and if at least one of these conditions is not satisfied, the control output unit 33 continues to output the subtitle display position (X, Y) that has been output up to now.

[0083] As described above, according to the subtitle display position inference device 2-1 of the first embodiment, the inference unit 32 infers the subtitle display position (X, Y) from image data (without subtitles) by using the trained model generated by the subtitle display position learning device 1-1 of the first embodiment.

[0084] The control output unit 33 performs a predetermined control so that the caption display position (X, Y) is not frequently changed, and outputs the caption display position (X, Y) after the control.

[0085] As a result, the subtitle display position (X, Y) is inferred using a trained model that follows the conventional method of manually adjusting the subtitle display position, so the subtitle display position (X, Y) can be automatically adjusted to an appropriate position according to the video.

[0086] Example 2 Next, a description will be given of Example 2. Example 2 is an example in which image data (without subtitles) is extracted from video data (without subtitles), a trained model is generated using the image data (without subtitles) and the upper left coordinates (x, y) of the subtitles as training data, and the trained model is used to infer the upper left coordinates (x, y) of the subtitles.

[0087] (Example 2 / Subtitle display position learning device) First, a description will be given of a subtitle display position learning device according to the second embodiment. Fig. 10 is a block diagram showing a configuration example of the subtitle display position learning device according to the second embodiment, and Fig. 11 is a schematic diagram showing an overall process flow in the subtitle display position learning device according to the second embodiment.

[0088] The subtitle display position learning device 1-2 includes an image clipping unit 10, a subtitle area extraction unit 11, a storage unit 17, a learning unit 18, and a memory unit 19.

[0089] The subtitle display position learning device 1-2 inputs video data (without subtitles) and video data (with subtitles) (the video data (with subtitles) in FIG. 11 shows an area γ where subtitles are added). Then, the subtitle display position learning device 1-2 generates teacher data consisting of image data (without subtitles) and the upper left subtitle coordinates (x, y) (for example, (x, y)=(420, 900) with reference to FIG. 11) by generating the upper left subtitle coordinates (x, y) in the image cropping unit 10 and the subtitle area extraction unit 11 (step S1101), and stores this in the storage unit 17.

[0090] The subtitle display position learning device 1-2 reads the teacher data from the storage unit 17 in the learning unit 18, learns the relationship between the image data (without subtitles) and the upper left coordinates (x, y) of the subtitles using the teacher data, and generates a learned model (step S1102). Then, the subtitle display position learning device 1-2 stores the learned model in the storage unit 19.

[0091] 12 is a flowchart showing an example of processing of the subtitle display position learning device 1-2 of Example 2. Hereinafter, an example of processing of the subtitle display position learning device 1-2 will be described with reference to FIGS.

[0092] Similar to steps S401 and S402 in FIG. 4, the image cropping unit 10 inputs video data (without subtitles) and video data (with subtitles) (step S1201), crops out still images, and generates image data (without subtitles) and image data (with subtitles) (step S1202).

[0093] The image cropping unit 10 outputs the image data (without subtitles) and the image data (with subtitles) to a subtitle region extraction unit 11, and also stores the image data (without subtitles) in a storage unit 17.

[0094] The subtitle region extraction unit 11 inputs the image data (without subtitles) and the image data (with subtitles) from the image cropping unit 10. Then, similar to step S403 in Fig. 4, the subtitle region extraction unit 11 extracts a subtitle region using the image data (without subtitles) and the image data (with subtitles), and obtains the upper left coordinates (x, y) of the subtitle from the subtitle region (step S1203).

[0095] The subtitle region extraction unit 11 stores the subtitle upper left coordinates (x, y) in the storage unit 17. That is, the image cropping unit 10 and the subtitle region extraction unit 11 store the image data (without subtitles) and the corresponding subtitle upper left coordinates (x, y) in the storage unit 17 as teacher data (step S1204).

[0096] Fig. 13 is a diagram showing an example of teacher data stored in the storage unit 17. As shown in Fig. 13, the storage unit 17 stores a plurality of sets of data (a plurality of teacher data) with each set being a single teacher data set consisting of image data (without subtitles) and the corresponding upper left coordinates (x, y) of the subtitles.

[0097] 10 and 12, the learning unit 18 reads out teacher data consisting of image data (without subtitles) and the corresponding upper left coordinates (x, y) of the subtitles from the storage unit 17. Then, using the teacher data, the learning unit 18 learns the relationship between the input data and the output data by deep learning for a learning model in which the image data (without subtitles) is used as input data and the upper left coordinates (x, y) of the subtitles are used as output data (step S1205).

[0098] The learning unit 18 generates a trained model by performing learning using a predetermined number of teacher data, and stores the trained model in the storage unit 19 (step S1206).

[0099] As described above, according to the subtitle display position learning device 1-2 of the second embodiment, the subtitle area extraction unit 11 extracts a subtitle area using image data (without subtitles) and image data (with subtitles), and obtains the upper left subtitle coordinates (x, y) from the subtitle area.

[0100] In the storage unit 17, image data (without subtitles) and the upper left coordinates (x, y) of the subtitles are stored as teacher data.

[0101] The learning unit 18 learns these relationships using training data consisting of image data (without subtitles) and the upper left coordinates (x, y) of the subtitles, and generates a learned model.

[0102] This provides the same effect as the subtitle display position learning device 1-1 of the first embodiment, and the subtitle display position inference device 2-2 of the second embodiment described later can automatically adjust the subtitle display position (X, Y) to an appropriate position according to the video by using this trained model.

[0103] (Example 2 / subtitle display position inference device) Next, a description will be given of a subtitle display position inference device according to a second embodiment. Fig. 14 is a block diagram showing a configuration example of the subtitle display position inference device according to the second embodiment, and Fig. 15 is a schematic diagram showing an overall process flow in the subtitle display position inference device according to the second embodiment.

[0104] This subtitle display position inference device 2-2 includes an image cropping unit 30, a storage unit 34, an inference unit 35, a table 36, a coordinate conversion unit 37, and a control output unit 33. The storage unit 34 stores a trained model generated by the subtitle display position learning device 1-2 shown in FIG.

[0105] The subtitle display position inference device 2-2 inputs video data (without subtitles) and cuts out the image data (without subtitles) in the image cutout unit 30 (step S1501). Then, the subtitle display position inference device 2-2 infers the upper left subtitle coordinates (x, y) (for example, (x, y)=(200, 100) with reference to FIG. 15) in the inference unit 35 using the trained model stored in the storage unit 34 (step S1502).

[0106] The subtitle display position inference device 2-2 converts the upper left subtitle coordinates (x, y) into the subtitle display position (X, Y) (for example, (X, Y)=(center, fifth row) with reference to FIG. 15) using the table 36 and the coordinate conversion unit 37 (step S1503), and controls the subtitle display position (X, Y) using the control output unit 33.

[0107] 16 is a flowchart showing an example of processing of the subtitle display position inference device 2-2 of Example 2. Hereinafter, an example of processing of the subtitle display position inference device 2-2 will be described with reference to FIGS.

[0108] 9, the image cutout unit 30 receives video data (without subtitles) (step S1601), cuts out still images, and generates image data (without subtitles) (step S1602). The image cutout unit 30 outputs the image data (without subtitles) to the inference unit 35.

[0109] The inference unit 35 inputs the image data (without subtitles) from the image cropping unit 30, reads the trained model from the storage unit 34, and infers the upper left subtitle coordinates (x, y) from the image data (without subtitles) using the trained model (step S1603). Then, the inference unit 35 outputs the upper left subtitle coordinates (x, y) to the coordinate conversion unit 37.

[0110] The coordinate conversion unit 37 inputs the upper left subtitle coordinates (x, y) from the inference unit 35, and converts the upper left subtitle coordinates (x, y) to a subtitle display position (X, Y) by reading the subtitle display position (X, Y) corresponding to the upper left subtitle coordinates (x, y) from the table 36 (step S1604). An example of the data configuration of the table 36 is similar to the example of the data configuration of the table 12 shown in FIG. 5. The coordinate conversion unit 37 outputs the subtitle display position (X, Y) to the control output unit 33.

[0111] The coordinate conversion unit 37 may perform the conversion process using an arithmetic expression in which the conversion rule is defined, instead of the table 36 in which the conversion rule is stored. The same applies to the area conversion unit 42 shown in Fig. 21, the area conversion unit 45 shown in Fig. 30, and the area conversion unit 50 shown in Fig. 35, which will be described later.

[0112] The control output unit 33 receives the subtitle display position (X, Y) from the coordinate conversion unit 37, performs the same control as the control output unit 33 in FIG. 7, and outputs the controlled subtitle display position (X, Y).

[0113] As described above, according to the subtitle display position inference device 2-2 of the second embodiment, the inference unit 35 infers the upper left subtitle coordinates (x, y) from image data (without subtitles) using the trained model generated by the subtitle display position learning device 1-2 of the second embodiment.

[0114] The coordinate conversion unit 37 converts the upper left subtitle coordinates (x, y) into the subtitle display position (X, Y) using the table 36. The control output unit 33 performs a predetermined control so that the subtitle display position (X, Y) is not frequently changed, and outputs the controlled subtitle display position (X, Y).

[0115] This provides the same effect as the subtitle display position inference device 2-1 of the first embodiment, and makes it possible to automatically adjust the subtitle display position (X, Y) to an appropriate position according to the video.

[0116] Example 3 Next, a description will be given of Example 3. Example 3 is an example in which image data (without subtitles) is cut out from video data (without subtitles), a trained model is generated using the image data (without subtitles), the subtitle display position (X, Y), and the subtitle display area (cx, cy, w, h) as training data, two subtitle display areas (cx, cy, w, h) are inferred in descending order of score using the trained model, one of the two subtitle display areas (cx, cy, w, h) is selected based on the number of characters C of the subtitles, and the subtitle display position (X, Y) is obtained.

[0117] In the above-mentioned first and second embodiments, learning is performed using the subtitle display position (X, Y) and the upper left coordinates (x, y) of the subtitle, which indicate the position where the subtitle is displayed, but learning is not performed using data of the rectangular area where the subtitle is displayed. Therefore, in the first and second embodiments, when the subtitle display position (X, Y) is calculated, there is a possibility that the subtitle may overlap with the background.

[0118] Therefore, in the third embodiment, learning is performed using the subtitle display area (cx, cy, w, h), which is data on the rectangular area in which the subtitles are displayed, so that the subtitles do not overlap with the background. This makes it possible to find an appropriate subtitle display position (X, Y) according to the number of characters C in the subtitles.

[0119] (Example 3 / Subtitle display position learning device) First, a description will be given of a subtitle display position learning device according to a third embodiment. Fig. 17 is a block diagram showing a configuration example of the subtitle display position learning device according to the third embodiment, and Fig. 18 is a schematic diagram showing an overall process flow in the subtitle display position learning device according to the third embodiment.

[0120] The subtitle display position learning device 1-3 includes an image clipping unit 10, a subtitle area extraction unit 20, a table 12, a coordinate conversion unit 21, a storage unit 22, a learning unit 23, and a memory unit 24.

[0121] The subtitle display position learning device 1-3 inputs video data (without subtitles) and video data (with subtitles) (the video data (with subtitles) in FIG. 18 shows an area γ where subtitles are added). Then, the subtitle display position learning device 1-3 generates the upper left subtitle coordinates (x, y) and the subtitle display area (cx, cy, w, h) in the image clipping unit 10 and the subtitle area extraction unit 20, and generates the subtitle display position (X, Y) in the table 12 and the coordinate conversion unit 21, thereby generating teacher data consisting of the image data (without subtitles), the subtitle display position (X, Y) (for example, (X, Y)=(right-justified, 7th row) with reference to FIG. 18), and the subtitle display area (cx, cy, w, h) (for example, (cx, cy, w, h)=(960, 950, 1080, 100) with reference to FIG. 18) (step S1801), and stores this in the storage unit 22.

[0122] The subtitle display position learning device 1-3 reads the training data from the storage unit 22 in the learning unit 23, and uses the training data to learn the relationship between the image data (without subtitles) and the subtitle display position (X, Y) and the subtitle display area (cx, cy, w, h) to generate a trained model (step S1802). Then, the subtitle display position learning device 1-3 stores the trained model in the storage unit 24.

[0123] 19 is a flowchart showing an example of processing of the subtitle display position learning device 1-3 of Example 3. Hereinafter, an example of processing of the subtitle display position learning device 1-3 will be described with reference to FIGS.

[0124] Similar to steps S401 and S402 in FIG. 4, the image cropping unit 10 inputs video data (without subtitles) and video data (with subtitles) (step S1901), crops out still images, and generates image data (without subtitles) and image data (with subtitles) (step S1902).

[0125] The image cropping unit 10 outputs the image data (without subtitles) and the image data (with subtitles) to a subtitle region extraction unit 20, and also stores the image data (without subtitles) in a storage unit 22.

[0126] The subtitle region extraction unit 20 inputs image data (without subtitles) and image data (with subtitles) from the image cropping unit 10. Then, the subtitle region extraction unit 20 extracts a subtitle region using the image data (without subtitles) and image data (with subtitles), for example, by a background subtraction method, and obtains the upper left subtitle coordinates (x, y) and the subtitle display area (cx, cy, w, h) from the subtitle region (step S1903). The subtitle region extraction unit 20 outputs the upper left subtitle coordinates (x, y) and the subtitle display area (cx, cy, w, h) to the coordinate conversion unit 21.

[0127] The coordinate conversion unit 21 receives the upper left subtitle coordinates (x, y) and the subtitle display area (cx, cy, w, h) from the subtitle area extraction unit 20. Then, similar to step S404 in Fig. 4, the coordinate conversion unit 21 reads out the subtitle display position (X, Y) corresponding to the upper left subtitle coordinates (x, y) from table 12, thereby converting the upper left subtitle coordinates (x, y) into the subtitle display position (X, Y) (step S1904).

[0128] The coordinate conversion unit 21 stores the subtitle display position (X, Y) and the subtitle display area (cx, cy, w, h) in the storage unit 22. That is, the image cropping unit 10 and the coordinate conversion unit 21 store the image data (without subtitles) and the corresponding subtitle display position (X, Y) and subtitle display area (cx, cy, w, h) in the storage unit 22 as teacher data (step S1905).

[0129] Fig. 20 is a diagram showing an example of teacher data stored in the storage unit 22. As shown in Fig. 20, the storage unit 22 stores a plurality of sets of data (a plurality of teacher data) with one teacher data set being a set of image data (without subtitles) and the corresponding subtitle display position (X, Y) and subtitle display area (cx, cy, w, h).

[0130] 17 and 19, the learning unit 23 reads out training data consisting of image data (without subtitles) and the corresponding subtitle display position (X, Y) and subtitle display area (cx, cy, w, h) from the storage unit 22. Then, using the training data, the learning unit 23 learns the relationship between the input data and the output data by deep learning for a learning model in which the image data (without subtitles) is used as input data and the subtitle display area (cx, cy, w, h) is used as output data (step S1906).

[0131] The learning unit 23 performs learning using a predetermined number of teacher data to generate a trained model, and stores the trained model in the storage unit 24 (step S1907).

[0132] As described above, according to the subtitle display position learning device 1-3 of the third embodiment, the subtitle area extraction unit 20 extracts a subtitle area using image data (without subtitles) and image data (with subtitles), and obtains the upper left subtitle coordinates (x, y) and the subtitle display area (cx, cy, w, h) from the subtitle area, and the coordinate conversion unit 21 converts the upper left subtitle coordinates (x, y) into the subtitle display position (X, Y) using table 12.

[0133] In the storage unit 22, image data (without subtitles), the subtitle display position (X, Y) and the subtitle display area (cx, cy, w, h) are stored as teacher data.

[0134] The learning unit 23 performs learning using training data consisting of image data (without subtitles) and the subtitle display position (X, Y) and the subtitle display area (cx, cy, w, h), and generates a learned model.

[0135] This provides the same effect as the subtitle display position learning device 1-1 of the first embodiment, and the subtitle display position inference device 2-3 of the third embodiment described later can automatically adjust the subtitle display position (X, Y) to an appropriate position according to the video by using this trained model.

[0136] In particular, the subtitle display position learning device 1-3 of the third embodiment generates a trained model using image data (without subtitles) as input data and using the subtitle display position (X, Y) and the subtitle display area (cx, cy, w, h) as output data. As a result, the subtitle display position inference device 2-3 of the third embodiment described later can use this trained model to infer, for example, two subtitle display areas (cx, cy, w, h) with high scores, select one of the two subtitle display areas (cx, cy, w, h) based on the number of characters C of the subtitles actually displayed, and determine the subtitle display position (X, Y). This subtitle display area (cx, cy, w, h) includes the size at which the subtitles are displayed.

[0137] In contrast, in the first and second embodiments, the subtitle display position (X, Y) and the upper left coordinates (x, y) of the subtitle can be inferred by using a trained model, but the size at which the subtitles are displayed cannot be obtained. Therefore, the size at which the subtitles are displayed must be determined by the user, and for example, the user must specify the line break position and the length of the characters.

[0138] Therefore, according to the third embodiment, the inferred subtitle display area (cx, cy, w, h) includes the size at which the subtitles are displayed, eliminating the need for the user to determine and specify the size. This reduces the user's workload compared to the first and second embodiments, and increases the accuracy of the inferred subtitle display position (X, Y).

[0139] (Example 3 / subtitle display position inference device) Next, a description will be given of a subtitle display position inference device according to a third embodiment. Fig. 21 is a block diagram showing a configuration example of the subtitle display position inference device according to the third embodiment, and Fig. 22 is a schematic diagram showing an overall process flow in the subtitle display position inference device according to the third embodiment.

[0140] This subtitle display position inference device 2-3 includes an image clipping unit 30, a storage unit 39, an inference unit 40, a table 41, a region conversion unit 42, and a control output unit 33. The storage unit 39 stores a trained model generated by the subtitle display position learning device 1-3 shown in FIG.

[0141] The subtitle display position inference device 2-3 inputs video data (without subtitles) and cuts out the image data (without subtitles) in the image cutout unit 30 (step S2201). Then, the subtitle display position inference device 2-3 infers, in the inference unit 40, two subtitle display areas (cxA, cyA, wA, hA) and (cxB, cyB, wB, hB) in descending order of score, for example, using the trained model stored in the storage unit 39 (step S2202).

[0142] The subtitle display position inference device 2-3 inputs subtitle character data to be added to the video data (without subtitles) in the area conversion unit 42 to obtain the number of characters C, converts the subtitle display area (cxA, cyA, wA, hA), (cxB, cyB, wB, hB) to the subtitle display positions (XA, YA, CNA), (XB, YB, CNB) using the table 41, selects one of the subtitle display positions (XA, YA, CNA) and (XB, YB, CNB) based on the number of characters C, and obtains the subtitle display position (X, Y) (for example, (X, Y)=(center, fifth row) with reference to FIG. 22) (step S2203). Then, the subtitle display position inference device 2-3 controls the subtitle display position (X, Y) in the control output unit 33.

[0143] Here, (X,Y) in the subtitle display position (X,Y,CN) ((XA,YA,CNA),(XB,YB,CNB)) is the position on the X and Y axes where the subtitle is displayed in the subtitle coordinate system as explained in Fig. 1 (2). (CN) is the number of characters in the subtitle.

[0144] 23 is a flowchart showing an example of processing of the subtitle display position inference device 2-3 of Example 3. Hereinafter, an example of processing of the subtitle display position inference device 2-3 will be described with reference to FIGS.

[0145] 9, the image cutout unit 30 receives video data (without subtitles) (step S2301), cuts out still images, and generates image data (without subtitles) (step S2302). The image cutout unit 30 outputs the image data (without subtitles) to the inference unit 40.

[0146] The inference unit 40 inputs image data (without subtitles) from the image cropping unit 30, reads out the trained model from the storage unit 39, and infers the subtitle display area (cx, cy, w, h) from the image data (without subtitles) using the trained model, and selects the subtitle display area (cxA, cyA, wA, hA) with the highest score and the subtitle display area (cxB, cyB, wB, hB) with the next highest score (step S2303). Then, the inference unit 40 outputs the subtitle display areas (cxA, cyA, wA, hA) and (cxB, cyB, wB, hB) to the area conversion unit 42.

[0147] The area conversion unit 42 inputs the subtitle display areas (cxA, cyA, wA, hA), (cxB, cyB, wB, hB) from the inference unit 40, and reads out the subtitle display positions (X, Y, CN) corresponding to the subtitle display area (cx, cy, w, h) from table 41, thereby converting the subtitle display areas (cxA, cyA, wA, hA), (cxB, cyB, wB, hB) to subtitle display positions (XA, YA, CNA), (XB, YB, CNB) (step S2304).

[0148] Area conversion unit 42 inputs the subtitle character data to be added to the video data (without subtitles), and calculates the number of characters C based on the subtitle character data (step S2305).

[0149] In addition, since the size of the caption display area for input caption text data is uniquely determined according to the number of characters C, it is also possible to calculate the number of characters C from the size using the following formula. Let CM be the preset maximum number of characters, WM be the preset maximum caption display area width, and wM be the size of the character string width displaying the caption text data. [Number 3] C = (CM / WM) * wM (3)

[0150] The area conversion unit 42 selects one of the subtitle display positions (XA, YA, CNA) and (XB, YB, CNB) converted in step S2304 based on the number of characters C calculated in step S2305, and calculates the subtitle display position (X, Y) corresponding to the selected subtitle display position (step S2306). The area conversion unit 42 outputs the subtitle display position (X, Y) to the control output unit 33. The subtitle display position (X, Y) is one of (XA, YA) and (XB, YB).

[0151] Specifically, area conversion unit 42 compares the number of characters CNA, CNB at the subtitle display positions (XA, YA, CNA), (XB, YB, CNB) with the number of characters C. Then, area conversion unit 42 identifies the number of characters CNA or the number of characters CNB that matches the number of characters C from among the number of characters CNA, CNB, and sets the subtitle display position (XA, YA) or (XB, YB) corresponding to the identified number of characters CNA or the number of characters CNB as the subtitle display position (X, Y).

[0152] Note that, by weighting each of the character numbers CNA and CNB, one of the subtitle display positions (XA, YA) or (XB, YB) may be determined as the subtitle display position (X, Y). For example, when the character number C=8, the character number CNA=14, and the character number CNB=7, the area conversion unit 42 identifies the character number CNA=14, which is equal to or greater than the character number C=8 and is closest to the character number C=8, from the character number CNA=14 and the character number CNB=7, and determines the subtitle display position (XA, YA) corresponding to the identified character number CNA=14 as the subtitle display position (X, Y).

[0153] 24 is a diagram showing an example of the data configuration of table 41. In this table 41, a plurality of sets of data are stored, each set being a subtitle display area (cx, cy, w, h) and its corresponding subtitle display position (X, Y, CN).

[0154] The subtitle display area (cx, cy, w, h) corresponds to the subtitle display areas (cxA, cyA, wA, hA) and (cxB, cyB, wB, hB) shown in Figures 21 to 23, and the subtitle display position (X, Y, CN) corresponds to the subtitle display positions (XA, YA, CNA) and (XB, YB, CNB) shown in Figure 23. As mentioned above, CN is the number of characters.

[0155] The relationship between the subtitle display area (cx, cy, w, h) representing the area of ​​the subtitles added to the video data (with subtitles) and the corresponding subtitle display position (X, Y, CN) can be set in advance, and the correspondence data indicating this relationship is stored in table 41.

[0156] 21 and 23, the control output unit 33 inputs the subtitle display position (X, Y) from the area conversion unit 42, performs control similar to that of the control output unit 33 in FIG. 7 (step S2307), and outputs the controlled subtitle display position (X, Y) (step S2308).

[0157] As described above, according to the subtitle display position inference device 2-3 of the third embodiment, the inference unit 40 infers two subtitle display areas (cxA, cyA, wA, hA) and (cxB, cyB, wB, hB) in descending order of score from image data (without subtitles) using the trained model generated by the subtitle display position learning device 1-3 of the third embodiment.

[0158] The area conversion unit 42 converts the subtitle display areas (cxA, cyA, wA, hA), (cxB, cyB, wB, hB) into subtitle display positions (XA, YA, CNA), (XB, YB, CNB) using table 41. Then, the area conversion unit 42 selects one of the subtitle display positions (XA, YA, CNA), (XB, YB, CNB) based on the number of characters C of the subtitle character data, and obtains the subtitle display position (X, Y).

[0159] The control output unit 33 performs a predetermined control so that the caption display position (X, Y) is not frequently changed, and outputs the caption display position (X, Y) after the control.

[0160] This provides the same effect as the subtitle display position inference device 2-1 of the first embodiment, and makes it possible to automatically adjust the subtitle display position (X, Y) to an appropriate position according to the video.

[0161] In particular, the subtitle display areas (cxA, cyA, wA, hA) and (cxB, cyB, wB, hB) inferred by the subtitle display position inference device 2-3 of the third embodiment include the size in which the subtitles are displayed.

[0162] In contrast, the size at which the subtitles are displayed cannot be obtained from the subtitle display position (X, Y) and the upper left subtitle coordinates (x, y) inferred by the subtitle display position inference devices 2-1 and 2-2 in the first and second embodiments. Therefore, the user must determine and specify this size.

[0163] According to the third embodiment, since the user does not need to determine and specify the size at which the subtitles are displayed, the user's workload can be reduced compared to the first and second embodiments. In addition, since the subtitle display areas (cxA, cyA, wA, hA), (cxB, cyB, wB, hB) are inferred using a trained model according to the number of characters C of the subtitle size, it is possible to obtain a highly accurate subtitle display position (X, Y) where the subtitles do not overlap with the background.

[0164] In the third embodiment, the inference unit 40 of the subtitle display position inference device 2-3 infers two subtitle display areas (cxA, cyA, wA, hA) and (cxB, cyB, wB, hB) in descending order of score using a trained model. The area conversion unit 42 obtains the subtitle display position (X, Y) from the number of characters C of the subtitle character data and the two subtitle display areas (cxA, cyA, wA, hA) and (cxB, cyB, wB, hB).

[0165] In response to this, the inference unit 40 may infer a predetermined number of subtitle display areas (cxA, cyA, wA, hA) or the like, which are three or more, in order of highest score, and the area conversion unit 42 may determine the subtitle display position (X, Y) from the number of characters C of the subtitle character data and the predetermined number of subtitle display areas (cxA, cyA, wA, hA) or the like, which are three or more.

[0166] Example 4 Next, a fourth embodiment will be described. The fourth embodiment is an example in which image data (without subtitles) is cut out from video data (without subtitles), and a trained model is generated using the image data (without subtitles), the subtitle display position (X, Y), and two lines of subtitle display areas (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) as teacher data, and the trained model is used to infer the first line of subtitle display area (cx1, cy1, w1, h1) and determine the subtitle display position (X, Y, Z). Z indicates the line break position from the first line of subtitles to the second line of subtitles when the subtitles span two lines.

[0167] In the above-described third embodiment, the subtitle display position learning device 1-3 is configured to generate a trained model using one subtitle display area (cx, cy, w, h).

[0168] However, since this subtitle display area (cx, cy, w, h) is a single rectangular area, the trained model cannot handle an area that combines two rectangular areas of different sizes.

[0169] Therefore, in Example 4, learning is performed using the subtitle display areas (cx1, cy1, w1, h1) and (cx2, cy2, w2, h2) for each row. This makes it possible to generate a trained model that matches an area of ​​a complex shape other than the subtitle display area (cx, cy, w, h) consisting of a single rectangular area, and to use the trained model to find the subtitle display position (X, Y, Z) that matches the shape of the area where the subtitles are displayed.

[0170] Fig. 25 is a diagram for explaining the subtitle display position (X, Y, Z) in the subtitle coordinate system of Example 4. As shown in Fig. 25, the subtitles are configured across two lines, and are displayed in an area β4 on the first line and an area β5 on the second line, and the display start positions on the X axis are the same on the first line and the second line.

[0171] In the subtitle display position (X,Y,Z), (X,Y) indicates the position of the subtitle part on the first line where the subtitles are displayed, as explained in Fig. 1 (2). (Z) indicates the position where the line break occurs from the subtitle part on the first line to the subtitle part on the second line.

[0172] (Example 4 / Subtitle display position learning device) First, a description will be given of a subtitle display position learning device according to a fourth embodiment. Fig. 26 is a block diagram showing a configuration example of the subtitle display position learning device according to the fourth embodiment, and Fig. 27 is a schematic diagram showing an overall process flow in the subtitle display position learning device according to the fourth embodiment.

[0173] The subtitle display position learning device 1-4 includes an image clipping unit 10, a subtitle area extraction unit 25, a table 12, a coordinate conversion unit 26, a storage unit 27, a learning unit 28, and a memory unit 29.

[0174] The subtitle display position learning device 1-4 inputs video data (without subtitles) and video data (with subtitles) (the video data (with subtitles) in FIG. 27 shows area γ1 in the first line and area γ2 in the second line where subtitles are added). Then, the subtitle display position learning device 1-4 generates the upper left coordinates (x, y) of the subtitle in the first line and the subtitle display areas (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) for two lines in the image cropping unit 10 and the subtitle area extraction unit 25.

[0175] The subtitle display position learning device 1-4 generates a subtitle display position (X, Y) using the table 12 and the coordinate conversion unit 26, thereby generating teacher data consisting of image data (no subtitles), the subtitle display position (X, Y) (for example, (X, Y) = (right-justified, 7th line) with reference to FIG. 27), and the subtitle display area (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) (for example, (cx1, cy1, w1, h1) = (960, 950, 1080, 100), (cx2, cy2, w2, h2) = (960, 1000, 900, 50) with reference to FIG. 27) (step S2701), and stores this in the storage unit 27.

[0176] The subtitle display position learning device 1-4 reads the training data from the storage unit 27 in the learning unit 28, and uses the training data to learn the relationship between the image data (without subtitles) and the subtitle display position (X, Y) and the subtitle display area (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) to generate a trained model (step S2702). Then, the subtitle display position learning device 1-4 stores the trained model in the storage unit 29.

[0177] 28 is a flowchart showing an example of processing performed by the subtitle display position learning device 1-4 in the embodiment 4. Hereinafter, an example of processing performed by the subtitle display position learning device 1-4 will be described with reference to FIGS.

[0178] 4, the image cropping unit 10 inputs video data (without subtitles) and video data (with subtitles) (step S2801), crops still images, and generates image data (without subtitles) and image data (with subtitles) (step S2802). Here, the image data (with subtitles) includes subtitles that span two lines.

[0179] The image cropping unit 10 outputs the image data (without subtitles) and the image data (with subtitles) to the subtitle region extraction unit 25, and also stores the image data (without subtitles) in the storage unit 27.

[0180] The subtitle area extraction unit 25 inputs image data (without subtitles) and image data (with subtitles) from the image cropping unit 10. Then, the subtitle area extraction unit 25 extracts a subtitle area by, for example, a background subtraction method using the image data (without subtitles) and the image data (with subtitles), and obtains the upper left subtitle coordinates (x, y) and the subtitle display area for each of two lines (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) from the subtitle area (step S2803). The subtitle area extraction unit 25 outputs the upper left subtitle coordinates (x, y) and the subtitle display area (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) to the coordinate conversion unit 26.

[0181] The coordinate conversion unit 26 receives the subtitle upper left coordinates (x, y) and the subtitle display areas (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) from the subtitle area extraction unit 25. Then, similar to step S404 in Fig. 4, the coordinate conversion unit 26 reads out the subtitle display position (X, Y) corresponding to the subtitle upper left coordinates (x, y) from table 12, thereby converting the subtitle upper left coordinates (x, y) into the subtitle display position (X, Y) (step S2804).

[0182] The coordinate conversion unit 26 stores the subtitle display position (X, Y) and the subtitle display area (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) in the storage unit 27. That is, the image cropping unit 10 and the coordinate conversion unit 26 store the image data (without subtitles) and the corresponding subtitle display position (X, Y) and subtitle display area (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) in the storage unit 27 as teacher data (step S2805).

[0183] Fig. 29 is a diagram showing an example of teacher data stored in the storage unit 27. As shown in Fig. 29, the storage unit 27 stores a plurality of sets of data (a plurality of teacher data) with one teacher data set being a set of image data (without subtitles) and the corresponding subtitle display position (X, Y) and subtitle display area (cx1, cy1, w1, h1), (cx2, cy2, w2, h2).

[0184] 26 and 28, the learning unit 28 reads out training data consisting of image data (without subtitles) and the corresponding subtitle display position (X, Y) and subtitle display area (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) from the storage unit 27. Then, using the training data, the learning unit 28 learns the relationship between the input data and the output data by deep learning for a learning model in which the image data (without subtitles) is used as input data and the subtitle display position (X, Y) and the subtitle display area (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) are used as output data (step S2806).

[0185] The learning unit 28 performs learning using a predetermined number of teacher data to generate a trained model, and stores the trained model in the storage unit 29 (step S2807).

[0186] As described above, according to the subtitle display position learning device 1-4 of the fourth embodiment, the subtitle area extraction unit 25 extracts the subtitle area using image data (without subtitles) and image data (with subtitles), and obtains the upper left subtitle coordinates (x, y) and the subtitle display area (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) from the subtitle area, and the coordinate conversion unit 26 converts the upper left subtitle coordinates (x, y) into the subtitle display position (X, Y) using table 12.

[0187] In the storage unit 27, image data (without subtitles), the subtitle display position (X, Y), and the subtitle display area (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) are stored as teacher data.

[0188] The learning unit 28 performs learning using training data consisting of image data (without subtitles) and subtitle display positions (X, Y) and subtitle display areas (cx1, cy1, w1, h1), (cx2, cy2, w2, h2), and generates a learned model.

[0189] This provides the same effect as the subtitle display position learning device 1-1 of the first embodiment, and the subtitle display position inference device 2-4 of the fourth embodiment described below can automatically adjust the subtitle display position (X, Y, Z) to an appropriate position according to the video by using this trained model.

[0190] In particular, the subtitle display position learning device 1-4 of the fourth embodiment generates a trained model using image data (without subtitles) as input data and using the subtitle display position (X, Y) and the subtitle display area (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) as output data. As a result, the subtitle display position inference device 2-4 of the fourth embodiment described later can use this trained model to infer the subtitle display area (cx1, cy1, w1, h1) of the first line and determine the subtitle display position (X, Y, Z) that allows a line break to be made according to the shape of the area where two lines of subtitles are displayed.

[0191] In contrast, in Example 3, a trained model is generated using a subtitle display area (cx, cy, w, h) that is a single rectangular area, and the trained model is used to obtain the subtitle display position (X, Y) that corresponds to the single rectangular area. However, Example 3 cannot handle an area that combines two rectangular areas of different sizes.

[0192] Therefore, according to the fourth embodiment, a trained model that matches the shape of the area in which the subtitles are displayed can be generated, and the trained model can be used to determine a subtitle display position (X, Y, Z) that allows a line break to be inserted in accordance with the shape of the area in which the subtitles are displayed.

[0193] In addition, in Example 4, the subtitle area extraction unit 25 of the subtitle display position learning device 1-4 obtains two subtitle display areas (cx1, cy1, w1, h1) and (cx2, cy2, w2, h2) from the subtitle area, and the learning unit 28 generates a learned model using training data consisting of image data (without subtitles) and the subtitle display position (X, Y) and the two subtitle display areas (cx1, cy1, w1, h1) and (cx2, cy2, w2, h2).

[0194] In response to this, the subtitle area extraction unit 25 may obtain a predetermined number of subtitle display areas (cx1, cy1, w1, h1) that is three or more, and the learning unit 28 may generate a trained model using teacher data consisting of image data (without subtitles) and the subtitle display position (X, Y) and a predetermined number of subtitle display areas (cx1, cy1, w1, h1) that is three or more.

[0195] (Example 4 / subtitle display position inference device) Next, a description will be given of a subtitle display position inference device according to a fourth embodiment. Fig. 30 is a block diagram showing a configuration example of the subtitle display position inference device according to the fourth embodiment, and Fig. 31 is a schematic diagram showing an overall process flow in the subtitle display position inference device according to the fourth embodiment.

[0196] This subtitle display position inference device 2-4 includes an image cropping unit 30, a storage unit 43, an inference unit 40, a table 44, a region conversion unit 45, and a control output unit 46. The storage unit 43 stores a trained model generated by the subtitle display position learning device 1-4 shown in FIG.

[0197] By using the trained model by the subtitle display position inference device 2-4, subtitles are added to areas β4 and β5 spanning two lines in the video data (without subtitles), as shown in FIG.

[0198] The subtitle display position inference device 2-4 inputs video data (without subtitles) and cuts out the image data (without subtitles) in the image cutout unit 30 (step S3101). Then, the subtitle display position inference device 2-4 infers the subtitle display area (cx1, cy1, w1, h1) of the first line (for example, (cx1, cy1, w1, h1) = (960, 670, 800, 130) with reference to FIG. 31) in the inference unit 40 using the trained model stored in the storage unit 43 (step S3102).

[0199] The subtitle display position inference device 2-4 converts the subtitle display area (cx1, cy1, w1, h1) into the subtitle display position (X, Y, Z) (for example, (X, Y, Z) = (center, 5th line, 12th character) with reference to FIG. 31) using the table 44 and the area conversion unit 45 (step S3103), and controls the subtitle display position (X, Y, Z) using the control output unit 46.

[0200] Here, the subtitle display position (X, Y, Z) = (center, 5th line, 12th character) shown in Figure 31 indicates that the subtitles are displayed in an area spanning two lines, and the starting positions of the first and second lines are the same, so the subtitle portion of the first line is displayed at position (X, Y) = (center, 5th line), and the line break position from the subtitle portion of the first line to the subtitle portion of the second line is (Z) = 12th character.

[0201] 32 is a flowchart showing an example of processing by the subtitle display position inference device 2-4 of Example 4. Hereinafter, an example of processing by the subtitle display position inference device 2-4 will be described with reference to FIGS.

[0202] 9, the image cutout unit 30 receives video data (without subtitles) (step S3201), cuts out still images, and generates image data (without subtitles) (step S3202). The image cutout unit 30 outputs the image data (without subtitles) to the inference unit 40.

[0203] The inference unit 40 inputs the image data (without subtitles) from the image cropping unit 30, reads the trained model from the storage unit 43, and infers the subtitle display area (cx1, cy1, w1, h1) of the first line from the image data (without subtitles) using the trained model (step S3203). Then, the inference unit 40 outputs the subtitle display area (cx1, cy1, w1, h1) to the area conversion unit 45.

[0204] The area conversion unit 45 receives the subtitle display area (cx1, cy1, w1, h1) from the inference unit 40, and converts the subtitle display area (cx1, cy1, w1, h1) to the subtitle display position (X, Y, Z) by reading the subtitle display position (X, Y, Z) corresponding to the subtitle display area (cx1, cy1, w1, h1) from the table 44 (step S3204). The area conversion unit 45 outputs the subtitle display position (X, Y, Z) to the control output unit 46.

[0205] Fig. 33 is a diagram showing an example of the data configuration of table 44. This table 44 stores a plurality of sets of data, each set consisting of a subtitle display area (cx1, cy1, w1, h1) and its corresponding subtitle display position (X, Y, Z). As shown in Fig. 25, (X, Y) indicates the position where the subtitle portion of the first line is displayed when the subtitles are configured over two lines, and (Z) indicates the position where the line break occurs from the subtitle portion of the first line to the subtitle portion of the second line.

[0206] 30 and 32, the control output unit 46 inputs the subtitle display position (X, Y, Z) from the area conversion unit 45, performs control similar to that of the control output unit 33 in FIG. 7 (step S3205), and outputs the controlled subtitle display position (X, Y, Z) (step S3206).

[0207] As described above, according to the subtitle display position inference device 2-4 of the fourth embodiment, the inference unit 40 infers the subtitle display area (cx1, cy1, w1, h1) of the first line from image data (without subtitles) using the trained model generated by the subtitle display position learning device 1-4 of the fourth embodiment.

[0208] The area conversion unit 45 converts the subtitle display area (cx1, cy1, w1, h1) into the subtitle display position (X, Y, Z) using the table 44. Then, the control output unit 46 performs a predetermined control so that the subtitle display position (X, Y, Z) is not frequently changed, and outputs the controlled subtitle display position (X, Y, Z).

[0209] This provides the same effect as the subtitle display position inference device 2-1 of the first embodiment, and makes it possible to automatically adjust the subtitle display position (X, Y, Z) to an appropriate position according to the video.

[0210] As described above, the subtitle display position learning device 1-4 of the fourth embodiment generates a trained model that matches the shape of the area where two lines of subtitles are displayed. By using this trained model, the subtitle display position inference device 2-4 of the fourth embodiment can obtain a subtitle display position (X, Y, Z) that allows a line break to be performed in accordance with the shape of the area where two lines of subtitles are displayed.

[0211] In the fourth embodiment, when the subtitles span two lines, the area conversion unit 45 of the subtitle display position inference device 2-4 converts the subtitle display area (cx1, cy1, w1, h1) into the subtitle display position (X, Y, Z) by using the table 44. In this case, (Z) indicates the line break position from the subtitle part on the first line to the subtitle part on the second line.

[0212] In contrast, when the subtitles span a predetermined number of lines (three or more), the area conversion unit 45 may use a new table instead of table 44 to convert the subtitle display area (cx1, cy1, w1, h1) into the subtitle display position (X, Y, Z').

[0213] In this case, the new table stores multiple sets of data, each set consisting of a subtitle display area (cx1, cy1, w1, h1) and its corresponding subtitle display position (X, Y, Z'). (Z') indicates the line break position from the first subtitle line to the second subtitle line, the line break position from the second subtitle line to the third subtitle line, etc., and includes all line break positions up to a predetermined number of lines, 3 or more.

[0214] Example 5 Next, a fifth embodiment will be described. The fifth embodiment is an example in which image data (without subtitles) is cut out from video data (without subtitles), and a trained model is generated using the image data (without subtitles), the subtitle display position (X, Y), and the two-line subtitle display area (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) as teacher data, and the trained model is used to infer the two-line subtitle display area (cx1, cy1, w1, h1), (cx2, cy2, w2, h2), and to obtain the subtitle display position (X, Y, Z, XW). XW indicates the start position where the display of the second line of the subtitle portion begins.

[0215] In the above-mentioned Example 4, a trained model was generated using two lines of subtitle display areas (cx1, cy1, w1, h1) and (cx2, cy2, w2, h2) when the start positions of the subtitle portion on the first line and the subtitle portion on the second line are the same.

[0216] However, the fourth embodiment cannot be applied to a case where the start positions of the subtitle portion on the first line and the subtitle portion on the second line are different.

[0217] Therefore, in Example 5, a trained model is generated using two lines of subtitle display areas (cx1, cy1, w1, h1) and (cx2, cy2, w2, h2) when the start positions of the subtitle part on the first line and the subtitle part on the second line are different. This makes it possible to obtain a subtitle display position (X, Y, Z, XW) that matches the shape of the area where the subtitles are displayed, using the trained model, when the start positions of the subtitle part on the first line and the subtitle part on the second line are different.

[0218] Fig. 34 is a diagram for explaining the subtitle display position (X, Y, Z, XW) in the subtitle coordinate system of Example 5. As shown in Fig. 34, the subtitles are configured across two lines, and are displayed in an area β6 on the first line and an area β7 on the second line, and the display start positions on the X axis are different between the first line and the second line.

[0219] In the subtitle display position (X,Y,Z,XW), (X,Y) indicates the position of the subtitle portion on the first line where the subtitles are displayed, as explained in Fig. 1 (2). (Z) indicates the line break position from the subtitle portion on the first line to the subtitle portion on the second line, as explained in Fig. 25. (XW) indicates the start position of the subtitle portion on the second line (for example, the start position of the subtitle portion on the second line based on the start position of the subtitle portion on the first line).

[0220] (Example 5 / Subtitle display position learning device) A subtitle display position learning device 1-5 of the fifth embodiment is similar to the subtitle display position learning device 1-4 of the fourth embodiment shown in Fig. 26, and therefore a description thereof will be omitted here. The fifth embodiment is applied to a case where a subtitle is configured over two lines, and the start positions of the subtitle part on the first line and the subtitle part on the second line are different.

[0221] For this reason, the subtitle display position learning device 1-5 inputs, as video data (with subtitles), video data including subtitles that span two lines and have different start positions for the subtitle portion on the first line and the subtitle portion on the second line.Then, the image cropping unit 10 crops, as image data (with subtitles), image data including subtitles that span two lines and have different start positions for the subtitle portion on the first line and the subtitle portion on the second line.

[0222] As described above, according to the subtitle display position learning device 1-5 of the fifth embodiment, the learning unit 28 performs learning using teacher data consisting of image data (without subtitles), the subtitle display position (X, Y), and the subtitle display areas (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) of subtitles having different start positions in the subtitle part on the first line and the subtitle part on the second line, and generates a learned model.

[0223] This provides the same effect as the subtitle display position learning device 1-1 of the first embodiment, and the subtitle display position inference device 2-5 of the fifth embodiment described below can automatically adjust the subtitle display position (X, Y, Z, XW) to an appropriate position according to the video by using this trained model.

[0224] In particular, it is possible to generate a trained model that matches the shape of the area in which the subtitles are displayed, and by using this trained model, it is possible to determine a subtitle display position (X, Y, Z, XW) that allows line breaks to be made that match the shape of the area in which the subtitles are displayed, and that enables the second line to be displayed from an appropriate position.

[0225] (Example 5 / Subtitle display position inference device) Next, a description will be given of a subtitle display position inference device according to a fifth embodiment. Fig. 35 is a block diagram showing a configuration example of the subtitle display position inference device according to the fifth embodiment, and Fig. 36 is a schematic diagram showing an overall process flow in the subtitle display position inference device according to the fifth embodiment.

[0226] The subtitle display position inference device 2-5 includes an image cropping unit 30, a storage unit 47, an inference unit 48, a table 49, a region conversion unit 50, and a control output unit 51. The storage unit 47 stores a trained model generated by the subtitle display position learning device 1-5 of the fifth embodiment.

[0227] By using the trained model by the subtitle display position inference device 2-5, subtitles are added to areas β6 and β7 spanning two lines in the video data (without subtitles), as shown in FIG.

[0228] The subtitle display position inference device 2-5 inputs video data (without subtitles) and cuts out the image data (without subtitles) in the image cutout unit 30 (step S3601). Then, the subtitle display position inference device 2-5 infers the subtitle display area (cx1, cy1, w1, h1) of the first line and the subtitle display area (cx2, cy2, w2, h2) of the second line (for example, (cx1, cy1, w1, h1) = (900, 950, 1080, 100), (cx2, cy2, w2, h2) = (960, 1000, 900, 50) with reference to FIG. 36) in the inference unit 48 using the trained model stored in the storage unit 47 (step S3602).

[0229] The subtitle display position inference device 2-5 converts the subtitle display areas (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) into the subtitle display positions (X, Y, Z, XW) using table 49 and area conversion unit 50 (step S3603), and controls the subtitle display position (X, Y, Z, XW) using control output unit 51.

[0230] 37 is a flowchart showing an example of processing of the subtitle display position inference device 2-5 of Example 5. Hereinafter, an example of processing of the subtitle display position inference device 2-5 will be described with reference to FIGS.

[0231] 9, the image cutout unit 30 receives video data (without subtitles) (step S3701), cuts out still images, and generates image data (without subtitles) (step S3702). The image cutout unit 30 outputs the image data (without subtitles) to the inference unit .

[0232] The inference unit 48 inputs image data (without subtitles) from the image cropping unit 30, reads the trained model from the storage unit 47, and infers the subtitle display area of ​​the first line (cx1, cy1, w1, h1) and the subtitle display area of ​​the second line (cx2, cy2, w2, h2) from the image data (without subtitles) using the trained model (step S3703). Then, the inference unit 48 outputs the subtitle display areas (cx1, cy1, w1, h1) and (cx2, cy2, w2, h2) to the area conversion unit 50.

[0233] The area conversion unit 50 inputs the subtitle display areas (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) from the inference unit 48, and calculates the difference position (W) on the X-axis between the subtitle portion on the first line and the subtitle portion on the second line using the following formula. [Number 4] W=(cx1-w1 / 2)-(cx2-w2 / 2) ···(4) If W<0, the starting position of the subtitle part on the second line is shifted to the right of the starting position of the subtitle part on the first line, and if W>0, the starting position of the subtitle part on the second line is shifted to the left of the starting position of the subtitle part on the first line.

[0234] The area conversion unit 50 converts the subtitle display area (cx1, cy1, w1, h1) and the difference position (W) to the subtitle display position (X, Y, Z, XW) by reading the subtitle display position (X, Y, Z, XW) corresponding to the subtitle display area (cx1, cy1, w1, h1) and the difference position (W) from the table 49 (step S3704). That is, the subtitle display areas (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) are converted to the subtitle display position (X, Y, Z, XW).

[0235] 38 is a diagram showing an example of the data configuration of table 49. In this table 49, a plurality of sets of data are stored, each set being a subtitle display area (cx1, cy1, w1, h1), a differential position (W), and a corresponding subtitle display position (X, Y, Z, XW).

[0236] 34, (X,Y) indicates the position where the first subtitle line is displayed when the subtitles span two lines, (Z) indicates the line feed position from the first subtitle line to the second subtitle line, and (XW) indicates the start position where the display of the second subtitle line begins.

[0237] Incidentally, instead of the start position (XW) of the subtitle portion on the second line, the upper left coordinate value of the subtitle portion on the second line may be used as (XW).

[0238] 35 and 37, the control output unit 51 inputs the subtitle display position (X, Y, Z, XW) from the area conversion unit 50, performs control similar to that of the control output unit 33 in FIG. 7 (step S3705), and outputs the controlled subtitle display position (X, Y, Z, XW) (step S3706).

[0239] As described above, according to the subtitle display position inference device 2-5 of the fifth embodiment, the inference unit 48 infers the subtitle display areas (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) from the image data (without subtitles) using the trained model generated by the subtitle display position learning device 1-5 of the fifth embodiment.

[0240] The area conversion unit 50 calculates the difference position (W) on the X-axis between the subtitle portion on the first line and the subtitle portion on the second line from the subtitle display areas (cx1, cy1, w1, h1), (cx2, cy2, w2, h2), and converts the subtitle display area (cx1, cy1, w1, h1) and the difference position (W) into the subtitle display position (X, Y, Z, XW) using table 49. Then, the control output unit 51 performs a predetermined control so that the subtitle display position (X, Y, Z, XW) is not frequently changed, and outputs the controlled subtitle display position (X, Y, Z, XW).

[0241] This provides the same effect as the subtitle display position inference device 2-1 of the first embodiment, and makes it possible to automatically adjust the subtitle display position (X, Y, Z, XW) to an appropriate position according to the video.

[0242] In particular, by using the trained model generated by the subtitle display position learning device 1-5, it is possible to determine a subtitle display position (X, Y, Z, XW) that enables line breaks to be made according to the shape of the area in which the subtitles are displayed and that allows the second line to be displayed from an appropriate position.

[0243] In addition, in Example 5, the inference unit 48 of the subtitle display position inference device 2-5 infers the subtitle display areas (cx1, cy1, w1, h1), (cx2, cy2, w2, h2) from the image data (without subtitles) using the trained model generated by the subtitle display position learning device 1-5 of Example 5, and the area conversion unit 50 calculates the differential position (W) and converts the subtitle display area (cx1, cy1, w1, h1) and the differential position (W) into the subtitle display position (X, Y, Z, XW) using table 49.

[0244] In this case, (Z) indicates the position where a line break occurs from the subtitle portion on the first line to the subtitle portion on the second line, and (XW) indicates the start position of the subtitle portion on the second line.

[0245] In response to this, the inference unit 48 infers the subtitle display area (cx1, cy1, w1, h1) and the like from the image data (without subtitles) using the trained model generated by the subtitle display position learning device 1-5 of the fifth embodiment (the trained model generated using video data (with subtitles) in which the subtitles span a predetermined number of lines, which is three or more), and the area conversion unit 50 calculates the difference position (W') on the X-axis between the subtitle portion on the first line and each of the subtitle portions on the second line and thereafter, and may convert the subtitle display area (cx1, cy1, w1, h1) and the difference position (W') into the subtitle display position (X, Y, Z', XW') using a new table in place of table 49.

[0246] In this case, the new table stores multiple sets of data, each set consisting of a subtitle display area (cx1, cy1, w1, h1), a differential position (W'), and the corresponding subtitle display position (X, Y, Z', XW'). (Z') indicates the line break position from the first subtitle part to the second subtitle part, the line break position from the second subtitle part to the third subtitle part, etc., and includes all line break positions up to a predetermined number of lines (3 or more). (XW') indicates the start position of the subtitle part on the second line, the start position of the subtitle part on the third line, etc., and includes all start positions up to a predetermined number of lines (3 or more).

[0247] [Another First Example] Next, a first alternative embodiment will be described. The first alternative embodiment is an example in which a broadcast TS (Transport Stream) is input, video data (without subtitles) and subtitle data are extracted from the broadcast TS, image data (without subtitles) cut out from the video data (without subtitles) and subtitle display position data extracted from the subtitle data are used as teacher data to generate a trained model, and the trained model is used to infer the subtitle display position data.

[0248] The broadcast TS includes various data such as video, audio, subtitles, data broadcasting, EPG, etc. The first alternative embodiment is applicable to the first, second, third, fourth and fifth embodiments described above.

[0249] In another first embodiment, the subtitle display position learning device inputs a broadcast TS and extracts video data (without subtitles) and subtitle data from the broadcast TS. Then, the subtitle display position learning device extracts image data (without subtitles) from the video data (without subtitles) in the same manner as the image extraction unit 10 in FIG. 2 processes the video data (without subtitles).

[0250] The subtitle display position learning device extracts subtitle display position data (e.g., the upper left subtitle coordinates (x, y) in the image coordinate system, the subtitle coordinate values ​​(x', y') in the subtitle coordinate system, and the subtitle display area (cx, cy, w, h)) from the subtitle data. The subtitle coordinate values ​​(x', y') will be described later. Note that the process of extracting the subtitle display area (cx, cy, w, h) from the subtitle data includes the process of generating the subtitle display area (cx, cy, w, h) based on the subtitle data.

[0251] When the subtitle display position learning device extracts, for example, the upper left subtitle coordinate (x, y) as subtitle display position data from the subtitle data, it converts the upper left subtitle coordinate (x, y) to the subtitle display position (X, Y) in the same manner as in the first embodiment, and generates a trained model using the image data (without subtitles) and the subtitle display position (X, Y) as training data.

[0252] Then, the subtitle display position inference device performs the same process as the subtitle display position inference device 2-1 of the first embodiment shown in Fig. 7. That is, the subtitle display position inference device infers the subtitle display position (X, Y) by using the learned model generated by the above-mentioned subtitle display position learning device, and outputs the controlled subtitle display position (X, Y).

[0253] In addition, when the subtitle display position learning device extracts, for example, the upper left subtitle coordinates (x, y) as subtitle display position data from the subtitle data, it generates a trained model using the image data (without subtitles) and the upper left subtitle coordinates (x, y) as training data, as in the second embodiment.

[0254] Then, the subtitle display position inference device performs the same process as the subtitle display position inference device 2-2 of the second embodiment shown in Fig. 14. That is, the subtitle display position inference device infers the upper left subtitle coordinates (x, y) using the trained model generated by the subtitle display position learning device described above, converts the upper left subtitle coordinates (x, y) to a subtitle display position (X, Y), and outputs the controlled subtitle display position (X, Y).

[0255] Furthermore, when the subtitle display position learning device extracts, for example, the upper left subtitle coordinates (x, y) and the subtitle display area (cx, cy, w, h) as subtitle display position data from the subtitle data, it converts the upper left subtitle coordinates (x, y) to the subtitle display position (X, Y) in the same manner as in the third embodiment, and generates a trained model using the image data (without subtitles) as well as the subtitle display position (X, Y) and the subtitle display area (cx, cy, w, h) as training data.

[0256] Then, the subtitle display position inference device performs the same process as the subtitle display position inference device 2-3 of the third embodiment shown in Fig. 21. That is, the subtitle display position inference device infers the subtitle display area (cx, cy, w, h) using the trained model generated by the above-mentioned subtitle display position learning device, converts the subtitle display area (cx, cy, w, h) to a subtitle display position (X, Y), and outputs the subtitle display position (X, Y) after control. For the fourth and fifth embodiments, too, a trained model is generated by extracting subtitle display position data according to the fourth and fifth embodiments from the subtitle data, and inference is performed using the trained model to output the subtitle display positions (X, Y, Z), (X, Y, Z, XW) after control.

[0257] This allows a trained model to be generated using the broadcast TS, and this trained model can be used to obtain the subtitle display position (X, Y), etc., thereby achieving the same effects as in Examples 1, 2, 3, 4, and 5.

[0258] [Another Second Example] Next, another second embodiment will be described. The other second embodiment is an example in which video data (without subtitles) and an ARIB subtitle file are input, image data (without subtitles) cut out from the video data (without subtitles) and subtitle display position data extracted from the ARIB subtitle file are used as teacher data to generate a trained model, and the trained model is used to infer the subtitle display position data.

[0259] The ARIB subtitle file includes information on the timing of transmission, the contents of the subtitles, the color, the position, etc. The second embodiment is applicable to the first, second, third, fourth and fifth embodiments described above.

[0260] In another second embodiment, the subtitle display position learning device inputs video data (without subtitles) and the corresponding ARIB subtitle file, and extracts image data (without subtitles) from the video data (without subtitles) in the same manner as the image extraction unit 10 in Figure 2 processes the video data (without subtitles).

[0261] The subtitle display position learning device extracts subtitle display position data (e.g., the upper left subtitle coordinates (x, y) in the image coordinate system, the subtitle coordinate values ​​(x', y') in the subtitle coordinate system, and the subtitle display area (cx, cy, w, h)) from the ARIB subtitle file. The subtitle coordinate values ​​(x', y') will be described later. Note that the process of extracting the subtitle display area (cx, cy, w, h) from the ARIB subtitle file includes the process of generating the subtitle display area (cx, cy, w, h) based on the data contained in the ARIB subtitle file.

[0262] The subsequent processing of the subtitle display position learning device is similar to that of the first embodiment. The processing of the subtitle display position inference device is also similar. The second embodiment can be applied to the fourth and fifth embodiments as well.

[0263] This allows a trained model to be generated using video data (without subtitles) and an ARIB subtitle file, and this trained model can be used to obtain the subtitle display position (X, Y), etc., achieving the same effects as in Examples 1, 2, 3, 4, and 5.

[0264] The present invention has been described above by giving Examples 1, 2, 3, 4, 5, another first embodiment, and another second embodiment. However, the present invention is not limited to the above-mentioned Examples 1, 2, 3, 4, 5, etc., and various modifications are possible without departing from the technical concept thereof.

[0265] For example, in the example of FIG. 1, the number of patterns of the subtitle display position (X, Y) where the subtitle is displayed is set to 24, but this is merely an example and the number may be more than 24 or less than 24.

[0266] Furthermore, the subtitle display position inference device 2-1 of the first embodiment shown in FIG. 7, the subtitle display position inference device 2-2 of the second embodiment shown in FIG. 14, and the subtitle display position inference device 2-3 of the third embodiment shown in FIG. 21 are each equipped with a control output unit 33, but may not be equipped with the control output unit 33.

[0267] Similarly, the subtitle display position inference device 2-4 of the fourth embodiment shown in FIG. 30 has a control output unit 46, but does not have to have the control output unit 46, and the subtitle display position inference device 2-5 of the fifth embodiment shown in FIG. 35 has a control output unit 51, but does not have to have the control output unit 51.

[0268] In the subtitle display position learning device 1-1 of the first embodiment shown in Fig. 2, the subtitle area extraction unit 11 extracts a subtitle area from the image data (without subtitles) and the image data (with subtitles), and obtains the upper left subtitle coordinate (x, y) from the subtitle area. The upper left subtitle coordinate (x, y) is the upper left coordinate of the area α where the subtitle is displayed in the image coordinate system, as shown in Fig. 1 (1).

[0269] On the other hand, when the X-axis and Y-axis of the subtitle coordinate system are divided into 21 and 8 regions, respectively, the subtitle region extraction unit 11 may determine the subtitle coordinate values ​​(x', y') that are the upper left coordinate values ​​where the subtitle is displayed from the extracted subtitle region, with 21×8=168 regions as the subtitle coordinate values ​​(x', y'). x'=1,...,21 and y'=1,...,8.

[0270] The subtitle region extraction unit 11 outputs the subtitle coordinate values ​​(x', y') to the coordinate conversion unit 13, and the coordinate conversion unit 13 converts the subtitle coordinate values ​​(x', y') into subtitle display positions (X, Y) using a table 12. In this case, a plurality of pairs of data, each pair being a subtitle coordinate value (x', y') and a corresponding subtitle display position (X, Y), are stored in advance in the table 12. The storage unit 14 and the learning unit 15 perform the same process as in the first embodiment. Then, the subtitle display position inference device performs the same process as the subtitle display position inference device 2-1 of the first embodiment shown in FIG. 7.

[0271] Also in the subtitle display position learning device 1-2 of the second embodiment shown in FIG. 10, the subtitle area extraction unit 11 may obtain the subtitle coordinate values ​​(x', y') similar to those described above.

[0272] In this case, the teacher data is composed of image data (without subtitles) and subtitle coordinate values ​​(x', y'). This teacher data is stored in the storage unit 17, and the learning unit 18 learns the relationship between the input data and the output data for a learning model in which the image data (without subtitles) is used as input data and the subtitle coordinate values ​​(x', y') are used as output data, thereby generating a learned model.

[0273] In the subtitle display position inference device 2-2 of the second embodiment shown in Fig. 14, an inference unit 35 infers subtitle coordinate values ​​(x', y') from image data (without subtitles) using the above-mentioned trained model. A coordinate conversion unit 37 converts the subtitle coordinate values ​​(x', y') into subtitle display positions (X, Y) using a table 36. In this case, a plurality of pairs of data, each pair consisting of a subtitle coordinate value (x', y') and a corresponding subtitle display position (X, Y), are stored in advance in the table 36.

[0274] Moreover, the above-mentioned embodiments 1, 2, 3, 4, 5, etc. are applicable not only to the production of live subtitles in which closed captions are added in real time to live broadcast programs, but also to the production of fully-produced subtitles that are recorded in advance and closed captions for web videos. Furthermore, by implementing the functions of embodiments 1, 2, 3, 4, 5, etc. in playback software that plays back video, it is possible to develop a universal player that adjusts and displays the subtitle position without depending on the service of the distribution source.

[0275] A normal computer can be used as the hardware configuration of the subtitle display position learning devices 1-1, 1-2, 1-3, 1-4, 1-5, etc. and the subtitle display position inference devices 2-1, 2-2, 2-3, 2-4, 2-5, etc. according to the first, second, third, fourth, fifth, etc. embodiments of the present invention. The subtitle display position learning devices 1-1, 1-2, 1-3, 1-4, 1-5, etc. and the subtitle display position inference devices 2-1, 2-2, 2-3, 2-4, 2-5, etc. are configured by a computer equipped with a CPU, a GPU, a volatile storage medium such as a RAM, a non-volatile storage medium such as a ROM, an interface, etc.

[0276] The functions of the image cropping unit 10, the subtitle area extraction unit 11, the table 12, the coordinate conversion unit 13, the storage unit 14, the learning unit 15 and the memory unit 16 provided in the subtitle display position learning device 1-1 of Example 1 are each realized by causing a CPU or the like to execute a program describing these functions.

[0277] In addition, each function of the image cropping unit 10, the subtitle area extraction unit 11, the storage unit 17, the learning unit 18 and the memory unit 19 provided in the subtitle display position learning device 1-2 of Example 2 is realized by causing a CPU or the like to execute a program describing these functions.

[0278] In addition, each function of the image cropping unit 10, the subtitle area extraction unit 20, the table 12, the coordinate conversion unit 21, the storage unit 22, the learning unit 23 and the memory unit 24 provided in the subtitle display position learning device 1-3 of the third embodiment is realized by causing a CPU or the like to execute a program describing these functions.

[0279] Moreover, the functions of the image cropping unit 10, the subtitle area extraction unit 25, the table 12, the coordinate conversion unit 26, the storage unit 27, the learning unit 28 and the memory unit 29 provided in the subtitle display position learning devices 1-4 and 1-5 of the fourth and fifth embodiments are also realized by causing a CPU to execute a program describing these functions. The same applies to the subtitle display position learning devices of the first embodiment and the second embodiment.

[0280] Furthermore, each function of the image cropping unit 30, memory unit 31, inference unit 32 and control output unit 33 provided in the subtitle display position inference device 2-1 of Example 1 is also realized by having a CPU or the like execute a program that describes these functions.

[0281] In addition, each function of the image cropping unit 30, memory unit 34, inference unit 35, table 36, coordinate conversion unit 37 and control output unit 33 provided in the subtitle display position inference device 2-2 of Example 2 is realized by causing a CPU or the like to execute a program describing these functions.

[0282] In addition, each function of the image cropping unit 30, memory unit 39, inference unit 40, table 41, area conversion unit 42 and control output unit 33 provided in the subtitle display position inference device 2-3 of Example 3 is realized by having a CPU or the like execute a program describing these functions.

[0283] In addition, each function of the image cropping unit 30, memory unit 43, inference unit 40, table 44, area conversion unit 45 and control output unit 46 provided in the subtitle display position inference device 2-4 of Example 4 is realized by having a CPU or the like execute a program describing these functions.

[0284] In addition, each function of the image cropping unit 30, memory unit 47, inference unit 48, table 49, area conversion unit 50 and control output unit 51 provided in the subtitle display position inference device 2-5 of Example 5 is realized by having a CPU or the like execute a program describing these functions.

[0285] These programs are stored in the storage medium and are read and executed by a CPU, etc. In addition, these programs can be distributed by being stored in a storage medium such as a magnetic disk (floppy disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), or a semiconductor memory, and can also be transmitted and received via a network. [Explanation of symbols]

[0286] 1. Subtitle display position learning device 2 Subtitle display position inference device 10,30 Image extraction section 11,20,25 Subtitle area extraction part 12,36,41,44,49 Table 13,21,26,37 Coordinate conversion section 14,17,22,27 Storage section 15,18,23,28 Learning Department 16,19,24,29,31,34,39,43,47 Storage section 32,35,40,48 Reasoning part 33, 46, 51 Control output section 42,45,50 Domain conversion section (x,y) Subtitle top left coordinates (X,Y),(X,Y,CN),(X,Y,Z),(X,Y,Z,XW) Subtitle display position (cx,cy,w,h) Subtitle display area C Number of characters (W) Difference position

Claims

1. A subtitle display position learning device that learns a display position of subtitles added to a video, a subtitle area extraction unit that extracts a position of a subtitle area where the subtitle is displayed as a subtitle top left coordinate indicating a top left coordinate of the subtitle area in an image coordinate system where an image is displayed, based on data including the subtitle; a coordinate conversion unit that converts the upper left coordinate of the subtitle extracted by the subtitle region extraction unit into a subtitle display position in a subtitle coordinate system in which the subtitle is displayed, using a preset conversion rule; a learning unit that uses teacher data consisting of the image data (without subtitles) and the subtitle display positions converted by the coordinate conversion unit to learn a relationship between the input data and the output data for a learning model in which the image data (without subtitles) is used as input data and the subtitle display positions at which the subtitles are displayed within the image data (without subtitles) are used as output data, thereby generating a learned model; and A subtitle display position learning device comprising:

2. A subtitle display position inference device that infers a display position of a subtitle added to a video using a trained model, A storage unit in which the trained model generated by the subtitle display position learning device according to claim 1 is stored; an inference unit that inputs the image data to which no subtitles are added as image data (without subtitles) and infers a subtitle display position where the subtitles are to be displayed, using the trained model stored in the storage unit; A subtitle display position inference device comprising:

3. 3. The subtitle display position inference device according to claim 2, Further, the subtitle display position inferred by the inference unit is input, and the subtitle display position is continuously output for a preset time period, or Alternatively, the subtitle display position is input, and if the subtitle display position is the same a preset number of times in succession, the subtitle display position is output; Alternatively, the subtitle display position is input, the currently input subtitle display position is set as a first subtitle display position, the previously input subtitle display position is set as a second subtitle display position, and if a difference between the first subtitle display position and the second subtitle display position is equal to or greater than a preset threshold, the first subtitle display position is output, and if the difference is less than the threshold, the previously output subtitle display position is continued to be output; or or a subtitle display position inference device comprising a control output unit that inputs the subtitle display position and inputs the image data (without subtitles) corresponding to the subtitle display position, sets the currently input image data as first image data and the image data input immediately before as second image data, and determines that it is time for a cut change if a difference between the first image data and the second image data is equal to or greater than a preset threshold, and outputs the subtitle display position input at the time of the cut change, and controls whether to continue outputting the subtitle display position that was being output if the difference is less than the threshold.

4. A computer constituting a subtitle display position learning device that learns the display position of subtitles added to a video, a subtitle area extraction unit that extracts a position of a subtitle area in which the subtitle is displayed as a subtitle top left coordinate indicating a top left coordinate of the subtitle area in an image coordinate system in which an image is displayed, based on data including the subtitle; a coordinate conversion unit that converts the upper left coordinate of the subtitle extracted by the subtitle region extraction unit into a subtitle display position in a subtitle coordinate system in which the subtitle is displayed, using a preset conversion rule; and A program for functioning as a learning unit that uses image data to which no subtitles are added as image data (without subtitles) and teacher data consisting of the image data (without subtitles) and the subtitle display position converted by the coordinate conversion unit, and learns the relationship between the input data and the output data for a learning model in which the image data (without subtitles) is used as input data and the subtitle display position at which the subtitles are displayed within the image data (without subtitles) as output data, and generates a learned model.

5. A computer constituting a subtitle display position inference device, comprising: a storage unit in which the trained model generated by the subtitle display position learning device according to claim 1 is stored; and the computer infers a display position of a subtitle added to a video by using the trained model stored in the storage unit, A program for functioning as an inference unit that inputs image data to which no subtitles are added as image data (without subtitles) and infers a subtitle display position where the subtitles will be displayed, using the trained model stored in the memory unit.

Images