Subtitle display position adjustment device and program

Abstract

To automatically adjust a subtitle display position to an appropriate position in accordance with a video.SOLUTION: An object detection unit 10 of a subtitle display position adjustment apparatus 1-1 detects objects from a base video to determine positions and types of the objects. A coordinate score addition unit 11 adds, based on the positions and types of the objects, according to scores set in advance for each of the types of the objects, scores for each object, maps the objects to score coordinates in which scores for the objects are set in advance, on the basis of the positions of the objects, and adds the scores for each score coordinate. A display region score addition unit 12 maps the scores for each score coordinate onto subtitle display regions set in advance, and adds scores for each subtitle display region. A subtitle display position determination unit 13 determines, as a subtitle display position, a subtitle display region having the minimum score, out of all the subtitle display regions.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a subtitle display position adjustment device and program for adjusting the display position of subtitles added to video. [Background technology]

[0002] Traditionally, television broadcasting has offered closed captioning, a service for the hearing impaired that displays the audio of broadcast programs as text on the screen. Subtitles for live broadcasts are manually transcribed from the audio of the live program.

[0003] Methods for creating subtitles for live broadcast programs include methods where a person inputs the subtitles while listening to the audio, and semi-automatic methods where the results of speech recognition are manually corrected (see, for example, Non-Patent Document 1).

[0004] Furthermore, the on-screen position of subtitles for live broadcasts is manually adjusted by the staff. In other words, in the subtitle production room that produces and broadcasts subtitles for live broadcasts, the staff manually adjusts the subtitle display position by using buttons such as height, right alignment, and left alignment on the control box while watching the broadcast screen, so that the text does not overlap with the on-screen CG supers.

[0005] Figure 16 is a diagram illustrating the operation box used in the conventional subtitle display position adjustment method, and Figure 17 is a diagram showing examples of subtitle display positions before and after adjustment in the conventional subtitle display position adjustment method.

[0006] The control box shown in Figure 16 is equipped with a height adjustment button B1 for adjusting the height position of the subtitles on the screen, and a left / right adjustment button B2 for adjusting the left / right position of the subtitles on the screen.

[0007] The height adjustment button B1 consists of buttons such as "2" for moving the subtitle to the position of the second line on the screen, "3" for moving it to the position of the third line, and so on. Also, the left - right adjustment button B2 consists of the button "Left" for moving the subtitle arranged on the screen only a predetermined distance to the left, the button "Center" for moving the subtitle to a predetermined position in the center of the screen, and the button "Right" for moving the subtitle only a predetermined distance to the right.

[0008] In the pre - adjustment broadcast screen shown in Fig. 17(1), the subtitle "This town is said to be the entrance to Europe" overlaps with the CG super. Therefore, viewers watching the broadcast program as it is cannot correctly obtain the information of the CG super.

[0009] Therefore, while looking at the pre - adjustment broadcast screen shown in Fig. (1), the person in charge in the subtitle production room uses the operation box to change the display position of the subtitle to an appropriate position so that the subtitle displayed on the broadcast screen does not overlap with the CG super. That is, the person in charge moves the display position of the subtitle shown in Fig. 17(1) to the position shown in Fig. 17(2) by operating the height adjustment button B1 and the left - right adjustment button B2 of the operation box.

[0010] As a result, the subtitle moves to a position where it does not overlap with the CG super, so viewers can accurately obtain not only the information of the subtitle but also the information of the CG super.

Prior Art Documents

Non - Patent Documents

[0011]

Non - Patent Document 1

Summary of the Invention

[0012] As mentioned above, in creating subtitles for live broadcasts, in order to determine the appropriate display position of subtitles on the screen, the person in charge needs to adjust the height and horizontal position of the subtitles by operating the buttons on the control box while closely watching the broadcast screen. However, there was a problem in that the manual adjustment work performed by the person in charge was burdensome.

[0013] Therefore, a less burdensome new method was desired to replace the manual adjustment work performed by the person in charge. This is because reducing the burden on the person in charge would improve the efficiency of the subtitling production workflow in the production process.

[0014] This applies not only to the production of live subtitles, where closed captions are added in real time to live broadcast programs, but also to the production of subtitles for pre-recorded, non-live broadcast programs and for video services provided through online content. Furthermore, even when display position information is not added during subtitle production or transmission, there has been a demand for a new method to display subtitles in a position that is easy for viewers to see when playing the video on the receiver or online content client.

[0015] Therefore, the present invention has been made to solve the above-mentioned problems, and its objective is to provide a subtitle display position adjustment device and program that can automatically adjust the subtitle display position to an appropriate position according to the video. [Means for solving the problem]

[0017] In order to solve the aforementioned problem, Claim 1The subtitle display position adjustment device is a subtitle display position adjustment device that adjusts the display position of subtitles added to a video, and includes an object detection unit that detects one or more objects from the video and determines the position and type of each object, and uses the coordinates that represent each of the multiple regions into which the screen area of ​​the video is divided as score coordinates, and assigns a score indicating the importance of the information of the object to each of the one or more objects detected by the object detection unit, based on the position and type of each object determined by the object detection unit, according to a preset score for each type of object, thereby determining a score for each object, and based on the position of each object determined by the object detection unit, The system is characterized by comprising: a coordinate score assignment unit that maps the score for each object to the score coordinates and assigns a score to the score coordinates to obtain a score for each score coordinate; a display area score assignment unit that assigns a score to each of the multiple subtitle display areas based on the score for each score coordinate obtained by the coordinate score assignment unit, treating the multiple areas in which the subtitles are displayed within the screen area of ​​the video as multiple subtitle display areas; and a subtitle display position determination unit that determines one of the multiple subtitle display areas as the display position for the subtitles, based on the score for each subtitle display area obtained by the display area score assignment unit, such that the higher the score, the less likely the subtitles are to be displayed.

[0018] Furthermore, claims 2 The subtitle display position adjustment device is as described in the claim. 1 The subtitle display position adjustment device described above is characterized in that the type of each object includes at least a character superimposition, and the setting score of the character superimposition is set to a value greater than that of other objects.

[0019] Furthermore, claims 3 The subtitle display position adjustment device is as described in the claim. 1The subtitle display position adjustment device described above is characterized in that the coordinate score assignment unit determines a score for each of the one or more objects such that the value of the score decreases as time elapses after the object is detected, maps the score for each object to the score coordinates based on the position of each object, and assigns a score to the score coordinates to determine a score for each score coordinate.

[0020] Furthermore, claims 4 The subtitle display position adjustment device is as described in the claim. 3 The subtitle display position adjustment device described above is characterized in that the coordinate scoring unit determines a score for each of the one or more objects based on the position and type of each object, based on a preset initial score for each type of object, an initial time for which the initial score is maintained, and a reduction coefficient, such that the value of the score decreases as time elapses after the object is detected, and the type of each object includes at least a text superimposition, and the reduction coefficient for the text superimposition is set to a larger value than that for other objects.

[0021] Furthermore, claims 5 The program includes a computer that constitutes a subtitle display position adjustment device that adjusts the display position of subtitles added to a video, an object detection unit that detects one or more objects from the video and determines the position and type of each object, The coordinates representing each of the multiple regions into which the screen area of ​​the aforementioned video is divided are used as score coordinates. For each of the one or more objects detected by the object detection unit, a score indicating the importance of the information of that object is assigned based on the position and type of each object determined by the object detection unit, according to a pre-set score for each type of object, thereby determining the score for each object. - A coordinate score assignment unit that maps the score for each object to the score coordinates based on the position of each object determined by the object detection unit and assigns a score to the score coordinates to determine a score for each score coordinate; a display area score assignment unit that assigns a score to each of the multiple subtitle display areas based on the score for each score coordinate determined by the coordinate score assignment unit, treating the multiple areas in the video screen area where the subtitles are displayed as multiple subtitle display areas; and a display area score assignment unit that, based on the score for each subtitle display area determined by the display area score assignment unit, sets one of the multiple subtitle display areas as the subtitle display position, such that the higher the score, the less likely the subtitles are to be displayed. It is characterized by functioning as a subtitle display position determination unit. [Effects of the Invention]

[0022] As described above, according to the present invention, the subtitle display position can be automatically adjusted to an appropriate position according to the video. [Brief explanation of the drawing]

[0023] [Figure 1] This is a schematic diagram showing the overall system including the subtitle display position adjustment device of Examples 1 and 2. [Figure 2] This is a block diagram showing an example configuration of the subtitle display position adjustment device of Example 1. [Figure 3] This flowchart shows an example of the processing of the subtitle display position adjustment device of Example 1. [Figure 4] This figure shows an example of a base video. [Figure 5] This figure shows examples of the position and type of each object. [Figure 6] This is a block diagram showing an example configuration of the coordinate score assignment unit provided in the subtitle display position adjustment device of Example 1. [Figure 7] This figure shows examples of setting scores for each object type in Example 1. [Figure 8] This figure shows an example of the score for each object. [Figure 9] This figure shows an example of score coordinate data. [Figure 10] This figure shows an example of scores for each score coordinate. [Figure 11] This figure shows an example of a subtitle display area. [Figure 12] This figure shows an example of the score for each subtitle display area. [Figure 13] This figure shows an example of a base video with subtitles displayed in the optimal position. [Figure 14] This is a block diagram showing an example configuration of the subtitle display position adjustment device of Example 2. [Figure 15] This figure shows an example of a score that decreases according to the display duration of an object in Example 2. [Figure 16] This diagram illustrates the control box used in conventional subtitle display position adjustment methods. [Figure 17] This figure shows examples of subtitle display positions before and after adjustment using conventional subtitle display position adjustment methods. [Modes for carrying out the invention]

[0024] The embodiments for carrying out the present invention will be described in detail below with reference to the drawings. Figure 1 is a schematic diagram showing the overall system including the subtitle display position adjustment device of Embodiments 1 and 2, which will be described later. This system comprises a subtitle display position adjustment device 1 and a subtitle transmission system 2.

[0025] Subtitle display position adjustment device 1 is a device that adjusts the display position of subtitles when they are displayed on the screen with subtitles added to the base video. Subtitle display position adjustment device 1 takes the base video as input, detects one or more background video objects from the base video that should avoid displaying subtitles, and determines the position and type of each object. Then, according to the pre-set score for each type of object, subtitle display position adjustment device 1 assigns a score to each of the detected one or more objects.

[0026] The subtitle display position adjustment device 1 determines the subtitle display position based on the position and score of each object, so as to objects with higher scores, it avoids displaying subtitles in the area of ​​that object, and outputs the subtitle display position to the subtitle transmission system 2.

[0027] This allows us to determine the position and type of each object, thereby recognizing what kind of video the base video is. Then, based on the base video, we can control the subtitle transmission system 2 so that subtitles are displayed in a position that avoids the detected objects as much as possible.

[0028] The subtitle transmission system 2 receives the subtitle display position from the subtitle display position adjustment device 1 and performs subtitle transmission processing using the subtitle display position.

[0029] Here, the base video is, for example, the video of the program being broadcast. In the case of a live broadcast, this includes the video output from the studio, the video transmitted to the transmission station, and the video received from the broadcast. Furthermore, the base video includes not only the video of a live broadcast, but also the recorded video of a pre-recorded program, and the video content of an online video.

[0030] Objects extracted from the base video are partial images representing information with a predetermined meaning. Examples of object types include text overlays using the open captioning method (hereinafter simply referred to as "text overlays"), CG (Computer Graphics) overlays (such as map overlays), flip charts (such as handheld boards in the studio), human faces, program logos, etc.

[0031] Furthermore, the types of objects include not only text overlays and CG overlays, but also breaking news overlays that display things like earthquake information and news alerts, L-shaped overlays, inverted L-shaped overlays, clocks, and more.

[0032] The set score and score are weighted values ​​indicating the importance of the information. A higher set score and score value indicates that the information is important, while a lower value indicates that the information is not important. In other words, the set score and score represent the degree to which you want to avoid overlapping (overlap) with subtitles.

[0033] As a result, the higher the score, the less likely subtitles are to appear in the object's area (and its vicinity), and the lower the score, the more likely subtitles are to appear in the object's area (and its vicinity). Consequently, the subtitle display position is determined in a way that prevents subtitles from overlapping objects with high informational importance.

[0034] Therefore, since the subtitle display position can be automatically adjusted to the appropriate position according to the video without manual intervention, it becomes possible to streamline the subtitle production workflow.

[0035] [Subtitle display position adjustment device 1 / Example 1] First, let's describe Example 1. Figure 2 is a block diagram showing an example of the configuration of the subtitle display position adjustment device 1 in Example 1, and Figure 3 is a flowchart showing an example of the processing of the subtitle display position adjustment device 1 in Example 1.

[0036] The subtitle display position adjustment device 1-1 of this embodiment 1 includes an object detection unit 10, a coordinate score assignment unit 11, a display area score assignment unit 12, and a subtitle display position determination unit 13.

[0037] (Object detection unit 10) The object detection unit 10 receives the base video as input (step S301) and extracts still image frames from the base video at predetermined intervals (e.g., 1 second). The object detection unit 10 then detects background video objects from the frames that should be avoided when displaying subtitles, and determines the position and type of each object (step S302).

[0038] Specifically, the object detection unit 10 performs object detection processing for each type of object in the frame, and determines the position and type of each object by identifying the coordinates of the object (for example, the coordinates of the top-left and bottom-right vertices).

[0039] As mentioned above, the types of objects include text overlays, CG overlays, flip charts, human faces, program logos, etc. Existing processes are used for detection processing according to the type of object (processing to detect text overlay objects, processing to detect CG overlay objects, etc.).

[0040] The object detection unit 10 outputs the position and type of each object to the coordinate score assignment unit 11.

[0041] Figure 4 shows an example of a base video. The object detection unit 10 receives the base video shown in Figure 4, for example, and detects objects from the frames that make up the base video.

[0042] In the example in Figure 4, the types of objects are a CG superimposed map of Europe (bottom left of Figure 4), a human face (center), a program logo consisting of the word "Trade" (top right), and a subtitle superimposed text consisting of the word "Gateway to Europe" (bottom right).

[0043] Figure 5 shows an example of the position and type of each object. The object detection unit 10 detects four objects OB1 to OB4 from, for example, the frames of the base video shown in Figure 4, and determines the position and type of each of the objects OB1 to OB4.

[0044] In the example in Figure 5, for object OB1, its position is determined as PS1 (coordinates to identify the position of object OB1 on the base video screen), and its type is determined as "CG Super" (a map of Europe in this example). Similarly, for object OB2, its position is determined as PS2 (coordinates to identify the position of object OB2 on the base video screen), and its type is determined as "human face".

[0045] Furthermore, for object OB3, its position is determined as PS3 (coordinates to identify the position of object OB3 on the base video screen), and its type is determined as "program logo" (in this example, the word "trade"). In addition, for object OB4, its position is determined as PS4 (coordinates to identify the position of object OB4 on the base video screen), and its type is determined as "text superimposition" (in this example, the word "gateway to Europe").

[0046] (Coordinate scoring unit 11) Returning to Figures 2 and 3, the coordinate scoring unit 11 receives the position and type of each object from the object detection unit 10, as well as the pre-set set score for each type of object and the pre-set score coordinate data.

[0047] Here, the score coordinate data consists of coordinates (score coordinates) that indicate each region when the screen area of ​​the base video is divided into a predetermined number of regions.

[0048] The coordinate scoring unit 11 assigns a score to each of the one or more objects detected by the object detection unit 10, based on its position and type, according to the set score for each type of object (step S303). This provides a score for each object for each frame of the base video.

[0049] The coordinate scoring unit 11 maps the score for each object to the score coordinates indicated by the score coordinate data, based on the position of each object, and assigns a score to each score coordinate (step S304). This provides a score for each score coordinate for each frame of the base video. In this case, the coordinate scoring unit 11 maps the score for each object to the score coordinates, setting the score for areas other than the object's position to 0.

[0050] Figure 6 is a block diagram showing an example configuration of the coordinate scoring unit 11. This coordinate scoring unit 11 includes an object score processing unit 20 and a coordinate score processing unit 21.

[0051] The object score processing unit 20 receives the position and type of each object from the object detection unit 10, as well as a preset score for each type of object.

[0052] Figure 7 shows examples of setting scores for each object type in Example 1. As mentioned above, the setting score is a weighted value indicating the importance of the information, and is pre-set according to the importance of the information corresponding to the object type.

[0053] As shown in Figure 7, for example, the setting score is 80 when the object type is "text superimposed," 80 when the object type is "CG superimposed," 70 for "flip," 30 for "human face," and 10 for "program logo." These setting scores can be changed by the user.

[0054] When the object type is "text superimposition" or "CG superimposition," the information displayed on the object is considered more important than in the case of "flip," etc. Therefore, the setting score for "text superimposition" and "CG superimposition" is set to a higher value than for "flip," etc., reflecting the intention to prevent subtitles from overlapping.

[0055] Here, it is desirable to set the "text overlay" setting score to a higher value than the setting scores for other elements such as "CG overlays." This is because "text overlays" have a higher ability to convey information to the viewer than other elements such as "CG overlays" and "flip charts."

[0056] This ensures that when an object is of the "text superimposed" type, the subtitles do not overlap with the object, and the meaning of the information displayed in the "text superimposed" object is reliably conveyed to the viewer along with the subtitles. In other words, the viewer can reliably obtain the content of the information displayed in the "text superimposed" object along with the content of the subtitles.

[0057] Furthermore, if the object type is a high-priority item such as a "breaking news superimposed," an "L-shaped superimposed," or an "inverted L-shaped superimposed," or a "clock," it is desirable that the setting score be the highest possible value. This ensures that subtitles do not overlap with "breaking news superimposed," etc., and that viewers can reliably obtain the information from "breaking news superimposed," etc.

[0058] Returning to Figure 6, the object score processing unit 20 identifies the same type of setting score as the detected object from the set scores for each type of object shown in Figure 7 for each of the one or more objects detected by the object detection unit 10 (for each object at each location), and assigns that setting score as the score for the detected object.

[0059] The object score processing unit 20 outputs the position and score of each object to the coordinate score processing unit 21.

[0060] Figure 8 shows an example of the score for each object. For example, suppose the object detection unit 10 detects objects OB1 to OB4 shown in Figure 5. Then, the object score processing unit 20 identifies 80 for object OB1 (object position "PS1" and type "CG Super") from the set scores for each object type shown in Figure 7, which is the same type of set score as object OB1's type "CG Super". The object score processing unit 20 then assigns a score of 80 to object OB1.

[0061] Similarly, the object score processing unit 20 identifies 30 as the same type of setting score as object OB2's type "human face" and assigns 30 as the score for object OB2. Similarly, the object score processing unit 20 assigns 10 as the same type of setting score as "program logo" for object OB3. In addition, the object score processing unit 20 assigns 80 as the same type of setting score as "text superimposition" for object OB4.

[0062] In other words, as shown in Figure 8, within the base video screen, objects OB1 to OB4 are assigned scores of 80, 30, 10, and 80, respectively.

[0063] Returning to Figure 6, the coordinate score processing unit 21 receives the position and score of each object from the object score processing unit 20, as well as pre-set score coordinate data.

[0064] Figure 9 shows an example of score coordinate data. (X,Y) represents the score coordinates within the base video screen, and (x,y) represents the coordinates within the base video screen (base coordinates). In the example in Figure 9, the score coordinates are coordinates that represent divided regions obtained by dividing the base video screen into 10 equal parts vertically (m=10) and 7 equal parts horizontally (n=7), i.e., meshed coordinates.

[0065] Furthermore, the score coordinate data consists of 10 × 7 = 70 score coordinates represented as (X1, Y1), ..., (X10, Y7), and base coordinates used to identify those score coordinates. For example, for the score coordinate (X1, Y1), the corresponding base coordinates are (x0, y0) and (x1, y1). The base coordinate (x0, y0) indicates the coordinate of the top-left vertex of the region of the score coordinate (X1, Y1), and the base coordinate (x1, y1) indicates the coordinate of the bottom-right vertex of the region of the score coordinate (X1, Y1).

[0066] The data showing the correspondence between score coordinates and base coordinates, as shown in Figure 9, is input to the coordinate score processing unit 21 as pre-set score coordinate data. In the example in Figure 9, the number of score coordinates is set to m=10 and n=7 (total 70), but m and n can be integers of 1 or greater and are pre-set by the user.

[0067] Returning to Figure 6, the coordinate scoring processing unit 21 maps the score for each object to the score coordinates by associating the position of each object with the score coordinate data, and assigns a score to each score coordinate.

[0068] For example, the coordinate scoring processing unit 21 maps the scores of objects OB1 to OB4 shown in Figure 8 to the score coordinates shown in Figure 9, thereby assigning scores to each of the score coordinates (X1, Y1), ..., (X10, Y7).

[0069] The coordinate score processing unit 21 outputs the score for each score coordinate to the display area score assignment unit 12.

[0070] Figure 10 shows an example of scores for each score coordinate. These scores for each score coordinate were obtained by mapping the scores of objects OB1 to OB4 shown in Figure 8 to the score coordinates shown in Figure 9.

[0071] For example, the score of object OB1, 80, shown in Figure 8, is mapped to the score coordinates (X1,Y4),...,(X1,Y7),(X2,Y4),...,(X4,Y7) shown in Figure 9, which correspond to this position (PS1 shown in Figure 5). The scores of object OB2, 30, object OB3, 10, and object OB4, 80, are also mapped to their corresponding score coordinates, as shown in Figure 10.

[0072] Here, the coordinate score processing unit 21 determines the degree of overlap between, for example, the region representing the object's position and the region representing the score coordinates during the mapping process. Then, if the entire region of the score coordinates overlaps with the region of the object, or if even a part of it overlaps, the coordinate score processing unit 21 maps the object's score to the score coordinates.

[0073] Furthermore, the coordinate score processing unit 21 may map the object's score to the score coordinates if more than half of the score coordinate region overlaps with the object's region. If less than half of the score coordinate region overlaps with the object's region, or if the score coordinate region does not overlap with the object's region, the coordinate score processing unit 21 sets the score of the score coordinates to 0.

[0074] (Display area score assignment unit 12) Returning to FIGS. 2 and 3, the display area score assigning unit 12 inputs the score for each score coordinate from the coordinate score assigning unit 11 and also inputs the preset subtitle display area data.

[0075] FIG. 11 is a diagram showing an example of a subtitle display area. FIG. 11(1) shows the left subtitle display area R 1,2 ~R 1,6 in the screen of the base video. Hereinafter, in the subtitle display area R M,N the subscript M indicates the starting column number, which is M = 1 for the left side, M = 3 for the center described later, and M = 6 for the right side described later. The subscript N indicates the starting row number.

[0076] Specifically, the subtitle display area R 1,2 corresponds to the score coordinates (X1, Y2), (X1, Y3), ···, (X5, Y3) in the second and third rows and the first to fifth columns on the left side.

[0077] Also, the subtitle display area R 1,3 corresponds to the score coordinates (X1, Y3), (X1, Y4), ···, (X5, Y4) in the third and fourth rows and the first to fifth columns, which are moved one row down with respect to the subtitle display area R 1,2 .

[0078] Similarly, the subtitle display area R 1,6 corresponds to the score coordinates (X1, Y6), (X1, Y,7), ···, (X5, Y7) in the sixth and seventh rows and the first to fifth columns, which are moved one row down with respect to the subtitle display area R 1,5 .

[0079] FIG. 11(2) shows the central subtitle display area R 3,2 ~R 3,6 in the screen of the base video. Specifically, the subtitle display area R 3,2 corresponds to the score coordinates (X3, Y2), (X3, Y3), ···, (X8, Y3) in the second and third rows and the third to eighth columns in the center.

[0080] ]Also, subtitle display area R 3,3 This is the subtitle display area R 3,2 This corresponds to the score coordinates (X3,Y3), (X3,Y4), ..., (X8,Y4) in the 3rd and 4th rows and columns 3-8, which are shifted down one row.

[0081] Similarly, subtitle display area R 3,6 This is the subtitle display area R 3,5 This corresponds to the score coordinates (X3,Y6), (X3,Y7),...,(X8,Y7) in the 6th and 7th rows and columns 3-8, which are shifted down one row.

[0082] Figure 11(3) shows the right-aligned subtitle display area R within the base video screen. 6,2 ~R 6,6 This indicates the subtitle display area R. 6,2 These correspond to the score coordinates (X6,Y2), (X6,Y3), ..., (X10,Y3) in the right-aligned 2nd and 3rd rows and 6th to 10th columns.

[0083] Also, subtitle display area R 6,3 This is the subtitle display area R 6,2 This corresponds to the score coordinates (X6,Y3), (X6,Y4), ..., (X10,Y4) in the 3rd and 4th rows and columns 6-10, which are shifted down one row.

[0084] Similarly, subtitle display area R 6,6 This is the subtitle display area R 6,5 This corresponds to the score coordinates (X6,Y6), (X6,Y7), ..., (X10,Y7) in the 6th and 7th rows and columns 6-10, which are shifted down one row.

[0085] The data showing the correspondence between the subtitle display area and score coordinates as shown in Figures 11(1) to (3) is input to the display area score assignment unit 12 as pre-set subtitle display area data. In the example of Figures 11(1) to (3), a total of 15 subtitle display areas R 1,2 ~R 1,6 ,R 3,2 ~R 3,6 ,R6,2 ~R 6,6 However, the number and location of subtitle display areas are arbitrary and can be pre-configured by the user.

[0086] Returning to Figures 2 and 3, the display area score assignment unit 12 assigns a score to each subtitle display area by summing the scores of all score coordinates corresponding to the subtitle display area, according to the correspondence between the subtitle display area and the score coordinates indicated by the pre-set subtitle display area data, and using the sum as the score for that subtitle display area (step S305).

[0087] The display area score assignment unit 12 outputs a score for each subtitle display area to the subtitle display position determination unit 13.

[0088] In the examples shown in Figures 10 and 11, the display area score assignment unit 12 assigns a subtitle display area R with the starting column being the Mth column and the starting row being the Nth row. M,N Total score S M,N This is calculated using the following formula.

number

[0089] In equation (1) above, when M=1 and M'=5, the left-aligned subtitle display area R 1,2 ~R 1,6 Total score S 1,2 ~S 1,6 The following is calculated: If M=3 and M'=8, the central subtitle display area R 3,2 ~R 3,6 Total score S 3,2 ~S 3,6 This is calculated. Also, when M=6 and M'=10, the right-aligned subtitle display area R 6,2 ~R 6,6 Total score S 6,2 ~S 6,6 This is calculated. Xi,YN This indicates the score at the score coordinates (Xi, YN), and s Xi,Y(N+1) This indicates the score at the score coordinates (Xi, Y(N+1)).

[0090] For example, the subtitle display area R on the left side of the second line1,2 The total score S is calculated using the following formula. 1,2 The result is 60.

number

[0091] Also, the subtitle display area R in the center of the 6th line 3,6 The total score S is calculated using the following formula. 3,6 =800 is calculated.

number

[0092] Also, the subtitle display area R, which is aligned to the right of the fourth line. 6,4 The total score S is calculated using the following formula. 6,4 The result is 30.

number

[0093] Figure 12 shows an example of the score for each subtitle display area. These scores for each subtitle display area correspond to the examples in Figures 10 and 11, and the total score S calculated by formula (1) above is the same as the total score S. M,N This indicates that.

[0094] Furthermore, the display area score assignment unit 12 assigns a score to each subtitle display area by summing the scores of all score coordinates corresponding to the subtitle display area and setting the total score S as the score for that subtitle display area. Alternatively, the display area score assignment unit 12 may calculate the average or median value of the scores of all score coordinates corresponding to the subtitle display area and set this as the score for the subtitle display area to assign a score to each subtitle display area.

[0095] (Subtitle display position determination unit 13) Returning to Figures 2 and 3, the subtitle display position determination unit 13 receives a score for each subtitle display area from the display area score assignment unit 12, identifies the subtitle display area with the smallest score among all subtitle display areas, and determines that subtitle display area as the subtitle display position (step S306). Then, the subtitle display position determination unit 13 outputs the subtitle display position to the subtitle transmission system 2 (step S307).

[0096] This controls the subtitle display position so that, within the base video screen, the subtitle is displayed in the subtitle display area with the smallest score among all pre-set subtitle display areas. In the example in Figure 12, the right-aligned subtitle display area R of the fourth row. 6,4 Total score S 6,4 Since =30 is the minimum score, the subtitles are displayed in the right-aligned subtitle display area R of the fourth line within the base video screen. 6,4 It is controlled to be positioned accordingly.

[0097] Figure 13 shows an example of a base video with subtitles displayed in the optimal position, corresponding to the example in Figure 12. In the example in Figure 12, the total score is S 6,4 Since =30 is the minimum score, the subtitle "This city is said to be the gateway to Europe" is displayed in the right-aligned subtitle display area R of the fourth line within the base video screen, as shown in Figure 13. 6,4 It will be placed in that location.

[0098] Furthermore, if the subtitle display position determination unit 13 identifies multiple subtitle display areas with the minimum score among all subtitle display areas, it identifies one subtitle display area according to a pre-set default area or priority and determines that subtitle display area as the subtitle display position.

[0099] As described above, according to the subtitle display position adjustment device 1-1 of Embodiment 1, the object detection unit 10 detects one or more objects from the base video frame and determines the position and type of each object.

[0100] The coordinate scoring unit 11 determines the score for each object by assigning a score to each of one or more objects based on its position and type, according to a pre-set score for each type of object. Then, the coordinate scoring unit 11 maps the score for each object to score coordinates based on the position of each object, and assigns a score to each score coordinate.

[0101] The display area score assignment unit 12, according to the pre-set correspondence between subtitle display areas and score coordinates, sums the scores of all score coordinates in the subtitle display area and assigns a score to each subtitle display area.

[0102] The subtitle display position determination unit 13 determines the subtitle display position to be the subtitle display area with the smallest score among all subtitle display areas.

[0103] This allows subtitles to be automatically adjusted to the appropriate position according to the video without manual intervention, reducing the burden of adjustment work. For example, it can streamline the subtitle production workflow in a subtitle production environment.

[0104] Although the subtitle display position adjustment device 1-1 shown in Figure 2 includes an object detection unit 10, a coordinate score assignment unit 11, a display area score assignment unit 12, and a subtitle display position determination unit 13, it may also include only the object detection unit 10, the coordinate score assignment unit 11, and the subtitle display position determination unit 13.

[0105] In this case, the coordinate scoring unit 11 calculates a score for each object using the same process as described above, and outputs the position, type, and score of each object to the subtitle display position determination unit 13.

[0106] The subtitle display position determination unit 13 receives the position, type, and score for each object from the coordinate score assignment unit 11. Based on the position and score of each object, the subtitle display position determination unit 13 determines the subtitle display position so as to avoid displaying subtitles in the positions (areas) of objects with higher scores (for example, displaying subtitles in an area other than the object's position, or, if it is not possible to display subtitles in an area other than the object's position, displaying subtitles in a way that overlaps the position of the object with the lowest score and a part of it).

[0107] In this case, the subtitle display position determination unit 13 may identify the position of the object with the smallest score and determine the subtitle display position based on that position (displaying the subtitle in an area other than the object's position, or, if it is not possible to display the subtitle in an area other than the object's position, displaying the subtitle with part of the object overlapping it).

[0108] [Subtitle display position adjustment device 1 / Example 2] Next, we will describe Example 2. In Example 1 described above, the subtitle display position was determined by setting the score of the object detected from the base video frame to a constant value. In contrast, Example 2 is characterized by treating objects that are displayed for a long time as having low information freshness, making the object's score variable (decreasing over time), and determining the subtitle display position accordingly.

[0109] Figure 14 is a block diagram showing an example configuration of the subtitle display position adjustment device 1 of Embodiment 2. The subtitle display position adjustment device 1-2 of Embodiment 2 includes an object detection unit 10, a coordinate score assignment unit 14, a display area score assignment unit 12, and a subtitle display position determination unit 13.

[0110] Comparing the subtitle display position adjustment device 1-1 of Embodiment 1 shown in Figure 2 with the subtitle display position adjustment device 1-2 of this Embodiment 2, both subtitle display position adjustment devices 1-1 and 1-2 are common in that they include an object detection unit 10, a display area score assignment unit 12, and a subtitle display position determination unit 13. On the other hand, the subtitle display position adjustment device 1-2 differs from the subtitle display position adjustment device 1-1 in that it includes a coordinate score assignment unit 14 that is different from the coordinate score assignment unit 11 provided in the subtitle display position adjustment device 1-1. In Figure 14, parts common to Figure 2 are denoted by the same reference numerals as in Figure 2, and their detailed explanation is omitted.

[0111] The coordinate scoring unit 14 receives the position and type of each object from the object detection unit 10, as well as the initial score, initial time, and reduction coefficient for each type of object, and the pre-set score coordinate data.

[0112] Here, the initial score is a parameter that indicates the initial value of the score when an object is detected, and the initial time is a parameter that indicates the duration for which the initial score is maintained. The decrement coefficient is a parameter that decreases the score value the longer the object is continuously displayed on the screen. These parameters are pre-set for each type of object.

[0113] The coordinate scoring unit 14 assigns a score to each of the one or more objects detected by the object detection unit 10, based on its position and type, according to a preset initial score, initial time, and reduction coefficient for each type of object, such that the value decreases as time elapses after the object is detected. As a result, for each frame of the base video, an object-specific score is obtained that decreases according to the duration of display of the object after it is detected.

[0114] The coordinate scoring unit 14 maps the score for each object to the score coordinates indicated by the score coordinate data, based on the position of each object, and assigns a score to each score coordinate. The coordinate scoring unit 14 outputs the score for each score coordinate to the display area scoring unit 12.

[0115] An example configuration of the coordinate score assignment unit 14 will be described. Similar to the example configuration of the coordinate score assignment unit 11 shown in Figure 6, the coordinate score assignment unit 14 includes an object score processing unit 20' and a coordinate score processing unit 21.

[0116] The object score processing unit 20' receives the position and type of each object from the object detection unit 10, as well as the initial score, initial time, and reduction coefficient for each type of object, which have been set in advance.

[0117] The object score processing unit 20' identifies, for each of the one or more objects detected by the object detection unit 10 (for each object at each location), the initial score, initial time, and reduction coefficient of the same type as the detected object from among the pre-set initial score, initial time, and reduction coefficient for each type of object.

[0118] The object score processing unit 20' assigns the initial score to an object as its score when the object is detected, based on the identified initial score, initial time, and reduction coefficient, and calculates the display duration as the time during which the object's detection continues. The object score processing unit 20' then maintains the initial score as the score until the display duration reaches the initial time.

[0119] The object score processing unit 20' decreases the score according to a reduction coefficient when the display duration exceeds the initial time.

[0120] As a result, for each of the one or more objects detected by the object detection unit 10, a score is obtained that decreases according to the display duration of that object.

[0121] The object score processing unit 20' outputs the position and score (a score whose value decreases over time) for each object to the coordinate score processing unit 21. The processing of the coordinate score processing unit 21 is the same as that shown in Figure 6, so the explanation is omitted.

[0122] Figure 15 shows an example of the score decreasing according to the duration of object display in Example 2. Figure 15 shows the score for each type of object, according to the elapsed time since the object was detected.

[0123] If the object type is "Text Super", the initial score is set to 80, the initial time to 10 seconds, and the decrease factor to 2 (score / second). If the object type is "CG Super", the initial score is set to 80, the initial time to 10 seconds, and the decrease factor to 0.8 (score / second). If the object type is "Flip", the initial score is set to 70, the initial time to 20 seconds, and the decrease factor to 1 (score / second).

[0124] In this case, if the object type is "text superimposed," and the object detection unit 10 detects an object corresponding to that type, the score will have the characteristics shown in Figure 15 as time progresses, as long as the object continues to be displayed. Specifically, when an object is detected, an initial score of 80 is assigned, and this initial score of 80 is maintained until the display duration reaches the initial time of 10 seconds. After the display duration exceeds the initial time of 10 seconds, the score decreases according to a decrease coefficient of 2. That is, the score becomes 60 when the display duration is 20 seconds, and 40 when the display duration is 30 seconds.

[0125] Similarly, when the object type is "CG Super" or "Flip," the score will also have the characteristics shown in Figure 15, according to the pre-set initial score, initial time, and reduction coefficient, once an object corresponding to that type is detected.

[0126] Furthermore, if the object type is "Text Super," the score will remain at 20 after the display duration reaches 40 seconds. This indicates that, in addition to the initial value, initial time, and decrease coefficient parameters, other parameters such as the decrease stop time or minimum score are pre-set to achieve this characteristic.

[0127] Here, it is desirable that the reduction factor for "text superimposed" objects be set to a larger value than the reduction factor for other objects such as "CG superimposed" objects. This is because "text superimposed" objects can be understood immediately by reading them, unlike "CG superimposed" objects, "flips," etc., so even if the score value is reduced after a certain amount of time has passed and it overlaps with the subtitles, it does not cause any problems.

[0128] As a result, when the object type is "text overlay," viewers have already understood the meaning of the information displayed on the "text overlay" object in a short time when the object is detected. Therefore, even if the score is reduced afterward and subtitles are superimposed on the object, there is no problem in terms of information transmission.

[0129] Furthermore, it is desirable that the reduction factor for "CG Super" objects be set to a smaller value than the reduction factor for other objects such as "Text Super". This is because "CG Super" is shown to the viewer while they are listening to the dialogue, so the score value should not decrease significantly over time and should not overlap with the subtitles.

[0130] Furthermore, if the object type is a highly urgent "breaking news superimposed," "L-shaped superimposed," or "inverted L-shaped superimposed," or a "clock," it is desirable that the reduction coefficient be the lowest possible value close to 0. This ensures that the score maintains its initial value or a value close to it even as time passes, that subtitles do not overlap with "breaking news superimposed," etc., and that viewers can reliably obtain information from "breaking news superimposed," etc.

[0131] As described above, according to the subtitle display position adjustment device 1-2 of Embodiment 2, the object detection unit 10 detects one or more objects from the base video frame and determines the position and type of each object.

[0132] The coordinate scoring unit 14 assigns a score to each of one or more objects, based on its position and type, according to a preset initial score, initial time, and decrease coefficient for each type of object, such that the value decreases as time elapses after the object is detected. The coordinate scoring unit 14 then maps the score for each object to score coordinates based on the position of each object, and assigns a score to each score coordinate.

[0133] The display area score assignment unit 12, according to the pre-set correspondence between subtitle display areas and score coordinates, sums the scores of all score coordinates in the subtitle display area and assigns a score to each subtitle display area.

[0134] The subtitle display position determination unit 13 determines the subtitle display position to be the subtitle display area with the smallest score among all subtitle display areas.

[0135] As a result, similar to Example 1, the subtitle display position can be automatically adjusted to an appropriate position according to the video without manual intervention, thereby reducing the burden of adjustment work. For example, it can improve the efficiency of the subtitle production workflow in a subtitle production environment.

[0136] Furthermore, the score of an object decreases in proportion to its display duration, so the subtitle display position is determined according to the freshness of the information displayed by the object. In other words, since the freshness of the information of an object gradually decreases over time, by gradually changing its score to a smaller value, even if subtitles overlap the object as the score decreases after the viewer has understood the meaning of the information the object represents, there is no problem in terms of conveying the information of that object.

[0137] [Subtitle display position adjustment device 1 / Other examples] Regarding the subtitle display position adjustment device 1 shown in Figure 1, another example will be described. In this other example, existing subtitled program content is used as training data to machine-learn the subtitle display position adjustment patterns, and the resulting learned model is used.

[0138] Another example uses the relationship between base video and subtitle display position extracted from recorded program footage as training data for pre-recorded programs. The learning device extracts the base video and subtitle display position from the recorded program footage for pre-recorded programs and uses these as training data to build a learning model. The input data of the learning model is the base video frame, and the output data is the subtitle display position.

[0139] The subtitle display position adjustment device 1 is equipped with a learning model constructed by the aforementioned learning device, and receives a base video as input, inputs frames of the base video into the learning model, obtains the subtitle display position from the learning model, and outputs the subtitle display position.

[0140] This enables manual adjustment of the subtitle display position by the person in charge, as well as control equivalent to the automatic adjustment process by the subtitle display position adjustment devices 1-1 and 1-2 in Examples 1 and 2.

[0141] Another example uses the relationship between the base video and log data obtained from manual adjustments of the subtitle display position by staff in the subtitling production room (operation data for the height adjustment button B1 and left / right adjustment button B2 on the control box shown in Figure 16) as training data; in other words, it uses the base video and log data on the time axis.

[0142] The learning device constructs a learning model using the frames and log data of the base video as training data. The input data of the learning model is the frames of the base video, and the output data is subtitle display position control information.

[0143] The subtitle display position adjustment device 1 is equipped with a learning model constructed by the aforementioned learning device. The subtitle display position adjustment device 1 receives a base video as input, inputs frames of the base video into the learning model, obtains subtitle display position control information from the learning model, determines the subtitle display position based on the subtitle display position control information, and outputs the subtitle display position.

[0144] This enables manual adjustment of the subtitle display position by a person in charge, as well as control equivalent to the automatic adjustment process by the subtitle display position adjustment devices 1-1 and 1-2 in Examples 1 and 2. In particular, it enables control of the subtitle display position equivalent to that when using the operation box shown in Figure 16.

[0145] Although the present invention has been described above with reference to Examples 1 and 2, the present invention is not limited to Examples 1 and 2, and can be modified in various ways without departing from the technical concept.

[0146] For example, the subtitle display position adjustment devices 1-1 and 1-2 described above are applicable not only to the production of live subtitles, where closed captions are added in real time to live broadcast programs, but also to the production of subtitles added to pre-recorded, non-live broadcast programs. They are also applicable to the production of subtitles added to video services provided via the internet. Furthermore, they are applicable when adjusting the position of subtitles in receivers that receive and play back programs, or in internet content clients that receive and play back video content.

[0147] By using online video services as a universal service, it is possible to improve the services provided to viewers.

[0148] Furthermore, a standard computer can be used as the hardware configuration for the subtitle display position adjustment devices 1-1, 1-2, etc., according to Embodiments 1, 2, etc., of the present invention. The subtitle display position adjustment devices 1-1, 1-2, etc., are composed of a computer equipped with a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, and an interface, etc.

[0149] The functions of the object detection unit 10, coordinate scoring unit 11, display area scoring unit 12, and subtitle display position determination unit 13 provided in the subtitle display position adjustment device 1-1 are each realized by having the CPU execute a program that describes these functions.

[0150] Furthermore, the functions of the object detection unit 10, coordinate scoring unit 14, display area scoring unit 12, and subtitle display position determination unit 13 provided in the subtitle display position adjustment device 1-2 are each realized by having the CPU execute a program that describes these functions. The same applies to other examples.

[0151] These programs are stored in the aforementioned storage medium and are read and executed by the CPU. These programs can also be stored and distributed on storage media such as magnetic disks (floppy disks, hard disks, etc.), optical disks (CD-ROMs, DVDs, etc.), and semiconductor memory, and can be transmitted and received via a network. [Explanation of Symbols]

[0152] 1,1-1,1-2 Subtitle display position adjustment device 2. Subtitle transmission system 10 Object detection unit 11,14 Coordinate Score Assignment Section 12 Display area score assignment unit 13 Subtitle display position determination section 20,20' Object Score Processing Unit 21 Coordinate Score Processing Unit

Claims

1. In a subtitle display position adjustment device that adjusts the display position of subtitles added to a video, An object detection unit that detects one or more objects from the aforementioned video and determines the position and type of each object, The coordinates representing each of the multiple regions into which the screen area of ​​the aforementioned video is divided are used as score coordinates. For each of the one or more objects detected by the object detection unit, a score indicating the importance of the information of that object is assigned based on the position and type of each object determined by the object detection unit, according to a pre-set score for each type of object, thereby determining the score for each object. A coordinate score assignment unit calculates a score for each score coordinate by mapping the score for each object to the score coordinates based on the position of each object determined by the object detection unit, and assigning a score to the score coordinates. Multiple areas within the screen area of ​​the aforementioned video where the subtitles are displayed are defined as multiple subtitle display areas. A display area score assigning unit that assigns a score to each of the multiple subtitle display areas based on the score for each score coordinate obtained by the coordinate score assigning unit, thereby determining a score for each subtitle display area, A subtitle display position determination unit determines one of the multiple subtitle display areas as the display position for the subtitle, based on the score for each subtitle display area obtained by the display area score assignment unit, such that the higher the score, the less likely the subtitle is to be displayed. A subtitle display position adjustment device characterized by having the following features.

2. In the subtitle display position adjustment device according to claim 1, The type of each object includes at least a character superscript, A subtitle display position adjustment device characterized by setting the character superimposition setting score to a value greater than that of other objects.

3. In the subtitle display position adjustment device according to claim 1, The aforementioned coordinate scoring unit, For each of the one or more objects, a score is calculated such that the value of the score decreases as time elapses after the object has been detected. A subtitle display position adjustment device characterized by mapping the score for each object to the score coordinates based on the position of each object, and assigning a score to the score coordinates to obtain a score for each score coordinate.

4. In the subtitle display position adjustment device according to claim 3, The aforementioned coordinate scoring unit, For each of the one or more objects, a score is determined based on the position and type of each object, a preset initial score for each type of object, an initial time for which the initial score is maintained, and a reduction coefficient, such that the value of the score decreases as time elapses after the object is detected. The type of each object includes at least a character superscript, A subtitle display position adjustment device characterized in that the reduction coefficient of the character superimposition is set to a larger value than that of other objects.

5. A computer that constitutes a subtitle display position adjustment device, which adjusts the display position of subtitles added to a video, An object detection unit detects one or more objects from the aforementioned video and determines the position and type of each object. The coordinates representing each of the multiple regions into which the screen area of ​​the aforementioned video is divided are used as score coordinates. For each of the one or more objects detected by the object detection unit, a score indicating the importance of the information of that object is assigned based on the position and type of each object determined by the object detection unit, according to a pre-set score for each type of object, thereby determining the score for each object. Based on the position of each object determined by the object detection unit, the coordinate score assignment unit maps the score of each object to the score coordinates and assigns a score to the score coordinates, thereby determining the score for each score coordinate. Multiple areas within the screen area of ​​the aforementioned video where the subtitles are displayed are defined as multiple subtitle display areas. A display area score assigning unit that assigns a score to each of the multiple subtitle display areas based on the score for each score coordinate obtained by the coordinate score assigning unit, and A program to function as a subtitle display position determination unit, which determines one of the multiple subtitle display areas as the display position for the subtitle, based on the score for each subtitle display area obtained by the display area score assignment unit, such that the higher the score, the less likely the subtitle is to be displayed.

Images