Learning device, representative image extraction device and program

The learning device and representative image extraction device use a neural network trained with genre-specific labels to calculate image importance, addressing the failure of conventional methods to represent program content effectively by extracting images that reflect genre-specific characteristics.

JP7594474B2Active Publication Date: 2024-12-04NIPPON HOSO KYOKAI
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2021046325
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-03-19
Publication Date
2024-12-04
Estimated Expiration
2041-03-19

AI Technical Summary

Technical Problem

Conventional methods for extracting representative images from program videos fail to account for genre-specific characteristics and program production know-how, leading to ineffective representation of program content on websites.

Method used

A learning device and representative image extraction device that utilize a neural network trained with genre-specific labels to calculate the importance of images based on their suitability as representative images, incorporating genre codes and labels to generate a genre vector, thereby adjusting the neural network parameters to reflect the characteristics of each program genre.

Benefits of technology

The solution enables the extraction of representative images that accurately reflect the genre-specific characteristics of program videos, enhancing the effectiveness of program website content presentation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007594474000001
    Figure 0007594474000001
  • Figure 0007594474000002
    Figure 0007594474000002
  • Figure 0007594474000003
    Figure 0007594474000003
Patent Text Reader

Abstract

To extract a representative image reflecting characteristics of programs for every genre from program videos.SOLUTION: A program learning data generation unit 10 of a learning device 1 reads a genre label corresponding to a genre code from a table 21, gives the genre label to a program image for learning, and generates learning data consisting of the program image for learning, representative image suitability, and the genre label. A learning unit 11 generates a genre vector on the basis of the genre label included in the learning data, and calculates importance from the program image for learning included in the learning data and the generated genre vector by using an NN unit 31. The learning unit 11 gives a correct answer score from the representative image suitability included in the learning data, and updates a parameter of the NN unit 31 so that an error between the importance and the correct answer score becomes small.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a learning device, a representative image extraction device, and a program used in the field of video analysis for extracting a representative image from a video. [Background technology]

[0002] Traditionally, broadcasting stations have been improving their program homepages (HPs) in order to increase viewer contact rates. In many cases, the HPs contain multiple representative images extracted from the program footage to allow viewers to get a rough idea of ​​the program content.

[0003] However, extracting a representative image from a program video requires a lot of effort, so methods for automatically extracting a representative image from a program video have been proposed (see, for example, Patent Document 1 and Non-Patent Document 1).

[0004] For example, the method of Patent Document 1 calculates the relevance between images based on the results of discrimination of human faces, scenes and objects from a collection of images, GPS (Global Positioning System) information and similarity, and extracts a representative image based on the relevance and the date of shooting.

[0005] In addition, the method of Non-Patent Document 1 uses a pre-trained GoogleNet neural network to judge the artistic quality of an image. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 6149015 [Non-patent literature]

[0007] [Non-Patent Document 1] Xin Jin, et al., “ILGNet:Inception modules with connected local and global features for efficient image aesthetic quality classification using domain adaptation.”, IET Computer Vision 13.2 (2018):206-212. Summary of the Invention [Problem to be solved by the invention]

[0008] However, when extracting a representative image from a program video, the method of Patent Document 1 requires special information such as GPS information, shooting date, etc. In addition, the method of Non-Patent Document 1 does not take into account the know-how of program production.

[0009] For this reason, the conventional methods had the problem that they were unable to extract representative images that reflected the characteristics of each genre of program footage, and were also unable to extract representative images that reflected the know-how of program production.

[0010] In general, program images have characteristics according to the genre, and the representative image should reflect the characteristics of the program for each genre. Here, the characteristics of the program for each genre refer to the inherent properties of the programs belonging to the genre. For example, the video of a drama program has a characteristic that there are many images that show the face of the main character, and the video of a news program has a characteristic that there are many images that have CG or the like embedded therein.

[0011] For this reason, it is desirable for the representative image of a drama program to be an image showing the face of the main character, and for the representative image of a news program to be an image in which CG or the like is embedded.

[0012] Furthermore, when a representative image extracted using conventional methods was used on a program homepage, the program homepage was not necessarily effective, and there was a problem that the program content may not be effectively presented to viewers.

[0013] Therefore, the present invention has been made to solve the above-mentioned problems, and its object is to provide a learning device, a representative image extraction device, and a program capable of extracting representative images that reflect the characteristics of programs of each genre from program footage. [Means for solving the problem]

[0014] In order to solve the above problem, the learning device of claim 1 is a learning device that trains a neural network used to extract a representative image from a program video, the learning device comprising: a program learning data generation unit that inputs a training program image to which a representative image suitability has been assigned, and inputs one or more genre codes for the program of the training program image as a training genre code, reads out one or more genre labels corresponding to the training genre code from a table that defines a correspondence between the genre codes and genre labels, and generates learning data consisting of the training program image, the representative image suitability, and the one or more genre labels; and a learning unit that generates a genre vector by vectorizing the one or more genre labels included in the training data generated by the program learning data generation unit, calculates an importance from the training program image and the genre vector included in the training data using the neural network, assigns a correct score from the representative image suitability included in the training data, and updates parameters of the neural network so as to reduce an error between the importance and the correct score, thereby learning the neural network.

[0015] In addition, the learning device of claim 2 is characterized in that, in the learning device described in claim 1, the table defines a correspondence between one of the genre codes and one of the genre labels, and a correspondence between multiple of the genre codes and one of the genre labels.

[0016] The learning device of claim 3 is the learning device of claim 1 or 2, wherein the learning unit includes a genre vector generation unit that generates the genre vector, which is a vector consisting of elements with a number of dimensions equal to the number of all the genre labels defined in the table, and includes elements reflecting one or more of the genre labels included in the learning data; a neural network unit that uses the neural network to calculate the importance from the training program images included in the learning data and the genre vector generated by the genre vector generation unit; a correct answer score assignment unit that assigns the correct answer score from the representative image suitability included in the learning data; and a parameter update unit that updates parameters of the neural network so that an error between the importance calculated by the neural network unit and the correct answer score assigned by the correct answer score assignment unit is reduced.

[0017] The learning device of claim 4 is the learning device of claim 3, wherein the neural network unit comprises an image feature extraction neural network which inputs the learning program images included in the learning data and outputs an image feature vector by performing neural network operations, a genre feature extraction neural network which inputs the genre vector and outputs a genre feature vector by performing neural network operations, an addition unit which adds the image feature vector output by the image feature extraction neural network and the genre feature vector output by the genre feature extraction neural network to obtain a feature vector, and an importance calculation neural network which inputs the feature vector obtained by the addition unit and outputs the importance by performing neural network operations.

[0018] Further, a representative image extraction device of claim 5 is a representative image extraction device for extracting a representative image from a program video, comprising: an image sampling unit that inputs the program video and samples the program video to acquire a group of program images; a genre label assignment unit that inputs one or more genre codes for the program video as genre codes for extraction, reads out one or more genre labels corresponding to the genre codes for extraction from a table that defines a correspondence between the genre codes and genre labels, and assigns one or more of the genre labels to each of the group of program images acquired by the image sampling unit; an importance calculation unit that generates a genre vector for each of the group of program images by vectorizing the one or more genre labels assigned by the genre label assignment unit, and calculates importance from the program images and the genre vector using a neural network trained by the learning device of any one of claims 1 to 4; and an image sorting unit that sorts the group of program images based on the importance for each of the group of program images calculated by the importance calculation unit, and extracts the representative image.

[0019] The program of claim 6 is characterized in that the program causes a computer constituting a learning device that trains a neural network used to extract a representative image from a program video to function as a program learning data generation unit that inputs a training program image to which a representative image suitability has been assigned and one or more genre codes for the program of the training program image as a training genre code, reads out one or more genre labels corresponding to the training genre code from a table that defines a correspondence between the genre codes and genre labels, and generates learning data consisting of the training program image, the representative image suitability, and the one or more genre labels, and also functions as a learning unit that generates genre vectors by vectorizing the one or more genre labels included in the learning data generated by the program learning data generation unit, calculates importance from the training program image and the genre vector included in the learning data using the neural network, assigns a correct score from the representative image suitability included in the learning data, and updates parameters of the neural network so as to reduce an error between the importance and the correct score, thereby learning the neural network.

[0020] The program of claim 7 causes a computer constituting a representative image extraction device that extracts a representative image from a program video to function as an image sampling unit that inputs the program video and samples the program video to acquire a group of program images, a genre label assignment unit that inputs one or more genre codes for the program of the program video as genre codes for extraction, reads out one or more genre labels corresponding to the genre codes for extraction from a table that defines the correspondence between the genre codes and genre labels, and assigns one or more of the genre labels to each of the group of program images acquired by the image sampling unit, an importance calculation unit that generates a genre vector for each of the group of program images by vectorizing the one or more genre labels assigned by the genre label assignment unit, and calculates importance from the program images and the genre vector using a neural network trained by the learning device of any one of claims 1 to 4, and an image sorting unit that sorts the group of program images based on the importance for each of the group of program images calculated by the importance calculation unit and extracts the representative image. Effect of the Invention

[0021] As described above, according to the present invention, a representative image that reflects the characteristics of a program for each genre can be extracted from a program video. [Brief description of the drawings]

[0022] [Figure 1] 1 is a block diagram showing an example of the configuration of a learning device according to an embodiment of the present invention. [Diagram 2] 13 is a block diagram showing a configuration example of a program learning data generating unit. [Diagram 3] 13 is a flowchart showing an example of processing by a program learning data generating unit. [Figure 4] 4 is a block diagram showing an example of the configuration of a learning unit; FIG. [Diagram 5] 13 is a flowchart showing an example of processing by a learning unit. [Figure 6] 1 is a block diagram showing an example of the configuration of a representative image extraction device according to an embodiment of the present invention. [Figure 7] 5 is a flowchart illustrating an example of processing performed by the representative image extraction device according to the embodiment of the present invention. [Figure 8] 1 is a diagram for explaining an example of a major classification code of a program, and 2 is a diagram for explaining an example of a medium classification code of a program. [Figure 9] FIG. 4 is a diagram illustrating an example of a data configuration of a table. [Figure 10] FIG. 1 is a diagram for explaining a specific configuration example of an image feature extraction NN. [Figure 11] 1A is a diagram for explaining a specific example of the configuration of a genre feature extraction NN, and FIG. 1B is a diagram for explaining a specific example of the configuration of an importance calculation NN. [Figure 12] FIG. 1 is a diagram illustrating a program homepage creation system according to a first embodiment using a representative image extraction device. [Figure 13] FIG. 13 is a diagram illustrating a program DVD sales homepage creation system according to a second embodiment using a representative image extraction device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. [Learning device] First, a learning device according to an embodiment of the present invention will be described below. Fig. 1 is a block diagram showing an example of the configuration of a learning device according to an embodiment of the present invention.

[0024] This learning device 1 includes a program learning data generating unit 10 and a learning unit 11. The learning device 1 is a device that learns a neural network (hereinafter referred to as "NN") used by a representative image extracting device 2, which will be described later, using learning program images, genre codes, etc., so that the representative image extracting device 2, which will be described later, can extract representative images that reflect the characteristics of programs for each genre from program videos.

[0025] The program learning data generating unit 10 inputs a learning program image and one or more genre codes (learning genre codes) for the program of the learning program image. The learning program images are frame images sampled from program videos of various genres. A label (representative image suitability) indicating the degree of suitability as a representative image is assigned to the learning program image. For example, when the number of label stages is 3, the representative image suitability is one of great (very high suitability), good (high), and bad (low).

[0026] It is assumed that the program learning data generating unit 10 holds the genre codes of all programs that have been broadcast. There may be cases where one program corresponds to multiple genre codes.

[0027] The program learning data generating unit 10 integrates genres indicated by genre codes according to user operations and generates a table that defines the correspondence between genre codes and genre labels. This table is used by the representative image extracting device 2, which will be described later.

[0028] The program learning data generating unit 10 assigns one or more genre labels to the learning program images, and generates learning data consisting of the learning program images, the representative image suitability, and one or more genre labels. The program learning data generating unit 10 outputs the learning data to the learning unit 11. The program learning data generating unit 10, the table, the representative image suitability, the genre code, and the genre label will be described in detail later.

[0029] As a result, learning data is generated that includes learning program images, representative image suitability, and one or more genre labels.

[0030] The learning unit 11 is provided with a plurality of NNs to be trained. The learning unit 11 receives training data from the program training data generating unit 10 and trains the plurality of NNs using the training data.

[0031] This allows obtaining optimal parameters (weighting coefficients, etc.) for use with each of the multiple NNs. The multiple trained NNs are networks that reflect the characteristics of programs for each genre, and the parameters obtained by training are values ​​that reflect the characteristics of programs for each genre, and are used by the representative image extraction device 2, which will be described later.

[0032] (Program learning data generation unit 10) Next, the program learning data generating unit 10 shown in Fig. 1 will be described in detail. Fig. 2 is a block diagram showing an example of the configuration of the program learning data generating unit 10, and Fig. 3 is a flowchart showing an example of the processing of the program learning data generating unit 10. This program learning data generating unit 10 includes a genre integrating unit 20, a table 21, and a genre label assigning unit 22.

[0033] The genre integration unit 20 inputs a learning program image to which a representative image suitability has been assigned, and one or more genre codes (major category code and medium category code) for the program of the learning program image (step S301).

[0034] Prior to the learning process, the genre integration unit 20 integrates the genres indicated by the genre codes (major category codes and medium category codes) of all broadcasted programs in accordance with a user operation (step S302). In this case, the genre integration unit 20 calculates the number of learning program images for each genre code, for example, and displays them on the screen. The user performs an operation to integrate genres while referring to the number of learning program images for each genre code displayed on the screen.

[0035] The genre integration unit 20 generates a table 21 that defines the correspondence between one or more genre codes and one genre label through integration processing according to user operations (step S303). The number of genre labels defined in the table 21 is the number of genres after integration N, which is smaller than (or may be the same as) the number of original genre codes.

[0036] The table 21 generated by the genre integration unit 20 is output to a representative image extraction device 2 shown in FIG.

[0037] The genre integration unit 20 outputs to the genre label assignment unit 22 the learning program image to which the representative image suitability has been assigned, and one or more genre codes of the program of the learning program image.

[0038] FIG. 8(1) is a diagram for explaining examples of major classification codes of programs, and FIG. 8(2) is a diagram for explaining examples of medium classification codes of programs.

[0039] As shown in FIG. 8(1), there are 14 major category codes for programs. For example, the major category code "0" indicates "news / reporting" and the major category code "1" indicates "sports."

[0040] Also, as shown in Figure 8 (2), the number of program medium classification codes is determined for each major classification code. For example, when the major classification code is "0" (in the case of "News / Report"), the medium classification code "0" indicates "Regular / General", when the major classification code is "0", the medium classification code "1" indicates "Weather", and when the major classification code is "1" (in the case of "Sports"), the medium classification code "0" indicates "Sports News".

[0041] For details on the major and subgenres of programs, please refer to Part 2 of the standard ARIB STD-B10 "Program Arrangement Information for Use in Digital Broadcasting."

[0042] The genre integration unit 20 calculates the number of learning program images for each major classification code shown in FIG. 8(1), for example, and displays this on a display device not shown in FIG.

[0043] The genre integration unit 20 integrates, for example, "drama" with major category code "3", "movie" with major category code "6", "anime / special effects" with major category code "7", "theater / performance" with major category code "9", "extension" with major category code "14", and "other" with major category code "15" into one genre in order to eliminate genres with an extremely small number of study program images according to key operations (user operations) by the user who has referred to the number of study program images for each major category code. The genre integration unit 20 also integrates "documentary / culture" with major category code "8" and "welfare" with major category code "11" into another genre according to key operations by the user. Then, the genre integration unit 20 generates the table 21.

[0044] 9 is a diagram showing an example of the data structure of table 21, and shows table 21 generated by the above example. Table 21 is made up of major category codes, medium category codes, and genre labels corresponding to the major category codes and medium category codes.

[0045] In the above example, the major category code "0" "News / Reporting", the major category code "1" "Sports", the major category code "2" "Information / Variety Show", the major category code "4" "Music", the major category code "5" "Variety", and the major category code "10" "Hobbies / Education" are not integrated. For this reason, the genre label a is defined for the major category code "0" (and all the intermediate category codes "0-9, 10, 15" of the major category code "0"). Similarly, the genre labels b, c, d, e, and h are defined for the major category codes "1, 2, 4, 5, 10".

[0046] In addition, the major category code "3" "Drama", the major category code "6" "Movies", ..., and the major category code "15" "Other" are integrated, and the genre label f ​​is defined for these. Furthermore, the major category code "8" "Documentary / Educational" and the major category code "11" "Welfare" are integrated, and the genre label g is defined for these.

[0047] In this manner, the genre integration unit 20 generates the table 21 so as to reduce the number of genres indicated by the genre codes, thereby eliminating genres with an extremely small number of study-use program images.

[0048] This eliminates any genre with an extremely small number of training program images. Therefore, the training unit 11 described later can train an NN that is used by the representative image extraction device 2 described later to extract desired representative images from program videos.

[0049] Returning to FIG. 2 and FIG. 3, the genre label assignment unit 22 receives, from the genre integration unit 20, the training program images to which the representative image suitability has been assigned, and one or more genre codes of the programs of the training program images.

[0050] The genre label assigning unit 22 reads out a genre label corresponding to the genre code of the program of the study-purpose program image from the table 21 (step S304), and assigns the read-out genre label to the study-purpose program image (step S305).

[0051] Here, the genre label assignment unit 22 may assign multiple genre labels to one learning program image when multiple genre codes for one learning program image are input from the genre integration unit 20. In other words, the genre label assignment unit 22 assigns one genre label or multiple genre labels to one learning program image.

[0052] The genre label assignment unit 22 generates learning data consisting of the learning program images, the representative image suitability, and one or more genre labels, and outputs the learning data to the learning unit 11 (step S306).

[0053] (Learning Section 11) Next, a detailed description will be given of the learning unit 11 shown in Fig. 1. Fig. 4 is a block diagram showing an example of the configuration of the learning unit 11, and Fig. 5 is a flowchart showing an example of the processing performed by the learning unit 11.

[0054] The learning unit 11 includes a genre vector generation unit 30, an NN unit 31, a correct answer score assignment unit 36, an error calculation unit 37, and a parameter update unit 38. The NN unit 31 includes an image feature extraction NN 32, a genre feature extraction NN 33, an addition unit 34, and an importance calculation NN 35. The learning unit 11 performs the processes of steps S501 to S506 for each set consisting of a plurality of learning data until a termination condition is satisfied in the process of step S507.

[0055] The learning unit 11 receives learning data made up of a learning program image, a representative image suitability, and one or more genre labels from the program learning data generating unit 10 (step S501).

[0056] The genre vector generation unit 30 inputs one or more genre labels included in the learning data, and generates an N-dimensional genre vector based on the one or more genre labels (step S502).

[0057] As described above, N is the number of genres, specifically, the number of genre labels defined in table 21 generated by genre integration unit 20 shown in Fig. 2 in the processing of step S303 in Fig. 3. An N-dimensional genre vector is a vector that is made up of elements equal to the number of genres, N (elements corresponding to genre labels a, . . . , h in the example of table 21 shown in Fig. 9), and includes elements that reflect one or more genre labels included in the training data.

[0058] In the example of table 21 shown in Fig. 9, the number of genres N = 8. For example, if the genre label included in the training data is a, the genre vector generation unit 30 generates a genre vector (1,0,0,0,0,0,0,0,0). Also, if the genre labels included in the training data are e and f, the genre vector generation unit 30 generates a genre vector (0,0,0,0,1,1,0,0).

[0059] The NN unit 31 inputs the training program images included in the training data, and also inputs the genre vectors from the genre vector generation unit 30. Then, the NN unit 31 calculates a score from the training program images and the genre vectors using the image feature extraction NN 32, genre feature extraction NN 33, and importance calculation NN 35, the parameters of which have been updated by the parameter update unit 38, and the adder 34, and sets this score as the importance (step S503). The NN unit 31 outputs the importance to the error calculation unit 37. The image feature extraction NN 32, genre feature extraction NN 33, adder 34, and importance calculation NN 35 will be described in detail later.

[0060] Prior to processing, the correct score assigning unit 36 ​​presets a score assignment rule according to the number of levels of the representative image suitability label in accordance with a user operation. The correct score assigning unit 36 ​​inputs the representative image suitability included in the learning data, and assigns a correct score within the range of 0.0 to 1.0 according to the number of levels of the representative image suitability label using the preset score assignment rule (step S504). The correct score assigning unit 36 ​​outputs the correct score to the error calculation unit 37.

[0061] For example, if the representative image suitability labels are great / good / bad and the number of stages is 3, the correct score assigning unit 36 ​​assigns 1.0 / 0.5 / 0.0 to the representative image suitability labels great / good / bad, respectively, in accordance with a user operation. In this case, the correct scores do not necessarily have to be at equal intervals in the range of 0.0 to 1.0, and may be, for example, 1.0 / 0.7 / 0.0, as long as they are assigned at appropriate intervals.

[0062] The error calculation unit 37 inputs the importance from the NN unit 31 and also inputs the correct score from the correct score assignment unit 36. Then, the error calculation unit 37 calculates the error between the importance and the correct score (step S505), and outputs the error to the parameter update unit 38. For example, as a function for calculating the error, a function that outputs a larger value as the error increases, such as MSE (mean square error), is used.

[0063] The parameter update unit 38 holds parameters of the image feature extraction NN 32, the genre feature extraction NN 33, and the importance calculation NN 35. The parameter update unit 38 inputs the error from the error calculation unit 37, and updates the held parameters so as to reduce the error (step S506). Then, the parameter update unit 38 sets the updated parameters to the image feature extraction NN 32, the genre feature extraction NN 33, and the importance calculation NN 35 of the NN unit 31.

[0064] The parameter update unit 38 uses a general NN optimization method such as Adam, SGD (Stochastic Gradient Descent), or backpropagation as a process for updating parameters.

[0065] The parameter update unit 38 proceeds from step S506 and determines whether or not a parameter update termination condition is satisfied (step S507).

[0066] If it is determined in step S507 that the termination condition is not satisfied (step S507: N), the parameter update unit 38 returns to step S501 and performs the processes of steps S501 to S506 on the next learning data. That is, the processes of steps S501 to S506 are performed on each set of learning data until the termination condition is satisfied.

[0067] On the other hand, if it is determined in step S507 that the termination condition is satisfied (step S507: Y), the parameter update unit 38 outputs the parameters updated in the process of step S506 as optimal parameters (step S508). The optimal parameters output by the parameter update unit 38 are set in the NN of the importance calculation unit 43 provided in the representative image extraction device 2 shown in FIG. 6, which will be described later.

[0068] Here, the termination condition in step S507 is, for example, a condition as to whether or not a parameter has been updated a preset number of times, whether or not the amount of parameter update is smaller than a preset threshold, or the like.

[0069] In this way, the importance calculated by the NN unit 31 of the learning unit 11 is a value that reflects the characteristics of the image and the genre, and the higher the representative image suitability, the larger the value, and the lower the representative image suitability, the smaller the value. In other words, the NN unit 31 learned by the learning unit 11 becomes a network that calculates importance that reflects the characteristics of programs for each genre and the tendency of program images as representative images.

[0070] 6, which will be described later, uses an NN with the same configuration as the NN unit 31 in which the parameters of the NN unit 31 are set, so that if a program image has a high tendency to be a representative image, a high importance value can be obtained. On the other hand, if a program image has a low tendency to be a representative image, a low importance value can be obtained.

[0071] (NN Division 31) Next, the image feature extraction NN 32, the genre feature extraction NN 33, the addition unit 34, and the importance calculation NN 35 of the NN unit 31 shown in FIG. 4 will be described in detail.

[0072] FIG. 10 is a diagram for explaining a specific example of the configuration of the image feature extraction NN32, FIG. 11(1) is a diagram for explaining a specific example of the configuration of the genre feature extraction NN33, and FIG. 11(2) is a diagram for explaining a specific example of the configuration of the importance calculation NN35.

[0073] In Figure 10, Figure 11 (1) and (2), "Conv" indicates a convolution layer, "MaxPool" indicates a pooling layer that extracts the maximum value, and "LocalResponseNorm" indicates a normalization layer. "Inception Module" is a technology included in "GoogLeNet" and indicates a convolution layer and a pooling layer. "AveragePool" indicates a pooling layer that calculates the average value, "FC" indicates a fully connected layer, "Concat" indicates a concatenation layer, and "Sigmoid" indicates a layer that uses a sigmoid function. "Kernel" indicates the filter size, and "dim" indicates the number of dimensions.

[0074] As shown in Fig. 10, the image feature extraction NN32 inputs data of learning program images to the convolutional layer α1 of "Conv" and performs NN calculations from the convolutional layer α1 of "Conv" to the concatenation layer α2 of "Concat". Then, the image feature extraction NN32 outputs a 1024-dimensional image feature vector from the concatenation layer α2 of "Concat". The 1024-dimensional image feature vector output by the image feature extraction NN32 is input to the addition unit 34 of the NN unit 31 shown in Fig. 4.

[0075] As shown in FIG. 11(1), the genre feature extraction NN33 inputs an N-dimensional genre vector to the fully connected layer α3 of "FC", and performs NN calculations on the fully connected layer α3 of "FC" and the fully connected layer α4 of "FC". Then, the genre feature extraction NN33 outputs a 1024-dimensional genre feature vector from the fully connected layer α4 of "FC". The 1024-dimensional genre feature vector output by the genre feature extraction NN33 is input to the adder 34 of the NN unit 31 shown in FIG. 4. Here, the dimension (size) of the genre feature vector output by the genre feature extraction NN33 is equal to the dimension (size) of the image feature vector output by the image feature extraction NN32.

[0076] 4 inputs the 1024-dimensional image feature vector from the image feature extraction NN 32, and also inputs the 1024-dimensional genre feature vector from the genre feature extraction NN 33. The addition unit 34 then adds the 1024-dimensional image feature vector and the 1024-dimensional genre feature vector, and outputs the 1024-dimensional feature vector resulting from the addition to the importance calculation NN 35.

[0077] As shown in Fig. 11(2), the importance calculation NN35 inputs a 1024-dimensional feature vector to the fully connected layer α5 of "FC", performs NN calculations from the fully connected layer α5 of "FC" to the layer α6 of the sigmoid function of "Sigmoid", and calculates a one-dimensional score. Then, the importance calculation NN35 outputs the one-dimensional score from the layer α6 of the sigmoid function of "Sigmoid" to the error calculation unit 37 shown in Fig. 4 as importance.

[0078] Thus, the NN unit 31 is composed of an image feature extraction NN 32 that calculates a 1024-dimensional image feature vector from the training program image, a genre feature extraction NN 33 that calculates a 1024-dimensional genre feature vector from the genre vector, an addition unit 34 that adds the 1024-dimensional image feature vector and the 1024-dimensional genre feature vector, and an importance calculation NN 35 that calculates a one-dimensional score (importance) from the 1024-dimensional feature vector resulting from the addition.

[0079] This NN unit 31 calculates a one-dimensional score regardless of the number of stages of the correct score when the correct score assigning unit 36 ​​generates the correct score from the representative image suitability. In other words, as the NN unit 31, it is not necessary to prepare a different NN according to the number of stages of the correct score of the representative image suitability, and it is sufficient to prepare an NN with a fixed configuration that is not dependent on the number of stages.

[0080] As described above, according to the study device 1 of the embodiment of the present invention, the program study data generation unit 10 inputs a study program image and a genre code, and reads out a genre label corresponding to the genre code from the table 21 generated in accordance with a user operation. The program study data generation unit 10 then assigns a genre label to the study program image, and generates study data consisting of the study program image, the representative image suitability, and the genre label.

[0081] The learning unit 11 uses the learning data to train the NN unit 31. Specifically, the learning unit 11 generates a genre vector based on a genre label included in the learning data, and uses the NN unit 31 to calculate importance from the learning program image and the genre vector included in the learning data.

[0082] The learning unit 11 assigns a correct answer score based on the representative image suitability included in the learning data, and updates the parameters of the NN unit 31 so as to reduce the error between the importance and the correct answer score, thereby training the NN unit 31.

[0083] Here, the representative image suitability is an index showing whether or not an image is suitable as a representative image of a program, and reflects the know-how of program production, but does not reflect the characteristics of programs of each genre. In the embodiment of the present invention, in addition to the representative image suitability, genre labels are used as learning data, and thus, learning incorporating genres is performed, and the NN unit 31 (parameters of the NN unit 31) that reflects the characteristics of programs of each genre is generated.

[0084] The representative image extraction device 2 described later can extract representative images that reflect the characteristics of programs of each genre from program footage by using an NN (an NN with the same configuration as NN unit 31) in which the parameters of the NN unit 31 are set.

[0085] [Representative Image Extraction Device] Next, a representative image extraction device that extracts representative images from program videos using the NN trained by the learning device 1 shown in Fig. 1 will be described. Fig. 6 is a block diagram showing an example of the configuration of the representative image extraction device according to an embodiment of the present invention, and Fig. 7 is a flowchart showing an example of the processing performed by the representative image extraction device.

[0086] The representative image extraction device 2 includes an image sampling section 40, a table 41, a genre label assignment section 42, an importance calculation section 43, and an image sorting section 44.

[0087] The representative image extraction device 2 receives a program video (all frames constituting the program video) and one or more genre codes (genre codes for extraction) for the program of the program video (step S701). The image sampling unit 40 receives the program video, samples the program video at regular time intervals, and acquires a group of program images after sampling (step S702).

[0088] Instead of inputting all the frames constituting the program video, the representative image extraction device 2 may input only the frames obtained by sampling the program video (sampled program images). In this case, the representative image extraction device 2 does not need to include the image sampling unit 40, and the calculation processing load can be reduced.

[0089] Furthermore, the image sampling unit 40 may select a predetermined number of program images from a group of program images acquired by sampling the program video, and output only the selected predetermined number of program images to the genre label assignment unit 42. This makes it possible to reduce the calculation processing load in the genre label assignment unit 42, importance calculation unit 43, and image sorting unit 44 in the subsequent stages.

[0090] The table 41 stores the same data as the table 21 generated by the genre integration unit 20 provided in the program learning data generation unit 10 of the learning device 1. Prior to processing by the representative image extraction device 2, the table 41 is set with the same data as the table 21.

[0091] The genre label assignment unit 42 inputs the program image group from the image sampling unit 40, and also inputs one or more genre codes of the program of the program video. Then, for each of the program image group, the genre label assignment unit 42 reads out a genre label corresponding to the program genre code of the program image from the table 41 (step S703), and assigns the read genre label to the program image (step S704).

[0092] Here, when multiple genre codes are input for one program image, the genre label assignment unit 42 may assign multiple genre labels to one program image. That is, the genre label assignment unit 42 assigns one or multiple genre labels to one program image. The genre label assignment unit 42 outputs the program image and one or multiple genre labels for each of the program image group to the importance calculation unit 43.

[0093] As a result, one or more genre labels are assigned to each of the program images.

[0094] The importance calculation unit 43 includes an NN having the same configuration as the NN unit 31 that has been trained by the learning device 1. The NN included in the importance calculation unit 43 is set with the same parameters as those in the NN unit 31 prior to processing by the representative image extraction device 2.

[0095] The importance calculation unit 43 inputs the program image and one or more genre labels for each of the program image groups from the genre label assignment unit 42. Then, the importance calculation unit 43 calculates the importance from the program image and one or more genre labels using a NN.

[0096] Specifically, importance calculation section 43 generates an N-dimensional genre vector based on one or more genre labels, similar to the process of step S502 by genre vector generation section 30 shown in FIGS. 4 and 5 (step S705).

[0097] The importance calculation unit 43 calculates the importance from the program image and the genre vector using a trained NN similar to that of the NN unit 31 shown in Fig. 4 (step S706). Then, the importance calculation unit 43 outputs the program image and the importance for each of the program image group to the image sorting unit 44.

[0098] The image sorting unit 44 inputs the program images and the importance for each of the program images from the importance calculation unit 43. Then, the image sorting unit 44 sorts the program images in descending order of importance, and extracts a predetermined number (an integer equal to or greater than 1) of program images with high importance from among all the program images as representative images (candidates) (step S707).

[0099] The image sorting unit 44 sorts the predetermined number of representative images in chronological order, and outputs the predetermined number of representative images in chronological order (step S708).

[0100] The image sorting unit 44 may input all program images constituting the program video and their corresponding importance, classify all program images into, for example, three classes based on the importance by threshold processing, and extract program images in the higher classes as representative images. The image sorting unit 44 does not necessarily need to classify program images by dividing the importance into equal intervals.

[0101] For example, the image sorting unit 44 performs threshold processing using a preset threshold (e.g., 0.25, 0.75) to classify program images of the importance into a first class level if 0.00≦importance≦threshold 0.25. Also, the image sorting unit 44 classifies program images of the importance into a second class level if threshold 0.25<importance<threshold 0.75, and classifies program images of the importance into a third class level if threshold 0.75≦importance≦1.00. Then, the image sorting unit 44 extracts program images of the third class level as representative images.

[0102] As described above, according to the representative image extraction device 2 of the embodiment of the present invention, the image sampling unit 40 inputs a program video and samples the program video to obtain a group of program images. The genre label assignment unit 42 inputs a program genre code of the program video for each of the group of program images, reads a genre label corresponding to the genre code from table 41, which is the same as table 21 generated by the learning device 1, and assigns the genre label to the program image.

[0103] The importance calculation unit 43 assigns a correct answer score based on the genre label, and calculates the importance of each of the program images from the program image and the genre label using the same NN as the NN unit 31 trained by the learning device 1.

[0104] The image sorting unit 44 sorts the program images in descending order of importance, extracts a predetermined number of program images with high importance as representative images, and outputs them.

[0105] Here, the NN used by the representative image extraction device 2 has the same configuration as the NN unit 31 trained by the learning device 1, and is a network that reflects the characteristics of programs for each genre. Therefore, by using this NN, it is possible to extract representative images that reflect the characteristics of programs for each genre from program videos. And, for example, a homepage creator who creates a program homepage using the representative images can greatly reduce the amount of work and improve the homepage.

[0106] [Example using representative image extraction device 2] Next, an embodiment using the representative image extraction device 2 shown in Fig. 6 will be described. Fig. 12 is a diagram illustrating a program HP creation system of a first embodiment using the representative image extraction device 2. This program HP creation system 3 is a system that creates a program HP using program EPG (Electronic Programming Guide) information and program video for the program for which the program HP is to be created.

[0107] The program homepage creation system 3 comprises a representative image extraction device 2, a summary video generation unit 100, and an automatic layout processing unit 101. The summary video generation unit 100 is a component that generates a summary video from program video by conventional processing, and the representative image extraction device 2 is a device according to the embodiment of the present invention shown in Fig. 6, and extracts, for example, three representative images from the program video.

[0108] The automatic arrangement processing unit 101 arranges the program EPG information, summary video, and three representative images at preset positions, and creates a program HP as shown in FIG.

[0109] 13 is a diagram illustrating a program DVD sales website creation system of a second embodiment using a representative image extraction device 2. This program DVD sales website creation system 4 is a system that creates a program DVD sales website using DVD promotional comments, DVD package images, and program DVD videos about the program DVD for which the program DVD sales website is to be created.

[0110] The program DVD sales website creation system 4 comprises a representative image extraction device 2 and an automatic layout processing unit 102. The representative image extraction device 2 is a device according to the embodiment of the present invention shown in Fig. 6, and extracts, for example, six representative images from a program DVD video.

[0111] The automatic arrangement processing unit 102 arranges the DVD promotional comments, the DVD package image, and the six representative images at preset positions, and creates a program DVD sales homepage as shown in FIG.

[0112] In this way, the program homepage creation system 3 and the program DVD sales homepage creation system 4 can create a substantial homepage by using the representative image extracted by the representative image extraction device 2.

[0113] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical concept thereof.

[0114] For example, in the learning device 1 shown in Fig. 1, the image feature extraction NN 32 shown in Fig. 10 obtains output data for a 1024-dimensional image feature vector, and the genre feature extraction NN 33 shown in Fig. 11(1) obtains output data for a 1024-dimensional genre feature vector. Also, the importance calculation NN 35 shown in Fig. 11(2) inputs the 1024-dimensional feature vector obtained by the addition unit 34. The number of dimensions in the 1024-dimensional image feature vector, genre feature vector, and feature vector is merely an example, and the present invention does not limit the number of dimensions to 1024.

[0115] A normal computer can be used as the hardware configuration of the learning device 1 according to the embodiment of the present invention. The learning device 1 is configured by a computer equipped with a CPU, a volatile storage medium such as a RAM, a non-volatile storage medium such as a ROM, an interface, etc. The same applies to the representative image extraction device 2 according to the embodiment of the present invention.

[0116] Each function of the program learning data generating unit 10 and the learning unit 11 provided in the learning device 1 is realized by causing a CPU to execute a program in which these functions are written.

[0117] In addition, each function of the image sampling unit 40, table 41, genre label assignment unit 42, importance calculation unit 43 and image sorting unit 44 provided in the representative image extraction device 2 is realized by having the CPU execute a program that describes these functions.

[0118] These programs are stored in the storage medium and are read and executed by the CPU. These programs can also be distributed by storing them in a storage medium such as a magnetic disk (floppy disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), or a semiconductor memory, and can also be transmitted and received via a network. [Explanation of symbols]

[0119] 1 Learning device 2. Representative image extraction device 3 Program homepage creation system 4 Program DVD sales website creation system 10 Program learning data generation unit 11 Learning Department 20 Genre Integration Department 21,41 Table 22,42 Genre label assignment section 30 Genre Vector Generation Unit 31 NN (Neural Network) Department 32 Image feature extraction NN 33 Genre feature extraction NN 34 Addition section 35 Importance calculation NN 36 Correct answer score assignment section 37 Error calculation section 38 Parameter Update Section 40 Image Sampling Section 43 Importance calculation part 44 Image sorting section

Claims

1. A learning device for learning a neural network used to extract a representative image from a program video, inputting a learning program image to which a representative image suitability has been assigned, and inputting one or more genre codes for the program of the learning program image as a learning genre code; read out one or more genre labels corresponding to the learning genre code from a table that defines a correspondence relationship between the genre code and the genre label; a program learning data generation unit that generates learning data including the learning program images, the representative image suitability, and one or more of the genre labels; generating a genre vector by vectorizing one or more of the genre labels included in the learning data generated by the program learning data generation unit; calculating an importance level from the training program images and the genre vectors included in the training data using the neural network; assigning a correct answer score based on the representative image suitability included in the learning data; a learning unit that learns the neural network by updating parameters of the neural network so that an error between the importance and the correct score becomes small; A learning device comprising:

2. 2. The learning device according to claim 1, The table includes: A learning device, characterized in that a correspondence relationship between one of the genre codes and one of the genre labels, and a correspondence relationship between a plurality of the genre codes and one of the genre labels are defined.

3. 3. The learning device according to claim 1, The learning unit is a genre vector generation unit that generates the genre vector, the genre vector being a vector consisting of elements of the number of dimensions, the number of all the genre labels defined in the table being a number of dimensions, the genre vector including elements reflecting one or more of the genre labels included in the training data; a neural network unit that uses the neural network to calculate the importance from the training program images included in the training data and the genre vector generated by the genre vector generation unit; a correct answer score assigning unit that assigns the correct answer score based on the representative image suitability included in the learning data; a parameter update unit that updates parameters of the neural network so that an error between the importance calculated by the neural network unit and the correct score assigned by the correct score assigning unit becomes small; A learning device comprising:

4. 4. The learning device according to claim 3, The neural network unit includes: an image feature extraction neural network that receives the training program images included in the training data and outputs an image feature vector by performing a neural network operation; a genre feature extraction neural network that receives the genre vector and performs neural network calculations to output a genre feature vector; an adder that adds the image feature vector output by the image feature extraction neural network and the genre feature vector output by the genre feature extraction neural network to obtain a feature vector; an importance calculation neural network that receives the feature vector calculated by the adding unit and outputs the importance by performing a neural network operation; A learning device comprising:

5. A representative image extraction device for extracting a representative image from a program video, comprising: an image sampling unit that receives the program video and samples the program video to acquire a group of program images; a genre label assigning unit that inputs one or more genre codes for the program of the program video as genre codes for extraction, reads out one or more genre labels corresponding to the genre codes for extraction from a table that defines a correspondence between the genre codes and genre labels, and assigns one or more of the genre labels to each of the program images acquired by the image sampling unit; generating a genre vector by vectorizing the one or more genre labels assigned by the genre label assigning unit for each of the program images; an importance calculation unit that calculates an importance from the program image and the genre vector by using a neural network trained by the learning device according to any one of claims 1 to 4; an image sorting unit that sorts the group of program images based on the importance of each of the group of program images calculated by the importance calculation unit, and extracts the representative image; A representative image extraction device comprising:

6. A computer constituting a learning device for learning a neural network used to extract a representative image from a program video, inputting a learning program image to which a representative image suitability has been assigned, and inputting one or more genre codes for the program of the learning program image as a learning genre code; read out one or more genre labels corresponding to the learning genre code from a table that defines a correspondence relationship between the genre code and the genre label; a program learning data generating unit that generates learning data including the learning program image, the representative image suitability, and one or more of the genre labels; and generating a genre vector by vectorizing one or more of the genre labels included in the learning data generated by the program learning data generation unit; calculating an importance level from the training program images and the genre vectors included in the training data using the neural network; assigning a correct answer score based on the representative image suitability included in the learning data; A program for functioning as a learning unit that learns the neural network by updating parameters of the neural network so as to reduce an error between the importance and the correct answer score.

7. A computer constituting a representative image extraction device for extracting a representative image from a program video, an image sampling unit that inputs the program video and samples the program video to acquire a group of program images; a genre label assigning unit that inputs one or more genre codes for the program of the program video as genre codes for extraction, reads out one or more genre labels corresponding to the genre codes for extraction from a table that defines a correspondence between the genre codes and genre labels, and assigns one or more of the genre labels to each of the group of program images acquired by the image sampling unit; generating a genre vector by vectorizing the one or more genre labels assigned by the genre label assigning unit for each of the program images; an importance calculation unit that calculates an importance from the program image and the genre vector by using a neural network trained by the learning device according to any one of claims 1 to 4; a program for functioning as an image sorting section that sorts the group of program images based on the importance of each of the group of program images calculated by the importance calculation section, and extracts the representative image;

Citation Information

Patent Citations

  • Shaking platform

    JP1986049015A

  • Recording apparatus, scene classification method, computer program and storage medium

    JP2008252886A

  • Information transmission system and program

    JP2020177373A