Representative image extraction device and program

Abstract

To enable the most appropriate representative image to be extracted, after determining whether or not to reflect a characteristic of a program for each genre, based on a trend of the representative image, when extracting the representative image from a program video.SOLUTION: A representative image extraction device 2 comprises: an NN selection unit 50 which inputs program image data in which a program image, a genre label and a correct answer score are made a set, and determines whether to select a model with genre information, or whether to select the model without the genre information; an image selection unit 51 which calculates a score from the program image and the genre label, when the selection result is the model with the genre information, and calculates the score from the program image, when the selection result is the model without the genre information; and an image sort unit 52 which sorts a program image group, on the basis of a calculated score, and extracts a representative image.SELECTED DRAWING: Figure 17

Description

[Technical field]

[0001] The present invention relates to a representative image extraction device and a program for use in the field of video analysis for extracting a representative image from a video. [Background technology]

[0002] Traditionally, broadcasting stations have been improving their program homepages (HPs) in order to increase viewer contact rates. In many cases, the HPs contain multiple representative images extracted from the program footage to allow viewers to get a rough idea of ​​the program content.

[0003] However, extracting a representative image from a program video requires a lot of effort, so methods for automatically extracting a representative image from a program video have been proposed (see, for example, Patent Document 1 and Non-Patent Document 1).

[0004] For example, the method of Patent Document 1 calculates the relevance between images based on the results of discrimination of human faces, scenes and objects from a collection of images, GPS (Global Positioning System) information and similarity, and extracts a representative image based on the relevance and the date of shooting.

[0005] Moreover, the method of Non-Patent Document 1 uses a pre-trained GoogleNet neural network (hereinafter, abbreviated as "NN") to judge the artistic quality of an image. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 6149015 [Non-patent literature]

[0007] [Non-Patent Document 1] Xin Jin, et al., “ILGNet:Inception modules with connected local and global features for efficient image aesthetic quality classification using domain adaptation.”, IET Computer Vision 13.2 (2018):206-212. Summary of the Invention [Problem to be solved by the invention]

[0008] However, when extracting a representative image from a program video, the method of Patent Document 1 requires special information such as GPS information, shooting date, etc. In addition, the method of Non-Patent Document 1 does not take into account the know-how of program production.

[0009] For this reason, the conventional methods had the problem that they were unable to extract representative images that reflected the characteristics of each genre of program footage, and were also unable to extract representative images that reflected the know-how of program production.

[0010] In general, program images have characteristics according to the genre, and the representative image should reflect the characteristics of the program for each genre. Here, the characteristics of the program for each genre refer to the inherent properties of the programs belonging to the genre. For example, the video of a drama program has a characteristic that there are many images that show the face of the main character, and the video of a news program has a characteristic that there are many images that have CG or the like embedded therein.

[0011] For this reason, it is desirable for the representative image of a drama program to be an image showing the face of the main character, and for the representative image of a news program to be an image in which CG or the like is embedded.

[0012] Therefore, the same inventor as the present patent application invented a learning device, a representative image extraction device, and a program that can extract representative images that reflect the characteristics of programs of each genre from program footage by using genre information (genre code, genre label, etc.) (Patent Application No. 2021-46325, unpublished at the time of filing this patent application).

[0013] However, when extracting a representative image that reflects the characteristics of a program for each genre, for example, when one program contains various genres such as a variety show, the genre information can have a negative effect, and ultimately, a representative image suitable for the program video may not be extracted.

[0014] That is, there are cases where a representative image reflecting the characteristics of a program for each genre is more suitable as a representative image for a program video, and cases where a representative image reflecting the know-how of program production is more suitable as a representative image regardless of the genre. Therefore, if a representative image reflecting the characteristics of a program for each genre is extracted for all program videos, when the extracted representative image is used on a program homepage, the program homepage is not necessarily effective, and there are cases where the program content cannot be effectively presented to viewers.

[0015] Therefore, the present invention has been made to solve the above-mentioned problems, and its object is to provide a representative image extraction device and program that, when extracting a representative image from program footage, is capable of taking into account the tendency of the representative image and determining whether or not to reflect the characteristics of programs of each genre, and then extracting the most appropriate representative image. [Means for solving the problem]

[0016] In order to solve the above problem, the representative image extraction device of claim 1 defines a model in which the genre label of a program image is included in input data as a model with genre information, and a model in which the genre label is not included in input data as a model without genre information, and uses a neural network of the model with genre information (NN with genre information) and a neural network of the model without genre information (NN without genre information) trained by a learning device to extract representative images from program video. The representative image extraction device inputs program image data in which the program image, the genre label, and a correct score are set, calculates scores using the model with genre information and the model without genre information, calculates errors between the scores and the correct score, and extracts representative images from the model with genre information based on the respective errors. an image selection unit that inputs the program video and a genre code of the program video, as well as the selection result determined by the NN selection unit, and, if the selection result indicates the model with genre information, calculates a first score based on the program image obtained by sampling the program video and the genre label corresponding to the genre code, and, if the selection result indicates the model without genre information, calculates a second score based on the program image obtained by sampling the program video; and an image sorting unit that sorts the program images obtained by sampling the program video based on the first score or the second score calculated by the image selection unit, and extracts the representative image.

[0017] A representative image extraction device according to claim 2 is the representative image extraction device according to claim 1, further comprising: a first sampling processing unit which, when the selection result input by the image selection unit indicates the model with genre information, samples the program video input by the image selection unit to generate a group of program images; a genre label assignment unit which reads out the genre label corresponding to the genre code input by the image selection unit from a table that defines a correspondence relationship between the genre code and the genre label, and assigns the genre label to each of the group of program images generated by the first sampling processing unit; The present invention is characterized in that it comprises a first score calculation unit that generates a genre vector by vectorizing the genre label assigned by the label assignment unit, and calculates the first score from the program image and the genre vector using the NN with genre information trained by the learning device; a second sampling processing unit that, when the selection result input by the image selection unit indicates the model without genre information, samples the program video input by the image selection unit to generate a group of program images; and a second score calculation unit that calculates the second score from the program image for each of the group of program images generated by the second sampling processing unit, using the NN without genre information trained by the learning device.

[0018] The representative image extraction device of claim 3 is the representative image extraction device of claim 1 or 2, wherein the NN selection unit comprises: a first model error calculation unit that inputs the program image data in which the program image, the genre label, and the correct score are set, calculates a score using the NN with genre information, calculates the error between the score and the correct score, and calculates the sum of the errors for each of the genre labels; a second model error calculation unit that inputs the program image and the correct score from the program image data, calculates a score using the NN without genre information, calculates the error between the score and the correct score, and calculates the sum of the errors for each of the genre labels; and a comparison unit that compares the sum of the errors calculated by the first model error calculation unit and the sum of the errors calculated by the second model error calculation unit for each of the genre labels, and determines the selection result indicating that a model with a smaller sum of the errors should be selected.

[0019] Further, the program of claim 4 defines a model in which a genre label of a program image is included in input data as a model with genre information, and a model in which the genre label is not included in input data as a model without genre information, and uses the neural network of the model with genre information (NN with genre information) and the neural network of the model without genre information (NN without genre information) trained by a learning device to extract representative images from program video. The program inputs program image data in which the program image, the genre label, and a correct score are set, calculates scores using the model with genre information and the model without genre information, calculates errors between the scores and the correct score, and selects the model with genre information based on the errors. the program video and the genre code of the program video, as well as the selection result determined by the NN selection unit, and, if the selection result indicates the model with genre information, calculates a first score based on the program image obtained by sampling the program video and the genre label corresponding to the genre code, and, if the selection result indicates the model without genre information, calculates a second score based on the program image obtained by sampling the program video; and the image sorting unit sorts the program images obtained by sampling the program video based on the first score or the second score calculated by the image selection unit, and extracts the representative image. Effect of the Invention

[0020] As described above, according to the present invention, when extracting representative images from program footage, a neural network incorporating genre information (NN with genre information) and a neural network not incorporating genre information (NN without genre information) are used, which makes it possible to take into account the tendency of the representative image and determine whether or not to reflect the characteristics of programs of each genre, and then extract the most appropriate representative image. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a learning device in a model with genre information according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a block diagram showing an example of the configuration of a learning device in a model without genre information according to an embodiment of the present invention. [Diagram 3] 13 is a block diagram showing an example of the configuration of a program learning data generating unit in a model with genre information. FIG. [Figure 4] 13 is a block diagram showing an example of the configuration of a program learning data generating unit in a model without genre information. FIG. [Diagram 5] 13 is a flowchart showing an example of processing by a program learning data generating unit in a model with genre information. [Figure 6] 13 is a flowchart showing an example of processing by a program learning data generating unit in a model without genre information. [Figure 7] FIG. 13 is a block diagram showing an example of the configuration of a learning unit in a model with genre information. [Figure 8] FIG. 13 is a block diagram showing an example of the configuration of a learning unit in a model without genre information. [Figure 9] 13 is a flowchart showing an example of processing by a learning unit in a model with genre information. [Figure 10] 13 is a flowchart illustrating an example of processing by a learning unit in a model without genre information. [Figure 11] 1 is a diagram for explaining an example of a major classification code of a program, and 2 is a diagram for explaining an example of a medium classification code of a program. [Figure 12] FIG. 13 is a diagram illustrating an example of a data configuration of a table in a model with genre information. [Figure 13] FIG. 13 is a diagram illustrating a specific example of the configuration of an image feature extraction NN in a model with genre information. [Figure 14]1A is a diagram for explaining a specific example of the configuration of a genre feature extraction NN, and FIG. 1B is a diagram for explaining a specific example of the configuration of an importance calculation NN. [Figure 15] FIG. 13 is a block diagram showing an example of the configuration of an NN unit in a model without genre information. [Figure 16] FIG. 13 is a diagram illustrating a specific example of the configuration of the NN unit in a model without genre information. [Figure 17] 1 is a block diagram showing an example of the configuration of a representative image extraction device according to an embodiment of the present invention. [Figure 18] FIG. 2 is a block diagram showing a configuration example of an NN selection unit. [Figure 19] 4 is a block diagram showing an example of the configuration of an image selection unit; FIG. [Figure 20] 5 is a flowchart illustrating an example of processing performed by the representative image extraction device according to the embodiment of the present invention. [Figure 21] FIG. 1 is a diagram illustrating a program homepage creation system according to a first embodiment using a representative image extraction device. [Figure 22] FIG. 13 is a diagram illustrating a program DVD sales homepage creation system according to a second embodiment using a representative image extraction device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. [Learning device] First, a learning device according to an embodiment of the present invention will be described. Fig. 1 is a block diagram showing an example of the configuration of a learning device in a model with genre information according to an embodiment of the present invention, and Fig. 2 is a block diagram showing an example of the configuration of a learning device in a model without genre information according to an embodiment of the present invention.

[0023] The model with genre information refers to a model that incorporates genre information, for example, a model in which a genre label is included in the input data (see the learning unit 11A of the learning device 1A shown in FIG. 1 and FIG. 7 described later). The model without genre information refers to a model that does not incorporate genre information, for example, a model in which a genre label is not included in the input data (see the learning unit 11B of the learning device 1B shown in FIG. 2 and FIG. 8 described later).

[0024] The learning device 1A in the model with genre information includes a program learning data generating unit 10A and a learning unit 11A. The learning device 1A in the model with genre information is a device that learns a "neural network incorporating genre information (hereinafter referred to as "NN with genre information")" used by a representative image extracting device 2, which will be described later, using learning program images and genre codes, etc., so that the representative image extracting device 2, which will be described later, can extract representative images that reflect the characteristics of programs for each genre from program videos.

[0025] Learning device 1B in the model without genre information includes program learning data generation unit 10B, memory 12, and learning unit 11B. Learning device 1B in the model without genre information is a device that uses learning program footage to learn a "neural network that does not incorporate genre information (hereinafter, referred to as "NN without genre information")" used by representative image extraction device 2, which will be described later, so that representative image extraction device 2 can extract representative images from program footage that take into account program production know-how.

[0026] The program learning data generation unit 10A in the genre information-included model inputs a learning program image and one or more genre codes (learning genre codes) for the program of the learning program image. The learning program images are frame images sampled from program videos of various genres. A label (representative image suitability) indicating the degree of suitability as a representative image is assigned to the learning program image. For example, when the number of label stages is 3, the representative image suitability is one of great (very high suitability), good (high), and bad (low).

[0027] It is assumed that the program learning data generating unit 10A holds the genre codes of all programs that have been broadcast. There may be cases where one program corresponds to multiple genre codes.

[0028] The program learning data generating unit 10A integrates genres indicated by genre codes according to user operations and generates a table that defines the correspondence between genre codes and genre labels. This table is used by the representative image extracting device 2, which will be described later.

[0029] The program learning data generating unit 10A assigns one or more genre labels to the learning program images, and generates program learning data consisting of the learning program images, the representative image suitability, and one or more genre labels. The program learning data generating unit 10A outputs the program learning data to the learning unit 11A. The program learning data generating unit 10A, the table, the representative image suitability, the genre code, and the genre label will be described in detail later.

[0030] As a result, program learning data is generated that includes learning program images, representative image suitability, and one or more genre labels.

[0031] The learning unit 11A in the model with genre information includes a plurality of NNs (NNs with genre information) to be learned. The learning unit 11A receives program learning data from the program learning data generating unit 10A and learns the plurality of NNs (NNs with genre information) using the program learning data.

[0032] This allows obtaining optimal parameters (weighting coefficients, etc.) to be used for each of the multiple NNs (NNs with genre information). The multiple trained NNs (NNs with genre information) are networks that reflect the characteristics of programs for each genre, and the parameters obtained by training are values ​​that reflect the characteristics of programs for each genre, and are used in the representative image extraction device 2 described later.

[0033] On the other hand, in the model without genre information, the program learning data generator 10B inputs a learning program video and a URL (Uniform Resource Locator) of a program homepage corresponding to the learning program video. Then, the program learning data generator 10B calculates the similarity between each of a plurality of frame images (hereinafter referred to as "program images") obtained by sampling the learning program video and each of a plurality of still images obtained by accessing the URL of the program homepage.

[0034] The program learning data generating unit 10B assigns a correct answer score to the program image based on the similarity, and stores the program learning data consisting of the program image and the correct answer score in the memory 12.

[0035] As a result, the memory 12 stores program learning data consisting of program images and correct answer scores for all program images obtained by sampling the learning program video.

[0036] Here, the still images posted on the program homepage can be said to be representative images selected from the program footage by the program production staff using their know-how. Therefore, the similarity between the program images and the still images is a value that reflects the know-how of the program production staff, and as a result, the accuracy score of the program images is a value that reflects the know-how of the program production staff.

[0037] The learning unit 11B in the model without genre information includes a learning target NN (NN without genre information). The learning unit 11B reads out program learning data consisting of program images and correct answer scores from the memory 12. The learning unit 11B then uses the program learning data to learn the NN (NN without genre information). This NN (NN without genre information) is a model in which a program image is input and a one-dimensional score (importance) is output.

[0038] This allows obtaining optimal parameters (weighting coefficients, etc.) to be used in the NN (NN without genre information). These parameters are values ​​that reflect the know-how of the program production staff, and are used in the NN (NN without genre information) provided in the representative image extraction device 2 described later.

[0039] (Program learning data generation unit) Next, the program learning data generating units 10A and 10B shown in Fig. 1 and Fig. 2 will be described in detail. Fig. 3 is a block diagram showing an example of the configuration of the program learning data generating unit 10A in a model with genre information, and Fig. 4 is a block diagram showing an example of the configuration of the program learning data generating unit 10B in a model without genre information. Fig. 5 is a flowchart showing an example of the processing of the program learning data generating unit 10A in a model with genre information, and Fig. 6 is a flowchart showing an example of the processing of the program learning data generating unit 10B in a model without genre information.

[0040] The program learning data generation unit 10A in the model with genre information includes a genre integration unit 20, a table 21, and a genre label assignment unit 22. The program learning data generation unit 10B in the model without genre information includes a sampling processing unit 23, a download processing unit 24, a similarity calculation unit 25, and a correct score assignment unit 26.

[0041] The genre integration unit 20 inputs a learning program image to which a representative image suitability has been assigned, and one or more genre codes (major category code and medium category code) for the program of the learning program image (step S501).

[0042] Prior to the learning process, the genre integration unit 20 integrates genres indicated by the genre codes (major category codes and medium category codes) of all broadcasted programs in accordance with a user operation (step S502). In this case, the genre integration unit 20 calculates the number of learning program images for each genre code, for example, and displays them on the screen. The user performs an operation to integrate genres while referring to the number of learning program images for each genre code displayed on the screen.

[0043] The genre integration unit 20 generates a table 21 that defines the correspondence between one or more genre codes and one genre label through integration processing according to user operations (step S503). The number of genre labels defined in the table 21 is the number of genres after integration N, which is smaller than (or may be the same as) the number of original genre codes.

[0044] The table 21 generated by the genre integration unit 20 is output to a representative image extraction device 2 shown in FIGS. 17 to 19, which will be described later, and is used as a table 71.

[0045] The genre integration unit 20 outputs to the genre label assignment unit 22 the learning program image to which the representative image suitability has been assigned, and one or more genre codes of the program of the learning program image.

[0046] FIG. 11(1) is a diagram for explaining examples of major classification codes of programs, and FIG. 11(2) is a diagram for explaining examples of medium classification codes of programs.

[0047] As shown in FIG. 11(1), there are 14 major category codes for programs. For example, the major category code "0" indicates "news / reporting" and the major category code "1" indicates "sports."

[0048] Also, as shown in Figure 11 (2), the number of program middle classification codes is determined for each major classification code. For example, when the major classification code is "0" (in the case of "News / Report"), the middle classification code "0" indicates "Regular / General", when the major classification code is "0", the middle classification code "1" indicates "Weather", and when the major classification code is "1" (in the case of "Sports"), the middle classification code "0" indicates "Sports News".

[0049] For details on the major and subgenres of programs, please refer to Part 2 of the standard ARIB STD-B10 "Program Arrangement Information for Use in Digital Broadcasting."

[0050] The genre integration unit 20 calculates the number of learning program images for each major classification code shown in FIG. 11(1), for example, and displays this on a display device not shown in FIG.

[0051] The genre integration unit 20 integrates, for example, "drama" with major category code "3", "movie" with major category code "6", "anime / special effects" with major category code "7", "theater / performance" with major category code "9", "extension" with major category code "14", and "other" with major category code "15" into one genre in order to eliminate genres with an extremely small number of study program images according to key operations (user operations) by the user who has referred to the number of study program images for each major category code. The genre integration unit 20 also integrates "documentary / culture" with major category code "8" and "welfare" with major category code "11" into another genre according to key operations by the user. Then, the genre integration unit 20 generates the table 21.

[0052] 12 is a diagram showing an example of the data configuration of table 21 in a model with genre information, and shows table 21 generated by the above example. Table 21 is made up of major category codes, medium category codes, and genre labels corresponding to the major category codes and medium category codes.

[0053] In the above example, the major category code "0" "News / Reporting", the major category code "1" "Sports", the major category code "2" "Information / Variety Show", the major category code "4" "Music", the major category code "5" "Variety", and the major category code "10" "Hobbies / Education" are not integrated. For this reason, the genre label a is defined for the major category code "0" (and all the intermediate category codes "0-9, 10, 15" of the major category code "0"). Similarly, the genre labels b, c, d, e, and h are defined for the major category codes "1, 2, 4, 5, 10".

[0054] In addition, the major category code "3" "Drama", the major category code "6" "Movies", ..., and the major category code "15" "Other" are integrated, and the genre label f ​​is defined for these. Furthermore, the major category code "8" "Documentary / Educational" and the major category code "11" "Welfare" are integrated, and the genre label g is defined for these.

[0055] In this manner, the genre integration unit 20 generates the table 21 so as to reduce the number of genres indicated by the genre codes, thereby eliminating genres with an extremely small number of study-use program images.

[0056] This eliminates any genre with an extremely small number of learning program images. Therefore, the learning unit 11A described later can learn an NN (NN with genre information) that the representative image extraction device 2 described later uses to extract desired representative images from program videos.

[0057] Returning to FIG. 3 and FIG. 5, the genre label assignment unit 22 receives from the genre integration unit 20 the training program images to which the representative image suitability has been assigned, and one or more genre codes of the programs of the training program images.

[0058] The genre label assigning unit 22 reads out a genre label corresponding to the genre code of the program of the study-purpose program image from the table 21 (step S504), and assigns the read genre label to the study-purpose program image (step S505).

[0059] Here, the genre label assignment unit 22 may assign multiple genre labels to one learning program image when multiple genre codes for one learning program image are input from the genre integration unit 20. In other words, the genre label assignment unit 22 assigns one genre label or multiple genre labels to one learning program image.

[0060] The genre label assignment unit 22 generates program learning data consisting of the learning program image, the representative image suitability, and one or more genre labels, and outputs the program learning data to the learning unit 11A (step S506).

[0061] On the other hand, in the model without genre information, the program learning data generator 10B inputs learning program video stored in a hard disk recorder or the like and the URL of a program homepage corresponding to the learning program video (step S601). The sampling processor 23 inputs the learning program video, and the download processor 24 inputs the URL of the corresponding program homepage.

[0062] The sampling processing unit 23 samples program images, which are frame images, from the learning program video at regular intervals (step S602). All the sampled program images are denoted as P1, . . . , P N Here, N is an integer equal to or greater than 2. The sampling processing unit 23 samples program images P1, . . . , P N to the similarity calculation unit 25.

[0063] The download processing unit 24 accesses the URL of the program homepage and downloads all still images posted on the program homepage (step S603). All downloaded still images are stored as P'1, . . . , P' M Here, M is an integer equal to or greater than 2. The download processing unit 24 downloads still images P'1, . . . , P' M to the similarity calculation unit 25.

[0064] The similarity calculation unit 25 receives the program images P1, . . . , P N , P' are input from the download processing unit 24. M Then, the similarity calculation unit 25 inputs the program image P n About still image P' m The similarity S between n,m (Step S604), where n=1, . . . , N and m=1, . . . , M.

[0065] The similarity calculation unit 25 calculates the similarity of the program image P n and the program image P n Similarity S n,m (S n,1 ,···,S n,M ) to the correct answer score assigning unit 26.

[0066] The correct answer score assigning unit 26 receives the program image P n and the program image P n Similarity S n,m (S n,1 ,···,S n,M ) is input. Then, the correct answer score assigning unit 26 inputs the program image P n For similarity S n,1 ,···,S n,M The maximum value of B=max m S n,m is calculated (step S605).

[0067] The correct answer score assigning unit 26 judges whether the maximum value B is equal to or greater than a preset threshold value (step S606). When the correct answer score assigning unit 26 judges in step S606 that the maximum value B is equal to or greater than the threshold value (step S606: Y), the correct answer score assigning unit 26 assigns the program image P n A positive example correct answer score (=1) is assigned to the positive example (step S607).

[0068] On the other hand, when it is determined in step S606 that the maximum value B is not equal to or greater than the threshold value (step S606: N), the correct answer score assigning unit 26 n A negative example correct answer score (=0) is assigned to the corresponding example (step S608).

[0069] In addition, the correct answer score assigning unit 26 n In the above embodiment, two levels of correct answer scores (positive example (=1) or negative example (=0)) are assigned to the program image P in the range of 0 to 1. However, three or more levels of correct answer scores may be assigned. For example, in the case of three levels of correct answer scores, the correct answer score assigning unit 26 performs threshold processing on the maximum value B to obtain the program image P nEach answer is assigned one of three correct answer scores (e.g., 0.0, 0.5, 1.0).

[0070] In this case, the correct score does not necessarily have to be at equal intervals in the range from 0 to 1, and may be, for example, 0.0, 0.7, 1.0, or any other appropriate interval. Also, the correct score is set to a level in the range from 0 to 1, for example.

[0071] The correct answer score assigning unit 26 may assign the similarity input from the similarity calculation unit 25 as a correct answer score to the program image.

[0072] The correct answer score assigning unit 26 proceeds from step S607 or S608 and stores the program learning data consisting of the program images and the correct answer scores in the memory 12 (step S609). n This is performed for (n=1, . . . , N), and the memory 12 stores N pieces of program learning data.

[0073] This results in the program image P n The similarity S between the still images downloaded from the program website and n,m A correct answer score reflecting the similarity S is assigned, and the program learning data is stored in the memory 12. n,m The higher the score, the closer to 1 the image is to the representative image. n,m The lower the score (the less suitable the image is to be a representative image), the correct score will be 0 or a value closer to 0.

[0074] (Learning Department) Next, the learning units 11A and 11B shown in Fig. 1 and Fig. 2 will be described in detail. Fig. 7 is a block diagram showing a configuration example of the learning unit 11A in a model with genre information, and Fig. 8 is a block diagram showing a configuration example of the learning unit 11B in a model without genre information. Fig. 9 is a flowchart showing a processing example of the learning unit 11A in a model with genre information, and Fig. 10 is a flowchart showing a processing example of the learning unit 11B in a model without genre information.

[0075] The learning unit 11A in the model with genre information includes a genre vector generation unit 30, an NN unit 31A, a correct answer score assignment unit 36, an error calculation unit 37A, and a parameter update unit 38A. The NN unit 31A includes an image feature extraction NN 32, a genre feature extraction NN 33, an addition unit 34, and an importance calculation NN 35. The learning unit 11A performs the processes of steps S901 to S906 for each set of multiple program learning data until a termination condition is satisfied in the process of step S907.

[0076] The learning unit 11A receives program learning data including a learning program image, a representative image suitability, and one or more genre labels from the program learning data generating unit 10A (step S901).

[0077] The genre vector generating unit 30 inputs one or more genre labels included in the program learning data, and generates an N-dimensional genre vector based on the one or more genre labels (step S902).

[0078] As described above, N is the number of genres, specifically, the number of genre labels defined in table 21 generated by genre integration unit 20 shown in Fig. 3 in the process of step S503 in Fig. 5. An N-dimensional genre vector is a vector that is made up of elements equal to the number of genres, N (elements corresponding to genre labels a, . . . , h in the example of table 21 shown in Fig. 12), and includes elements that reflect one or more genre labels included in the program learning data.

[0079] In the example of table 21 shown in Fig. 12, the number of genres N = 8. For example, when the genre label included in the program learning data is a, the genre vector generation unit 30 generates a genre vector (1,0,0,0,0,0,0,0,0). Also, when the genre labels included in the program learning data are e and f, the genre vector generation unit 30 generates a genre vector (0,0,0,0,1,1,0,0).

[0080] The NN unit 31A in the model with genre information inputs the learning program images included in the program learning data, and also inputs the genre vectors from the genre vector generation unit 30. Then, the NN unit 31A calculates a score from the learning program images and the genre vectors using the image feature extraction NN 32, the genre feature extraction NN 33, and the importance calculation NN 35, in which parameters updated by the parameter update unit 38A are set, and the adder 34, and sets this as the importance (step S903). The NN unit 31A outputs the score (importance) to the error calculation unit 37A. The image feature extraction NN 32, the genre feature extraction NN 33, the adder 34, and the importance calculation NN 35 will be described in detail later.

[0081] Prior to processing, the correct score assigning unit 36 ​​presets a score assignment rule according to the level of the label of the representative image suitability in accordance with a user operation. The correct score assigning unit 36 ​​inputs the representative image suitability included in the program learning data, and assigns a correct score within the range of 0.0 to 1.0 according to the level of the label of the representative image suitability using the preset score assignment rule (step S904). The correct score assigning unit 36 ​​outputs the correct score to the error calculation unit 37A.

[0082] For example, if the representative image suitability labels are great / good / bad and the number of stages is 3, the correct score assigning unit 36 ​​assigns 1.0 / 0.5 / 0.0 to the representative image suitability labels great / good / bad, respectively, in accordance with a user operation. In this case, the correct scores do not necessarily have to be at equal intervals in the range of 0.0 to 1.0, and may be, for example, 1.0 / 0.7 / 0.0, as long as they are assigned at appropriate intervals.

[0083] The error calculation unit 37A inputs the score (importance) from the NN unit 31A and also inputs the correct score from the correct score assignment unit 36. Then, the error calculation unit 37A calculates the error between the score (importance) and the correct score (step S905), and outputs the error to the parameter update unit 38A. For example, as a function for calculating the error, a function that outputs a larger value as the error increases, such as MSE (mean square error), is used.

[0084] Parameter update unit 38A holds parameters of image feature extraction NN 32, genre feature extraction NN 33, and importance calculation NN 35. Parameter update unit 38A inputs the error from error calculation unit 37A, and updates the held parameters so that the error (sum of errors) becomes smaller (step S906). Then, parameter update unit 38A sets the updated parameters to the image feature extraction NN 32, genre feature extraction NN 33, and importance calculation NN 35 of NN unit 31A.

[0085] The parameter update unit 38A uses a general NN optimization method such as Adam, SGD (Stochastic Gradient Descent), or backpropagation as a process for updating parameters.

[0086] The parameter update unit 38A proceeds from step S906 and determines whether or not a parameter update termination condition is satisfied (step S907).

[0087] When the parameter update unit 38A determines in step S907 that the termination condition is not satisfied (step S907: N), the parameter update unit 38A returns to step S901 and performs the processes of steps S901 to S906 on the next program learning data. That is, the processes of steps S901 to S906 are performed on each set of program learning data until the termination condition is satisfied.

[0088] On the other hand, if it is determined in step S907 that the termination condition is satisfied (step S907: Y), the parameter update unit 38A outputs the parameters updated in the process of step S906 as optimal parameters (step S908). The optimal parameters output by the parameter update unit 38A are set in the NN with genre information 61A and the NN of the score calculation unit 74A provided in the representative image extraction device 2 shown in Figs. 17 to 19, which will be described later.

[0089] Here, the termination condition in step S907 is, for example, a condition as to whether or not a parameter has been updated a preset number of times, whether or not the amount of parameter update is smaller than a preset threshold, or the like.

[0090] In this way, the score (importance) calculated by the NN unit 31A of the learning unit 11A is a value that reflects the characteristics of the image and the genre, and the higher the representative image suitability, the larger the value, and the lower the representative image suitability, the smaller the value. In other words, the NN unit 31A learned by the learning unit 11A is a network that calculates a score (importance) that reflects the characteristics of the programs for each genre and the tendency of the program images as representative images.

[0091] 17 to 19, a NN with the same configuration as the NN unit 31A in which the parameters of the NN unit 31A are set can be used to obtain a large score (importance) when a program image has a high tendency to be a representative image. On the other hand, a small score (importance) can be obtained when a program image has a low tendency to be a representative image.

[0092] On the other hand, the learning unit 11B in the model without genre information includes an NN unit 31B, an error calculation unit 37B, and a parameter update unit 38B. The learning unit 11B performs the processes of steps S1001 to S1004 for each program learning data until the end condition is satisfied in the process of step S1005.

[0093] The NN unit 31B in the model without genre information reads out program learning data consisting of program images and correct scores from the memory 12 (step S1001). Then, the NN unit 31B calculates a one-dimensional score from the program image using a neural network whose parameters are set by the parameter update unit 38B, and outputs the score to the error calculation unit 37B (step S1002). This neural network is the feature extraction NN 40 and the score calculation NN 41 shown in FIG. 15, which will be described later.

[0094] The error calculation unit 37B inputs the score from the NN unit 31B and the correct score of the program learning data, calculates the error between them, and outputs the error to the parameter update unit 38B (step S1003). For example, as a function for calculating the error, a function that outputs a larger value as the error increases, such as MSE (mean square error), is used.

[0095] Parameter update unit 38B receives the error of the program image from error calculation unit 37B, and updates the parameters held therein so as to reduce the error (sum of errors) (step S1004). Then, parameter update unit 38B sets the updated parameters in NN unit 31B.

[0096] Here, it is assumed that parameter update unit 38B holds the parameters set in NN unit 31B.

[0097] The parameter update unit 38B uses a general neural network optimization method such as Adam, SGD (Stocastic Gradient Descent), or backpropagation as a process for updating parameters.

[0098] The parameter update unit 38B proceeds from step S1004 and determines whether or not a parameter update termination condition is satisfied (step S1005).

[0099] If it is determined in step S1005 that the termination condition is not satisfied (step S1005: N), the parameter update unit 38B returns to step S1001 and performs the processes of steps S1001 to S1004 on the next program learning data. That is, the processes of steps S1001 to S1004 are performed on each program learning data until the termination condition is satisfied.

[0100] On the other hand, if it is determined in step S1005 that the termination condition is satisfied (step S1005: Y), the parameter update unit 38B outputs the parameters updated in the process of step S1004 as optimal parameters (step S1006). The optimal parameters output by the parameter update unit 38B are set in the neural networks of the NN 61B without genre information and the score calculation unit 74B provided in the representative image extraction device 2 shown in Figs. 17 to 19, which will be described later.

[0101] Here, the termination condition in step S1005 is, for example, a condition as to whether or not a parameter has been updated a preset number of times, whether or not the amount of parameter update is smaller than a preset threshold, or the like.

[0102] (NN Department) Next, the image feature extraction NN 32, the genre feature extraction NN 33, the addition unit 34, and the importance calculation NN 35 of the NN unit 31A shown in FIG. 7 will be described in detail.

[0103] FIG. 13 is a diagram illustrating a specific example of the configuration of the image feature extraction NN32 in a model with genre information, FIG. 14(1) is a diagram illustrating a specific example of the configuration of the genre feature extraction NN33, and FIG. 14(2) is a diagram illustrating a specific example of the configuration of the importance calculation NN35.

[0104] In Figure 13, Figure 14 (1) and (2), "Conv" indicates a convolution layer, "MaxPool" indicates a pooling layer that extracts the maximum value, and "LocalResponseNorm" indicates a normalization layer. "Inception Module" is a technology included in "GoogLeNet" and indicates a convolution layer and a pooling layer. "AveragePool" indicates a pooling layer that calculates the average value, "FC" indicates a fully connected layer, "Concat" indicates a concatenation layer, and "Sigmoid" indicates a layer that uses a sigmoid function. "Kernel" indicates the filter size, and "dim" indicates the number of dimensions.

[0105] 13, the image feature extraction NN32 inputs data of learning program images to the convolutional layer α1 of "Conv" and performs NN calculations from the convolutional layer α1 of "Conv" to the concatenation layer α2 of "Concat". Then, the image feature extraction NN32 outputs a 1024-dimensional image feature vector from the concatenation layer α2 of "Concat". The 1024-dimensional image feature vector output by the image feature extraction NN32 is input to the addition unit 34 of the NN unit 31A shown in FIG. 7.

[0106] As shown in FIG. 14(1), the genre feature extraction NN33 inputs an N-dimensional genre vector to the fully connected layer α3 of "FC", and performs NN calculations on the fully connected layer α3 of "FC" and the fully connected layer α4 of "FC". Then, the genre feature extraction NN33 outputs a 1024-dimensional genre feature vector from the fully connected layer α4 of "FC". The 1024-dimensional genre feature vector output by the genre feature extraction NN33 is input to the addition unit 34 of the NN unit 31A shown in FIG. 7. Here, the dimension (size) of the genre feature vector output by the genre feature extraction NN33 is equal to the dimension (size) of the image feature vector output by the image feature extraction NN32.

[0107] 7 inputs the 1024-dimensional image feature vector from the image feature extraction NN 32, and also inputs the 1024-dimensional genre feature vector from the genre feature extraction NN 33. Then, the addition unit 34 adds the 1024-dimensional image feature vector and the 1024-dimensional genre feature vector, and outputs the 1024-dimensional feature vector resulting from the addition to the importance calculation NN 35.

[0108] As shown in Fig. 14(2), the importance calculation NN35 inputs a 1024-dimensional feature vector to the fully connected layer α5 of "FC", performs NN calculations from the fully connected layer α5 of "FC" to the layer α6 of the sigmoid function of "Sigmoid", and calculates a one-dimensional score. Then, the importance calculation NN35 outputs the one-dimensional score from the layer α6 of the sigmoid function of "Sigmoid" to the error calculation unit 37A shown in Fig. 7 as importance.

[0109] Thus, the NN unit 31A is composed of an image feature extraction NN 32 that calculates a 1024-dimensional image feature vector from the training program image, a genre feature extraction NN 33 that calculates a 1024-dimensional genre feature vector from the genre vector, an addition unit 34 that adds the 1024-dimensional image feature vector and the 1024-dimensional genre feature vector, and an importance calculation NN 35 that calculates a one-dimensional score (importance) from the 1024-dimensional feature vector resulting from the addition.

[0110] This NN unit 31A calculates a one-dimensional score regardless of the number of stages of the correct score when the correct score assigning unit 36 ​​generates the correct score from the representative image suitability. In other words, as the NN unit 31A, it is not necessary to prepare a different NN according to the number of stages of the correct score of the representative image suitability, and it is sufficient to prepare an NN with a fixed configuration that is not dependent on the number of stages.

[0111] As described above, according to the learning device 1A in the model with genre information, the program learning data generating unit 10A inputs a learning program image and a genre code, and reads out a genre label corresponding to the genre code from the table 21 generated in accordance with a user operation. Then, the program learning data generating unit 10A assigns a genre label to the learning program image, and generates program learning data consisting of the learning program image, the representative image suitability, and the genre label.

[0112] The learning unit 11A in the model with genre information uses the program learning data to learn the NN unit 31A in the model with genre information. Specifically, the learning unit 11A generates a genre vector based on a genre label included in the program learning data, and calculates a score (importance) from the learning program image and the genre vector included in the program learning data using the NN unit 31A.

[0113] The learning unit 11A assigns a correct answer score based on the representative image suitability included in the program learning data, and updates the parameters of the NN unit 31A so as to reduce the error between the score (importance) and the correct answer score, thereby learning the NN unit 31A.

[0114] Here, the representative image suitability is an index showing whether or not an image is suitable as a representative image of a program, and reflects the know-how of program production, but does not reflect the characteristics of programs of each genre. In the embodiment of the present invention, in addition to the representative image suitability, a genre label is used as program learning data, and thus, learning incorporating the genre is performed, and the NN unit 31A (parameters of the NN unit 31A) that reflects the characteristics of programs of each genre is generated.

[0115] Then, when the representative image extraction device 2 described later uses an NN in which the parameters of the NN unit 31A are set (an NN having the same configuration as the NN unit 31A), it can extract a representative image that reflects the characteristics of programs of each genre from the program video.

[0116] Next, a detailed description will be given of the NN unit 31B in the model without genre information shown in Fig. 8. Fig. 15 is a block diagram showing an example of the configuration of the NN unit 31B in the model without genre information. This NN unit 31B is configured with an NN 40 for feature extraction and an NN 41 for score calculation.

[0117] The feature extraction NN 40 inputs a program image, and determines output data of a 1024-dimensional image feature vector by calculation of a neural network whose parameters are set by parameter update unit 38B, using the program image as input data. Then, the feature extraction NN 40 outputs the 1024-dimensional image feature vector to the score calculation NN 41.

[0118] The score calculation NN 41 inputs the 1024-dimensional image feature vector from the feature extraction NN 40, and uses this as input data to obtain one-dimensional score output data through the calculation of a neural network whose parameters are set by the parameter update unit 38B. The score calculation NN 41 then outputs the score.

[0119] FIG. 16 is a diagram for explaining a specific configuration example of the NN unit 31B in a model without genre information, and shows in detail the configuration of the NN unit 31B shown in FIG. 15. In FIG. 16, "Conv" indicates a convolution layer, "MaxPool" indicates a pooling layer that extracts the maximum value, and "LocalResponseNorm" indicates a normalization layer. "Inception Module" is a technology included in "GoogLeNet" and indicates a convolution layer and a pooling layer. "AveragePool" indicates a pooling layer that calculates the average value, "FC" indicates a fully connected layer, "Concat" indicates a concatenation layer, and "Sigmoid" indicates a layer that uses a sigmoid function. "Kernel" indicates a filter size, and "dim" indicates the number of dimensions.

[0120] The feature extraction NN 40 is composed of layers from the pooling layer β1 of “Conv” to which the program image or artistic evaluation image is input, to the concatenation layer β2 of “Concat” which outputs a 1024-dimensional image feature vector.

[0121] Further, the score calculation NN41 is composed of each layer from the fully connected layer β3 of “FC” where a 1024-dimensional image feature vector is input, to the output layer of the sigmoid function β4 of “Sigmoid” where a one-dimensional score is output.

[0122] Thus, the NN unit 31B is composed of a feature extraction NN 40 that calculates a 1024-dimensional image feature vector of a program image from the image, and a score calculation NN 41 that calculates a one-dimensional score from the 1024-dimensional image feature vector of the image.

[0123] This NN unit 31B calculates a one-dimensional score regardless of the number of stages of the correct score given to the program image. In other words, as the NN unit 31B, it is not necessary to prepare different neural networks according to the number of stages of the program image, and it is sufficient to prepare a neural network of a fixed configuration that does not depend on the number of stages.

[0124] As described above, in the learning device 1B in the model without genre information, the program learning data generating unit 10B calculates the similarity between the program image obtained by sampling the learning program video and the still image obtained by accessing the URL of the program homepage. Then, the program learning data generating unit 10B assigns a correct answer score to the program image based on the similarity, and generates program learning data consisting of the program image and the correct answer score.

[0125] In the model without genre information, the learning unit 11B learns a neural network using program learning data. Specifically, the NN unit 31B uses the neural network to calculate a one-dimensional score from a program image included in the program learning data, and the error calculation unit 37B calculates an error between the score of the program image and the correct score included in the program learning data.

[0126] Parameter update unit 38B updates the parameters of the neural network so as to reduce errors in the program image, and outputs the parameters when a predetermined termination condition is satisfied as optimal parameters.

[0127] Here, since the still images on the program homepage are images generated by making use of the know-how of the program production staff, the accuracy score of the program image calculated from the similarity between the program image and the still image is a score that takes into account the know-how of program production.

[0128] As a result, the neural network trained using the correct score of the program image also takes into account the know-how of program production. Therefore, when the representative image extraction device 2 described later uses a neural network trained by the learning device 1B, it can extract a representative image that takes into account the know-how of program production from the program video. In addition, since the representative image can be extracted without using special data other than the program video, the processing load can be reduced. And, when creating a program homepage using the representative image, the amount of work can be significantly reduced.

[0129] [Representative Image Extraction Device] Next, a representative image extraction device that extracts representative images from program videos using the NN trained by the learning devices 1A and 1B shown in Figures 1 and 2 will be described. Figure 17 is a block diagram showing an example of the configuration of a representative image extraction device 2 according to an embodiment of the present invention.

[0130] This representative image extraction device 2 includes an NN selection unit 50, an image selection unit 51, and an image sorting unit 52. Fig. 18 is a block diagram showing an example of the configuration of the NN selection unit 50, and Fig. 19 is a block diagram showing an example of the configuration of the image selection unit 51. Also, Fig. 20 is a flowchart showing an example of processing by the representative image extraction device 2.

[0131] As shown in FIG. 18, the NN selection unit 50 includes an NN with genre information 61A, a model error calculation unit with genre information 60A having an error calculation unit 62A and an error addition unit 63A, an NN without genre information 61B, a model error calculation unit without genre information 60B having an error calculation unit 62B and an error addition unit 63B, a comparison unit 64 and a memory 65.

[0132] The NN with genre information 61A is an NN trained by the learning device 1A in a model with genre information, and the same parameters as those of the NN unit 31A are set prior to processing by the representative image extraction device 2. The NN with genre information 61A inputs a program image and a genre label from program image data in a format in which the program image, the genre label, and the correct answer score are set, and outputs a score.

[0133] The error calculation unit 62A is a calculation unit that calculates the error for each genre label based on the score output by the genre information-present NN 61A and the correct score of the original program image data. The error addition unit 63A is an addition unit that accumulates the errors calculated by the error calculation unit 62A for each genre label and calculates the sum of the errors.

[0134] The NN without genre information 61B is an NN trained by the learning device 1B in a model without genre information, and the same parameters as those of the NN unit 31B are set prior to processing by the representative image extraction device 2. The NN without genre information 61B inputs a program image from the program image data, and outputs a score.

[0135] The error calculation unit 62B is a calculation unit that calculates the error for each genre label based on the score output by the NN without genre information 61B and the correct score of the original program image data. The error addition unit 63B is an addition unit that accumulates the errors calculated by the error calculation unit 62B for each genre label to calculate the sum of the errors. The sum of the errors calculated by the error addition unit 63B corresponds to the sum of the errors calculated by the error addition unit 63A, and becomes data for each genre label.

[0136] The comparison unit 64 is a comparison unit that compares the sum of errors calculated by the error addition unit 63A of the model error calculation unit 60A with genre information with the sum of errors calculated by the error addition unit 63B of the model error calculation unit 60B without genre information, that is, compares the sum of errors accumulated for each genre in the model with genre information with the sum of errors accumulated for each genre in the model without genre information, and determines which sum of errors is smaller. Then, the comparison unit 64 determines which sum of errors is smaller for each genre label, that is, when the model with genre information is used or when the model without genre information is used, and selects (determines to use) the one with the smaller sum of errors and stores it in the memory 65. That is, the comparison unit 64 stores the genre label and the selection result of the one with the smaller sum of errors (selection result indicating whether to use the model with genre information or the model without genre information) in the memory 65.

[0137] The data input to the NN with genre information 61A and the data input to the NN without genre information 61B are program image data (program image group) in the same format as the program learning data used in the learning process of the model with genre information, in which the program image, genre label, and correct score are set. The sums of errors calculated for each genre label are compared, and it is decided which model to adopt for each genre.

[0138] The memory 65 is a memory for storing the result of selection by the comparison unit 64, i.e., the selection result indicating whether to use the model with genre information or the model without genre information, in association with the genre label of the original program image data. Then, the switching unit 70 of the image selection unit 51 reads out the selection result corresponding to the genre label from the memory 65.

[0139] Prior to processing by the representative image extraction device 2, the NN selection unit 50 determines a selection result indicating whether to use a model with genre information or a model without genre information for each genre label, and stores the result in the memory 65.

[0140] As shown in FIG. 19, the image selection unit 51 includes a switching unit 70, a table 71, a genre label assignment unit 72, sampling processing units 73A and 73B, and score calculation units (NN) 74A and 74B.

[0141] The switching unit 70 inputs a program video and the genre code of the program video, reads out a genre label corresponding to the genre code from the table 71, and reads out a selection result corresponding to the genre label from the memory 65 of the NN selection unit 50 (step S2001).

[0142] Here, when multiple genre codes are input as the genre code of the program video, the switching unit 70 may read out multiple different genre labels from the table 71. When the selection results corresponding to multiple different genre labels indicate a model with genre information and a model without genre information, the switching unit 70 compares the number of models with genre information and the number of models without genre information in the selection results, identifies the selection result with the greater number, and performs the determination process of step S2002 described later. When the number of models with genre information and the number of models without genre information in the selection results are the same, a preset selection result is identified.

[0143] The table 71 stores the same data as that stored in the table 21 generated by the genre integration unit 20 provided in the program learning data generation unit 10A of the learning device 1A in the model with genre information. Prior to processing by the representative image extraction device 2, the table 71 is set with the same data as that in the table 21.

[0144] If the selection result input from the NN selection unit 50 is to use a model with genre information (step S2002: Y), the switching unit 70 outputs the input program video to the sampling processing unit 73A and outputs the input genre code to the genre label assignment unit 72.

[0145] The sampling processing unit 73A receives the program video from the switching unit 70, samples program images, which are frame images, from the program video at regular intervals to generate a group of program images, and outputs each of the group of program images to the genre label assignment unit 72.

[0146] The sampling processing unit 73A may preselect a predetermined number of program images from among all program images obtained by sampling the program video, and output only the selected predetermined number of program images to the genre label assignment unit 72. This can reduce the processing load on the genre label assignment unit 72 and the score calculation unit 74A.

[0147] The genre label assignment unit 72 inputs each of the program images from the sampling processing unit 73A and also inputs the genre code from the switching unit 70, reads out a genre label corresponding to the genre code from the table 71 (step S2003), and assigns the read genre label to each of the program images (step S2004).

[0148] Here, when multiple different genre codes are input, the genre label assignment unit 72 may assign multiple genre labels to one program image. That is, the genre label assignment unit 72 assigns one or multiple genre labels to one program image. The genre label assignment unit 72 outputs the program image and one or multiple genre labels for each of the program image group to the score calculation unit 74A.

[0149] As a result, one or more genre labels are assigned to each of the program images.

[0150] The genre label assignment unit 72 may select a predetermined number of program images from among all program images obtained by sampling the program video, and output the selected program images and one or more genre labels to the score calculation unit 74A. This can reduce the processing load on the score calculation unit 74A.

[0151] The score calculation unit 74A includes an NN with genre information having the same configuration as the NN unit 31A trained by the learning device 1A in the genre-included information model shown in Figures 1 and 7. The NN included in the score calculation unit 74A is set with the same parameters as those in the NN unit 31A prior to processing by the representative image extraction device 2. In other words, the score calculation unit 74A inputs the optimal parameters output by the learning device 1A and sets the NN with genre information.

[0152] The score calculation unit 74A inputs the program image and one or more genre labels for each of the program image groups from the genre label assignment unit 72. Then, the score calculation unit 74A calculates a score from the program image and one or more genre labels using the genre presence information NN.

[0153] Specifically, score calculation section 74A generates an N-dimensional genre vector based on one or more genre labels, similar to the process of step S902 by genre vector generation section 30 shown in FIGS. 7 and 9 (step S2005).

[0154] The score calculation unit 74A calculates a score from the program image and the genre vector using a trained NN with genre information similar to the NN with genre information unit 31A shown in Fig. 7 (step S2006). Then, the score calculation unit 74A outputs the program image and the score for each of the program image group to the image sorting unit 52.

[0155] The image sorting unit 52 inputs the program images and scores for each of the program image groups from the score calculation unit 74A of the image selection unit 51. Then, the image sorting unit 52 sorts the program images in descending order of score, and extracts a predetermined number (an integer equal to or greater than 1) of program images with high scores from among all the program images as representative images (candidates) (step S2008).

[0156] The image sorting unit 52 sorts the predetermined number of representative images in chronological order, and outputs the predetermined number of representative images in chronological order (step S2009).

[0157] On the other hand, if the selection result input from the NN selection unit 50 is not a selection result of using a model with genre information (step S2002:N), that is, if the selection result is a selection result of using a model without genre information, the switching unit 70 outputs the program video to the sampling processing unit 73B.

[0158] Similar to sampling processing unit 73A, sampling processing unit 73B inputs a program video, samples program images, which are frame images, from the program video at regular intervals to generate a group of program images, and outputs each of the group of program images to score calculation unit 74B.

[0159] Note that, like the sampling processing unit 73A, the sampling processing unit 73B may select a predetermined number of program images in advance from all program images obtained by sampling the program video, and output only the selected predetermined number of program images to the score calculation unit 74B. This can reduce the processing load on the score calculation unit 74B.

[0160] The score calculation unit 74B includes an NN without genre information having the same configuration as the NN unit 31B trained by the learning device 1B in the genre-free information model shown in Figures 2 and 8. The NN included in the score calculation unit 74B is set with the same parameters as those in the NN unit 31B prior to processing by the representative image extraction device 2. In other words, the score calculation unit 74B inputs the optimal parameters output by the learning device 1B and sets the NN without genre information.

[0161] The score calculation unit 74B receives each of the program images from the sampling processing unit 73B and calculates a score from the program image using the NN without genre information (step S2007). Then, the score calculation unit 74B outputs the program image and the score of the program image to the image sorting unit 52.

[0162] As a result, for each of a group of program images obtained by sampling the program video, a score for that program image is calculated, and the program image and the score for that program image are output to the image sorting unit 52 .

[0163] The image sorting unit 52 inputs the program images and scores from the score calculation unit 74B for each of the program images obtained by sampling by the sampling processing unit 73B. Then, the image sorting unit 52 sorts the program images in descending order of score, and extracts a predetermined number (an integer equal to or greater than 1) of program images with high scores from among all the program images as representative images (candidates) (step S2008).

[0164] The image sorting unit 52 sorts the predetermined number of representative images in chronological order, and outputs the predetermined number of representative images in chronological order (step S2009).

[0165] The image sorting unit 52 may input all program images constituting a program video and their corresponding scores, classify all program images into, for example, three classes based on the scores by threshold processing, and extract program images in the higher classes as representative images. The image sorting unit 52 does not necessarily need to classify program images by dividing the scores into equal intervals.

[0166] For example, the image sorting unit 52 performs threshold processing using a preset threshold (e.g., 0.25, 0.75) and classifies the program image of the score into a first class level if 0.00≦score≦threshold 0.25. Also, the image sorting unit 52 classifies the program image of the score into a second class level if threshold 0.25<score<threshold 0.75, and classifies the program image of the score into a third class level if threshold 0.75≦score≦1.00. Then, the image sorting unit 52 extracts the program image of the third class level as a representative image.

[0167] Here, the NN with genre information used by the representative image extraction device 2 has the same configuration as the NN section 31A trained by the learning device 1A, and is a network that reflects the characteristics of programs for each genre. Therefore, by using this NN with genre information, it is possible to extract representative images that reflect the characteristics of programs for each genre from program video. And, for example, a homepage creator who creates a program homepage using the representative images can greatly reduce the amount of work and improve the homepage.

[0168] Moreover, the NN without genre information used by the representative image extraction device 2 has the same configuration as the NN section 31B trained by the learning device 1B, and is a model generated taking into consideration the know-how of program production. Therefore, by using this NN without genre information, it is possible to extract representative images from program footage that take into consideration the know-how of program production. Furthermore, since it is possible to extract representative images without using special data other than the program footage, it is possible to reduce the processing load. And when creating a program homepage using representative images, it is possible to significantly reduce the amount of work.

[0169] As described above, according to the representative image extraction device 2, when extracting representative images from program footage, an NN that incorporates genre information (NN with genre information) and an NN that does not incorporate genre information (NN without genre information) are used, and the most appropriate representative image can be extracted after taking into account the tendency of the representative image and determining whether or not to reflect the characteristics of programs for each genre.

[0170] [Example using representative image extraction device 2] Next, an embodiment using the representative image extraction device 2 shown in Fig. 17 to Fig. 19 will be described. Fig. 21 is a diagram illustrating a program HP creation system of a first embodiment using the representative image extraction device 2. This program HP creation system 3 is a system that creates a program HP using program EPG (Electronic Programming Guide) information and program video for a program for which the program HP is to be created.

[0171] The program homepage creation system 3 comprises a representative image extraction device 2, a summary video generation unit 100, and an automatic layout processing unit 101. The summary video generation unit 100 is a component that generates a summary video from program video by conventional processing, and the representative image extraction device 2 is a device according to the embodiment of the present invention shown in Figures 17 to 19, and extracts, for example, three representative images from the program video.

[0172] The automatic arrangement processing unit 101 arranges the program EPG information, summary video, and three representative images at preset positions, and creates a program HP as shown in FIG.

[0173] 22 is a diagram illustrating a program DVD sales website creation system of a second embodiment using a representative image extraction device 2. This program DVD sales website creation system 4 is a system that creates a program DVD sales website using DVD promotional comments, DVD package images, and program DVD videos about the program DVD for which the program DVD sales website is to be created.

[0174] The program DVD sales homepage creation system 4 comprises a representative image extraction device 2 and an automatic layout processing unit 102. The representative image extraction device 2 is a device according to the embodiment of the present invention shown in Figs. 17 to 19, and extracts, for example, six representative images from a program DVD video.

[0175] The automatic arrangement processing unit 102 arranges the DVD promotional comments, the DVD package image, and the six representative images at preset positions, and creates a program DVD sales homepage as shown in FIG.

[0176] In this way, the program homepage creation system 3 and the program DVD sales homepage creation system 4 can create a substantial homepage by using the representative image extracted by the representative image extraction device 2.

[0177] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above-described embodiments and can be modified in various ways without departing from the technical concept thereof.

[0178] For example, in the learning device 1A shown in Fig. 1, the image feature extraction NN 32 shown in Fig. 13 obtains output data for a 1024-dimensional image feature vector, and the genre feature extraction NN 33 shown in Fig. 14(1) obtains output data for a 1024-dimensional genre feature vector. Also, the importance calculation NN 35 shown in Fig. 14(2) inputs the 1024-dimensional feature vector obtained by the addition unit 34. The number of dimensions in the 1024-dimensional image feature vector, genre feature vector, and feature vector is merely an example, and the present invention does not limit the number of dimensions to 1024.

[0179] A normal computer can be used as the hardware configuration of the learning devices 1A and 1B according to the embodiment of the present invention. The learning devices 1A and 1B are configured by a computer equipped with a CPU, a volatile storage medium such as RAM, a non-volatile storage medium such as ROM, and an interface. The same is true for the representative image extraction device 2 according to the embodiment of the present invention.

[0180] The functions of the program learning data generating units 10A, 10B and the learning units 11A, 11B provided in the learning devices 1A, 1B are respectively realized by causing a CPU to execute a program in which these functions are described.

[0181] Furthermore, the functions of the NN selection unit 50, the image selection unit 51, and the image sorting unit 52 provided in the representative image extraction device 2 are also realized by causing the CPU to execute a program describing these functions.

[0182] These programs are stored in the storage medium and are read and executed by the CPU. These programs can also be distributed by storing them in a storage medium such as a magnetic disk (floppy disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), or a semiconductor memory, and can also be transmitted and received via a network. [Explanation of symbols]

[0183] 1A, 1B Learning device 2. Representative image extraction device 3 Program homepage creation system 4 Program DVD sales website creation system 10A, 10B Program learning data generation unit 11A, 11B Learning section (NN) 12,65 Memory 20 Genre Integration Department 21,71 Table 22,72 Genre label assignment section 23, 73A, 73B Sampling processing section 24 Download processing section 25 Similarity calculation unit 26,36 Correct answer score assignment section 30 Genre Vector Generation Unit 31A, 31B NN (Neural Network) Section 32 Image feature extraction NN 33 Genre feature extraction NN 34 Addition section 35 Importance calculation NN 37A,37B Error calculation section 38A, 38B Parameter update section 40 Feature Extraction NN 41 NN for score calculation 50 NN selection department 51 Image Selection Unit (NN) 52 Image sorting section 60A Genre information included model error calculation section 60B Genre information-free model error calculation section 61A Genre information availableNN 61B No genre informationNN 62A,62B Error calculation section 63A, 63B Error addition section 64 Comparison section 70 Switching section 74A, 74B Score calculation unit (NN) 100 Summary video generation unit 101, 102 Automatic placement processing unit

Claims

1. A representative image extraction device extracts representative images from program video using a neural network of the model with genre information (NN with genre information) and a neural network of the model without genre information (NN without genre information) trained by a learning device, the model being a model with genre information, in which the genre label of a program image is included in input data, and a model without genre information, in which the genre label is not included in input data, the representative image extraction device an NN selection unit that inputs program image data including a set of the program image, the genre label, and a correct score, calculates scores using the model with genre information and the model without genre information, calculates errors between the scores and the correct score, and determines a selection result indicating whether to select the model with genre information or the model without genre information based on the errors; an image selection unit that inputs the program video and a genre code of the program video, and inputs the selection result determined by the NN selection unit, and calculates a first score based on the program image obtained by sampling the program video and the genre label corresponding to the genre code when the selection result indicates a model with genre information, and calculates a second score based on the program image obtained by sampling the program video when the selection result indicates a model without genre information; and an image sorting unit that sorts the program images obtained by sampling the program video based on the first score or the second score calculated by the image selection unit, and extracts the representative image.

2. 2. The representative image extraction device according to claim 1, The image selection unit includes: a first sampling processing unit that samples the program video input by the image selection unit to generate a program image group when the selection result input by the image selection unit indicates the genre information-present model; a genre label assigning unit that reads out the genre label corresponding to the genre code input by the image selecting unit from a table that defines a correspondence relationship between the genre code and the genre label, and assigns the genre label to each of the program images generated by the first sampling processing unit; a first score calculation unit that generates a genre vector by vectorizing the genre label assigned by the genre label assignment unit for each of the program image group generated by the first sampling processing unit, and calculates the first score from the program image and the genre vector using the genre information-present NN trained by the training device; a second sampling processing unit that samples the program video input by the image selection unit to generate a program image group when the selection result input by the image selection unit indicates the model without genre information; a second score calculation unit that calculates the second score from the program images for each of the group of program images generated by the second sampling processing unit using the genre-information-free NN trained by the learning device.

3. 3. The representative image extraction device according to claim 1, The NN selection unit is a first model error calculation unit that inputs the program image data including the program image, the genre label, and the correct score as a set, calculates a score using the NN with genre information, calculates an error between the score and the correct score, and obtains a sum of the errors for each of the genre labels; a second model error calculation unit that inputs the program image and the correct score from the program image data, calculates a score using the NN without genre information, calculates an error between the score and the correct score, and obtains a sum of the errors for each genre label; a comparison unit that compares, for each genre label, the sum of errors calculated by the first model error calculation unit with the sum of errors calculated by the second model error calculation unit, and determines the selection result indicating that the model having the smaller sum of errors should be selected.

4. a computer constituting a representative image extraction device that extracts representative images from program video using a neural network of the model with genre information (NN with genre information) and a neural network of the model without genre information (NN without genre information) trained by a learning device, the model being a model in which the genre label of a program image is included in input data and a model being a model without genre information, the model being a model in which the genre label is not included in input data; an NN selection unit that inputs program image data including a set of the program image, the genre label, and a correct score, calculates scores using the model with genre information and the model without genre information, calculates errors between the scores and the correct score, and determines a selection result indicating whether to select the model with genre information or the model without genre information based on the errors; an image selection unit that inputs the program video and a genre code of the program video, and inputs the selection result determined by the NN selection unit, and calculates a first score based on the program image obtained by sampling the program video and the genre label corresponding to the genre code when the selection result indicates a model with genre information, and calculates a second score based on the program image obtained by sampling the program video when the selection result indicates a model without genre information; and A program for functioning as an image sorting unit that sorts the program images obtained by sampling the program video based on the first score or the second score calculated by the image selection unit, and extracts the representative image.

Images