Caption generation program, caption generation method, and caption generation device

Abstract

To provide a caption generation program, caption generation method, and caption generation device, which enable efficient content search even when common content is generated for individual content or for multiple pieces of content.SOLUTION: A caption generation program disclosed herein causes a computer to execute classification processing (S201-S203) for classifying each of multiple pieces of content into multiple groups having an inclusion relationship, and generation processing (S204) for generating captions of the multiple groups.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a caption generation program, a caption generation method, and a caption generation device. [Background technology]

[0002] 2. Description of the Related Art A large number of contents, such as video, still images, audio, or a combination of video, still images, and audio, exist in the memory of storage devices such as server devices on the Internet, smartphones, and personal computers.

[0003] There are also technologies such as Dense Video Captioning that use machine learning to automatically generate captions (explanatory text) for video content (for example, Non-Patent Document 1). In Dense Video Captioning, a video is divided into clips that correspond to one event, and a caption is generated for each clip.

[0004] There is also a technology that uses machine learning to obtain video features from a video and generate captions based on the obtained video features.

[0005] Video features are extracted using, for example, a method based on a convolutional neural network (CNN) (e.g., Non-Patent Document 2) or a method based on a transformer (e.g., Non-Patent Document 3). In addition, a method for generating captions from video features based on a long short-term memory (LSTM) is also being researched (e.g., Non-Patent Document 4).

[0006] These technologies make it possible to generate captions for individual videos, such as "Adults dancing in front of a mirror" for video A and "Elementary school girls dancing in a gym" for video B.

[0007] On the other hand, it is also possible to generate a common caption for multiple videos (for example, Non-Patent Document 5). The technology in Non-Patent Document 5 generates an abstract caption that is common to all of the multiple videos. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] Krishna R., Hata K., Ren F., Fei-Fei L. and Carlos Niebles J.: "Dense-Captioning Events in Videos", Proceedings of the IEEE International Conference on Computer Vision, pp.706-715(2017) [Non-Patent Document 2] Tran D., Bourdev L., Fergus R., Torresani L. and Paluri M.: "Learning Spatiotemporal Features with 3D Convolutional Networks", Proceedings of the IEEE International Conference on Computer Vision, pp.4489-4497(2015) [Non-Patent Document 3] Dosovitskiy A., Beyer L., Kolesnikov A., Weissenborn D., Zhai X., Unterthiner T., Dehghani M., MindererM., Heigold G., Gelly S., Uszkoreit J., Houlsby N.: "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale", arXiv: 2010.11929(2020) [Non-Patent Document 4] Hochreiter, S. and Schmidhuber, J.: "Long Short-Term Memory", Neural Computation, Vol.9, No.8, pp.1735-1780(1997) [Non-Patent Document 5] Rikito Takahashi, Kanichi Kiyomaru, Chu Chenhui, and Yoshio Kurohashi, “A study on a basic model for generating abstract captions for multiple videos,” Proceedings of the 29th Annual Conference of the Association for Natural Language Processing, pp.1963-1867 (March 2023) Summary of the Invention [Problem to be solved by the invention]

[0009] However, while the above-mentioned conventional technology makes it possible to generate a common caption for an individual piece of content or for multiple pieces of content, it is difficult to efficiently search for content by matching the generated caption with a search word.

[0010] For example, when generating captions for individual content, the search words must be checked against the captions of all content, which poses the problem of not being able to perform high-speed searches.

[0011] In addition, when generating a common caption for multiple pieces of content, searches can be performed using search words that are somewhat abstract, but it is further desirable to be able to efficiently perform flexible searches that take into account the content itself.

[0012] An object of the present disclosure is to provide a caption generation program, a caption generation method, and a caption generation device that enable efficient search for content. [Means for solving the problem]

[0013] A caption generation program according to one aspect of the present disclosure causes a computer to execute a classification process for classifying a plurality of contents into a plurality of groups having an inclusion relationship, and a generation process for generating captions for the plurality of groups.

[0014] A method for generating captions for content in one aspect of the present disclosure includes a step of a computer classifying each of a plurality of contents into a plurality of groups having an inclusion relationship, and a step of the computer generating captions for the plurality of groups.

[0015] A caption generation device according to one aspect of the present disclosure includes a classification unit that classifies each of a plurality of contents into a plurality of groups having an inclusion relationship, and a generation unit that generates captions for the plurality of groups. Effect of the Invention

[0016] According to an embodiment of the present disclosure, it becomes possible to efficiently search for content. [Brief description of the drawings]

[0017] [Figure 1] A diagram showing an example of an information processing device [Diagram 2] FIG. 13 is a flowchart showing an example of a caption generation process that is first executed by the control device. [Diagram 3] FIG. 13 is a flowchart showing an example of a caption generation process executed by the control device for the second or subsequent times. [Figure 4] A diagram showing the concept of grouping each piece of content by each feature amount [Diagram 5] A diagram showing the concept of grouping processing using feature quantity X and feature quantity Y simultaneously. [Figure 6] A diagram showing the relationships between hierarchical groups [Figure 7] A diagram showing a concrete example of hierarchical clustering [Figure 8] Diagram showing an example of content being included in multiple lowest level groups [Figure 9] An example of a new subgroup being created. [Figure 10] An example of a window that displays content categorized into groups. [Figure 11] An example of a window displaying content DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] (Embodiment) Hereinafter, embodiments of the present disclosure will be described in detail with appropriate reference to the drawings.

[0019] [Configuration of information processing device 100] FIG. 1 is a diagram illustrating an example of an information processing device 100.

[0020] The information processing device 100 includes a control device 110, a storage device 120, an input device 130, an output device 140, and a communication device 150. Here, the input device 130 and the output device 140 may be provided outside the information processing device 100. The information processing device 100 is, for example, a terminal device such as a personal computer, a smartphone, or a tablet terminal, or a server device. The information processing device 100 may exist on a cloud.

[0021] The control device 110 is composed of circuits such as a central processing unit (CPU), an application specific integrated circuit (ASIC), and a field programmable gate array (FPGA), and includes an extraction unit 110a, a classification unit 110b, a generation unit 110c, and a search unit 110d.

[0022] The extraction unit 110a extracts feature amounts of multiple contents. The contents are moving images, still images, audio, or a combination of moving images, still images, and audio. The feature amount is an amount that represents the features of the content. The feature amount usually includes multiple variables, and the multiple variables are represented by vectors.

[0023] For extracting video features, known feature extraction methods such as those described in Non-Patent Document 2 or Non-Patent Document 3 can be used. For extracting audio features or still image features, known methods can also be used. When the content is a combination of video, still image, and audio, the video features, still image features, and audio features may be combined. Furthermore, in order to perform grouping based on the creation date and time or the shooting location, the creation date and time or the shooting location may be extracted as features.

[0024] The classification unit 110b classifies each of the plurality of contents into a plurality of groups having an inclusive relationship based on the feature amounts of the plurality of contents.

[0025] For example, the classification unit 110b classifies a plurality of contents by hierarchical clustering. Hierarchical clustering is one of unsupervised learning in machine learning, and is a process of classifying a plurality of elements into hierarchical groups (clusters) based on a specific rule. In this process, for example, classification is performed based on the similarity of the feature amount of the contents. In addition, for example, nearest neighbor search is used to calculate the similarity.

[0026] In hierarchical clustering, all contents are included in at least one group. In addition, in hierarchical clustering, the number of layers of hierarchical clustering increases as the number of contents increases. In addition, the number of layers of hierarchical clustering increases when the user specifies a small number of contents to be included in the lowest group.

[0027] Here, the content may be included in multiple groups. Each group may include a number of contents within a predetermined range. For a group that includes contents exceeding the upper limit of the predetermined range, it is necessary to create subgroups that are included in the group. Also, a group that includes contents exceeding the upper limit of the predetermined range may be divided into multiple groups. The predetermined range may be input by the user via the input device 130.

[0028] In addition, a group that includes fewer contents than the lower limit of the predetermined range may be merged with another group. Furthermore, a group that includes fewer contents than the lower limit of the predetermined range may be dissolved, and the contents included in that group may be directly included in a group above that group.

[0029] In hierarchical clustering, a higher-level group includes multiple lower-level groups, but as will be described below, a lower-level group may also be included in multiple higher-level groups.

[0030] Each upper group may include a number of lower groups within a predetermined range. In this case, an upper group that includes lower groups exceeding the upper limit of the predetermined range is divided into multiple groups. An upper group that includes fewer lower groups than the lower limit of the predetermined range is merged with another group. The set range for the number of lower groups included in an upper group may be different from the set range for the number of contents included in each group.

[0031] The generating unit 110c generates captions for the multiple groups into which the contents are classified by the classifying unit 110b.

[0032] For example, the generating unit 110c uses a known technique described in Non-Patent Document 5 to generate, for each group, an abstract caption that is common to all of the multiple contents included in the group.

[0033] In this case, more abstract captions are generated for higher-level groups, and more specific captions are generated for lower-level groups.

[0034] The captions of the higher-level group may be generated based on the content included in the higher-level group, or may be generated based on the captions of the lower-level group included in the higher-level group.

[0035] The search unit 110d searches for content based on information about the content being viewed by the user or a search word input by the user.

[0036] For example, the search unit 110d searches for other contents included in a group to which the content previously viewed or listened to by the user belongs, and displays information recommending the other contents to the user on the output device 140. It is assumed that history information of the contents previously viewed or listened to by the user is stored in the storage device 120.

[0037] Furthermore, the search unit 110d searches for a group that corresponds to a search word input by a user from the input device 130 and a caption generated for the group, among the multiple groups, and displays the search result on the output device 140. An example of the search process by the search unit 110d will be described in detail later.

[0038] The storage device 120 is a device such as an SSD (Solid State Drive) or a hard disk device, and stores various programs, content, feature amounts of the content, captions generated by the generation unit 110c, etc. The storage device 120 may exist on an external device connected to the information processing device 100 via a network.

[0039] The input device 130 is a device such as a touch panel or a keyboard, and receives information input from the user of the information processing device 100 .

[0040] The output device 140 is a device such as a display that displays information, and displays, for example, contents classified into groups, etc. The output device 140 may be a touch panel display integrated with the input device 130.

[0041] The communication device 150 communicates with external devices via a network. The network may be a LAN (Local Area Network) or the Internet.

[0042] [Caption generation process flowchart] 2 is a diagram showing an example of a flowchart of the caption generation process that is first executed by the control device 110. The control device 110 executes the caption generation process by, for example, executing a caption generation program.

[0043] The extraction unit 110a of the control device 110 extracts a feature amount of each content, such as a moving image, stored in the storage device 120 by one method (step S201). Note that the extraction unit 110a may read out a feature amount previously stored in the storage device 120.

[0044] For example, this method is a known method for extracting facial features of people contained in a video, and by performing hierarchical clustering using these features, it is possible to classify people into multiple groups with hierarchical relationships, such as women, men, Japanese men, and Russian women.

[0045] Then, the extraction unit 110a judges whether to extract features by other methods (step S202). The extraction unit 110a may make the above judgment by receiving an input from the user as to whether to extract features by other methods, or may make the above judgment by judging whether features have been extracted by all of the methods that are set in advance.

[0046] For example, the other method is a known method of extracting features about the movements of a person appearing in a video, and by performing hierarchical clustering using these features, it is possible to classify the movements of the person into multiple groups having a hierarchical relationship, such as playing a game, dancing, and dancing.

[0047] By changing the perspective for extracting features in this way, different groups with hierarchical relationships are generated, such as multiple groups with hierarchical relationships for people appearing in the video, or multiple groups with hierarchical relationships for the movements of people shown in the video.

[0048] For example, the "woman dancing" content in FIG. 7 described later is classified into the "woman" group from the perspective of the person, and into the "dancing" group from the perspective of the action.

[0049] Among the groups of women, there is also a group called "women playing games," but the group called "dancing" has a hierarchical relationship only with the group called "women dancing" among the groups of "women," and does not have a hierarchical relationship with the group called "women playing games."

[0050] In this way, by extracting features from different perspectives and combining different groups obtained as a result of performing hierarchical clustering, it is possible to classify data into a plurality of groups that are not necessarily in a hierarchical relationship.

[0051] In step S202, if it is determined that the feature amount is to be extracted by another method (Yes in step S202), the process returns to step S201, and the extracting unit 110a extracts the feature amount of each content by another method.

[0052] In step S202, when it is determined that the feature amount is not to be extracted by another method (step S202, No), the classification unit 110b applies hierarchical clustering to the multiple contents based on the extracted feature amount (step S203).

[0053] The hierarchical clustering here is a process of generating groups having a hierarchical relationship from features when the features are extracted using only one method, but when the features are extracted using multiple methods, as described above, it is a process of generating different groups having a hierarchical relationship from each of the features extracted using the multiple methods and combining those groups.

[0054] Then, the generating unit 110c generates a caption for each group obtained as a result of the hierarchical clustering (step S204), and this caption generating process ends.

[0055] For example, the generating unit 110c uses a known technique disclosed in Patent Document 5 to generate a common caption for the content included in each group, that is, a caption for the group.

[0056] Here, the captions in the higher group are more abstract than the captions of the contents contained in the higher group or the captions of the lower groups, i.e., the captions in the lower groups are more specific than the captions in the higher group.

[0057] 3 is a diagram showing an example of a flowchart of the caption generation process executed from the second time onward by the control device 110. This process is executed, for example, when one or more contents are added or deleted, or at preset regular or irregular timing.

[0058] First, the extraction unit 110a of the control device 110 judges whether or not new content has been stored in the storage device 120 (step S301). New content also includes cases where the stored content has been changed.

[0059] When new content is stored in the storage device 120 (step S301, Yes), the extraction unit 110a extracts features of the new content (step S302). Then, the classification unit 110b executes hierarchical clustering based on the features of the new content and the existing content (step S303). In step S303, the added content may be classified into an existing group, or a new group may be generated.

[0060] In step S301, if new content is not stored in storage device 120 (step S301, No), and when step S303 is executed, the process of step S304 is executed.

[0061] Thereafter, the extraction unit 110a determines whether or not there is a group that includes content that exceeds the upper limit of the predetermined range (step S304). When counting the number of contents included in a group, a subgroup may be counted as one piece of content, or only content that is not included in a subgroup may be counted. The predetermined range is a range of an appropriate number of contents to be included in a group. Since the number of contents to be included in a group is set to a predetermined range, the number of groups is determined according to the number of contents.

[0062] When it is determined that there is a group that includes content exceeding the upper limit of the predetermined range due to the addition of content (Yes at step S304), the extracting unit 110a extracts feature amounts of each of the content included in the group (step S305).

[0063] Then, the classification unit 110b executes hierarchical clustering based on the feature amount extracted in step S305 (step S306), and the generation unit 110c further generates a caption for each group (step S307). By the processes of steps S306 and S307, the contents of a group whose number of contents exceeds the upper limit number are classified into a subgroup of the group, and the number of contents is adjusted so that it is equal to or less than the upper limit number.

[0064] After the processing of step S307, or if it is determined in step S304 that there is no group containing content exceeding the upper limit number (step S304, No), extraction unit 110a determines whether there is a group containing content less than the lower limit number of a predetermined range (step S308).

[0065] If it is determined that the deletion of content results in a group containing less content than the lower limit of the specified range (Yes in step S308), the classification unit 100b dissolves the group and merges the content contained in the group into a higher-level group (step S309), and the caption generation process ends.

[0066] If it is determined in step S308 that there is no content group with fewer than the lower limit number (step S308, No), the caption generation process ends.

[0067] [Example of hierarchical clustering] Fig. 4 is a diagram showing the concept of a process for extracting a feature amount X corresponding to viewpoint A and a feature amount Y corresponding to viewpoint B for each content, and grouping the content by each feature amount. In Fig. 4, each content is indicated by a black circle, and a feature amount X corresponding to viewpoint A and a feature amount Y corresponding to viewpoint B are extracted. Note that features may be extracted from three or more viewpoints. Furthermore, the feature amount X and the feature amount Y are usually a combination of multiple variables.

[0068] Contents that are close to each other in terms of the feature amount X are classified into one group. For example, the contents are grouped into three groups, group 411, group 412, and group 420, using the feature amount X. In addition, for example, group 411 and group 412 are combined into upper group 410.

[0069] Contents that are close to each other in terms of the feature amount Y are also classified into one group. For example, the contents are grouped into five groups, group 450, group 461, group 462, group 463, and group 470, using the feature amount Y. Group 470 does not need to be grouped because it contains a small number of contents. Also, for example, groups 461, 462, and 463 are grouped into upper group 460.

[0070] A piece of content classified into a group using feature quantity X may also belong to a group classified using feature quantity Y. That is, a piece of content may belong to two groups classified based on different features.

[0071] In addition, in the example of Fig. 4, a case has been described in which hierarchical clustering for feature amount X and hierarchical clustering for feature amount Y are performed in sequence, but hierarchical clustering may be performed using feature amount X and feature amount Y simultaneously. Fig. 5 is a diagram showing the concept of a process of grouping using feature amount X and feature amount Y simultaneously.

[0072] Contents that are close to each other in terms of both the feature amount X and the feature amount Y are classified into one group. For example, the contents are grouped into groups 511-513, 521-522, 531-536, and 541-542 using the feature amount X and the feature amount Y.

[0073] In this case, the higher-level group is created, for example, as follows: Figure 6 is a diagram showing the relationship between the hierarchical groups.

[0074] For example, groups 511, 512, and 521 are grouped together because they are close in distance with respect to feature amount Y, and can be set as a top group 610. Groups 521, 541, 542, and content 522 are grouped together because they are close in distance with respect to feature amount X, and can be set as a top group 670.

[0075] By repeating such processing, hierarchical groups are generated. Then, when a plurality of contents are classified into each group, the control device 110 generates a caption for each group.

[0076] A specific example will be described below. Fig. 7 is a diagram showing a specific example of hierarchical clustering. As described above, hierarchical clustering is performed based on the similarity of feature amounts extracted from different perspectives, and nearest neighbor search is used to detect the similarity.

[0077] The group from which the caption "Women playing games" was generated (hereinafter referred to as the "Women playing games" group) contains seven pieces of content. The "Women dancing" group contains six pieces of content. The "Dancing Awa Odori" group contains nine pieces of content. The "Russian men dancing the Cossack" group contains seven pieces of content.

[0078] The groups "Women playing games" and "Women dancing" are included in the superordinate "Women" group. On the other hand, the groups "Dancing the Awa Odori Dance" and "Russian men dancing the Cossack" are included in the superordinate "Traditional dances from around the world" group.

[0079] Furthermore, the group "Women dancing" and the group "Traditional dances from around the world" are included in the superordinate group "Dancing."

[0080] Here, the "women" group and the "dancing" group are higher-level groups of the "women dancing" group, but there is no relationship between the "women" group and the "dancing" group as a higher-level group and a lower-level group.

[0081] However, the group "women dancing", which is a subgroup of the group "women", is contained in another group, "dancing". In this way, groups do not have a simple tree structure, and a group may have multiple superordinate groups.

[0082] This classification is performed by clustering from different perspectives, for example, the perspective of the person included in the content, such as "woman," and the perspective of the action performed in the content, such as "dancing." In the example of Figure 4, perspective A is "person" and perspective B is "action."

[0083] The viewpoints for clustering include, for example, the people included in the content, the actions performed in the content, the location where the content was shot, the date and time of content creation, the length of the content, the amount of data in the content, the type of content (video, audio, etc.), etc. Clustering can also be performed from various other viewpoints.

[0084] The feature to be extracted can be changed by changing the method of extracting the feature, such as a method of extracting feature focusing on people, a method of extracting feature focusing on actions, etc. The creation date and time or the location where the image was taken may also be extracted as a feature.

[0085] By repeatedly extracting features from various viewpoints, i.e., by changing the method of extracting features, it is possible to perform clustering from various viewpoints and create upper and lower groups. As a result, each piece of content will be included in various groups.

[0086] It is also possible to combine multiple perspectives to create new groups.

[0087] For example, a "Koenji Awa Odori" group and a "Tokushima Awa Odori" group can be created by combining "Koenji" or "Tokushima" as the location where the content was shot with "Awa Odori" clustered in terms of the action performed in the content. There may also be a "Koenji" group or a "Tokushima" group clustered in terms of the location where the content was shot. There may also be two higher-level groups of the "Tokushima Awa Odori" group: a "dancing Awa Odori" group and a "Tokushima" group.

[0088] FIG. 7 shows an example in which each piece of content is included in one lowest level group, but as will be described below, each piece of content may be included in multiple lowest level groups.

[0089] FIG. 8 is a diagram showing an example in which content is included in multiple lowest level groups.

[0090] FIG. 8 shows an example in which two pieces of content related to a woman dancing the Awa Odori dance are included in both the "Dancing the Awa Odori" group and the "Women Dancing" group.

[0091] The number of contents contained in the "Women dancing Awa Odori" group may decrease, causing the "Women dancing Awa Odori" group to be dissolved and merged with the "Dancing Awa Odori" group, which is a higher-level group of the "Women dancing Awa Odori" group, and the "Women dancing" group, resulting in the situation shown in Figure 8.

[0092] FIG. 9 is a diagram showing an example in which a new lower group is created.

[0093] For example, a case will be described in which each group can contain up to 15 pieces of content. If each group contains a lowest-level group, one lowest-level group is counted as one piece of content. If new content is added and the "Dancing Awa Odori" group contains, for example, 30 pieces of content, it is necessary to create a new lowest-level group.

[0094] In this case, for example, clustering is performed from the perspective of location to generate a new lowest-level group of "Dancing the Awa Odori in Koenji" and a group of "Dancing the Awa Odori in Tokushima" as shown in Figure 9. Furthermore, the contents included in the group "Dancing the Awa Odori in Tokushima" are clustered from the perspective of people, for example, to generate new lowest-level groups of "Aharen dancing the Awa Odori" and "Ahoren dancing the Awa Odori."

[0095] As a result, the "Dancing Awa Odori" group contains three lowest-ranking groups and nine pieces of content, meeting the condition of a maximum of 15 pieces of content.

[0096] On the other hand, when content is deleted and the number of contents included in a group falls below the lower limit of a predetermined range, the group may be deleted and the contents included in the group may be directly included in a higher-level group, or a new group may be generated that includes the contents included in the group and the contents included in other groups.

[0097] [Example of Classified Content Display] The clustered videos are displayed, for example, for each group on the output device 140 of the information processing device 100. Fig. 10 is a diagram showing an example of a window 1000 displaying the contents classified into groups. The window 1000 is displayed on the output device 140 under the control of the control device 110.

[0098] A window 1000 displays a group 1010, a caption 1020 of the current group, and a higher-level group 1030. The group 1010 is, for example, a group "dancing the Awa Odori dance in Tokushima."

[0099] Group 1010 includes contents 1011 to 1015 and subgroups 1016 and 1017. For example, subgroup 1016 is the "Awa-ren dancing the Awa Odori dance" group, and subgroup 1017 is the "Aho-ren dancing the Awa Odori dance" group. Subgroups 1016 and 1017 are displayed together with their captions.

[0100] In addition, stomach In the window 1000, all of the contents and subgroups contained in the group 1010 may be displayed, or only some of the contents and subgroups may be displayed.

[0101] Furthermore, captions for the contents 1011 to 1015 may be displayed, or contents with captions and contents without captions may be mixed.

[0102] Furthermore, by clicking on the contents 1011 to 1015, the details of the contents 1011 to 1015 may be displayed, and by clicking on the lower groups 1016 and 1017, the windows of the lower groups 1016 and 1017 may be displayed.

[0103] Current group caption 1020 is the caption generated for the group 1010 displayed in window 1000 .

[0104] The top group 1030 is information indicating a top group relative to the group 1010 displayed in the window 1000. The top group 1030 may display only the top group immediately above, or may display an even higher group. Also, multiple top groups may be displayed. They may be displayed in order from the top group, such as "Dancing - Traditional dances from around the world - Dancing the Awa Odori dance," or from the top group to the top group, such as "Tokushima - Shikoku - Japan."

[0105] When multiple upper groups are displayed, clicking on any one of the upper groups may cause a window corresponding to that upper group to be displayed.

[0106] In addition, either one of the lower groups 1016 and 1017 and the upper group 1030 may be displayed, or both may be displayed.

[0107] Moreover, the window 1000 may be displayed by clicking on a group corresponding to the window 1000 on a screen displaying the relationships between groups as shown in FIG.

[0108] FIG. 11 is a diagram showing an example of a window 1100 that displays content.

[0109] For example, when content 1015 in FIG. 10 is clicked, a window 1100 is displayed on output device 140 under the control of control device 110 .

[0110] In the window 1100, for example, a content 1110, a caption 1120, an upper group 1130, a lower group 1140, and a next content 1150 are displayed. Note that in the window 1100, other information may be displayed, or some information may not be displayed.

[0111] The content 1110 is the content selected in the window 1000 of Fig. 10. Note that the content 1110 may be the content selected and played back in the window 1000 of Fig. 10. Furthermore, the content 1110 may be played back by clicking on the content 1110.

[0112] Caption 1120 is the caption of the group corresponding to content 1110. Caption 1120 may also display the lowest level group among the groups to which content 1110 belongs. Furthermore, by clicking on caption 1120, other content or subgroups included in the current group may be displayed.

[0113] Top group 1130 displays information about a top group of the group to which content 1110 belongs. An even higher-ranking top group may also be displayed, or multiple top groups may be displayed. Clicking on top group 1130 may display information about the contents included in top group 1130 or groups below top group 1130 (including the current group).

[0114] Subgroups contained in the group to which content 1110 belongs are displayed in subgroup 1140. Note that multiple groups may be displayed in subgroup 1140. Furthermore, by clicking on subgroup 1140, the contents contained in subgroup 1140 or groups further below subgroup 1140 may be displayed.

[0115] The next content included in the group to which the content 1110 belongs is displayed in the next content 1150. The next content may be determined according to an order preset by the user, or may be determined randomly. The next content 1150 may also display the group to which the next content belongs. Furthermore, the next content 1150 may be displayed by clicking the next content 1150.

[0116] In addition, by clicking on a group on a screen displaying the relationships between groups as shown in FIG. 7, any content contained in that group may be selected by the control device 110, and that content may be displayed in a window 1100 as shown in FIG. 11.

[0117] In this manner, the output device 140 displays the caption 1120 containing the content 1110, as well as the upper group 1130 and the lower group 1140 together with their captions, thereby enabling the user to easily find the desired content from among a large amount of content.

[0118] [Example of content search processing] As described above, the search unit 110d of the control device 110 may refer to history information that stores the user's past content browsing history, and search for and recommend content or groups that the user is interested in. The content that the search unit 110d recommends to the user may be content included in the lowest group of content previously viewed by the user, or content included in a higher group of content previously viewed by the user.

[0119] For example, it is assumed that a "women" group exists as a higher-level group than the "women playing games" group, the "women dancing" group, and the "women playing soccer" group (not shown) in Fig. 7. The search unit 110d acquires from the storage device 120 history information of content previously viewed by the user.

[0120] In this case, the search unit 110d determines whether the content previously viewed by the user is included in the "Women playing games" group, and if the content previously viewed by the user is included in the "Women playing games" group, searches for other content included in the "Women playing games" group and displays information recommended to the user on the output device 140.

[0121] In addition, if the content previously viewed by the user is content included in the group "women playing games" and content included in the group "women dancing", the search unit 110d may display on the output device 140 information recommending to the user other subgroups included in the "women" group or content included in the subgroup (for example, content included in the group "women playing soccer").

[0122] Furthermore, as described above, the search unit 110d may search for a group that corresponds to a search word inputted by a user, among a plurality of classified groups, based on the search word inputted by the user and the generated caption.

[0123] [Using hashtags in captions] The generation unit 110c of the control device 110 generates a caption for the group classified by the classification unit 110b, and may further store the generated caption as a hashtag in the storage device 120 in association with the content included in the group corresponding to the caption.

[0124] For example, on many video sharing sites, when posting a video, the poster adds a hashtag, but it is common for the poster to forget to add a hashtag or to add an unnecessary hashtag.

[0125] Therefore, the generation unit 110c adds the generated caption as a hashtag to an information input field of the video posting site. This allows an appropriate hashtag to be added to the video. Note that the generation unit 110c performs hierarchical clustering from various perspectives and generates captions, so that hashtags can be added from various perspectives.

[0126] The present disclosure makes it possible to perform appropriate clustering of a large amount of content and generate a common caption for the content included in each group. Also, the present disclosure makes it possible to quickly search for a desired video from a large amount of videos.

[0127] In the above embodiment, the extraction unit 100a of the information processing device 100 extracts features of multiple contents, and the classification unit 100b classifies each of the multiple contents into multiple groups having an inclusion relationship based on the features of the multiple contents. However, the classification unit 100b may obtain information already obtained about the features of the multiple contents from another device and store it in the storage device 120, and classify each of the multiple contents into the above-mentioned multiple groups based on the information.

[0128] Although the embodiment of the present invention has been described in detail above with reference to the drawings, the functions of the information processing device 100 described above can be realized by a computer program. [Industrial Applicability]

[0129] The information processing device of the present disclosure is useful for generating captions for content. [Explanation of symbols]

[0130] 100 Information processing device 110 Control device 120 Storage device 130 Input Device 140 Output Device 150 Communication Equipment 1000, 1100 window 1010 Group 1011, 1012, 1013, 1014, 1015, 1110 Contents 1016, 1017, 1140 Lower Group 1020, 1120 Current group captions 1030, 1130 Top group 1150 Next Content

Claims

1. A classification process for classifying each of the plurality of contents into a plurality of groups having an inclusion relationship; a generation process for generating captions for the plurality of groups; A caption generation program that causes a computer to execute the above.

2. further executing an extraction process for extracting feature quantities of the plurality of contents on the computer; the classification process classifying each of the plurality of contents into the plurality of groups based on feature amounts of the plurality of contents; The caption generating program according to claim 1 .

3. The extraction of the feature amount is repeatedly performed by changing a method for extracting the feature amount. The caption generating program according to claim 2 .

4. In the extraction process, different feature amounts corresponding to different viewpoints of the plurality of contents are extracted. The caption generating program according to claim 2 .

5. In the classification process, the plurality of contents are classified into the plurality of groups by hierarchical clustering. The caption generating program according to claim 1 .

6. In the classification process, some of the plurality of contents are classified into a plurality of groups that are not in a hierarchical relationship. The caption generating program according to claim 1 .

7. and causing the computer to execute a search process for searching for a group corresponding to the search word, among the plurality of classified groups, based on a search word input by a user and the generated caption. The caption generating program according to claim 1 .

8. and causing the computer to execute a search process of searching for other content included in a group to which the content previously viewed or listened to by the user belongs, and recommending the other content to the user. The caption generating program according to claim 1 .

9. The generation process associates the generated caption as a hashtag with content included in a group corresponding to the caption. The caption generating program according to claim 1 .

10. A step of classifying each of the plurality of contents into a plurality of groups having an inclusion relationship by a computer; generating captions for the plurality of groups by the computer; A caption generation method including:

11. a classification unit that classifies each of a plurality of contents into a plurality of groups having an inclusion relationship; a generation unit for generating captions for the plurality of groups; A caption generating device comprising:

Images