Content Processing Device

Abstract

To simply provide content paying attention to one person in a group that is a subject of a moving image.SOLUTION: An input reception part receives an input of content in which a moving image is reproduced with a voice. A section detection part detects a section of reproducing a voice similar to a voice corresponding to voice data of a predetermined length given in advance, from the input content. An object detection part detects an object from an image reproduced immediately after the detected section. A labeling part labels the detected object.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a content processing device, and more particularly to a content processing device that enables easy provision of content focusing on one person in a group of subjects of a moving image. [Background technology]

[0002] 2. Description of the Related Art Content such as concerts and plays has been distributed via a network by live distribution or on-demand distribution (see, for example, Non-Patent Document 1).

[0003] In recent years, the content that has been distributed has tended to focus on the appearances of specific idols or actors, rather than on the enjoyment of having fans focus on their favorite performers among the many idols, actors, and other performers.

[0004] Furthermore, with the recent development of VR (Virtual Reality) technology, it is now possible to create virtual spaces that combine avatars with real-life idols or actors, increasing the need for image processing that focuses on specific performers displayed within the content.

[0005] Furthermore, a technique has been proposed in which a plurality of people are extracted from an image and each of the extracted people is tracked (for example, see Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] https: / / livr.jp / app-banner [Non-Patent Document 2] https: / / www.programmersought.com / article / 17005126187 / Summary of the Invention [Problem to be solved by the invention]

[0007] However, many idol groups are made up of a large number of members, and for example, several dozen members may belong to the same group. In such cases, creating content that focuses on only one idol among several dozen idols dancing vigorously on stage can be time-consuming and costly.

[0008] It is also difficult to automatically detect a particular idol from among the content being played. For example, even if you try to identify each member of an idol group using facial image recognition, in the case of a large idol group, the number of pixels in the facial area for each person is significantly reduced, making it difficult to obtain a clear image. Furthermore, when factors such as changes in image quality due to stage lighting and changes in costume design are also taken into account, it is difficult to qualify the characteristics of each person.

[0009] In addition, the members of an idol group are usually of the same age, sex, and country, and there is little difference in characteristics such as skin tone and body type, so it is difficult to automatically identify each individual.

[0010] An object of one aspect of the present invention is to realize a technology that enables content that focuses on one person in a group of subjects of a video to be easily provided. [Means for solving the problem]

[0011] A content processing device according to one embodiment of the present invention includes a content input receiving unit that receives input of content in which moving images are played along with audio, a section detection unit that detects a section from the input content in which audio similar to audio corresponding to a predetermined length of audio data is played, an object detection unit that detects an object from an image played immediately after the detected section, and a label assignment unit that assigns a label to the detected object.

[0012] In one aspect of the present invention, the label assignment unit may estimate a label related to a name of the object by a calculation using a model parameter obtained by machine learning. Each aspect of the present invention may be realized by a computer. In this case, the present invention also includes a program for realizing the system on a computer by causing the computer to operate as each part (software element) of the above-mentioned system, and a computer-readable recording medium on which the program is recorded. Effect of the Invention

[0013] According to one aspect of the present invention, it is possible to realize a technique that enables content that focuses on one person in a group of subjects of a moving image to be easily provided. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing an example of a functional configuration of a content processing device according to a first embodiment. [Diagram 2] FIG. 13 is a diagram showing an example of an image when a song A of an idol group is being performed. [Diagram 3] FIG. 1 shows an image of a scene from an idol group's performance. [Figure 4] FIG. 11 is a diagram showing an example of a GUI that is displayed on a display of a content processing device when an object is designated. [Diagram 5] 13 is a diagram showing an example of an image that has been subjected to image processing by an object image processing unit; FIG. [Figure 6] 13 is a diagram showing another example of an image that has been subjected to image processing by the object image processing unit. FIG. [Figure 7] 11 is a flowchart illustrating an example of the flow of a content playback process. [Figure 8] FIG. 11 is a block diagram showing an example of a functional configuration of a content processing device according to a second embodiment. [Figure 9] FIG. 1 is a diagram illustrating an example of a virtual space. [Figure 10] FIG. 1 is a diagram showing an example of the configuration of a computer that executes instructions of a program, which is software that realizes each function of a content processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. First Embodiment (Content processing device) 1 is a block diagram showing an example of a functional configuration of a content processing device 10 according to a first embodiment. As shown in the figure, the content processing device 10 has a content input receiving unit 11, a content playback unit 12, and an operation input receiving unit 13.

[0016] The content input receiving unit 11 receives input of content data. The content data is supplied via a network such as the Internet. In addition, the content processing device may access a wide area wireless communication network such as a 5G communication system, and the content data may be supplied via the wide area wireless communication network.

[0017] As an example, the content may be MP4 format data obtained by filming a concert, a play, or the like. The content data includes moving images and audio, and may also include information such as subtitles as necessary. That is, the content input to the content processing device is a content in which moving images are reproduced together with audio. For example, the audio may be a piece of music played at a concert.

[0018] Here, the content will be mainly described as MP4 format data filmed at an idol group concert. At an idol group concert, a group consisting of multiple idols (people) sings and dances to the music being played. The singing and dancing to the music is also called a performance.

[0019] The content playback unit 12 plays back the content whose input has been accepted by the content input accepting unit 11. The played back content is displayed, for example, on a display 50. The display 50 is also equipped with a speaker, and the sound of the played back content is output from the speaker of the display 50.

[0020] The operation input receiving unit 13 receives an operation input from a user to the content processing device 10. The operation input receiving unit 13 may be configured with a keyboard, a mouse, etc. Alternatively, the display 50 may be configured to include a touch sensor, and the operation input receiving unit 13 may be configured by detecting an operation input of a GUI (Graphical User Interface) displayed on the display 50.

[0021] As shown in FIG. 1, the content processing apparatus 10 further includes a section detection unit 31, an object detection unit 32, a label assignment unit 33, and an object image processing unit .

[0022] (Section detection section) The section detection unit 31 detects, from the input content, a section in which a sound similar to a sound corresponding to audio data of a given length is reproduced.

[0023] Here, the pre-given audio data corresponds to audio of a sufficiently short duration compared to the overall time length of the input content. For example, if the input content is a concert of an idol group, audio data of audio corresponding to a part of a song performed at the concert, such as an intro, interlude, or chorus, is pre-given. However, the audio of the audio data is not limited to these, and may be any number of consecutive bars of sound in a song. The audio data may be stored, for example, in a storage unit of the content processing device 10 (not shown).

[0024] The audio may include instrumental sounds, singing voices, sound effects, and the like.

[0025] The section detection unit 31, for example, calculates the similarity between two audio signals by comparing the feature amount of the audio signal of pre-given audio data with the audio playback signal of the content being played. The section detection unit 31 continuously extracts audio signals of the same time length as the audio data while the content is being played, and detects a section where the similarity is equal to or greater than a threshold by comparing the feature amount of the extracted audio signal with the feature amount of the audio signal of the audio data. The section detected here has the same time length as the audio data.

[0026] (Object detection section) The object detection unit 32 detects an object from an image played back immediately after the detected section.

[0027] The image played immediately after the section detected by section detection unit 31 is an image played temporally after the section. For example, it may be one or more images from among images 1 to 30 frames after the final frame of the multiple frames included in the video of the section. For example, if the input content is the content of an idol group concert, several idols (people) appear in the image played immediately after the detected section, and the images of these people are detected as objects by object detection unit 32.

[0028] As an example, object detection can be performed by the graph cut method. In the graph cut method, the boundary of the region that constitutes the foreground object image to be cut out is calculated from the color distribution and pixel color gradient of two types of images, a foreground object image including the object to be cut out and a background image. Then, the image is cut out along the calculated boundary, and the foreground object image to be cut out is extracted.

[0029] (Label assignment section) The label assignment unit 33 assigns a label to the object detected by the object detection unit 32 .

[0030] Here, the process of the label assignment unit 33 will be described using an example of content of an idol group concert. In this case, the detected object is a person, and the label assignment unit 33 assigns a label related to the person's name. In other words, the person's name is assigned to the object as a label.

[0031] The label assignment unit 33 is configured by, for example, a convolutional neural network (CNN) and executes a process of estimating the label of an object in an input image. At this time, the label assignment unit 33 estimates the label of the object using model parameters obtained by machine learning executed in advance.

[0032] In this machine learning, for example, data in which correct labels are added to objects detected from images of a previously recorded concert of an idol group is used as training data. In general, idol group concerts are often held frequently at the same venue. In this case, there may be a large amount of video data of the same idol group taken from the same viewpoint.

[0033] Also, when idol groups hold concerts frequently, they usually play the same songs at each concert. Idol groups often dance in a fixed formation and according to a fixed choreography for each song. Therefore, for example, when song A is played, immediately after the interlude, a fixed number of people will stand in the front row and take the same pose.

[0034] 2 is a diagram showing an example of an image during the performance of a song A by an idol group. The diagram shows an image 101 displayed on a display 50, and the image 101 is an image taken immediately after the interlude of the song A. The diagram shows three people, a person 111, a person 112, and a person 113, each of whom is posing with their right hand raised.

[0035] While a song is being played, the members of an idol group often move vigorously, but there are also scenes in the middle of the song where the members of the idol group are almost still. For example, the image in Figure 2 corresponds to such a scene. In a performance of an idol group that dances according to a set choreography, the relative positions of each member and the poses they will take in such a scene are known in advance, making it relatively easy to estimate the labels of objects in the image.

[0036] In Fig. 2, person 111, person 112, and person 113 are each detected as an object in the image. Data to which the names (first name, nickname, etc.) of these people 111, person 112, and person 113 are assigned as labels is used as training data in machine learning of model parameters used by label assignment unit 33. If there is a large amount of data of moving images of the same idol group shot from the same viewpoint, a large amount of such training data can be generated. Machine learning using a large amount of training data also improves the accuracy of the estimation results of label assignment unit 33.

[0037] 2, the three people are all in the same pose, but if there is a scene in which each person is stationary in a different pose, it will be easier to identify each person. Machine learning using images of such scenes can further improve the accuracy of the estimation results of the labeling unit 33.

[0038] (Object Tracking) Furthermore, the object detection unit 32 tracks the object to which the label has been assigned within the video of the content that is played back temporally after the section detected by the section detection unit.

[0039] In object tracking, for example, the similarity between an object detected in an image of a certain frame and an object detected in an image one frame later is calculated. For example, the similarity of the objects is calculated by comparing the features related to the color, shape, and pattern of the objects. Then, the objects are tracked by identifying the objects having a similarity equal to or greater than a threshold as the same object.

[0040] By tracking objects in this manner, once a label has been assigned to a person, the same label will continue to be assigned to that person in the video images displayed during playback of the content.

[0041] For example, the object may be tracked more accurately by detecting the object multiple times while playing the same piece of music. For example, first audio data corresponding to the sound played in the intro and second audio data corresponding to the sound played in the interlude may be provided in advance in the same piece of music.

[0042] In this case, first audio data corresponding to a first audio, which is an audio of a section that is played earlier in time in the content, and second audio data corresponding to a second audio, which is an audio of a section that is played later in time in the content, are provided in advance. Then, the section detection unit 31 detects the first section in which an audio similar to the first audio is played, and the second section in which an audio similar to the second audio is played, respectively.

[0043] In this case, the labeling unit 33 estimates the name of the object detected by the object detection unit 32 from the image immediately after the first section and labels it, and re-labels the object detected by the object detection unit from the image immediately after the second section. Then, the object detection unit 32 tracks the object labeled in accordance with the first section and the object labeled in accordance with the second section.

[0044] For example, when an idol group has a large number of members, it is difficult to obtain an image that shows all of the members in one scene. Fig. 3 is a diagram showing an image 131 displayed on the display 50, which is an image of one scene in a performance of an idol group consisting of eight members. Fig. 3 corresponds to, for example, a scene immediately after the intro of a certain song.

[0045] In the scene shown in Fig. 3, all eight members are almost still and in the same pose, but person 142 is located behind person 141, and person 144 is located behind person 143. It is difficult to detect and track person 142 and person 144 as objects from the image of the scene shown in Fig. 3. In addition, when the labeling unit 33 estimates the names of the detected people in the scene shown in Fig. 3, it is difficult to correctly estimate the names of person 142 and person 144.

[0046] On the other hand, in idol group performances, there are often multiple scenes where all the members are almost still during the performance of one song. Since the formation changes depending on each scene, it is possible to select scenes in which a person who is not in one scene is in another scene.

[0047] For example, if audio data corresponding to the audio played when an idol group changes formation in a song is provided in advance, the names of multiple people positioned in the front row of the stage can be estimated each time the formation changes. Also, each of the multiple people positioned in the front row of the stage can be tracked each time the formation changes.

[0048] That is, a plurality of pieces of audio data (first audio data, second audio data, ...) may be provided in advance, and a label may be added each time a first scene, a second scene, ... is played immediately after a section corresponding to each piece of audio data. For example, if a person is detected and a label is added in each of a plurality of scenes, it becomes possible to track each person more accurately during playback of the content.

[0049] Furthermore, for example, if the same audio is played repeatedly within a piece of music, even if only one piece of audio data is provided, it is still possible to assign a label each time in the first scene, the second scene, etc.

[0050] (Object image processing unit) The object image processing unit 34 performs a predetermined image processing on an object designated by a user among the multiple objects detected by the object detection unit 34. The object is designated based on, for example, a label attached to the object.

[0051] 4 is a diagram showing an example of a GUI displayed on the display 50 of the content processing device 10 when an object is specified. In this example, "Idol group XXX 8th generation member list" is displayed. Here, "XXX" indicates the name of the idol group that will perform at the concert of the content being played back. This idol group may be made up of, for example, 20 members, and the member list is updated every time at least one of the members is replaced. The member list at the time of the group's formation is the first generation, and thereafter, the member list is updated to the second generation, third generation, ... every time at least one of the members is replaced.

[0052] In the member table shown in Fig. 4, the leftmost column shows "Members", and the names of the 20 members who make up the idol group XXX are written. Here, each person's name is shown as "AAA", "BBB", "CCC", .... Note that in reality, each person's name is associated with an identification number, and each identification number corresponds to a label assigned by the label assignment unit 33.

[0053] In the member table shown in FIG. 4, the central column shows "Profile", where the profile of each member is described.

[0054] In the member table shown in Fig. 4, the rightmost column shows "Attention", and this column is where the user makes their designation. For example, the user inputs an operation for designating a person via the operation input receiving unit 13, and designates a person on whom the user pays attention in the member table shown in Fig. 4. In this example, the user pays attention to the member "CCC", and a star mark indicating that this person ("CCC") has been designated is displayed in the "Attention" column.

[0055] The object image processing unit 34 performs a predetermined image processing on an image of a person that the user focuses on, among images of the content being played back. Fig. 5 shows an image 161 that is displayed on the display 50 and is an example of an image that has been subjected to image processing by the object image processing unit 34.

[0056] Here, for example, it is assumed that the user has specified person 113 as a person of interest using the GUI described above with reference to Fig. 4. In the example of Fig. 5, a mark (heart-shaped mark in this example) 171 is displayed superimposed near person 113. As an example of image processing by object image processing unit 34, as shown in Fig. 5, a predetermined image (mark 171 in this example) is displayed superimposed within a predetermined range near the specified object.

[0057] Fig. 6 is a diagram showing another example of an image that has been subjected to image processing by the object image processing unit 34. In the example of Fig. 6, an image 191 in which a person 113 is enlarged and displayed is displayed on the display 50. That is, only the person 113 designated by the user is enlarged and displayed on the display 50. As an example of image processing by the object image processing unit 34, a designated object is enlarged and displayed as shown in Fig. 6.

[0058] Furthermore, in an enlarged image as shown in FIG. 6, a mark as shown in FIG. 5 may be displayed in a superimposed manner.

[0059] The image processing by the object image processing unit 34 is continuously performed on the images of each frame constituting the moving image of the content being played. For example, when a mark 171 is superimposed as shown in Fig. 5, the mark 171 is always displayed near the person 113 dancing to the music while the content is being played. Also, when an object is displayed in an enlarged manner as shown in Fig. 6, an image showing almost only the person 113 is always displayed on the display while the content is being played.

[0060] Next, an example of the flow of content playback processing by the content processing device 10 will be described with reference to the flowchart in Fig. 7. This processing is executed after the content input receiving unit 11 receives input of content data. Here, an example will be described in which content data of a live-streamed concert by an idol group is input.

[0061] In step S101, the content playback unit 12 plays back the input content.

[0062] In step S102, the section detection unit 31 calculates the similarity between the sound corresponding to the pre-given sound data and the sound of the content being played. As described above, sound data of a part of a piece of music played at a concert, for example, sound corresponding to an intro, interlude, chorus, etc., is pre-given. The section detection unit 31 calculates the similarity between the two sound signals, for example, by comparing the feature amount of the sound signal of the pre-given sound data with the playback signal of the sound of the content being played.

[0063] In step S103, the section detection unit 31 judges whether a similar section has been detected. As described above, the section detection unit 31 continuously extracts audio signals having the same time length as the audio data during playback of the content, and compares the extracted audio signal features with the audio signal features of the audio data. Then, the section detection unit 31 detects, for example, a section whose similarity is equal to or greater than a threshold value as a similar section.

[0064] Also, when multiple pieces of audio data are provided in advance, the similarity of the audio signals is calculated by comparing each of the feature quantities of the audio signals of the multiple pieces of audio data with the audio signal of the audio of the content being played. In this way, for example, a section similar to the audio of the first audio data, a section similar to the audio of the second audio data, etc. are detected as similar sections.

[0065] If it is determined in step S103 that a similar section has been detected, the process proceeds to step S104.

[0066] In step S104, the object detection unit 32 detects an object from the image played immediately after the detected section. At this time, for example, an image of one or more people appearing in the image played immediately after the detected section is detected as an object by the object detection unit 32.

[0067] In step S105, the labeling unit 33 estimates the name of the object. At this time, the name of the person detected as the object in the process of step S104 is estimated. As described above, the labeling unit 33 is configured by, for example, a CNN, and executes a process of estimating the label of the object of the input image. At this time, the labeling unit 33 estimates the name of the object using model parameters obtained by machine learning executed in advance.

[0068] In step S106, the label assignment unit 33 assigns the name estimated in the process of step S105 as a label to the object detected in the process of step S104. That is, the name of the person is assigned to the object as a label.

[0069] In step S107, the object detection unit 32 tracks the object to which the label was assigned in the process of step S106. At this time, for example, the similarity between an object detected from an image of a certain frame and an object detected from an image one frame later is calculated, and objects having a similarity equal to or greater than a threshold value are identified as the same object, thereby tracking the object.

[0070] In step S108, the object image processing unit 34 performs a predetermined image processing on the object designated by the user among the multiple objects detected by the object detection unit 32.

[0071] As a result, for example, a mark is displayed near the specified object as described above with reference to Fig. 5. Alternatively, for example, only the person specified by the user is enlarged and displayed as described above with reference to Fig. 6. Furthermore, a mark as shown in Fig. 5 may be superimposed on the image enlarged as in Fig. 6.

[0072] The image processing by the object image processing unit 34 is successively performed on the images of each frame constituting the moving image of the content being played back.

[0073] If no object is specified, the process of step S108 is skipped.

[0074] If it is determined in step S103 that no similar section has been detected, steps S104 to S106 are skipped and step S107 is executed. If no label has been added yet, steps S107 and S108 cannot be executed, and so these steps are also skipped.

[0075] In step S109, the content playback unit 12 judges whether the content has been played to the end. If it is judged in step S109 that the content has not been played to the end, the process returns to step S102, and the section detection unit 31 compares the extracted feature quantity of the audio signal with the feature quantity of the audio signal of the audio data. Then, the section detection unit 31 detects a section whose similarity is equal to or greater than a threshold value as a similar section.

[0076] In this way, by repeatedly executing the processes of steps S102 to S109, a section corresponding to pre-given audio data is detected once or multiple times (step S103), and an object is detected from the image immediately following the detected section (step S104). Then, the name of the detected object is estimated (step S105), and a label is assigned (step S106). Once a label has been assigned to the object, it is tracked within the image of the content being played back (step S107), and image processing is performed on the designated object (step S108).

[0077] The processes of steps S104 to S106 are executed each time a section similar to the voice of pre-given voice data is detected, so that, for example, a person who could not be detected in a first scene can be detected in a second scene. Even if the estimation of a person's name is incorrect in the first scene, it is also possible to correct the label to the correct one in the second scene. Furthermore, even if a different object (person) is mistakenly tracked after the first scene, it is also possible to track the correct object (person) from the second scene onwards.

[0078] If it is determined in step S109 that the content has been reproduced to the end, the content reproduction process ends.

[0079] In this manner, the content playback process is carried out.

[0080] In the above explanation, the content is mainly MP4 format data of an idol group concert, but the same process may be performed on content of a theatrical performance, for example. A play usually features multiple actors, and if audio data corresponding to specific lines or sound effects is provided in advance, it also becomes easier to estimate the names of people, who are objects detected in the images of the content.

[0081] In other words, in a scene during a play where a particular line or sound effect is spoken, the relative positions of each person and the poses they will be in are known in advance, making it relatively easy to estimate the labels of objects in the images.

[0082] (Effects of the First Embodiment) As described above, according to this embodiment, it is possible to assign a label to an object detected in an image of a content to be played. At this time, the section detection unit 31 detects a section of an audio similar to the audio of pre-given audio data, so it becomes easier to predict in advance which object will be detected at which position in the image of the content to be played. It also becomes easier to predict in advance what pose a person, which is an object to be detected, will be taking.

[0083] Therefore, since it is possible to estimate the label of an object based on the characteristics such as the position and pose of a person, which is an object, it is possible to estimate the label with high accuracy even if the image quality of the content is low.Also, even if the lighting or costumes at a concert are changed, it is possible to estimate the label with high accuracy.

[0084] Furthermore, since the label estimation is performed for an object that is in the same positional relationship and in the same pose as an object in the training data, the possibility of estimating the label increases even when learning with a small amount of training data. Therefore, it is possible to estimate the label with high accuracy regardless of the degree of learning of the model parameters used by the label assignment unit 33.

[0085] In this way, by tracking objects with estimated labels, it becomes possible to perform image processing focusing on a specified person. In this way, for example, it becomes possible for fans to enjoy focusing on their favorite performers among many performers such as idols and actors.

[0086] Therefore, according to this embodiment, it is possible to easily provide content that focuses on one person in a group of people who are subjects of a moving image.

[0087] Second Embodiment 8 is a block diagram showing an example of a functional configuration of a content processing device 10 according to the second embodiment. Note that components having the same functions as those described in the first embodiment are denoted by the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0088] 8 includes a motion analysis unit 61 and an animation image generation unit 71. The other configuration is the same as that of the content processing device 10 shown in FIG.

[0089] (Motion Analysis) The motion analysis unit 61 analyzes the image of the content, thereby analyzing the motion of an object (e.g., a person) detected in the image. The motion analysis unit 61 also identifies, for example, the motion of a specified object. For example, it identifies the motion of the person's arms, legs, head, etc. in a section having a predetermined time length in the content. Note that the specified object may be, for example, an object specified as a person of interest by the user's operation described above with reference to FIG. 4.

[0090] As an example, the motion analysis unit 61 identifies the joint positions of a person, which is an object detected in an image, and analyzes how each part of the person, such as the arms, legs, head, etc., is moving. Note that in the processing by the motion analysis unit 61, the joint positions of the person and / or the movements of each part may be estimated using model parameters obtained by prior machine learning, for example.

[0091] The analysis result of the movement analysis unit 61 is supplied to the animation image generation unit 71 as information representing the movement of the person.

[0092] (Animation image generation section) The animation image generator 71 generates an animation image based on the analysis result of the motion analyzer 61. Here, the generated animation image is, for example, an image of a moving character. As an example, the character has arms, legs, and a head like a human body, and moves its joints like a human.

[0093] The animation image generator 71 generates an animation image for moving a character based on the information representing the movement of the person supplied from the motion analyzer 61. That is, an animation image for a character moving in the same manner as the movement of the person identified by the motion analyzer 61 is generated.

[0094] In the example of FIG. 8, the animation image generating unit 71 is provided inside the content processing device 10, but the animation image generating unit 71 may be provided in a device different from the content processing device 10, for example.

[0095] That is, the motion analysis unit 61 analyzes the movement of a person specified by the user from among the multiple people detected by the object detection unit 32, and the analysis result of the motion analysis unit 61 is supplied to the animation image generation unit 71 which generates an animation image that moves in the same manner as the specified person.

[0096] (Character display) The animation of the moving character may be displayed, for example, on the display 50. In this way, for example, the user can imitate the movements of a person of interest in an idol group. For example, the user can dance along with the music of the idol he or she likes.

[0097] Furthermore, the animation of a moving character may be used, for example, as an avatar displayed in a virtual space. That is, the animation image generated by the animation image generating unit 71 may be an image of a user's avatar. FIG. 9 is a diagram showing an example of a virtual space. The virtual space 200 shown in FIG. 9 is modeled after an event venue. Also shown in the figure are characters 211 to 216 used as avatars.

[0098] Each of the avatars is a character that represents a user in the virtual space 200, and is set to move within the virtual space 200 based on, for example, the user's operation. That is, each of the characters 211 to 216 is associated with a different user, and moves or moves its body within the virtual space 200 according to the user's operation.

[0099] For example, avatars of users who are fans of an idol group can gather on stage 200a in virtual space 200 and dance to the music. For example, each user analyzes the movements of a person in the idol group that the user pays attention to with movement analysis unit 61, and generates an animation image of the user's avatar character moving with animation image generation unit 71. By playing this animation image on stage 200a in virtual space 200, the user can enjoy the image of the avatar dancing to the same choreography as the idol group.

[0100] Furthermore, an image of an idol group's performance may be superimposed on the image of virtual space 200 and displayed.

[0101] (Effects of the Second Embodiment) As described above, according to this embodiment, it is possible to analyze the movement of a specified object among objects detected from an image of a content, and generate an animation image that moves in the same way as the specified object. In this way, for example, the variety of ways in which fans can enjoy the content increases, focusing on their favorite performers among a large number of performers such as idols and actors.

[0102] Therefore, according to this embodiment, it is possible to easily provide content that focuses on one person in a group of people who are subjects of a moving image.

[0103] <Other embodiments> In the above-described embodiment, the content processing device 10 may be configured, for example, by a personal computer, a game machine, etc., or may be configured by a smartphone, etc. Alternatively, some functions of the content processing device 10 may be realized by a personal computer, etc., and other functions may be realized by a smartphone, etc.

[0104] In the above-described embodiment, the camera used to capture the content to be played back by the content processing device 10 may be selected arbitrarily.

[0105] On the other hand, for example, a stereoscopic camera may be used to capture the content. In this way, an image with parallax can be captured, and the content reproduced by the content processing device 10 can be displayed in 3D. Note that the content may be captured using two cameras to obtain an image with parallax.

[0106] Also, for example, a 360-degree camera may be used to capture content. The 360-degree camera has a wide-angle lens, such as a fisheye lens, and can capture panoramic view images. In content generated from images captured using the 360-degree camera, for example, it is possible to display images viewed from any viewpoint.

[0107] In this way, for example, it is possible to play back an image from a viewpoint where a person specified by the user stands out. Also, by shooting the content using two 360-degree cameras, it is possible to display an image with parallax that focuses on the specified person.

[0108] <Example of software implementation> The above-mentioned content processing device 10 is a program for causing a computer to function, and can be realized by a program for causing a computer to function as the content processing device 10. In this case, the content processing device 10 includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the above-mentioned program. An example of such a computer is shown in FIG. 10.

[0109] The computer 500 includes at least one processor 501 and at least one memory 502. The memory 502 stores a program 520 for causing the computer 500 to operate as the content processing device 10. In the computer 500, the processor 501 reads the program 520 from the memory 502 and executes it to realize each function of the content processing device 10.

[0110] The processor 501 may be, for example, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating point number Processing Unit), a PPU (Physics Processing Unit), a microcontroller, or a combination of these.

[0111] As the memory 502, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination of these can be used.

[0112] The computer 500 may further include a RAM (Random Access Memory) for expanding the program 520 during execution and for temporarily storing various data. The computer 500 may further include a communication interface for transmitting and receiving data to and from other devices. The computer 500 may further include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0113] Furthermore, the program 520 for causing the computer 500 to operate as the content processing device 10 can be recorded on a non-transitory tangible recording medium 530 that is readable by the computer 500. For example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit can be used as such a recording medium 530. The computer 500 can obtain the program 520 via such a recording medium 530.

[0114] Furthermore, the program 520 for causing the computer 500 to operate as the content processing device 10 can be transmitted via a transmission medium. For example, a communication network or a broadcast wave can be used as such a transmission medium. The computer 500 can also obtain the program 520 via such a transmission medium.

[0115] Also, some or all of the functions of the content processing device 10 can be realized by logic circuits. For example, an integrated circuit in which a logic circuit that functions as each of the above control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the above control blocks can also be realized by, for example, a quantum computer.

[0116] According to each aspect of the present invention described above, by achieving the above-mentioned effects, it is possible to contribute to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote inclusive and sustainable industrialization and innovation."

[0117] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

[0118] 〔summary〕 A content processing device according to aspect 1 of the present invention includes a content input receiving unit that receives input of content in which moving images are played together with audio, a section detection unit that detects a section from the input content in which audio similar to audio corresponding to audio data of a predetermined length that has been given in advance is played, an object detection unit that detects an object from an image that is played immediately after the detected section, and a label assignment unit that assigns a label to the detected object.

[0119] A content processing device according to a second aspect of the present invention is the above-mentioned first aspect, in which the object is a person, and the label assignment unit assigns a label related to a name of the person.

[0120] A content processing device according to a third aspect of the present invention is the above-mentioned first or second aspect, wherein the label assignment unit estimates the label of the object by a calculation using model parameters obtained by machine learning.

[0121] A content processing device according to a fourth aspect of the present invention is one in which the sound reproduced together with the moving image is a piece of music in any one of the first to third aspects described above.

[0122] In a content processing device according to aspect 5 of the present invention, in any of aspects 1 to 4 above, the object detection unit further tracks the object to which the label has been assigned within a moving image of the content that is played back temporally after the section detected by the section detection unit.

[0123] A content processing device according to aspect 6 of the present invention, in accordance with aspect 5 above, is provided in advance with first audio data corresponding to a first audio, which is the audio of a section that is played earlier in time in the content, and second audio data corresponding to a second audio, which is the audio of a section that is played later in time in the content, and the section detection unit detects a first section in which audio similar to the first audio is played and a second section in which audio similar to the second audio is played, and the object detection unit tracks an object labeled corresponding to the first section and an object labeled corresponding to the second section, respectively.

[0124] A content processing device according to aspect 7 of the present invention, in accordance with aspect 5 or aspect 6 above, further includes an object image processing unit that applies a predetermined image processing to an object specified by a user from among the multiple objects detected by the object detection unit.

[0125] A content processing device according to an eighth aspect of the present invention is based on the seventh aspect, and the object image processing unit performs image processing to enlarge and display the specified object.

[0126] A content processing device according to aspect 9 of the present invention is based on aspect 7 or 8 above, and the object image processing unit performs image processing to superimpose and display a predetermined image within a predetermined range in the vicinity of the specified object.

[0127] The content processing device according to a tenth aspect of the present invention is based on any one of the above fifth to ninth aspects, and further includes a motion analysis unit that analyzes the motion of the detected object.

[0128] A content processing device according to aspect 11 of the present invention is, in aspect 10 above, a content processing device in which the object is a person, and the motion analysis unit analyzes the movement of a person specified by a user from among a plurality of people detected by the object detection unit, and the analysis result of the motion analysis unit is supplied to an animation image generation unit which generates an animation image which moves in the same manner as the specified person.

[0129] A content processing device according to a twelfth aspect of the present invention is the above-mentioned eleventh aspect, in which the animation image is an image of an avatar of the user.

[0130] A content processing method according to aspect 13 of the present invention includes the steps of accepting input of content in which moving images are played along with audio, detecting a section from the input content in which audio similar to audio corresponding to a predetermined length of audio data is played, detecting an object from an image played immediately after the detected section, and assigning a label to the detected object.

[0131] A program according to aspect 14 of the present invention causes a computer to function as a content processing device comprising an input receiving unit that receives input of content in which moving images are played together with audio, a section detection unit that detects a section from the input content in which audio similar to audio corresponding to a predetermined length of audio data is played, an object detection unit that detects an object from an image played immediately after the detected section, and a label assignment unit that assigns a label to the detected object. [Explanation of symbols]

[0132] 10 Content processing device 11 Content input reception section 12 Content playback section 13 Operation input reception section 31 Section detection unit 32 Object detection section 33 Labeling section 34 Object Image Processing Unit 50 Display 61 Motion Analysis Section 71 Animation Image Generation Unit

Claims

1. an input receiving unit that receives an input of a content for playing back a moving image of a plurality of people together with audio; a section detection unit that detects, from the input content, a section in which a sound similar to a sound corresponding to a given audio data of a predetermined length is reproduced; an object detection unit that detects, as objects, images of a plurality of people appearing in an image played back immediately after the detected section; a labeling unit that assigns a label corresponding to a name of the object to the object detected by the object detection unit by using model parameters obtained by machine learning executed in advance; A content processing device comprising:

2. The sound reproduced together with the moving image is a piece of music. The content processing device according to claim 1 .

3. The object detection unit further The object to which the label is assigned is tracked and detected in a moving image of the content that is played back after the section detected by the section detection unit. The content processing device according to claim 1 .

4. First audio data corresponding to a first sound, which is a sound of a section to be reproduced earlier in time in the content, and second audio data corresponding to a second sound, which is a sound of a section to be reproduced later in time in the content, are provided in advance; the section detection unit detects a first section in which a sound similar to the first sound is reproduced and a second section in which a sound similar to the second sound is reproduced; The labeling unit assigns a label to an object detected by the object detection unit from an image immediately after the first section, and assigns a label again to an object detected by the object detection unit from an image immediately after the second section. The content processing device according to claim 3 .

5. The apparatus further includes an object image processing unit that performs predetermined image processing on an object designated by a user among the plurality of objects detected by the object detection unit. The content processing device according to claim 3 .

6. The object image processing unit performs image processing for enlarging and displaying the specified object. The content processing device according to claim 5 .

7. The object image processing unit displays a predetermined image in a predetermined range in the vicinity of the specified object by superimposing the image. The content processing device according to claim 5 .

8. The apparatus further includes a motion analysis unit that analyzes the motion of the detected object. The content processing device according to claim 3 .

9. The motion analysis unit analyzes a motion of a person designated by a user among a plurality of people detected as an object by the object detection unit, The analysis result of the analysis unit is supplied to an animation image generation unit that generates an animation image that moves in the same manner as the specified person. The content processing device according to claim 8 .

Images