Dynamic image processing device, dynamic image processing method and recording medium

Abstract

The present invention provides a dynamic image processing device, a dynamic image processing method and a recording medium. The dynamic image is appropriately processed according to the characters contained in the dynamic image. The dynamic image processing device comprises: a character change detection unit, which detects the change of the state of the character recorded in the dynamic image from the dynamic image to be edited; and an editing unit, which, when the character change detection unit detects a given change in the state of the character, temporally edits the dynamic image corresponding to the cause of the given change in the dynamic image.

Description

[0001] This application is a divisional application of a patent application with application number 201810166264.2, application date February 28, 2018, and name “Dynamic image processing device, dynamic image processing method and recording medium”.

[0002] Citation to Related Applications

[0003] This application is based on and cites Japanese Patent Application No. 2017-050780 (filing date: March 16, 2017), the entire contents of which are incorporated herein by reference. Technical Field

[0004] The present invention relates to a dynamic image processing device, a dynamic image processing method and a recording medium. Background Art

[0005] In the past, when replaying moving images, unlike when replaying still images, even moving images deliberately shot by ordinary people tend to become monotonous and lack interest. In order to solve this problem, for example, Japanese Patent Publication No. 2009-288446 discloses the following technology: from a karaoke video in which a singer and a listener are shot, the listener's emotions are estimated, and a text or image is synthesized in the original karaoke video according to the listener's emotions.

[0006] However, the technology disclosed in Patent Document 1 is based on the premise that the singer and the listener are captured in advance, and thus has the problem of not being able to be used in images of people performing activities other than karaoke. Summary of the invention

[0007] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to appropriately process a moving image in accordance with a person included in the moving image.

[0008] One mode of the dynamic image processing device involved in the present invention comprises: an attention object determination unit, which determines from the dynamic image a plurality of attention objects contained in the dynamic image, that is, the plurality of attention objects of which at least one attention object is a person; and a processing execution unit, which executes given processing corresponding to an associating element that establishes an association between the plurality of attention objects determined by the attention object determination unit in the dynamic image.

[0009] In addition, one method of the dynamic image processing device involved in the present invention comprises: a character change detection unit, which detects a change in the state of a character recorded in the dynamic image from the dynamic image of the editing object; and an editing unit, which edits the dynamic image in time corresponding to the cause of the given change in the dynamic image when a given change is detected in the state of the character by the character change detection unit.

[0010] In addition, one embodiment of the dynamic image processing method involved in the present invention includes: an object of interest determination processing, which determines from the dynamic image a plurality of objects of interest contained in the dynamic image, that is, the plurality of objects of interest at least one of which is a person; and a processing execution processing, which executes a given processing corresponding to an associating element that establishes an associative element between the plurality of objects of interest determined by the object of interest determination processing within the dynamic image.

[0011] In addition, one embodiment of the dynamic image processing method involved in the present invention includes: character change detection processing, which detects changes in the state of a character recorded in the dynamic image from the dynamic image of the editing object; and editing processing, which, when a given change is detected in the state of the character by the character change detection processing, temporally edits the dynamic image corresponding to the cause of the given change in the dynamic image.

[0012] In addition, one mode of the recording medium involved in the present invention enables a computer to implement: an object of interest determination function, which determines from a dynamic image a plurality of objects of interest contained in the dynamic image, that is, a plurality of objects of interest in which at least one object of interest is a person; and a processing execution function, which executes given processing corresponding to an associating element that establishes an associative element between the plurality of objects of interest determined by the object of interest determination function in the dynamic image.

[0013] In addition, one method of the recording medium involved in the present invention enables a computer to implement: a character change detection function, which detects changes in the state of a character recorded in a dynamic image of an editing object from the dynamic image; and an editing function, which, when a given change is detected in the state of the character by the character change detection function, temporally edits the dynamic image corresponding to the cause of the given change in the dynamic image.

[0014] According to the present invention, a moving image can be appropriately processed according to a character included in the moving image.

[0015] Other purposes and advantages of the present invention will be set forth in the following description, and some of them will be obvious in the description, or can be learned through the practice of the invention. The purposes and advantages of the present invention can be understood and obtained through the means and combinations particularly pointed out below.

[0016] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate specific embodiments of the invention and, together with the general description given above and the detailed description of the specific embodiments given below, serve to explain the principles of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a diagram showing a schematic configuration of a moving image processing device according to Embodiment 1 of the present invention.

[0018] Figure 2A This is a diagram showing an example of a relationship table.

[0019] Figure 2B This is a diagram showing an example of an editing content table.

[0020] Figure 3 This is a flowchart showing an example of operations involved in the motion picture editing process.

[0021] Figure 4 This is a diagram showing a schematic configuration of a moving image processing device according to Embodiment 2 of the present invention.

[0022] Figure 5 This is a diagram showing an example of a relevance table in the second embodiment.

[0023] Figure 6 This is a flowchart showing an example of the operation involved in the dynamic image processing in the second embodiment.

[0024] Figure 7 This is a diagram showing a schematic configuration of a moving image processing device according to a third embodiment of the present invention.

[0025] Figure 8 This is a diagram showing an example of a factor identification table in the third embodiment.

[0026] Fig. 9 This is a diagram showing an example of an editing content table in the third embodiment.

[0027] Fig.10 This is a flowchart showing an example of the operations involved in the moving image editing process in the third embodiment. DETAILED DESCRIPTION

[0028] The present invention will be described in detail below using the drawings, but the scope of the invention is not limited to the drawings.

[0029] [Implementation Method 1]

[0030] Figure 1 This is a block diagram showing a schematic configuration of a moving image processing device 100 according to the first embodiment of the present invention.

[0031] like Figure 1 As shown, the moving image processing device 100 of the present embodiment includes a central control unit 101 , a memory 102 , a recording unit 103 , a display unit 104 , an operation input unit 105 , a communication control unit 106 , and a moving image processing unit 107 .

[0032] Furthermore, the central control unit 101 , the memory 102 , the recording unit 103 , the display unit 104 , the operation input unit 105 , the communication control unit 106 , and the moving image processing unit 107 are connected via a bus 108 .

[0033] The central control unit 101 controls each unit of the video processing device 100. Specifically, although not shown in the figure, the central control unit 101 includes a CPU (Central Processing Unit) and the like, and performs various control operations according to various processing programs (not shown) for the video processing device 100.

[0034] The memory 102 is composed of, for example, a DRAM (Dynamic Random Access Memory) and temporarily stores data processed by the central control unit 101 , the dynamic image processing unit 107 and the like.

[0035] The recording unit 103 is composed of, for example, an SSD (Solid State Drive) and records image data of still images or moving images encoded in a given compression format (e.g., JPEG format, MPEG format, etc.) by an image processing unit (not shown). In addition, the recording unit 103 may be, for example, composed of a recording medium (not shown) that can be freely assembled and disassembled, and controls the reading of data from the equipped recording medium and the writing of data to the recording medium. In addition, the recording unit 103 may also include a storage area of ​​a given server device in a state of being connected to a network via a communication control unit 106 described later.

[0036] The display unit 104 displays an image in a display area of ​​a display panel 104 a .

[0037] That is, the display unit 104 displays a moving image or a still image in a display area of ​​the display panel 104 a based on image data of a given size decoded by an image processing unit (not shown).

[0038] The display panel 104 a is formed of, for example, a liquid crystal display panel, an organic EL (Electro-Luminescence) display panel, etc. However, this is an example and the present invention is not limited thereto.

[0039] The operation input unit 105 is used to perform a given operation of the moving image processing device 100. Specifically, the operation input unit 105 includes a power button for turning on / off the power, buttons for selecting various modes or functions, etc. (all not shown).

[0040] When the user operates various buttons, the operation input unit 105 outputs an operation instruction corresponding to the operated button to the central control unit 101. The central control unit 101 causes each unit to perform a given action (such as editing a moving image) according to the operation instruction output from the operation input unit 105.

[0041] Furthermore, the operation input unit 105 includes a touch panel 105 a provided integrally with the display panel 104 a of the display unit 104 .

[0042] The communication control unit 106 transmits and receives data via the communication antenna 106 a and the communication network.

[0043] The moving image processing unit 107 includes a relevance table 107 a , an editing content table 107 b , an attention object specifying unit 107 c , a related element specifying unit 107 d , and an editing processing unit 107 e .

[0044] In addition, although each unit of the moving image processing unit 107 is constituted by, for example, a given logic circuit, the configuration is only an example and is not limited to this.

[0045] The association table 107a is as follows Figure 2A As shown, there are items including "ID" T11 for identifying related elements, "specific scene" T12 representing a specific scene, "object A" T13 representing an object, "object B" T14 representing other objects, and "related elements" T15 representing related elements.

[0046] Edit content table 107b as Figure 2B As shown, there are items such as "change in related element" T21 indicating whether or not the related element has changed, "amount of change per unit time" T22 indicating the amount of change per unit time, and "editing content" T23 indicating the editing content.

[0047] The attention object identification unit 107 c identifies a plurality of attention objects included in a moving image to be edited (for example, a panoramic moving image), at least one of which is a person.

[0048] Specifically, the focus object determination unit 107c performs target detection, character status analysis (e.g., line of sight analysis, heart rate analysis, expression analysis, etc.) and feature quantity analysis (estimation of focus area) on each frame image of the dynamic image constituting the editing object, and determines multiple focus objects contained in each frame image, that is, objects A and B, at least one of which is a character.

[0049] The related element identification unit 107d identifies related elements that relate the plurality of attention objects identified by the attention object identification unit 107c to each other in the editing target moving image. The related elements are also elements that change over time in the editing target moving image.

[0050] Specifically, when the attention object identification unit 107c identifies object A and object B in one frame image constituting the dynamic image to be edited, the related element identification unit 107d identifies the related elements of the IDs corresponding to object A and object B using the relevance table 107a.

[0051] For example, when the focus object determination unit 107c determines "parents" as object A and "children" as object B, the associated element determination unit 107d uses the association table 107a to determine the associated element "expressions of object A and object B" with ID number "2" which lists "parents" in the item of "object A" T13 and "children" in the item of "object B" T14.

[0052] The editing processing unit (processing execution unit, determination unit) 107e edits the moving image in accordance with the changes in the moving image of the related elements identified by the related element identification unit 107d.

[0053] Specifically, the editing processing unit 107e determines whether there is a change in the dynamic image of the related element determined by the related element determination unit 107d. Here, for example, based on a given number of frame images including the frame images of the related element determined by the related element determination unit 107d, it is determined whether the amount of change per unit time is greater than a given threshold value, thereby determining whether there is a change in the dynamic image of the related element.

[0054] Then, when it is determined that the amount of change per unit time in the dynamic image of the associated element determined by the associated element determination unit 107d is less than a given threshold, that is, there is no change in time, that is, it is determined to be an active element, the editing processing unit 107e uses the editing content table 107b to determine the editing content "normal time series replay" and performs normal time series replay processing (editing processing) on ​​a given number of frame images that are the objects of determination.

[0055] For example, when the associated element "expression of object A (parent) and object B (child)" with ID number "2" is determined by the associated element determination unit 107d, if it is determined that there is no change in the expression of object A (parent) and object B (child), the usual time series replay processing (editing processing) is implemented.

[0056] On the other hand, when it is determined that the amount of change per unit time in the dynamic image of the associated element determined by the associated element determination unit 107d is above a given threshold, that is, there is a change in time, that is, it is determined to be a passive element, the editing processing unit 107e determines whether the amount of change per unit time involved in the change is above a given threshold for determining the size of the change in order to further determine whether the amount of change is "large" or "small".

[0057] Then, when it is determined that the amount of change per unit time involved in the change is not greater than a given threshold for determining the size of the amount of change, that is, when it is determined to be "small", the editing processing unit 107e uses the editing content table 107b to determine one of the three types of editing content: "divide the screen into two and simultaneously replay object A and object B", "focus on object B and replay object A by wiping", and "replay the image by sliding the image from object B to object A", and performs editing processing of the determined editing content on a given number of frame images that are the objects of determination. In addition, the method of determining one of the three types of editing content may be determined, for example, according to the amount of change per unit time of the associated element, or may be determined randomly.

[0058] On the other hand, when the amount of change per unit time involved in the change is above a given threshold for determining the size of the amount of change, that is, when it is determined to be "large", the editing processing unit 107e uses the editing content table 107b to determine one of the three types of editing content: "focus on object A and replay it and then rewind in time, and focus on object B and replay it", "replay object A and object B by switching in slow motion or at high speed", and "replay it by changing the viewing angle to allow object A and object B to enter (for example, panoramic editing or asteroid editing (360° panoramic editing))", and performs editing processing of the determined editing content on a given number of frame images that serve as the determination object. For example, when the related element determination unit 107d determines the related element "expressions of object A (parent) and object B (child)" with ID number "2", if it is determined that the change in the expressions of object A (parent) and object B (child) is "large", "focus on object A and replay and then rewind in time, focus on object B and replay" is determined as the editing content, and the processing (editing processing) of focusing on object A, i.e., parents, rewinding in time and focusing on object B, i.e., children, is implemented. In addition, the method of determining one type of editing content from three types can be determined, for example, according to the amount of change per unit time of the related element, or it can be determined randomly.

[0059] <Motion picture editing process>

[0060] Next reference Figure 3 The moving image editing process performed by the moving image processing device 100 will be described. Figure 3 This is a flowchart showing an example of actions involved in the dynamic image editing process. Each function described in the flowchart is stored in the form of a readable program code, and the actions following the program code are executed one by one. In addition, the actions following the program code transmitted by the communication control unit 106 via a transmission medium such as a network can also be executed one by one. That is, the actions unique to this embodiment can also be executed using a program / data provided externally via a transmission medium in addition to a recording medium.

[0061] like Figure 3 As shown, first, based on the user operation, a designation operation of a dynamic image to be edited is performed from the dynamic image recorded in the recording unit 103. If the instruction related to the designated operation is input to the dynamic image processing unit 107 by the operation input unit 105 (step S1), the dynamic image processing unit 107 reads the designated dynamic image from the recording unit 103, and the focus object determination unit 107c performs target detection, character status analysis (such as line of sight analysis, heart rate analysis, expression analysis, etc.) and feature value analysis (estimation of the focus area) on each frame image constituting the dynamic image in turn, as an analysis of the content of the frame image (step S2).

[0062] Next, the related element identification unit 107d determines whether the object A and the object B are identified as the plurality of attention objects included in the frame image identified by the attention object identification unit 107c, that is, at least one of the attention objects is a person (step S3).

[0063] When it is determined in step S3 that the object A and the object B are identified (step S3: Yes), the related element identification unit 107d identifies the related elements of the ID numbers corresponding to the identified objects A and B using the relevance table 107a (step S4), and the process proceeds to step S5.

[0064] On the other hand, when it is determined in step S3 that the object A and the object B have been identified (step S3: No), the related element identification unit 107d skips step S4 and proceeds to step S5.

[0065] Next, the moving image processing unit 107 determines whether the attention object identification unit 107 c has performed content analysis up to the last frame image of the moving image (step S5 ).

[0066] When it is determined in step S5 that the content analysis is performed up to the last frame image (step S5 “No”), the process returns to step S2 and the subsequent processing is repeated.

[0067] On the other hand, when step S5 determines that the content is to be parsed until the last frame image (step S5 "yes"), the editing processing unit 107e takes each related element determined in step S4 as an object, and determines the editing content according to the changes in the related elements between a given number of frame images including the frame images in which each related element is determined (step S6).

[0068] Then, the editing processing unit 107e performs editing processing on a predetermined number of frame images including the frame images for which the related elements are specified, based on the editing contents specified in step S6 (step S7), and ends the moving image editing processing.

[0069] As described above, the dynamic image processing device 100 of this embodiment determines, from a dynamic image, a plurality of objects of interest contained in the dynamic image, at least one of which is a person. In addition, the dynamic image processing device 100 performs a given process corresponding to an association element that associates the plurality of objects of interest determined in the dynamic image with each other. Alternatively, the dynamic image processing device 100 determines an association element that associates the plurality of objects of interest determined in the dynamic image with each other, and performs a given process according to the determined association element.

[0070] To this end, when performing given processing on a dynamic image, since it is possible to focus on the associative elements that establish associations between multiple focus objects where at least one focus object is a person, the dynamic image can be appropriately processed according to the person as the focus object contained in the dynamic image.

[0071] In addition, the dynamic image processing device 100 of the present embodiment determines an element that establishes associations between multiple focus objects and changes over time in the dynamic image, namely, an associating element, and performs a given processing according to the temporal changes of the determined associating element in the dynamic image. Therefore, when performing a given processing on the dynamic image, processing around multiple focus objects can be appropriately performed.

[0072] Furthermore, the moving image processing device 100 according to the present embodiment edits the moving image in accordance with the temporal changes in the moving image of the identified related elements as a given process, and therefore can edit the moving image effectively.

[0073] Furthermore, the moving image processing device 100 of the present embodiment determines the amount of change in the moving image of the identified related element and edits the moving image according to the determination result, so that the moving image can be edited more effectively.

[0074] Furthermore, the moving image processing device 100 of the present embodiment identifies a plurality of attention objects based on at least two of object detection, analysis of a person's state, and analysis of feature amounts within a moving image, and can therefore identify a plurality of attention objects with high accuracy.

[0075] In addition, the dynamic image processing device 100 of this embodiment determines at least one of the heart rate, expression, action and line of sight of a person as a related factor, so when processing dynamic images, it can more appropriately process multiple focus objects surrounding at least one focus object which is a person.

[0076] [Implementation Method 2]

[0077] Next use Figures 4 to 6 Next, a moving image processing device 200 according to Embodiment 2 is described. Components identical to those in Embodiment 1 are denoted by the same reference numerals, and description thereof is omitted.

[0078] The dynamic image processing device 200 of this embodiment determines multiple focus objects (object A and object B) based on real-time dynamic images, and determines the temporally changing elements of each of the multiple focus objects, namely, the focus elements, and determines the association elements that establish associations between the multiple focus objects based on the respective focus elements of the determined multiple focus objects, and takes this as a feature.

[0079] like Figure 4 As shown, the moving image processing unit 207 of the present embodiment includes a relevance table 207a, an attention object identification unit 207b, an attention element identification unit 207c, and a related element identification unit 207d.

[0080] In addition, although each unit of the moving image processing unit 207 is constituted by, for example, a given logic circuit, the configuration is only an example and is not limited to this.

[0081] The association table 207a is as follows Figure 5 As shown, there are items including "ID" T31 for identifying related elements, "Object A" T32 representing an object, "Elements of Object A" T33 representing elements of object A that require attention, "Object B" T34 representing other objects, "Elements of Object B" T35 representing elements of object B that require attention, "Related Elements" T36 representing related elements, and "Specific Scene" T37 representing specific scene content.

[0082] The attention object identification unit 207 b identifies a plurality of attention objects included in a real-time moving image (for example, a panoramic moving image), at least one of which is a person.

[0083] Specifically, the focus object determination unit 207b performs target detection, character status analysis (e.g., line of sight analysis, heart rate analysis, expression analysis, etc.) and feature quantity analysis (estimation of focus area) on each frame image of a dynamic image that is successively captured by a live camera (camera unit) and obtained via the communication control unit 106, and determines multiple focus objects contained in each frame image, that is, multiple objects A and objects B at least one of which is a person.

[0084] The attention element identification unit 207 c identifies an attention element which is an element that changes over time in the moving image of each of the plurality of attention objects identified by the attention object identification unit 207 b .

[0085] Specifically, when object A and object B are determined in a frame image constituting a real-time dynamic image by the focus object determination unit 207b, the focus element determination unit 207c uses the association table 207a to determine the focus element of object A (the element of object A) and the focus element of object B (the element of object B) based on the results of target detection, analysis of the state of the character, and analysis of the feature value.

[0086] The related factor identification unit 207d identifies a related factor that relates the plurality of attention objects to each other in the real-time moving image based on the attention factors of each of the plurality of attention objects identified by the attention factor identification unit 207c.

[0087] Specifically, when object A and object B are determined in a frame image constituting a real-time dynamic image by the focus object determination unit 207b, and the focus elements of object A and object B are determined respectively by the focus element determination unit 207c, the associated element determination unit 207d uses the association table 207a to determine the associated elements of the IDs that correspond to the determined focus elements of object A and object B.

[0088] For example, when the focus element determination unit 207c determines "the line of sight and expression on object B" as the focus element of object A "person" and determines "the moving direction of object B" as the focus element of object B "car" in a frame image, the associated element determination unit 207d refers to the association table 207a to determine the associated element "change in line of sight and expression" with ID number "4" which lists "the line of sight and expression on object B" in item T33 of "elements of object A" and lists "the moving direction of object B" in item T35 of "elements of object B".

[0089] <Dynamic Image Processing>

[0090] Next reference Figure 6 Next, the moving image processing performed by the moving image processing device 200 will be described. Figure 6This is a flowchart showing an example of operations related to dynamic image processing.

[0091] like Figure 6 As shown, first, operations related to the acquisition of real-time dynamic images that are the objects of dynamic image processing are performed based on user operations. If instructions related to the operations are input to the dynamic image processing unit 207 through the operation input unit 105, the dynamic image processing unit 207 successively acquires real-time dynamic images via the communication control unit 106 (step S11).

[0092] Next, the focus object determination unit 207b performs target detection, character status analysis (such as line of sight analysis, heart rate analysis, expression analysis, etc.) and feature value analysis (estimation of the focus area) on each frame image constituting the acquired dynamic image in sequence as an analysis of the content of the frame image (step S12).

[0093] Next, the related element identification unit 207d determines whether the object A and the object B have been identified by the object identification unit 207b as a plurality of objects of interest included in the frame image, that is, at least one of the objects of interest is a person (step S13).

[0094] When it is determined in step S13 that the object A and the object B have been identified (step S3: Yes), the related element identification unit 207d determines whether the focus element identification unit 207c has identified the focus element of each of the object A and the object B (step S14).

[0095] When it is determined in step S14 that the respective focus elements of object A and object B have been determined (step S14 "Yes"), the associated element determination unit 207d uses the association table 207a to determine the associated elements of the ID numbers that correspond to the determined focus elements of object A and object B (step S15), and then proceeds to step S16.

[0096] On the other hand, when it is determined in step S13 that object A and object B have not been determined (step S13 "No"), or when it is determined in step S14 that the focus elements of each object A and object B have not been determined (step S14 "No"), the process proceeds to step S16.

[0097] Next, the moving image processing unit 207 determines whether acquisition of real-time moving images has been completed (step S16 ).

[0098] When it is determined in step S16 that the acquisition of the real-time moving image has not been completed (step S16: No), the process returns to step S12 and the subsequent processes are repeated.

[0099] On the other hand, when it is determined in step S16 that the acquisition of the real-time moving image has been completed (step S16: Yes), the moving image processing is completed.

[0100] As described above, the dynamic image processing device 200 of this embodiment determines a plurality of objects of interest contained in the dynamic image, at least one of which is a person, from a real-time dynamic image. In addition, the dynamic image processing device 200 performs processing around the plurality of objects of interest corresponding to the association elements that establish associations between the plurality of objects of interest determined in the dynamic image. Alternatively, the dynamic image processing device 200 determines the association elements that establish associations between the plurality of objects of interest determined in the dynamic image, and performs processing around the plurality of objects of interest according to the determined association elements.

[0101] Therefore, since it is possible to focus on the association factor that associates a plurality of attention objects with each other, when processing a real-time dynamic image, it is possible to appropriately process a plurality of attention objects surrounding at least one attention object that is a person.

[0102] In addition, the dynamic image processing device 200 of the present embodiment determines the temporally changing elements, i.e., the focus elements, within the dynamic image of each of the multiple focus objects that are determined, and determines the associated elements that establish associations between the multiple focus objects in the dynamic image based on the focus elements of each of the multiple focus objects that are determined, thereby being able to determine the associated elements with high accuracy.

[0103] Furthermore, the moving image processing device 200 of the present embodiment identifies a plurality of attention objects based on at least two of object detection, analysis of a person's state, and analysis of feature amounts within a moving image, and can therefore identify a plurality of attention objects with high accuracy.

[0104] In addition, the dynamic image processing device 200 of this embodiment determines at least any one of the heart rate, expression, action and line of sight of a person as a related element, so when processing the dynamic image, it can more appropriately process multiple focus objects surrounding at least one focus object which is a person.

[0105] [Implementation method 3]

[0106] Next use Figure 7 to Figure 10 Next, a moving image processing device 300 according to Embodiment 3 will be described. Components identical to those in Embodiments 1 and 2 are denoted by the same reference numerals, and description thereof will be omitted.

[0107] The moving image processing device 300 of this embodiment is characterized in that, when a given change is detected in the state of a person in a moving image recorded as an editing target, the device determines the cause of the given change and edits the moving image according to the determined cause.

[0108] like Figure 7 As shown, the moving image processing unit 307 of this embodiment includes a factor identification table 307a, an editing content table 307b, a person change detection unit 307c, a factor identification unit 307d, and an editing processing unit 307e.

[0109] In addition, each unit of the moving image processing unit 307 is constituted by, for example, a given logic circuit, but the configuration is only an example and is not limited to this.

[0110] Cause determination form 307a is as follows Figure 8 As shown, there are items including "ID" T41 for a method of identifying a factor, "type of change" T42 for indicating the type of change in a person's state, "determination of object" T43 for indicating a method of determining an object, and "determination of temporal position" T44 for indicating a method of determining the temporal position of the determined object.

[0111] Edit content table 307b Fig. 9 As shown, there are items including "Significant change in object" T51 indicating whether there is a meaningful change in the object, "Amount of change per unit time" T52 indicating the amount of change per unit time, "Emotion" T53 indicating the type of emotion, and "Editing content" T54 indicating the editing content.

[0112] The person change detection unit 307c detects a change in the state of a person recorded in a moving image (for example, a panoramic moving image) that is an editing target.

[0113] Specifically, the character change detection unit 307c detects changes in the state of a character recorded in a dynamic image from the dynamic image of the editing object by performing target detection, analysis of the character's state (such as line of sight analysis, heart rate analysis, expression analysis, etc.) and analysis of feature quantities (estimation of the area of ​​interest).

[0114] For example, when a scene is recorded in the dynamic image to be edited in which a smiling expression of a parent suddenly changes to a worried expression due to a child falling down, the character change detection unit 307c detects the change in the expression of the parent (character).

[0115] When a given change in the state of a character is detected by the character change detection unit 307c, the factor determination unit (determination unit, object determination unit, temporal position determination unit, object change detection unit) 307d determines the cause of the given change in the dynamic image of the editing object.

[0116] Specifically, each time the character change detection unit 307c detects a change in the state of a character recorded in a dynamic image, the factor determination unit 307d uses the factor determination table 307a to determine whether the detected change in the character's state meets any one of the "sudden change in line of sight" of ID number "1" and the "sudden change in heart rate or expression" of ID number "2".

[0117] For example, as in the above example, when the change in the expression of the parent (character) is detected by the character change detection unit 307c, the cause determination unit 307d determines that the change in the state of the detected character is consistent with the "sudden change in heart rate and expression" of ID number "2".

[0118] Then, when it is determined that the change in the state of the person detected by the person change detection unit 307c matches any one of the "sudden change in line of sight" of ID number "1" and the "sudden change in heart rate or expression" of ID number "2", the factor determination unit 307d determines the object by the determination method shown in the item "Determination of the object" T43 corresponding to the corresponding ID number. Specifically, when it is determined that it matches the "sudden change in line of sight" of ID number "1", the factor determination unit 307d determines as the object the target within the line of sight of the person in the same frame image as the frame image in which the person change detection unit 307c detects a given change in the state of the person. On the other hand, when it is determined that it matches the "sudden change in heart rate or expression" of ID number "2", the factor determination unit 307d determines the object based on the condition of the feature quantity in the same frame image as the frame image in which the person change detection unit 307c detects a given change in the state of the person.

[0119] Furthermore, the factor identifying unit 307d uses the identifying method shown in the item "identification of temporal position" T44 to retroactively identify the temporal position at which the object begins to change significantly.

[0120] In addition, the so-called meaningful change refers to the following situation: when the target within the sight line of the person in the same frame image as the frame image in which the person change detection unit 307c detects a given change in the state of the person is determined as the object, when the temporal position of the target within the sight line of the person is traced, for example, if the person suddenly falls down while running, suddenly runs when stopped, or an object placed on a table starts to fall, the change amount per unit time of the target within the sight line of the person exceeds a given threshold. In addition, it refers to the following situation: when the factor determination unit 307d determines the object based on the condition of the feature quantity within the same frame image as the frame image in which the person change detection unit 307c detects a given change in the state of the person, when the temporal position of the entire frame image is traced, the change amount per unit time of the feature quantity within the frame image exceeds a given threshold, such as a moving object such as a car enters at high speed, or the color tone within the frame image starts to change rapidly such as sunrise or sunset.

[0121] For example, as in the above example, when the change in the state of the parent (person) detected by the person change detection unit 307c is a sudden change in expression and is determined to be "sudden change in heart rate and expression" that matches the ID number "2", the factor determination unit 307d determines the object according to the first to third methods shown in the item "determination of object" T43 corresponding to the corresponding ID number "2". Specifically, the factor determination unit 307d detects a person in the object detection according to the first method and determines the detected person (child) as the object. In addition, the factor determination unit 307d detects an object other than a person in the object detection according to the second method and determines the detected object other than a person as the object. Here, when a person is determined as the object by the first method and an object other than a person is determined as the object by the second method, the object is determined based on the size of the object. On the other hand, when the object cannot be determined by the first method and the second method, the factor determination unit 307d determines the surrounding environment as the object according to the third method.

[0122] Then, the factor determination unit 307d traces back and determines the temporal position (e.g., the timing of falling) at which the object (e.g., child) determined by each method begins to undergo a significant change. Here, for example, when a person is determined as an object by the first method and a target other than a person is determined as an object by the second method as described above, the factor determination unit 307d first traces back and determines the temporal position at which the object begins to undergo a significant change by taking a larger target as an object. If the trace cannot be determined, the factor determination unit 307d traces back and determines the temporal position at which the object begins to undergo a significant change by taking a smaller target as an object.

[0123] The editing processing unit (editing unit) 307e edits the moving image in terms of time according to the determination result of the factor determination unit 307d.

[0124] Specifically, the editing processing unit 307e determines whether or not there is a significant change in the object specified by the factor specifying unit 307d.

[0125] Then, when it is determined that there is no meaningful change in the object determined by the factor determination unit 307d, the editing processing unit 107e uses the editing content table 307b to determine the editing content "normal time series replay" and performs normal time series replay processing (editing processing) on ​​a given number of frame images that are the objects of determination.

[0126] On the other hand, when it is determined that there is a significant change in the object specified by the factor specifying unit 307d, the editing processing unit 307e further determines whether the amount of change per unit time involved in the change is greater than a given threshold for determining the size of the change.

[0127] Then, when the amount of change per unit time involved in the change is not greater than a given threshold for determining the size of the amount of change, that is, when it is determined to be "small", the editing processing unit 307e determines the emotion of the person (the person detected by the person change detection unit 307c) at the temporal position determined by the factor determination unit 307d, determines the editing content corresponding to the emotion, and performs editing processing based on the determined editing content. More specifically, when it is determined that the emotion of the person at the temporal position determined by the factor determination unit 307d is "neutral (for example, "surprise")", the editing processing unit 307e refers to the editing content table 307b, determines "split the screen into two, and simultaneously replay object A (the person detected by the person change detection unit 307c, the same below) and object B (the object determined by the factor determination unit 307d, the same below)" as the editing content, and performs editing processing of the editing content. Furthermore, when it is determined that the emotion of the character at the temporal position determined by the factor determination unit 307d is "negative (e.g., "sadness," "fear," and "anger")," the editing processing unit 307e refers to the editing content table 307b, determines "focus on object B, and replay by wiping the display object A" as the editing content, and performs editing processing of the editing content. Furthermore, when it is determined that the emotion of the character at the temporal position determined by the factor determination unit 307d is "positive (e.g., "joy," "liking," and "calm")," the editing processing unit 307e refers to the editing content table 307b, determines "replay by sliding the image from object B to object A" as the editing content, and performs editing processing of the editing content.

[0128] On the other hand, when it is determined that the amount of change per unit time involved in the change is greater than a given threshold for determining the size of the amount of change, that is, when it is determined to be "large", the editing processing unit 307e also determines the emotion of the character at the temporal position determined by the factor determination unit 307d, and performs editing processing corresponding to the emotion. More specifically, when it is determined that the emotion of the character at the temporal position determined by the factor determination unit 307d is "neutral", the editing processing unit 307e refers to the editing content table 307b, determines "focus on object A and perform time reversal after replay, focus on object B and perform replay" as the editing content, and performs editing processing of the editing content. For example, when it is determined that the emotion of the person (parents) at the temporal position determined by the factor determination unit 307d is "surprise (neutral)" as in the above example, the editing processing unit 307e refers to the editing content table 307b, determines "focusing on the parents (object A) and replaying the video in time, and focusing on the children (object B) and replaying the video" as the editing content, and performs editing processing of the editing content. In addition, when it is determined that the emotion of the person at the temporal position determined by the factor determination unit 307d is "negative", the editing processing unit 307e refers to the editing content table 307b, determines "switching and replaying the object A and the object B in slow motion or high speed" as the editing content, and performs editing processing of the editing content. In addition, when it is determined that the emotion of the character at the temporal position determined by the factor determination unit 307d is "positive", the editing processing unit 307e refers to the editing content table 307b, determines "changing to a perspective that allows object A and object B to enter for playback (for example, panoramic editing or asteroid editing (360° panoramic editing))" as the editing content, and implements editing processing of the editing content.

[0129] In addition, the aforementioned character's emotions, namely "neutral (e.g., "surprise")", "negative (e.g., "sadness", "fear", "anger")", and "positive (e.g., "joy", "like", "calm")", can be determined using known audio analysis technology.

[0130] <Motion picture editing process>

[0131] Next reference Fig.10 The moving image editing process performed by the moving image processing device 300 will be described. Fig.10 This is a flowchart showing an example of operations involved in the motion picture editing process.

[0132] like Fig.10As shown in the figure, first, a dynamic image to be edited is designated from the dynamic images recorded in the recording unit 103 based on a user operation, and an instruction related to the designated operation is input to the dynamic image processing unit 307 through the operation input unit 105 (step S21), and then the dynamic image processing unit 307 reads the designated dynamic image from the recording unit 103. Then, the person change detection unit 307c sequentially performs target detection, person state analysis (for example, line of sight analysis, heart rate analysis, expression analysis, etc.) and feature value analysis (estimation of the attention area) as frame image content analysis for each frame image constituting the read dynamic image, thereby sequentially detecting changes in the state of the person recorded in the dynamic image from the read dynamic image (step S22).

[0133] Next, whenever the person change detection unit 307c successively detects a change in the state of the person recorded in the dynamic image, the factor determination unit 307d uses the factor determination table 307a to determine whether there is a given change in the detected state of the person, that is, to determine whether the change in the state of the person meets any one of the "sudden change in line of sight" of ID number "1" and the "sudden change in heart rate or expression" of ID number "2" (step S23).

[0134] If step S23 determines that there is no given change in the detected state of the person, that is, the change in the state of the person does not meet any of the "sudden change in line of sight" of ID number "1" and the "sudden change in heart rate and expression" of ID number "2" (step S23 "No"), transfer to step S29.

[0135] On the other hand, when step S23 determines that there is a given change in the state of the detected person, that is, the change in the state of the person meets any one of the "sudden change in line of sight" of ID number "1" and the "sudden change in heart rate and expression" of ID number "2" (step S23 "Yes"), the cause determination unit 307d determines the object that is the cause of the given change using the determination method indicated in the item "Determination of object" T43 corresponding to the corresponding ID number (step S24).

[0136] Next, the factor identification unit 307d traces back the temporal position of the moving image to determine whether there is a significant change in the object identified in step S24 (step S25).

[0137] When it is determined in step S25 that there is no significant change in the object (step S25 “No”), step S26 is skipped and the process proceeds to step S27 .

[0138] On the other hand, when it is determined in step S25 that there is a significant change in the object (step S25: Yes), the factor identification unit 307d identifies the temporal position where the object starts to change significantly (step S26), and the process proceeds to step S27.

[0139] Next, the editing processing unit 307e uses the editing content table 307b to determine the editing content according to the object determined by the factor determination unit 307d (step S27). Then, the editing processing unit 307e performs editing processing based on the editing content determined in step S27 (step S28).

[0140] Next, the dynamic image processing unit 307 determines whether the human change detection unit 307 c has performed content analysis up to the last frame image (step S29 ).

[0141] When it is determined in step S29 that the content analysis has not been performed up to the last frame image (step S29: No), the process returns to step S22 and the subsequent processes are repeated.

[0142] On the other hand, when it is determined in step S29 that the content analysis is performed up to the last frame image (step S29 “Yes”), the moving image processing unit 307 ends the moving image editing process.

[0143] As described above, the dynamic image processing device 300 of this embodiment detects the change of the state of the character recorded in the dynamic image from the dynamic image to be edited, and when a given change is detected in the state of the character, the dynamic image corresponding to the cause of the given change in the dynamic image is edited in time. Alternatively, when the dynamic image processing device 300 detects a given change in the state of the character, it determines the cause of the given change in the dynamic image, and edits the dynamic image in time according to the determination result of the cause.

[0144] Therefore, when a given change is detected in the state of a person recorded in a moving image to be edited, editing processing around the cause of the given change can be performed when editing the moving image, so the moving image can be edited effectively.

[0145] In addition, the motion image processing device 300 of the present embodiment determines an object within the motion image that is a cause of a given change when a given change is detected in the state of a character, and determines the temporal position of the cause of the given change within the motion image based on the determined object, and edits the motion image in time according to the determined temporal position, thereby being able to edit the motion image more effectively.

[0146] In addition, the dynamic image processing device 300 of the present embodiment detects changes in the state of the determined object within the dynamic image, and determines the temporal position when a given change is detected in the object as the temporal position of the cause of the given change within the dynamic image, thereby being able to determine the temporal position of the cause of the given change within the dynamic image with high accuracy.

[0147] In addition, the dynamic image processing device 300 of the present embodiment determines the object that is the cause of a given change in the dynamic image when a given change is detected in the state of the character based on the status of the feature quantity in the same frame image as the frame image when the given change is detected in the state of the character and at least one of the line of sight of the character, thereby determining the object that is the cause of the given change in the dynamic image with high accuracy.

[0148] In addition, the dynamic image processing device 300 of this embodiment selects a method for determining the factors of given changes that is pre-established corresponding to each given type of change to determine the factors of given changes in the dynamic image, so that the factors of given changes can be appropriately determined according to the given type of change.

[0149] Furthermore, the moving image processing device 300 of the present embodiment edits the moving image in time according to the type and magnitude of a given change in the detected state of the person, and thus can edit the moving image more effectively.

[0150] Furthermore, the moving image processing device 300 of the present embodiment edits the moving image in time according to the type of change in the state of the detected object in the moving image, and thus can edit the moving image more effectively.

[0151] In addition, the present invention is not limited to the embodiments, and various improvements and design changes can be made without departing from the scope of the present invention.

[0152] In the first to third embodiments, a panoramic moving image is described as an example of a moving image processed by the moving image processing unit, but the moving image may be a moving image that is usually shot.

[0153] In the second embodiment, the dynamic image processing unit 207 may include the same editing content table and editing processing unit as those in the first embodiment, and the editing processing unit may edit the dynamic image according to changes in the dynamic image (dynamic image of the editing object) of the related elements determined by the related element determination unit 207d.

[0154] Furthermore, it is needless to say that it can be provided as a dynamic image processing device that has a configuration for realizing the functions involved in the present invention in advance, and it is also possible to make an existing information processing device or the like function as the dynamic image processing device involved in the present invention by using a program. That is, by using a program for realizing the functional configurations of the dynamic image processing devices 100, 200, and 300 illustrated in the embodiments, the CPU or the like that controls the existing information processing device or the like can execute the program, and the dynamic image processing device involved in the present invention can be realized.

[0155] In addition, the method of using such a program is arbitrary. The program can be stored in a computer-readable storage medium such as a floppy disk, CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, memory card, etc. for use. Furthermore, the program can be superimposed on a carrier wave and used via a communication medium such as the Internet. For example, the program can be posted on a bulletin board (BBS: Bulletin Board System) on a communication network. And it can be configured to start the program and execute it in the same way as other applications under the control of the OS (Operating System), so that processing can be performed.

[0156] Although the embodiment of the present invention has been described, the scope of the present invention is not limited to the embodiment, but includes a scope equivalent to the scope of the invention described in the claims.

Claims

1. A dynamic image processing device, comprising: a character change detection unit for detecting, from a dynamic image of an editing target, a change in a state of a character recorded in the dynamic image; a determination unit that, when a given change is detected in the state of the person by the person change detection unit, determines an object in the dynamic image that is a cause of the given change and a change in the state of the object in the dynamic image; and an editing unit that edits the dynamic image in time according to the determination result determined by the determination unit, The determining unit has: an object determination unit that determines the object in the dynamic image that is the cause of the given change when the person change detection unit detects the given change in the state of the person, based on at least one of the state of the feature quantity in the same frame image of the dynamic image and the line of sight of the person; and an object change detection unit configured to detect a change in a state of the object determined by the object determination unit within the dynamic image, The editing unit temporally edits the moving image according to the change in the state of the object in the moving image detected by the object change detection unit.

2. The dynamic image processing device according to claim 1, wherein: The specifying unit further includes a temporal position specifying unit that specifies a temporal position of the factor of the given change in the dynamic image based on the object specified by the object specifying unit. The editing unit temporally edits the moving image according to the temporal position determined by the temporal position determination unit.

3. The dynamic image processing device according to claim 2, wherein: The temporal position specifying unit specifies the temporal position at which the object change detecting unit detects the given change in the object as the temporal position of the cause of the given change in the moving image.

4. The dynamic image processing device according to claim 1, wherein: The specifying unit selects a method for specifying the factor of the given change that is previously associated with each type of the given change, and specifies the factor of the given change in the moving image.

5. The dynamic image processing device according to claim 1, wherein: The editing unit temporally edits the moving image according to at least one of a type and a magnitude of the given change in the state of the person detected by the person change detection unit.

6. The dynamic image processing device according to claim 3, wherein: The editing unit temporally edits the moving image according to the type of the given change in the state of the object in the moving image detected by the object change detection unit.

7. A dynamic image processing method, comprising: Character change detection processing, detecting a change in the state of a character recorded in the dynamic image from the dynamic image of the editing object; determining processing for determining an object in the dynamic image that is a cause of the given change and a change in the state of the object in the dynamic image when a given change is detected in the state of the character by the character change detection processing; and an editing process of temporally editing the dynamic image according to the determination result determined by the determination process, The determination process comprises: An object determination process for determining the object in the dynamic image that is the cause of the given change when the given change in the state of the person is detected by the person change detection process, based on at least one of the condition of the feature quantity in the same frame image of the dynamic image and the line of sight of the person; and an object change detection process for detecting a change in a state of the object determined by the object determination process within the dynamic image, In the editing process, the dynamic image is edited temporally according to the change in the state of the object within the dynamic image detected by the object change detection process.

8. A recording medium storing a program that enables a computer to: A character change detection function detects changes in the state of a character recorded in a dynamic image of an editing object from the dynamic image; a determination function for, when a given change is detected in the state of the character by the character change detection function, determining an object in the dynamic image that is a cause of the given change and a change in the state of the object in the dynamic image; and an editing function for temporally editing the dynamic image according to the determination result determined by the determination function, The determining function comprises: An object determination function for determining the object in the dynamic image that is the cause of the given change when the given change is detected in the state of the person by the person change detection function, based on at least one of the condition of the feature quantity in the same frame image of the dynamic image and the line of sight of the person; and an object change detection function for detecting a change in the state of the object determined by the object determination function within the dynamic image, The editing function temporally edits the dynamic image in accordance with changes in the state of the object within the dynamic image detected by the object change detection function.

Images