Processing apparatus, processing method, and program

Abstract

To improve accuracy of determining whether or not a scene is a scene in which a subject is tracked in an imaging apparatus.SOLUTION: The information processing apparatus 101 extracts, from a moving image captured by the image capturing apparatus 107, a section in which the angle of view of the image capturing apparatus 107 continuously moves, and derives information indicating a content corresponding to a feature of an object in the section. Then, based on the information, the information processing apparatus 101 derives information for specifying that the section included in the moving image is a section in which the subject is tracked or information for specifying that the section included in the moving image is a section in which the subject is not tracked.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a processing device, a processing method, and a program. [Background technology]

[0002] Video images captured by an imaging device or the like may be displayed, edited, and managed. Patent Document 1 discloses a method for determining, from a video image, a scene in which a user (photographer) continuously moves the imaging device in a certain direction. Specifically, Patent Document 1 discloses a method for determining whether the scene is one in which the imaging device was moved to track a subject, or one in which the user moved the imaging device to shift their line of sight to another subject. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 8-191411 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the technology described in Patent Document 1, a correlation image between frames compensated by a motion vector is detected, and the above-mentioned determination is made based on the degree to which low-correlation areas are concentrated in the correlation image. Therefore, for example, if the subject's movement changes while the imaging device is tracking the subject, it may be determined that the imaging device has moved to shift the user's gaze to another subject, even though the subject is still being tracked. In this case, for example, a series of movements of the same subject may not be correctly determined as a single scene.

[0005] The present disclosure has been made in consideration of the above-described problems, and aims to improve the accuracy with which an imaging device can determine whether or not a scene is being tracked. [Means for solving the problem]

[0006] The processing device of the present disclosure is characterized by having an extraction means for extracting sections from a captured video where the angle of view of the imaging device moves continuously; a first derivation means for deriving subject information indicating content according to the characteristics of the subject in the section extracted by the extraction means; and a second derivation means for deriving, based on the subject information derived by the first derivation means, first section information for identifying the section included in the video as a section where the subject is being tracked, or second section information for identifying the section included in the video as a section where the subject is not being tracked. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to improve the accuracy of determining whether or not a scene is being tracked by an imaging device. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a processing system. [Figure 2] FIG. 1 is a diagram illustrating a configuration of an information processing device. [Figure 3] FIG. 1 is a diagram illustrating a scene in which multiple subjects are repeatedly imaged. [Figure 4] FIG. 10 is a diagram showing a scene list. [Figure 5] 10 is a flowchart illustrating an example of a processing method. [Figure 6] FIG. 10 is a diagram showing an image management screen. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. [System configuration example] FIG. 1 is a diagram showing an example of the configuration of a processing system according to this embodiment. In this embodiment, FIG. 1 illustrates a case where the processing system includes an information processing device 101. FIG. 1 illustrates a case where the information processing device 101 operates within a cloud 102. FIG. 1 also illustrates a case where the information processing device 101 is communicably connected to an external device via a network 103. In this embodiment, an example is shown where the network 103 includes the Internet. FIG. 1 also illustrates a case where the processing system includes PCs (Personal Computers) 104 and 105, a mobile device 106, and an imaging device 107. FIG. 1 also illustrates a case where the imaging device 107 is communicably connected to the PC 105, and information stored in the imaging device 107 is transmitted to an external device (such as the information processing device 101) via the PC 105. However, this is not necessarily the case. For example, the information stored in the imaging device 107 may be transmitted directly from the imaging device 107 to the external device. In this embodiment, an example is shown where the imaging device 107 is a device capable of capturing moving images, such as a video camera. Note that the processing system may not include at least one of the PCs (Personal Computers) 104 and 105, the mobile device 106, and the imaging device 107. If the processing system does not include the imaging device 107, for example, the information processing device 101 may read moving image data from a portable storage medium on which the moving image data captured by the imaging device 107 is stored.

[0010] In the processing system illustrated in FIG. 1 , moving images captured by the imaging device 107 may be uploaded to the information processing device 101, for example, as follows. First, a user 110, such as the owner of the imaging device 107, operates the imaging device 107 to cause the imaging device 107 to capture moving images. Further, for example, the user 110 operates the imaging device 107 and the PC 105 to store data of the moving images captured by the imaging device 107 in the PC 105. Then, the user 110 operates the PC 105 to upload the data of the moving images captured by the imaging device 107 to the information processing device 101 (image management application) via the network 103. Note that, as described above, uploading of data of moving images captured by the imaging device 107 may be performed by, for example, the imaging device 107.

[0011] In the processing system illustrated in Fig. 1, when using video uploaded to the information processing device 101, a user 110 may access an image management application on the information processing device 101 via a network 103 by operating a terminal device. In Fig. 1, the terminal device is, for example, at least one of PCs 104 and 105 and a mobile device 106. In this case, the user can use the terminal device to view and edit the video managed by the image management application.

[0012] In this embodiment, a case is illustrated in which a user 110, who is the owner of a video, uploads a video to an image management application installed on an information processing device 101 running in the cloud 102, and then views and edits the video. However, this is not necessarily the case. For example, the information processing device 101 (image management application) does not need to be located in the cloud 102, but may be located in an on-premise environment. The image management application may also be loaded and executed on a terminal device (such as a PC 104 or 105 or a mobile device 106). In this case, for example, the terminal device may directly import a video from an imaging device 107 and, by using the image management application, perform the same processing as that of the information processing device 101 and allow a user to view and edit the video. The network 103 is not limited to the Internet. The network 103 may include other networks, such as a LAN (Local Area Network), in addition to or instead of the Internet. The network 103 may be a wired communication network, a wireless communication network, or a network including both a wired communication network and a wireless communication network. In addition to or instead of the network 103, a communication line other than a network may be used.

[0013] Furthermore, the user and the owner of the image management application do not have to be the same person. For example, a user may be able to view and edit videos owned by other people (other users) by using a terminal device. In this case, for example, the information processing device 101 may perform user authentication for at least one of viewing and editing of the videos. In this case, the information processing device 101 may permit at least one of viewing and editing of the videos only for users who have successfully passed user authentication, within the scope of the authority granted to the user. However, such user authentication does not have to be performed.

[0014] [Example of information processing device (image management application) configuration] FIG. 2 is a diagram illustrating an example of the configuration of an information processing device 101 according to this embodiment. The information processing device 101 may be implemented by a single computer. Furthermore, the functions of the information processing device 101 may be distributed among multiple computers as necessary. When the functions of the information processing device 101 are implemented by multiple computers, the multiple computers are connected to each other via a communication line such as a LAN so that they can communicate with each other. Terminal devices other than the information processing device 101 (in the example shown in FIG. 1, PCs 104 and 105 and a mobile device 106) can also be implemented using the configuration shown in FIG. 2, for example. Therefore, detailed descriptions of devices other than the information processing device 101 will be omitted here. The imaging device 107 is implemented by, for example, including an imaging unit for capturing moving images in addition to the configuration shown in FIG. 2. The imaging unit includes, for example, an imaging optical system including an imaging lens and an imaging element. The configuration of the imaging unit itself can be implemented using known technology, and therefore detailed descriptions thereof will be omitted here.

[0015] In FIG. 2, the control unit 201 controls the entire information processing device 101. The control unit 201 has, for example, a Central Processing Unit (CPU). Furthermore, the control unit 201 may have one or more processors other than the CPU (for example, a Graphics Processing Unit (GPU)) in addition to or instead of the CPU. Furthermore, the processing by the control unit 201 may be shared among a plurality of hardware components. Furthermore, at least a portion of the processing by the control unit 201 may be performed by using dedicated hardware. Examples of the dedicated hardware include an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array). Note that, as described above, the processor is not limited to a specific processor (for example, a CPU), the processing may be shared among a plurality of hardware components, and dedicated hardware may be used, and this is also true for devices other than the information processing device 101.

[0016] The ROM (Read Only Memory) 202 stores programs and parameters that do not require modification. The RAM (Random Access Memory) 203 temporarily stores programs and data supplied from an external device or the like. The external storage device 204 is a hard disk, memory card, or the like fixedly installed in the information processing device 101. The external storage device 204 stores, for example, programs such as an operating system (OS). The input interface (I / F) 205 connects the information processing device 101 to an input device (not shown). The input device accepts user operations and inputs data. The input device includes, for example, at least one of a pointing device and a keyboard. The BMU (Bit Move Unit) 206 controls, for example, data transfer between memories (e.g., between the VRAM 207 and other memories) or between memories and each I / O device (e.g., a network interface (I / F) 208). The VRAM (Video RAM) 207 draws images to be displayed on the display unit 211. The VRAM 207 transmits images containing various information to the display unit 211 according to predetermined rules. As a result, the display unit 211 displays various types of information. The display unit 211 has a computer display such as a liquid crystal display. In this embodiment, an example is shown in which the information displayed on the display unit 211 includes information necessary for the user to view and edit moving images captured by the imaging device 107. Note that the display unit 211 may display different information between the moving images captured by the imaging device 107 and the information for editing the moving images. The network I / F 208 connects the information processing device 101 to a communication line. In this embodiment, an example is shown in which the network I / F 208 connects the information processing device 101 to the network 103. The system bus 209 connects each unit (control unit 201 to network I / F 208) included in the information processing device 101 so that they can communicate with each other.

[0017] [Explanation of video shooting example] FIG. 3 is a diagram illustrating an example of a scene in which multiple subjects are repeatedly imaged. FIG. 3 illustrates a scene in which subjects running along a straight section of a track at an athletics stadium are repeatedly imaged. Specifically, FIG. 3 illustrates an example in which subjects 302, 303, 304, and 305, who are runners, run in order along the track at the athletics stadium, and a photographer 301 tracks and images each of them with an imaging device 107. In the following description, imaging while tracking a subject will be referred to as tracking imaging as necessary. The period during which tracking imaging is performed includes a period during which the angle of view of the imaging device 107 continuously changes. Note that the continuous change in the angle of view of the imaging device 107 corresponds to a continuous change in the orientation of the imaging device 107.

[0018] When tracking imaging is being performed, the subject is usually in focus. However, there are cases where the subject is not in focus even though the photographer 301 intends to track the subject. In other words, there are cases where the subject is not in focus when tracking imaging is being performed. In this embodiment, a case where the subject to be tracked is in focus is exemplified. However, the subjects to be tracked may include subjects that are not in focus. In this case, it may be determined that tracking imaging is being performed if the subject is captured somewhere within the entire area of ​​the image. Also, in this embodiment, a case where the subject to be tracked is a person is exemplified. However, the subject to be tracked may be an object other than a person. For example, the subject may be a moving subject such as a living thing other than a human (e.g., an animal, a bird, an insect) or a vehicle (e.g., a car, an airplane, a train, etc.).

[0019] In FIG. 3, subjects 304 and 305 are waiting in pre-race waiting area 306. Subject 303 is at start position 307 of the track and field event. Subject 302 is in the middle of running from start position 307 to finish position 308 of the track and field event. After finishing the track and field event, the runners wait in post-race waiting area 309. Note that the circled numbers shown at waiting area 309 indicate the order of finish. FIG. 3 illustrates an example in which, after the race, runners wait in positions according to the order of finish of the runners.

[0020] 3, sections 310 to 313 each represent a section in which the angle of view of the image capture device 107 moves continuously. In sections 310 to 313, the light-density sections represent periods in which the subject is in focus, and the dark-density sections represent periods in which the subject is out of focus.

[0021] The direction in which the angle of view moves may be any direction. The direction in which the angle of view moves may be the left-right direction (pan direction) of the imaging device 107, the up-down direction (tilt direction) of the imaging device 107, or a direction different from these directions (for example, a diagonal direction). The direction in which the angle of view moves may be a constant direction. The constant direction does not have to be strictly the same direction. For example, if a captured image includes a subject to be tracked, the angle of view may be considered to be moving in a constant direction. The constant direction may include multiple directions. When tracking imaging is performed, the direction in which the angle of view moves may be the direction in which the subject moves. The direction in which the angle of view moves may not be strictly the same as the direction in which the subject moves. For example, if a captured image includes a subject to be tracked, the angle of view may be considered to be moving in the direction in which the subject moves. The direction in which the angle of view moves may be the opposite direction to the direction in which the subject moves. When tracking imaging is performed, the photographer 301 generally moves his or her line of sight to match the movement of the subject without changing his or her location. Therefore, when tracking imaging is performed, the angle of view of the imaging device 107 generally moves (the orientation of the imaging device 107 changes) while the position of the imaging device 107 does not change (greatly). Note that the imaging magnification may or may not change when the angle of view moves continuously.

[0022] 3, a situation is assumed in which the angle of view moves continuously from the start position 307 to the goal position 308 and the waiting area 309. Also, in FIG. 3, a situation is assumed in which the angle of view moves continuously from the goal position 308 to the start position 307 and the waiting area 306.

[0023] In addition, this embodiment assumes a case where the angle of view continuously changes as a result of the photographer 301 himself / herself changing the orientation of the image capture device 107 to track a subject. However, the method of this embodiment may be applied to a case where the angle of view continuously changes as a result of the image capture device 107 automatically tracking a subject, for example.

[0024] 3, first, with the image capture device 107 focused on the subject 302, tracking and imaging of the subject 302 is performed from the start position 307 to the goal position 308 (see subsection 310a (low-density portion) of section 310). After that, tracking of the subject 302 is temporarily interrupted at the goal position 308 (see subsection 310b (high-density portion) of section 310). In subsection 310b, the subject is out of focus. After that, the focus remains on the subject 302 until the subject 302 finishes the competition and begins waiting in the post-competition waiting area 309, and tracking and imaging of the subject 302 is performed (see subsection 310c of section 310). After that, in order to capture the next subject 303, the photographer 301 rapidly moves the angle of view of the image capture device 107 toward the start position 307 without tracking the subject. Then, the photographer 301 returns the angle of view of the image capturing device 107 to the start position 307 (see section 311). In section 311, the subject is not in focus.

[0025] Thereafter, with the image capturing device 107 focused on the subject 303, tracking and imaging of the subject 303 is performed from the start position 307 to the goal position 308 (see section 312). Thereafter, in order to capture the next subject 304, the photographer 301 rapidly moves the angle of view of the image capturing device 107 toward the start position 307 without tracking the subject. Then, the photographer 301 returns the angle of view of the image capturing device 107 to the start position 307 (see section 313). The subject 305 is also tracked and imaged in the same manner as when the subject 304 is tracked and imaged.

[0026] FIG. 3 illustrates an example in which a series of moving images in which subjects 302, 303, 304, and 305 are tracked and captured one after another in the above-described manner by the imaging device 107.

[0027] Below, an example of processing is described that includes determining, as one scene, portions of a video in which multiple subjects are continuously tracked in which the same subject is being tracked, and determining unnecessary scenes between scenes in which subjects are being tracked. Furthermore, unnecessary scenes (i.e., scenes in which subjects are not being tracked) may be deleted from the video. In this case, it is preferable to reduce the effort required to delete unnecessary scenes from the video. Therefore, below, a process for deleting unnecessary scenes from a video and a process for reducing the effort required to delete unnecessary scenes from a video are also described.

[0028] In the following description, the processing of the information processing device 101 is realized, for example, by the control unit 201 reading and executing an image management application from the ROM 202, the external storage device 204, or the network I / F 208. Furthermore, the processing of a terminal device other than the information processing device 101 is realized, for example, by a control unit of the terminal device reading and executing a program stored in the ROM or the external storage device. Furthermore, in the following description, a section in a moving image during which the angle of view of the imaging device 107 continuously moves is referred to as an angle of view movement section, as necessary. The angle of view movement section may include a section during which tracking imaging is performed, or a section during which tracking imaging is not performed.

[0029] [Example of a table that stores the analysis results of video images] FIG. 4 is a diagram showing an example of a scene list 400. The scene list 400 is an example of information used by the information processing device 101 (image management application) to manage angle of view movement sections (scenes) included in a video. In this embodiment, an example is shown in which the scene list 400 is a table that stores information about scenes included in a video. Specifically, this embodiment illustrates a case in which information about one angle of view movement section extracted from a video is stored in the scene list 400 as information about one scene. More specifically, this embodiment illustrates a case in which a scene ID 401, a start position 402, an end position 403, a subject ID 404, and a scene necessity 405 are included as one piece of scene information (column information).

[0030] A scene ID that is automatically assigned when the column is created is stored in the column of scene ID 401. The scene ID is an example of scene identification information. Information indicating the timing at which a scene identified by a scene ID starts is stored in the column of start position 402. For example, the column of start position 402 stores the time from the start of the video.

[0031] Information indicating the timing at which a scene identified by a scene ID ends is stored in the column of end position 403. For example, the column of end position 403 stores the time from the start of the video.

[0032] The subject ID 404 column stores information based on the result of identifying an individual subject focused on in a scene identified by the scene ID. The subject ID 404 column may store an ID that uniquely identifies the subject. The subject ID 404 column may store information associated with the subject's feature. In this case, the subject's feature may be, for example, a facial feature. The subject's feature may also be a feature of information that can identify the subject, such as the subject's bib or number. For example, the subject ID 404 column may store information that quantifies the analysis result of the subject's feature. In the following description, this result will be referred to as feature analysis information as needed. The feature analysis information may be, for example, information that quantifies the analysis result of a person's feature. In this case, the feature analysis information may be, for example, the ratio of the size of the eyes, nose, and mouth to the size of the face. The size may be area or length. The feature analysis information may also be information that quantifies the analysis result of the feature of an object other than a person. Furthermore, the analysis results of the feature amounts of a subject (person, object) may be quantified based on the analysis results of AI (Artificial Intelligence) that performs individual identification. Such feature analysis information may be linked to the feature amounts of the subject. Furthermore, an ID that uniquely identifies the subject may be linked to the feature amounts of the subject. As described above, subject information may be an ID, feature amounts, or other information as long as it indicates content according to the features of the subject.

[0033] The column of scene necessity 405 stores information indicating whether the scene identified by the scene ID is an unnecessary scene. Fig. 4 illustrates an example in which "FALSE" is stored in the column of scene necessity 405 when the scene identified by the scene ID is not an unnecessary scene. On the other hand, Fig. 4 illustrates an example in which "TRUE" is stored in the column of scene necessity 405 when the scene identified by the scene ID is an unnecessary scene. FIG. 4 illustrates an example in which scene information for sections 310 to 313 shown in FIG.

[0034] [Example of processing including video analysis] An example of a processing method by the information processing device 101 will be described with reference to the flowchart of FIG. 5. The flowchart of FIG. 5 illustrates a case in which each angle-of-view movement section included in a series of moving images captured by the imaging device 107 is treated as a single scene, and processing including analyzing each scene is performed. Scene analysis includes determining whether the scene is a scene in which tracking imaging is performed or a scene in which tracking imaging is not performed. As described above, the flowchart of FIG. 5 illustrates a case in which the control unit 201 reads and executes an image management application to perform the processing. The flowchart of FIG. 5 may also be started, for example, when an operation to instruct the launch of the image management application is performed on the input I / F 205. The flowchart of FIG. 5 may also be started, for example, when an operation to specify a moving image file to be analyzed is performed on the input I / F 205 after the image management application is launched. For simplicity's sake, the flowchart illustrates a case in which the moving image file to be analyzed has been uploaded to the information processing device 101 (image management application) before the processing according to the flowchart of FIG. 5 begins.

[0035] In S501, the control unit 201 reads data of the moving image to be analyzed. Next, in S502, the control unit 201 reads out the metadata of the video to be analyzed. In this embodiment, for example, the metadata of the video stores the start timing and end timing of a field of view movement section (a section in which the field of view of the image capture device 107 continuously moves) in the video to be analyzed. In this embodiment, the image capture device 107 creates metadata for the video and adds the metadata to the video. Note that the number of field of view movement sections included in the video to be analyzed may be one or more.

[0036] In this embodiment, the control unit 201 determines the start and end timings of the angle of view movement section based on information stored in metadata created by the image capture device 107. However, this is not necessarily required. For example, the control unit 201 may determine the start and end timings of the angle of view movement section based on the fact that the angle of view of the image capture device 107 is moving in accordance with the movement of the subject or the results of image analysis of each frame image.

[0037] Next, in S503, the control unit 201 extracts one angle of view movement section from the video to be analyzed. In this embodiment, the control unit 201 extracts one angle of view movement section with the earliest start timing from among unextracted angle of view movement sections included in the video to be analyzed based on the start timing and end timing of the angle of view movement section. However, this is not necessarily the case as long as an unextracted angle of view movement section included in the video to be analyzed is extracted in S503. Note that, in order to improve the accuracy of scene analysis, the extraction conditions for angle of view movement sections may include that the subject is moving in the same direction.

[0038] Next, in S504, the control unit 201 determines that the image in the angle of view movement section extracted in S503 is one scene, and stores the scene ID 401, start position 402, and end position 403 in the scene list 400 as information about the scene.

[0039] Next, in S505, the control unit 201 extracts a subject from the focused area in each frame image included in the angle of view shift section extracted in S503, for each frame image included in the angle of view shift section. The control unit 201 may extract the subject from the focused area in each frame image by analyzing the frame image using, for example, an image analysis library. Furthermore, for example, if the metadata of each frame image of the moving image to be analyzed includes identification information of the subject, the control unit 201 may extract the subject from the focused area in each frame image by using the identification information. Note that there may be cases where a subject does not exist in the focused area of ​​a frame image. In this case, the control unit 201 performs the subsequent processing assuming that no subject is to be extracted from the frame image. For example, if no subject exists in the focused area, a NO determination is made in S506 (described below). Furthermore, if no subject exists in the focused area, no feature information of the subject is derived (although it may be determined that no feature information of the subject does exist).

[0040] Next, in S506, the control unit 201 determines whether a specific subject is being tracked based on the subject extraction result in S505. The determination of whether a specific subject is being tracked may be a determination of whether a specific type of subject (e.g., a person) is being tracked. Alternatively, the determination of whether a specific subject is being tracked may be a determination of whether a pre-registered subject is being tracked.

[0041] When making the determination in S506, the control unit 201 may derive the feature amounts of the subject extracted in S505 and perform individual identification based on the feature amounts of the subject. In this case, the control unit 201 may issue a subject ID corresponding to the result of the individual identification. As described above, the feature amounts of the subject may be, for example, facial feature amounts or feature amounts of information that can identify the subject, such as the subject's bib or number. Furthermore, the individual identification may be performed, for example, by using an image analysis library included in an image management application or by using an image analysis library on an external server. Furthermore, the individual identification may be performed by using AI (for example, a machine learning model).

[0042] The control unit 201 may also compare a focused area of ​​a frame image with a template image to determine whether or not a subject corresponding to the template image is shown in that area. Alternatively, the control unit 201 may compare a frame image with a template image to determine whether or not a subject corresponding to the template image is shown in that frame image, regardless of whether or not the area is focused.

[0043] Here, for example, a process of dividing one angle of view movement section (scene) into a plurality of sections may be included between S505 and S506. For example, if the angle of view movement section extracted in S503 includes one or multiple consecutive frame images from which a subject was not extracted in S505, the control unit 201 may divide the angle of view movement section (scene) into multiple sections as follows. That is, the control unit 201 may treat the section of the frame image, the section of the previous frame image, and the section of the subsequent frame image as independent angle of view movement sections (scenes). In the example shown in FIG. 3, section 310 is divided into three partial sections 310a, 310b, and 310c. In this case, the angle of view movement section (section 310) is divided into three angle of view movement sections. The control unit 201 may perform this division of the angle of view movement section (scene) on all or part of one or multiple consecutive frame images from which a subject was not extracted in S505. When dividing the angle of view movement section (scene) as described above, the control unit 201 changes the information stored in S504 to information on each scene. In this case, the number of scene IDs (the number of records in the scene list 400) increases according to the number of divisions of the scene. Furthermore, in this case, the processing from S506 onward is performed not on the original angle of view movement section, but, for example, on the angle of view movement section with the first start timing among the multiple angle of view movement sections newly created by dividing the angle of view movement section. The remaining angle of view movement sections among the angle of view movement sections may be treated as unextracted angle of view movement sections and the processing of S503 may be performed on them. Note that, for the angle of view movement sections (scenes) from which no subject was extracted among the image movement sections (scenes) divided as described above, it is determined in S506 that a specific subject is not being tracked (determined as NO).

[0044] Furthermore, for example, in S506, the control unit 201 may divide one angle of view movement section (scene) into a plurality of sections based on the result of the individual discrimination described above. For example, when the angle of view shifting section extracted in S503 includes multiple different subjects (e.g., people) in the focused area, the control unit 201 may divide one angle of view shifting section (scene) into multiple sections as follows. That is, the control unit 201 may divide the angle of view shifting section extracted in S503 into multiple sections each in which the same subject is focused, and use the multiple divided sections as new angle of view shifting sections. When dividing the angle of view shifting section (scene) as described above, the control unit 201 changes the information stored in S504 to information about each scene. In this case, the number of scene IDs (the number of records in the scene list 400) increases according to the number of divided scenes. In this case, the processing from S507 onwards is not performed on the original angle of view shifting section, but on, for example, the angle of view shifting section with the earliest start timing among the multiple angle of view shifting sections newly created by dividing the angle of view shifting section. The remaining view angle movement sections among the view angle movement sections may be treated as unextracted view angle movement sections and subjected to the processing of S503.

[0045] Furthermore, if the same subject is in focus in all frame images included in the angle of view shifting section extracted in S503, it may be determined that the same subject is being tracked in that angle of view shifting section, and the processing from S507 onwards may be carried out. Furthermore, if the same subject is in focus in a predetermined percentage or more of all frame images included in the angle of view shifting section extracted in S503, it may be determined that the same subject is being tracked in that angle of view shifting section, and the processing from S507 onwards may be carried out. Furthermore, if the same subject is in focus in at least one frame in the first half of the angle of view shifting section extracted in S503 and at least one frame in the second half of the angle of view shifting section, it may be determined that the same subject is being tracked in that angle of view shifting section. For example, if the same subject is in focus in the first and last frames of the angle of view shifting section extracted in S503, it may be determined that the same subject is being tracked in that angle of view shifting section, and the processing from S507 onwards may be carried out. In this case, even if the same subject is not in focus in all of the frame images included in the angle of view movement section extracted in S503, the above-described division of the angle of view movement section does not need to be performed.

[0046] Furthermore, when the feature amounts of the individually identified subject are frequently used in an image management application, the control unit 201 may store the feature amounts as templates in the external storage device 204 or the like and reuse the templates. For example, the control unit 201 may create a template image based on the feature amounts and store it in the external storage device 204 or the like.

[0047] If the result of the determination in S506 is that a specific subject is being tracked (YES), the process of S507 is performed. In S507, the control unit 201 compares the subject ID issued for the subject as described above with the subject ID 404 for the field of view movement section (scene) immediately before the field of view movement section (scene) to be processed in the scene list 400. Based on the result of this comparison, the control unit 201 determines whether the subject to be tracked is the same in the field of view movement section (scene) to be processed and the field of view movement section (scene) immediately before the field of view movement section (scene).

[0048] As a result of the determination in S507, if the subject to be tracked is the same in the angle of view movement section (scene) to be processed and the angle of view movement section (scene) immediately before the angle of view movement section (scene) (YES), the process in S508 is performed. In S508, the control unit 201 includes, in the scene list 400, information about the angle of view movement section (scene) to be processed in the information about the angle of view movement section (scene) immediately before the angle of view movement section (scene). For example, the control unit 201 changes the information about the end position 403 for the angle of view movement section (scene) immediately before the angle of view movement section (scene) to the end timing of the angle of view movement section (scene) to be processed. In addition, the control unit 201 deletes the information (scene ID 401, start position 402, and end position 403) stored in S504 as information about the angle of view movement section (scene) to be processed.

[0049] Next, in S509, the control unit 201 determines whether or not all of the view angle movement sections (scenes) have been extracted from the video to be analyzed. If the result of this determination is that all of the view angle movement sections (scenes) have not been extracted (NO), the process of S503 described above is performed again.

[0050] As a result of the determination in S509, if all of the angle of view movement sections (scenes) have not been extracted (YES), the process of S510 is performed. In S510, the control unit 201 performs display processing, including displaying an image management screen for viewing and editing moving images on a display unit (such as a computer display) and performing processing and changing the display in accordance with operations on the image management screen. The image management screen may be displayed on a terminal device (such as the PC 104, 105, or the mobile device 106) or on the information processing device 101 (the display unit 211). When the image management screen is displayed on a terminal device (such as the PC 104, 105, or the mobile device 106), information on the image management screen may be transmitted (output) from the information processing device 101 to the terminal device. Information indicating the content of operations on the image management screen may also be transmitted (output) from the terminal device to the information processing device 101. The image management screen may be displayed when a display instruction is received from a user, or may be automatically displayed by the control unit 201. A specific example of image management will be described later with reference to FIG. 6. When the process of S510 is completed, the process according to the flowchart of Fig. 5 is completed. Note that the process of S510 may be performed as a process separate from the flowchart of Fig. 5 (i.e., the process of S510 may not be included in the flowchart of Fig. 5).

[0051] As a result of the determination in S507 described above, if the subject to be tracked is different between the view angle movement section (scene) to be processed and the view angle movement section (scene) immediately before the current view angle movement section (scene) (NO), the process in S510 is performed. In S510, the control unit 201 stores the subject ID 404 as information for the view angle movement section (scene) to be processed, in addition to the information stored in S504 (scene ID 401, start position 402, and end position 403). The information stored as the subject ID 404 may be the subject ID issued in the process in S506, or may be information linked to the feature amount described above. When the process in S510 ends, the process according to the flowchart in FIG. 5 ends.

[0052] If the result of the determination in S506 described above is that a specific subject is not being tracked (NO), the process in S512 is performed. In S512, the control unit 201 determines whether or not the subject to be tracked (e.g., the subject in the focused area) has changed between the angle of view movement sections (scenes) one before and one after the angle of view movement section (scene) to be processed. Note that, in order to perform the determination in S512, for example, the control unit 201 may extract the subject to be tracked and perform individual identification based on the feature amount of the subject for the angle of view movement section (scene) one after the angle of view movement section (scene) to be processed. The extraction of the subject to be tracked and the individual identification based on the feature amount of the subject may be performed, for example, as described in S505 and S506. Furthermore, the processes of S503, S504, S505, and S506 up to individual identification may be performed for all of the angle of view movement sections (scenes) included in the video, and then the processes of S507 and S512 may be performed. In this case, the processes of S506 to S512 may be repeated based on the result of the determination of S509. Furthermore, the determination of S512 may be performed if a process of dividing the angle of view movement section (scene) into multiple sections is performed between S505 and S506.

[0053] If the result of the determination in S512 is that the subject to be tracked has not changed in the angle of view movement sections (scenes) immediately before and after the angle of view movement section (scene) to be processed (NO), the control unit 201 determines that the angle of view movement section (scene) to be processed is not an unnecessary scene. In this case, the processing of S509 described above is performed. In this case, the control unit 201 may store "FALSE" as the scene necessary / unnecessary 405 as information for the angle of view movement section (scene) to be processed, in addition to the information (scene ID 401, start position 402, and end position 403) stored in S504.

[0054] Also, if the result of S512 is NO, the control unit 201 may treat the three angle of view movement sections (scene) to be processed, the angle of view movement section (scene) immediately before it, and the angle of view movement section (scene) immediately after it as one angle of view movement section (scene).

[0055] On the other hand, if the result of the determination in S512 is that the subject to be tracked has changed in the angle of view movement sections (scenes) one before and one after the angle of view movement section (scene) to be processed, the process of S513 is performed. In S513, the control unit 201 stores "TRUE" as the scene necessity 405 as information for the angle of view movement section (scene) to be processed, in addition to the information stored in S504 (scene ID 401, start position 402, and end position 403). Then, the process of S509 described above is performed.

[0056] Here, for example, the following determination may be made between S512 and S513. That is, the control unit 201 determines whether the difference between the direction in which the angle of view moves in the view angle movement section (scene) to be processed and the direction in which the angle of view moves in the view angle movement sections (scenes) immediately before and immediately after the view angle movement section to be processed is within a predetermined range. When making this determination, the control unit 201 may use, for example, only the view angle movement section (scene) immediately before the view angle movement section to be processed, instead of the view angle movement sections (scenes) immediately before and immediately after the view angle movement section to be processed. Furthermore, instead of determining whether the difference is within a predetermined range, the control unit 201 may determine whether there is a difference. If the result of this determination shows that the difference in the direction in which the angle of view moves is not within the predetermined range, the view angle movement section (scene) to be processed is deemed to be an unnecessary scene, and the processing of S513 may be performed. On the other hand, if the difference in the direction in which the angle of view moves is within a predetermined range, the angle of view movement section (scene) to be processed is determined not to be an unnecessary scene, and the process of S509 may be performed without performing the process of S513. Alternatively, it may be determined whether the difference in the direction in which the angle of view moves is within a predetermined range without performing the process of S512. In this case, for example, when the control unit 201 determines NO in S506, it may determine whether the difference in the direction in which the angle of view moves is within a predetermined range instead of the determination of S512.

[0057] [Image management screen example] Fig. 6 is a diagram showing an example of an image management screen 600. The image management screen 600 is a screen for viewing and editing moving images managed by, for example, an image management application. The image management screen 600 can clearly present to the user scenes in which a subject is being tracked and scenes in which the subject is not being tracked in moving images. As described above, the display process for the image management screen 600 shown in Fig. 6 is performed, for example, at S510 in Fig. 5.

[0058] 4, for example, the control unit 201 creates information necessary for displaying the image management screen 600, and causes the display unit to display the image management screen 600. The display unit may be the display unit 211, or a display unit of a terminal device (such as the PCs 104 and 105 or the mobile device 106). When the image management screen 600 is displayed on the display unit 211, the control unit 201 may cause the VRAM 207 to create the image management screen 600.

[0059] In FIG. 6, a display area 601 is an area where the moving image to be processed is played back and displayed. A display area 602 is an area where metadata information of the moving image file to be processed is displayed. An area 603 is an area that is in focus in a frame image of the moving image being processed that is being played back. Here, an example is shown in which the subject 302 shown in FIG.

[0060] The thumbnail list 604 is a list of thumbnails of individual frame images included in the video. The subject list 605 is a list of identification information (subject ID) of subjects that are in focus in the video. The information displayed in the subject list 605 is created, for example, based on the information stored in the subject ID 404 of the scene list 400. Fig. 6 illustrates an example in which the identification information of subjects 302, 303, 304, and 305 shown in Fig. 3 are A, B, C, and D, respectively.

[0061] The tracking section 610 is a section (scene) of moving angle of view during which tracking image capture is being performed on the subject 302 (subject A). FIG. 6 illustrates an example in which the tracking section 610 is displayed using a double-headed arrow. The tracking section 610 indicates that the thumbnail displayed above the double-headed arrow indicated as the tracking section 610 is a thumbnail of a scene in which the subject 302 is being tracked. The range of the tracking section 610 is derived based on the start position 402 and end position 403 associated with the subject ID 404 of the scene in the scene list 400. The tracking section 610 corresponds to the section 310 shown in FIG. 3.

[0062] In the example shown in FIG. 3, subject 302 (subject A) is tracked from start position 307 to goal position 308. Thereafter, while subject 302 moves from goal position 308 to post-race waiting area 309, the focus is temporarily interrupted. FIG. 6 illustrates an example in which this is displayed midway through tracking interval 610 as tracking interruption interval 611. Tracking interruption interval 611 corresponds to partial interval 310b shown in FIG. 3. The subject in focus in the interval immediately before and the interval immediately after tracking interruption interval 611 is subject 302 (subject A). In this case, as described with reference to the flowchart in FIG. 5, tracking interruption interval 611 is determined not to be an unnecessary scene (determined as NO in S511). Therefore, the control unit 201 displays the tracking interruption section 611 and the sections immediately before and after the tracking interruption section 611 as one section (a scene in which the subject 302 is being tracked).

[0063] The non-tracking section 612 is a section in a moving image where tracking imaging is not performed (a section in which the angle of view of the imaging device 107 is continuously moving without focusing on a specific subject). FIG. 6 illustrates an example in which the non-tracking section 612 is displayed as a black rectangle. The non-tracking section 612 is derived based on the scene necessity 405 in the scene list 400. Specifically, the range of the non-tracking section 612 is derived based on the start position 402 and end position 403 of a record (scene) for which the scene necessity 405 is "TRUE". The non-tracking section 612 corresponds to the section 311 shown in FIG. 3.

[0064] 3, after subject 302 (subject A) is tracked, subject 303 (subject B) is tracked in the next scene. In this case, the subject changes between tracking section 610 immediately before non-tracking section 612 and tracking section 613 immediately after non-tracking section 612. Therefore, as described with reference to the flowchart in FIG. 5, non-tracking section 612 is determined to be an unnecessary scene (YES is determined in S511).

[0065] 6, tracking section 613 is a section (scene) in which tracking image capture is performed on subject 303 (subject B). The range of tracking section 613 is derived based on start position 402 and end position 403 linked to subject ID 404 of the scene in scene list 400. Tracking section 613 corresponds to section 312 shown in FIG.

[0066] 3, subject 303 (subject B) is tracked from start position 307 to goal position 308. There is no subject in focus in the sections immediately before and after tracking section 613. For this reason, it is determined that the subject has not changed in the sections immediately before and after tracking section 613 (NO is determined in S511). Therefore, the series of scenes in which subject 303 is being tracked are not deemed to be unnecessary scenes.

[0067] The non-tracking section 614 is a section in the video where tracking imaging is not performed. The non-tracking section 614 is derived based on the scene necessity 405 in the scene list 400. Specifically, the range of the non-tracking section 614 is derived based on the start position 402 and end position 403 of a record (scene) for which the scene necessity 405 is "TRUE". The non-tracking section 614 corresponds to the section 313 shown in FIG. 3.

[0068] 3, after subject 303 (subject B) is tracked, subject 304 (subject C) is tracked in the next scene. In this case, the subject changes between tracking section 613 immediately before non-tracking section 614 and tracking section 615 immediately after non-tracking section 614. Therefore, as described with reference to the flowchart in FIG. 5, non-tracking section 614 is determined to be an unnecessary scene (YES is determined in S511).

[0069] Similarly, tracking intervals 615 and 617 are intervals in which tracking image capture is performed on subjects 304 (subject C) and 305 (subject D), respectively. Non-tracking interval 616 is an interval in which tracking image capture is not performed in the moving image. Since the subject in non-tracking interval 616 is different between the previous tracking interval 615 and the subsequent tracking interval 617, it is determined to be an unnecessary scene.

[0070] In this embodiment, an example is shown in which delete buttons 620a to 620c are displayed on the image management screen 600 shown in Fig. 6. Fig. 6 illustrates an example in which delete buttons 620a to 620c are displayed for each unnecessary scene. Specifically, Fig. 6 illustrates an example in which delete buttons 620a, 620b, and 620c are GUIs operated by the user when deleting non-tracking sections 612, 614, and 616, respectively. When delete button 620a, 620b, or 620c is pressed, control unit 201 deletes frame images of the angle of view change section (non-tracking section) corresponding to the delete button from the video. Control unit 201 may also delete information about the angle of view change section (non-tracking section) from the scene list 400. Using delete buttons 620a to 620c makes it easy to edit video.

[0071] Furthermore, by deleting the non-tracking sections 612, 614, and 616 from the video, it is possible to continuously present to the user a video in which the subject in focus changes while the subject remains in focus.

[0072] For example, the control unit 201 may automatically delete unnecessary scenes from the video. In this case, the control unit 201 may cause the image management screen 600 to display information indicating that the unnecessary scenes have been deleted from the video and information identifying the section in which the deleted unnecessary scenes existed. The control unit 201 may also cause the image management screen 600 to display a GUI that is operated by the user to return the deleted unnecessary scenes to the video. The control unit 201 may return the deleted unnecessary scenes to the video based on the operation of this GUI.

[0073] As described above, this embodiment illustrates a case where a processing device is realized by using the information processing device 101. Also, in the example shown in Fig. 3 as described above, the information of scene necessity 405 in the section where scene ID 401 is 310, 312 (tracking sections 610, 612) shown in Fig. 4 is an example of first section information. Also, the information of scene necessity 405 in the section where scene ID 401 is 311, 313 (tracking interruption sections 611, 613) shown in Fig. 4 is an example of second section information. Also, the delete buttons 609a to 609c are an example of operation information.

[0074] As described above, in this embodiment, the information processing device 101 extracts sections in which the angle of view of the imaging device 107 continuously changes from a video captured by the imaging device 107, and derives information (e.g., a subject ID) indicating content corresponding to the characteristics of the subject in the section. Based on the extracted information, the information processing device 101 derives information for identifying sections included in the video as sections in which the subject is being tracked, or information for identifying sections included in the video as sections in which the subject is not being tracked. Therefore, while it was previously impossible to distinguish individual sections (scenes) by using individual subject identification, this embodiment can associate individual sections with the characteristics of the subject. This improves the accuracy of determining whether a scene is tracking a subject in the imaging device. As a result, for example, the information processing device 101 can accurately determine, as a single scene, scenes in which the same subject is being tracked in a video, even if tracking of the subject is temporarily interrupted. Furthermore, for example, the information processing device 101 can automatically search for and filter out unnecessary scenes between scenes in which the subject is being tracked. In this case, the time and effort required for editing the moving image can be reduced.

[0075] Furthermore, in this embodiment, the information processing device 101 displays on the display unit an image management screen 600 that includes, for example, non-tracking sections 612, 614, and 616 as information for identifying sections included in a moving image as sections in which a subject is not being tracked. Furthermore, the information processing device 101 displays delete buttons 620a to 620c for deleting the non-tracking sections 612, 614, and 616 on the image management screen 600. This reduces the effort required for deleting unnecessary scenes.

[0076] Furthermore, in this embodiment, when the information indicating the content according to the characteristics of the subject in a section in which the angle of view continuously changes is the same as the information indicating the content according to the characteristics of the subject in the section immediately preceding the section, the information processing device 101 treats the section as one section, and therefore can automatically group together scenes of the same subject.

[0077] Furthermore, in this embodiment, when the information processing device 101 is unable to derive information indicating content corresponding to the characteristics of the subject in a section where the angle of view continuously changes, it refers to information indicating content corresponding to the characteristics of the subject in sections before and after the section. Then, based on the referred information, the information processing device 101 determines whether the section where information indicating content corresponding to the characteristics of the subject could not be derived is a section where the subject is being tracked. Therefore, it is possible to more accurately determine whether the section where information indicating content corresponding to the characteristics of the subject could not be derived is simply a section where the tracking of the subject is temporarily interrupted, but is part of a section where the tracking of the subject is continuing.

[0078] Furthermore, in this embodiment, when the subject being tracked changes in one section extracted as a section in which the angle of view moves continuously, the information processing device 101 divides the section into multiple sections. Therefore, a section set as a section in which the angle of view moves continuously can be automatically divided into sections for each subject being tracked.

[0079] In this embodiment, the information processing device 101 derives identification information of a subject based on the feature amount of the subject in a section where the angle of view changes continuously. Therefore, the user can easily identify the subject to be tracked in each section where the angle of view changes continuously.

[0080] (Other Examples) The present disclosure can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0081] Furthermore, a software program that realizes the functions of the above-described embodiments may be supplied to a system or device having a computer that can execute the program directly from a recording medium or via wired or wireless communication. In this case, the present disclosure also includes the case where the program is executed on the system or device.

[0082] Therefore, the program code itself that is supplied to and installed on a computer to realize the functional processing of the present disclosure also realizes the present disclosure, that is, the computer program itself for realizing the functional processing of the present disclosure is also included in the present disclosure.

[0083] In this case, as long as the program has the functionality, the form of the program is not important, such as object code, a program executed by an interpreter, or script data supplied to the OS. The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as a magnetic tape, an optical / magneto-optical storage medium, or a non-volatile semiconductor memory.

[0084] Another possible method of supplying the program is to store the computer program forming the present disclosure in a server on a computer network, and have a connected client computer download and execute the computer program.

[0085] Furthermore, the above-described embodiments of the present disclosure are merely examples of specific embodiments for carrying out the present disclosure, and the technical scope of the present disclosure should not be interpreted as being limited by these. In other words, the present disclosure can be embodied in various forms without departing from its technical concept or main features.

[0086] The disclosure of the present embodiment also includes, for example, the following configurations and methods. (Configuration 1) an extraction means for extracting a section in which the angle of view of the imaging device moves continuously from a captured moving image; a first derivation means for deriving subject information indicating content according to characteristics of the subject in the section extracted by the extraction means; a second derivation means for deriving, based on the subject information derived by the first derivation means, first section information for specifying that the section included in the video is a section where a subject is being tracked, or second section information for specifying that the section included in the video is a section where a subject is not being tracked; A processing device comprising: (Configuration 2) 2. The processing device according to configuration 1, further comprising output means for outputting the second section information. (Configuration 3) a deletion means for deleting a section of the video image identified by the second section information from the video image; the output means displays, on a display means, the second section information and operation information to be operated to delete the video in the section specified by the second section information; 3. The processing device according to configuration 2, wherein the deletion means deletes the video in the section specified by the second section information based on the operation on the operation information. (Configuration 4) The processing device described in any one of configurations 1 to 3, characterized in that the second derivation means derives the first section information or the second section information for at least one of the sections in which the angle of view of the imaging device moves continuously and the section before and after the section based on the subject information for the section in which the angle of view of the imaging device moves continuously and the subject information for at least one of the sections before and after the section in question. (Configuration 5) The processing device described in any one of configurations 1 to 4, characterized in that when the subject information in a section in which the angle of view of the imaging device continuously moves is the same as the subject information in the section immediately preceding the section, the processing device derives the first section information or the second section information in the one section by treating the section and the section immediately preceding the section as one section. (Configuration 6) a determination unit that determines whether or not the section is a section in which a subject is being tracked, based on the subject information in the section extracted by the extraction unit; The processing device of any one of configurations 1 to 5, wherein the second derivation means derives the first section information when the determination means determines that the section is a section in which a subject is being tracked, and creates the second section information when the determination means determines that the section is a section in which a subject is not being tracked. (Configuration 7) The processing device described in configuration 6, characterized in that when the subject information is not derived in a section where the angle of view of the imaging device moves continuously, the determination means determines whether the section where the subject information is not derived is a section where the subject is being tracked, based on the subject information in the sections before and after the section where the subject information is not derived. (Configuration 8) The processing device according to any one of configurations 1 to 7, further comprising a dividing means for dividing a section into a plurality of sections when the subject to be tracked changes in one section in which the angle of view of the imaging device moves continuously, based on the subject information derived by the first derivation means. (Configuration 9) The processing device according to any one of configurations 1 to 8, wherein the first derivation means derives identification information of the subject as the subject information based on the feature amount of the subject in the section extracted by the extraction means. (Configuration 10) 10. The processing device according to any one of configurations 1 to 9, wherein the subject information is information indicating content according to characteristics of the subject in an area including a position on which focus is placed in the moving image. (Method 1) an extraction step of extracting a section in which the angle of view of the imaging device moves continuously from the captured moving image; a first derivation step of deriving subject information indicating content according to the feature amount of the subject in the section extracted by the extraction step; a second derivation step of deriving, based on the subject information derived in the first derivation step, first section information for specifying that the section included in the video is a section where a subject is being tracked, and second section information for specifying that the section included in the video is a section where a subject is not being tracked; A processing method comprising the steps of: (Program 1) 11. A program for causing a computer to function as each means of the processing device according to any one of configurations 1 to 10. [Explanation of symbols]

[0087] 101: Information processing device, 103: Network, 104-105: PC, 107: Imaging device, 110: User

Claims

1. an extraction means for extracting a section in which the angle of view of the imaging device moves continuously from a captured moving image; a first derivation means for deriving subject information indicating content according to characteristics of the subject in the section extracted by the extraction means; a second derivation means for deriving, based on the subject information derived by the first derivation means, first section information for specifying that the section included in the moving image is a section where a subject is being tracked, or second section information for specifying that the section included in the moving image is a section where a subject is not being tracked; A processing device comprising:

2. 2. The processing device according to claim 1, further comprising output means for outputting the second section information.

3. a deletion means for deleting a section of the video image specified by the second section information from the video image; the output means displays, on a display means, the second section information and operation information to be operated to delete the video in the section specified by the second section information; The processing device according to claim 2 , wherein the deletion means deletes the video in the section specified by the second section information based on the operation in response to the operation information.

4. The processing device described in any one of claims 1 to 3, characterized in that the second derivation means derives the first section information or the second section information for at least one of the sections in which the angle of view of the imaging device moves continuously and the section before and after the section based on the subject information for the section in which the angle of view of the imaging device moves continuously and the subject information for at least one of the sections before and after the section in question.

5. The processing device described in any one of claims 1 to 3, characterized in that when the subject information in a section in which the angle of view of the imaging device continuously moves is the same as the subject information in the section immediately preceding the section in question, the second derivation means derives the first section information or the second section information for the one section by treating the section in question and the section immediately preceding the section in question as one section.

6. a determination unit that determines whether or not the section is a section in which a subject is being tracked, based on the subject information in the section extracted by the extraction unit; The processing device according to any one of claims 1 to 3, characterized in that the second derivation means derives the first section information when the determination means determines that the section is a section in which a subject is being tracked, and creates the second section information when the determination means determines that the section is a section in which a subject is not being tracked.

7. The processing device described in claim 6, characterized in that when the subject information is not derived in a section in which the angle of view of the imaging device moves continuously, the determination means determines whether the section in which the subject information is not derived is a section in which the subject is being tracked, based on the subject information in the sections before and after the section in which the subject information is not derived.

8. The processing device according to any one of claims 1 to 3, characterized in that it has a dividing means for dividing a section into a plurality of sections when the subject to be tracked changes within a section in which the angle of view of the imaging device moves continuously, based on the subject information derived by the first derivation means.

9. The processing device according to any one of claims 1 to 3, characterized in that the first derivation means derives identification information of the subject as the subject information based on the feature amount of the subject in the section extracted by the extraction means.

10. 4. The processing device according to claim 1, wherein the subject information is information indicating content according to characteristics of the subject in an area including a position where focus is placed in the moving image.

11. an extraction step of extracting a section in which the angle of view of the imaging device moves continuously from the captured moving image; a first derivation step of deriving subject information indicating content according to feature amounts of the subject in the section extracted by the extraction step; a second derivation step of deriving, based on the subject information derived in the first derivation step, first section information for specifying that the section included in the moving image is a section where a subject is being tracked, and second section information for specifying that the section included in the moving image is a section where a subject is not being tracked; A processing method comprising the steps of:

12. A program for causing a computer to function as each of the means of the processing device according to any one of claims 1 to 3.

Images