Image processing device, image processing system and image processing method

Abstract

Provided are an apparatus and method for calculating a predicted motion vector corresponding to the type and posture of a tracking subject and generating a camera control signal required for capturing an image of the tracking subject. The apparatus and method include a subject predicted motion vector calculation unit that detects a tracking subject of a pre-specified type in a captured image input from an imaging unit and calculates a predicted motion vector corresponding to the type and posture of the detected tracking subject, and a camera control signal generation unit that generates a camera control signal for capturing a tracking image of the tracking subject based on the predicted motion vector calculated by the subject predicted motion vector calculation unit. The subject predicted motion vector calculation unit performs processing for detecting a tracking subject of a type specified by a user from a captured image and predicted motion vector calculation processing by using a neural network or the like.

Description

Technical Field

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. More specifically, the present disclosure relates to an image processing device, an image processing system, and an image processing method that perform subject tracking and image capture. Background Art

[0002] When capturing an image of a moving subject by using an imaging device (camera), an automatic tracking imaging device capable of capturing a moving image while tracking the subject is used.

[0003] For example, the imaging device is attached to a camera pan-tilt device capable of freely setting an image capturing direction, and the camera pan-tilt device is driven to capture images following the movement of the subject.

[0004] Note that an automatic tracking imaging apparatus is described in, for example, Patent Document 1 (Japanese Patent Application Laid-Open No. 2010-154391) or the like.

[0005] With many automatic tracking imaging devices, in the case of capturing a tracking image (moving image) of a subject, a user (photographer) first performs a process of selecting a subject as a tracking target.

[0006] For example, before starting image capture with a camera, the user checks a live view image (through image) displayed on a display unit of the camera, and selects a tracking target object.

[0007] Specifically, for example, the user selects a subject as a tracking target in a live view image displayed on a touch panel type display screen, and sets a frame of an image area surrounding the selected subject with a finger. By regarding the image in the framed area as a template image, the control unit of the automatic tracking imaging device drives the camera pan / tilt to follow the image area similar to the template image. Through such processing, a tracking image is captured.

[0008] However, if the image area of ​​the tracking target subject cannot be correctly set, it is difficult to capture an image while following the tracking target subject.

[0009] For example, when the camera is not fixed, such as when a user (photographer) holds the camera by hand and attempts to capture a tracking image of the "bird" while following the "bird" as a subject, it is difficult for the user to draw a correct frame.

[0010] Therefore, when both the subject and the photographer are moving, it is extremely difficult to draw an accurate frame around the subject.

[0011] Citation List

[0012] Patent Literature

[0013] Patent Document 1: Japanese Patent Application Publication No. 2010-154391 Summary of the invention

[0014] Problems to be solved by the present invention

[0015] The present disclosure is produced, for example, in view of the above-mentioned problems, and an object of the present disclosure is to provide an image processing device, an image processing system and an image processing method that are capable of capturing a tracking image of a tracking object as an object without the user (photographer) performing processing such as border setting on a display screen.

[0016] Solution to the problem

[0017] A first aspect of the present disclosure is an image processing device comprising:

[0018] a subject prediction motion vector calculation unit that detects a tracking subject corresponding to a pre-specified type from a captured image and calculates a prediction motion vector corresponding to the type and posture of the detected tracking subject, and

[0019] A camera control signal generating unit generates a camera control signal for tracking the tracking object based on the predicted motion vector.

[0020] In addition, a second aspect of the present disclosure is an image processing system comprising

[0021] A camera mounted on a camera pan head, and a camera pan head control unit for controlling the camera pan head, wherein

[0022] The camera has

[0023] a subject prediction motion vector calculation unit that detects a tracking subject corresponding to a pre-specified type from a captured image and calculates a prediction motion vector corresponding to the type and posture of the detected tracking subject, and

[0024] a camera control signal generating unit, the camera control signal generating unit generating a camera control signal for tracking the tracking object based on the predicted motion vector, and

[0025] The camera pan tilt control unit performs camera pan tilt control for capturing an image of a tracking image of a tracking object based on a camera control signal.

[0026] In addition, a third aspect of the present disclosure is an image processing method executed in an image processing device,

[0027] wherein the subject prediction motion vector calculation unit performs a subject prediction motion vector calculation step of detecting a tracking subject corresponding to a pre-specified type from a captured image and calculating a prediction motion vector corresponding to the type and posture of the detected tracking subject, and

[0028] The camera control signal generating unit performs a camera control signal generating step of generating a camera control signal for tracking the tracking object based on the predicted motion vector.

[0029] Other objects, features and advantages of the present disclosure will become apparent from a more detailed description based on the embodiments of the present disclosure and the accompanying drawings described below. Note that in this specification, a system is a logical collection of multiple devices, and is not limited to a system in which various devices are in the same housing.

[0030] According to the configuration of the embodiment of the present disclosure, an apparatus and method are realized for calculating a predicted motion vector corresponding to the type and posture of a tracking object and generating a camera control signal required for capturing an image of the tracking object.

[0031] Specifically, for example, it includes a subject prediction motion vector calculation unit that detects a tracking subject of a pre-specified type in a captured image input from an imaging unit and calculates a prediction motion vector corresponding to the type and posture of the detected tracking subject, and a camera control signal generation unit that generates a camera control signal for capturing a tracking image of the tracking subject based on the prediction motion vector calculated by the subject prediction motion vector calculation unit. The subject prediction motion vector calculation unit performs processing for detecting a tracking subject of a type specified by a user from a captured image and prediction motion vector calculation processing by using a neural network or the like.

[0032] With this configuration, an apparatus and method are realized that calculates a predicted motion vector corresponding to the type and posture of a tracking object and generates a camera control signal required to capture an image of the tracking object.

[0033] Note that the effects described herein are merely examples and are not limited thereto, and there may be additional effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 : is a diagram illustrating a configuration example of an automatic tracking imaging device as an example of an image processing device according to the present disclosure.

[0035] Figure 2 2 is a diagram for explaining tracking subject type selection processing.

[0036] Figure 31 is a diagram for explaining automatic tracking mode setting and image capturing start processing.

[0037] Figure 4 2 is a diagram for explaining tracking subject type selection processing using a communication terminal.

[0038] Figure 5 1 is a diagram for explaining automatic tracking mode setting and image capturing start processing using a communication terminal.

[0039] Figure 6 is a flowchart illustrating a processing sequence executed by the image processing apparatus according to the present disclosure.

[0040] Figure 7 is a flowchart illustrating a processing sequence executed by the image processing apparatus according to the present disclosure.

[0041] Figure 8 1 is a diagram for explaining a specific example of tracking subject detection processing.

[0042] Fig. 9 1 is a diagram for explaining a specific example of tracking subject detection processing.

[0043] Fig.10 It is a diagram for explaining a specific example of a predicted motion vector for tracking a subject.

[0044] Fig.11 It is a diagram for explaining a specific example of a predicted motion vector for tracking a subject.

[0045] Fig.12 : is a diagram illustrating a configuration example of an image processing apparatus according to the present disclosure.

[0046] Fig.13 is a diagram explaining a specific example of the process performed by the subject prediction motion vector generation unit 110 of the image processing apparatus according to the present disclosure.

[0047] Fig.14 is a diagram explaining a specific example of the process performed by the subject prediction motion vector generation unit 110 of the image processing apparatus according to the present disclosure.

[0048] Fig.15 : is a diagram explaining a specific example of processing performed by the camera control signal generating unit of the image processing apparatus according to the present disclosure. DETAILED DESCRIPTION

[0049] Hereinafter, details of the image processing apparatus, the image processing system, and the image processing method according to the present disclosure will be described with reference to the drawings. Note that the description will be made according to the following items.

[0050] 1. Overview of the Configuration and Processing of the Image Processing Device According to the Present Disclosure

[0051] 2. Sequence of Processing Executed by the Image Processing Device According to the Present Disclosure

[0052] 3. Image Processing Device Configuration and Details of Subject Prediction Motion Vector Generation Processing

[0053] 4. Conclusion of this disclosure

[0054] [1. Overview of the Configuration and Processing of the Image Processing Device According to the Present Disclosure]

[0055] First, an overview of the configuration and processing of an image processing apparatus according to the present disclosure is described.

[0056] Figure 1 is a diagram illustrating a configuration example of an automatic tracking imaging device 10 as an example of an image processing device according to the present disclosure.

[0057] The automatic tracking imaging device 10 has a camera 11 , a camera platform 12 , and a camera platform control unit 13 .

[0058] The camera 11 is mounted on a camera platform 12. The camera 11 is configured so that the image capturing direction of the camera 11 can be freely changed on the camera platform 12.

[0059] The camera platform control unit 13 drives and controls the camera platform 12 so that the image capturing direction of the camera 11 faces a preset direction of tracking the subject.

[0060] Note that in order to capture an image of a tracking subject, it is necessary to detect the tracking subject from an image captured by a camera, analyze the direction of the detected tracking subject, and continuously perform processing to make the image capturing direction of the camera consistent with the analyzed direction.

[0061] This process requires, for example, the following process.

[0062] (1) a process of detecting a tracking object from an image captured by the camera 11,

[0063] (2) a process of calculating the direction of a camera for capturing an image of a detected tracking subject,

[0064] (3) a process for generating a camera pan / tilt drive control signal for driving the camera along the calculated camera direction,

[0065] (4) Camera pan / tilt drive processing based on the generated camera pan / tilt drive control signal.

[0066] These processes (1) to (4) are executed by the camera 11 or the camera pan / tilt control unit 13 .

[0067] For example, all of the above processes (1) to (4) may be performed in the camera pan / tilt control unit 13 , or the above processes (1) to (3) may be performed in the camera 11 , and only process (4) may be performed in the camera pan / tilt control unit 13 .

[0068] In the case where all of the above-described processes (1) to (4) are executed in the camera pan / tilt control unit 13 , an image captured by the camera 11 is input to the camera pan / tilt control unit 13 via the signal line 14 .

[0069] The camera pan / tilt control unit 13 executes the above-described processes (1) to (4) using the captured images.

[0070] On the other hand, when the above-mentioned processes (1) to (3) are performed in the camera 11 and only the process (4) is performed in the camera pan / tilt control unit 13, the camera pan / tilt drive control signal generated by the camera 11 is input to the camera pan / tilt control unit 13 via the signal line 14.

[0071] The camera pan control unit 13 performs the above-mentioned process (4) using the camera pan drive control signal generated by the camera 11 .

[0072] Note that in addition to the above, it may be set differently whether the camera 11 or the camera pan / tilt control unit 13 executes the above-mentioned processes (1) to (4).

[0073] Furthermore, an external device such as an external communication terminal or an external server (eg, a server on the cloud) may perform the above-described processes (1) to (3) while communicating with the camera 11 or the camera pan / tilt control unit 13 .

[0074] As described above, with many conventional automatic tracking imaging devices, in the case of capturing a tracking image (moving image) of a subject, a user (photographer) first performs a process of selecting a subject as a tracking target.

[0075] For example, the user selects a subject to be tracked in a live view image displayed on a touch panel type camera display screen, and sets a frame of an image area surrounding the selected subject with a finger.

[0076] Note that the subject selection process is not limited to the operation on the touch panel type camera display screen, and may be performed by using a physical operation unit such as an operation button.

[0077] However, as in Figure 1As in the use case (1) shown in the upper left corner illustration of , for example, when the camera is not fixed, such as when the user (photographer) holds the camera and attempts to capture a tracking image of the "bird" while following the "bird" as the subject, it is difficult for the user to draw the correct border.

[0078] Therefore, in a situation where both the subject and the photographer are moving, it is extremely difficult to draw an accurate frame around the subject.

[0079] The image processing apparatus according to the present disclosure solves this problem, for example, and can capture a tracking image of a target subject without the user (photographer) performing processing to set a frame around the tracking subject area.

[0080] In addition, many conventional methods including the above-mentioned patent document 1 (Japanese Patent Application Publication No. 2010-154391) are configured to predict the moving position of the subject by using multi-frame information, so that in order to improve the prediction accuracy, the occurrence of delay time is inevitable. Therefore, it is possible that the subject has left the screen before capturing a high-speed subject and starting tracking.

[0081] In contrast, the method according to the present disclosure is a method capable of capturing an image of a tracking object based on a result of estimating the posture of the tracking target object in one frame without using multi-frame information, thereby making it possible to capture an image of the tracking object without delay time.

[0082] Next, we will refer to Figure 2 and subsequent figures explain the overview of processing performed by the image processing apparatus according to the present disclosure.

[0083] Figure 2 and 3 is a description for using Figure 1 1 is an illustration of a processing step of capturing a moving image as a tracking image of a specific subject, the automatic tracking imaging device 10 being an example of an image processing device according to the present disclosure.

[0084] First, if Figure 2 As illustrated in FIG. 2 , by using the tracking subject type selection UI 16 displayed on the camera display unit 15 , the user (photographer) selects the type of subject to be tracked.

[0085] The UI illustrated in the figure displays icons that allow selection of the following subject types.

[0086] (1) People

[0087] (2) Dog

[0088] (3) Car

[0089] (4)Cat

[0090] (5) Bird

[0091] (6) Ball

[0092] Note that in addition to these subject types, there are other subject types that the user can select, and by sliding the screen left and right or up and down, the user can display icons indicating various other selectable subject types.

[0093] Figure 2 The diagram illustrates an example in which, in the process of (step S11 ), the user (photographer) selects “(5) Bird” as the tracking subject type.

[0094] The tracking subject type information selected by the user (photographer) is input to the data processing unit of the image processing device (automatic tracking imaging device 10 ).

[0095] Figure 3 The diagram illustrates the next processing steps.

[0096] Figure 3 : is a diagram illustrating a state in which an image is displayed on the camera display unit 15. The displayed image is a current image before starting video recording, a so-called live view image (LV image).

[0097] While viewing the live view image (LV image), the user first touches the automatic tracking (auto lock on) mode setting icon 17 at an arbitrary timing (step S12 ).

[0098] The touch operation of the automatic tracking (auto lock on) mode setting icon 17 is input to the data processing unit of the image processing device (automatic tracking imaging device 10), thereby starting the automatic tracking process. Note that the automatic tracking target subject is the tracking subject type = "bird" specified in the above (step S11).

[0099] Note that although Figure 3 In the example shown in the diagram, the image of a bird is captured on the live view image (LV image); however, it is not necessary to capture the image of the bird on the live view image (LV image) at this time; the automatic tracking (auto lock on) mode setting icon 17 may also be touched on the screen without capturing the image of the bird.

[0100] Next, in (step S13), the user (photographer) touches the image capturing (video recording) start instruction icon 18. This touch operation is input to the data processing unit of the image processing device (automatic tracking imaging device 10), thereby starting image capturing (video recording).

[0101] Note that the process of touching the image capturing (video recording) start instruction icon 18 can be performed at any timing desired by the user (photographer). Figure 3 In the example shown in the diagram, an image of a bird is captured on the live view image (LV image), but video recording can also be started by touching the image capturing (video recording) start instruction icon 18 on the screen without capturing the image of the bird.

[0102] By touching the process of the image capturing (video recording) start instruction icon 18, image capturing (video recording) is started.

[0103] The image capture process is performed in the automatic tracking mode, and image capture is performed so as to track the tracking subject type = "bird" specified in the above (step S11). That is, the camera pan control of the image capture direction of the mobile camera 11 is performed so as to track the tracking subject type = "bird", thereby performing image capture.

[0104] As reference Figure 2 and 3 As described above, the processing performed by the user (photographer) is the following processing.

[0105] (S11) Processing of selecting tracking subject type

[0106] (S12) Processing of setting the automatic tracking mode to on

[0107] (S13) Processing of Image Capture (Video Recording) Start Instruction

[0108] Only through these simple operations, it is possible to start capturing a tracking image of a subject of a type specified (selected) by the user.

[0109] That is, with the configuration according to the present disclosure, the user (photographer) does not need to perform the troublesome process of framing the tracking target object on the screen.

[0110] Furthermore, the method according to the present disclosure is a method capable of capturing an image of a tracking object based on a result of estimating the posture of the tracking target object in one frame without using multi-frame information, thereby enabling image capture of the tracking object without delay time.

[0111] Note that the processing performed by the image processing apparatus according to the present disclosure based on the user operations of ( S11 ) to ( S13 ) described above will be described in detail later.

[0112] Note that the reference Figure 4 and 5 Another specific example of the image processing apparatus according to the present disclosure is described.

[0113] Figure 4 A communication terminal 30 such as a smartphone (smartphone) owned by a user (photographer) is illustrated.

[0114] The communication terminal 30 has a configuration capable of communicating with at least the camera 11 or the camera pan / tilt control unit 13 .

[0115] The communication terminal 30 displays the same Figure 2 The tracking subject type selection UI 31 is similar to the tracking subject type selection UI described above.

[0116] Figure 4 The diagram illustrates an example in which, in the process of ( S11 ), the user (photographer) selects “(5) Bird” as the tracking subject type.

[0117] The tracking subject type information selected by the user (photographer) is transmitted from the communication terminal 30 to the camera 11 or the camera pan / tilt control unit 13 .

[0118] Figure 5 The following processing steps performed by using the communication terminal 30 are illustrated.

[0119] Figure 5 30 is a diagram illustrating a state in which an image captured by the camera 11 is displayed on the communication terminal 30. The image captured by the camera 11 is transmitted to the communication terminal 30 via the communication unit of the camera 11 to be displayed.

[0120] This displayed image is the current image before starting video recording, a so-called live view image (LV image).

[0121] While viewing the live view image (LV image) displayed on the communication terminal 30 , the user first touches the automatic tracking (auto lock on) mode setting icon 32 displayed on the communication terminal 30 at an arbitrary timing (step S12 ).

[0122] The information of the touch operation of the automatic tracking (auto lock on) mode setting icon 32 is transmitted from the communication terminal 30 to the camera 11 or the camera pan / tilt control unit 13, thereby starting the automatic tracking process in the automatic tracking imaging device 10. Note that the automatic tracking target subject is the tracking subject type = "bird" specified in the above (step S11).

[0123] Next, in (step S13), the user (photographer) touches the image capture (video recording) start instruction icon 33 displayed on the communication terminal 30. This touch operation information is sent from the communication terminal 30 to the camera 11 or the camera pan / tilt control unit 13, and is input to the data processing unit of the automatic tracking imaging device 10, thereby starting image capture (video recording).

[0124] The image capture process is performed in the automatic tracking mode, and image capture is performed so as to track the tracking subject type = "bird" specified in the above (step S11). That is, the camera pan control of the image capture direction of the mobile camera 11 is performed so as to track the tracking subject type = "bird", thereby performing image capture.

[0125] Thus, the user (photographer) can execute the following respective processes using a communication terminal such as a smartphone.

[0126] (S11) Processing of selecting tracking subject type

[0127] (S12) Processing of setting the automatic tracking mode to on

[0128] (S13) Processing of Image Capture (Video Recording) Start Instruction

[0129] Only through these simple operations, it is possible to start capturing a tracking image of a subject of a type specified (selected) by the user.

[0130] That is, with the configuration according to the present disclosure, the user (photographer) does not need to perform the troublesome process of framing the tracking target object on the screen.

[0131] Furthermore, the method according to the present disclosure is a method capable of capturing an image of a tracking object based on a result of estimating the posture of the tracking target object in one frame without using multi-frame information, thereby enabling image capture of the tracking object without delay time.

[0132] [2. Sequence of Processing Executed by Image Processing Device According to the Present Disclosure]

[0133] Next, the sequence of processing performed by the image processing apparatus according to the present disclosure will be described.

[0134] Figure 6 The flowchart illustrated in FIG. 1 is a flowchart illustrating a processing sequence executed by the image processing apparatus according to the present disclosure.

[0135] Note that execution follows Figure 6 The image processing device of the process of the flowchart shown in the diagram is a communication terminal 30 and the above reference Figures 1 to 3The automatic tracking imaging device 10 described above or the Figures 4-5 An apparatus for automatically tracking an imaging device 10 is described.

[0136] Below, the order is described in Figure 6 The processing of each processing step in the flowchart illustrated in FIG.

[0137] (Step S101)

[0138] First, in step S101 , the tracking subject type is specified.

[0139] This process corresponds to the above reference Figure 2 Or the processing described in 4.

[0140] By using Figure 2 4, that is, the tracking subject type selection UI displayed on the display unit of the camera 11 or the communication terminal 30, and the user (photographer) selects the type of subject to be tracked.

[0141] As described above Figure 2 and 4 The processing example in is an example in which the user (photographer) selects "(5) Bird" as the tracking subject type.

[0142] The tracking subject type information selected by the user (photographer) is input to the data processing unit of the image processing device (the automatic tracking imaging device 10 or the communication terminal 30).

[0143] (Step S102)

[0144] Next, in step S102 , the automatic tracking mode is set to on.

[0145] This process corresponds to the above reference Figure 3 Or the processing described in 5 (step S12).

[0146] The user (photographer) touches the automatic tracking mode setting icon displayed on the display unit of the camera 11 or the communication terminal 30 to set the automatic tracking mode to on.

[0147] Information indicating that the automatic tracking mode is set to on is input to the data processing unit of the image processing device (the automatic tracking imaging device 10 or the communication terminal 30).

[0148] The automatically tracked target subject is the tracking subject type specified in the above (step S101 ), for example, “bird”.

[0149] Note that despite the above description Figure 3 and 5In the embodiment, a live view image (LV image) is displayed on the display unit of the camera 11 or the communication terminal 30, but the live view image (LV image) may or may not be displayed.

[0150] (Step S103)

[0151] Next, an image capturing (video recording) start instruction is issued in step S103.

[0152] This process corresponds to the above reference Figure 3 Or the processing described in step 5 (step S13).

[0153] As reference Figure 3 and 5 As described above, the user (photographer) touches an image capturing (video recording) start instruction. This touch operation information is input to the data processing unit of the image processing device (automatic tracking imaging device 10 or communication terminal 30).

[0154] Note that the image capturing (video recording) start instruction can be executed at any timing desired by the user (photographer). Figure 3 and 5 In the example shown in the illustration, a captured image of a "bird" designated as a tracking subject is displayed on the live view image (LV image), however, video recording can also be started by issuing an image capture (video recording) start instruction at a screen position where the bird is not captured.

[0155] The processing in steps S101 to S103 is processing involving user operations on the UI, and these user operation information is input to the data processing unit of the image processing device (automatic tracking imaging device 10 or communication terminal 30), thereby starting image (moving image) capture processing in automatic tracking mode.

[0156] The tracking target is a subject that matches the subject type selected in step S101 , for example, a “bird”.

[0157] (Step S104)

[0158] When the following input processing is completed in steps S101 to S103 , the automatic tracking imaging device 10 starts an image (moving image) capturing process in the automatic tracking mode in step S104 and subsequent steps.

[0159] (S101) Processing of selecting tracking subject type

[0160] (S102) Setting the automatic tracking mode to on

[0161] (S103) Processing of Image Capture (Video Recording) Start Instruction

[0162] First, in step S104, a process of detecting a tracking subject from within a captured image is performed.

[0163] Here, the subject to be detected is a subject of a type specified by the user in step S101, for example, "bird".

[0164] The data processing unit of the image processing device performs processing to detect a subject that is consistent with the tracking subject type specified by the user (photographer) from a captured image. Note that the determination of whether a subject that is consistent with the tracking subject type specified by the user (photographer) is in the captured image can be performed as a process of determining whether the subject exists (1) or does not exist (0). Alternatively, a process of applying reliability can be performed in which a certain reliability (for example, a reliability of 0 to 100, etc.) that the subject is consistent with the tracking subject type is calculated, and when it is determined that it is equal to or greater than a predetermined reliability threshold, it is determined that the subject is detected.

[0165] Note that this subject recognition processing is performed by performing image analysis processing of a captured image. Specifically, for example, subject recognition processing using training data is performed. The details of this processing will be described later.

[0166] (Step S105)

[0167] In step S105 , it is determined whether the process of detecting the tracking object from within the captured image in step S104 has succeeded.

[0168] If it is determined to be successful, the process proceeds to step S106.

[0169] On the other hand, when it is determined that the process has failed, the process returns to step S104 and a process of detecting a tracking object from the continuously captured image frames is executed.

[0170] As described above, the subject to be detected is a subject of a type specified by the user in step S101 , for example, “bird”.

[0171] When the "bird" is not detected in the captured image, it is determined in step S105 that the process of detecting the tracking subject from within the captured image has failed, and the process returns to step S104 to start the detection process from the next image frame.

[0172] However, for example, there may be a case where a plurality of subjects of a type specified by the user, for example, a plurality of “birds”, are detected from one image frame.

[0173] In this case, the user sets the tracking mode in advance as to which "bird" is to be set as the tracking target.

[0174] An example of a sequence for setting tracking mode will be referred to Figure 7 The flowchart shown in the figure is used to illustrate.

[0175] Figure 7 The processing flow shown in the diagram is as follows: Figure 6 The flow shown in the diagram is a pre-processing before the image capturing process in the subject tracking mode, and a tracking mode setting process executed by input from the user (photographer).

[0176] according to Figure 7 The tracking mode information of the processing setting shown in the diagram is stored in the storage unit in the image processing device (automatic tracking imaging device 10), and according to the set tracking mode, the execution Figure 6 The success determination process of the process of detecting the tracking subject from within the captured image in steps S104 to S105 of the flow in FIG.

[0177] Will explain Figure 7 Processing in each step of the process shown in the diagram.

[0178] (Step S201)

[0179] First, in step S201, the user determines whether to limit the number of tracking objects to 1 and inputs selection information. Note that a UI for inputting selection information is displayed on the display unit of the camera 11 or the communication terminal 30, for example, and the user inputs selection information by using the displayed UI.

[0180] When the user selects the setting to limit the number of tracking subjects to 1 in step S201 , the process proceeds to step S202 .

[0181] On the other hand, when the user selects the setting not to limit the number of tracking subjects to 1 in step S201 , the process proceeds to step S211 .

[0182] (Step S202)

[0183] When the user selects the setting to limit the number of tracking subjects to 1 in step S201 , the process proceeds to step S202 .

[0184] In this case, in step S202, the user determines whether the tracking subject is a subject close to the center of the image and inputs selection information. Note that a UI for inputting selection information is also displayed on a display unit such as the camera 11 or the communication terminal 30, and the user inputs selection information by using the displayed UI.

[0185] When the user selects the setting that the tracking subject is a subject close to the center of the image in step S202 , the process proceeds to step S212 .

[0186] On the other hand, when the user selects a tracking subject other than a subject near the center of the image in step S202 , the process proceeds to step S213 .

[0187] The user's selection information in steps S201 and S202 is input to the data processing unit in the image processing device (automatic tracking imaging device 10 or communication terminal 30), and based on the input information, the data processing unit determines the subject tracking mode to be executed.

[0188] For example, there are the following three subject tracking modes.

[0189] (A) Multi-subject tracking mode

[0190] (B) Center subject tracking mode

[0191] (C) Maximum Subject Tracking Mode

[0192] According to the user's selection information in steps S201 and S202, the data processing unit in the image processing device (automatic tracking imaging device 10 or communication terminal 30) determines which tracking mode of the above-mentioned tracking modes (A) to (C) is executed.

[0193] In a case where the determination in step S201 is NO, the process proceeds to step S211 , and the tracking mode to be executed is set to (A) the multi-subject tracking mode.

[0194] If the determination in step S201 is YES, and the determination in step S202 is YES, the process proceeds to step S212 , and the tracking mode to be executed is set to (B) the center subject tracking mode.

[0195] If the determination in step S201 is YES, and the determination in step S202 is NO, the process proceeds to step S213, and the tracking mode to be executed is set to (C) the maximum subject tracking mode.

[0196] Next, the processing of steps S211 to S213 will be described.

[0197] (Step S211)

[0198] In a case where the determination in step S201 is NO, that is, when the user selects a setting not to limit the number of tracking subjects to 1 in step S201 , the process proceeds to step S211 .

[0199] In this case, the data processing unit of the image processing device (the automatic tracking imaging device 10 or the communication terminal 30 ) sets the tracking mode to be executed to (A) the multi-subject tracking mode.

[0200] (A) The multi-subject tracking mode is a mode in which the multi-subject tracking image is detected from the captured image. Figure 6 In the case where there are a plurality of subjects corresponding to the tracking subject type specified in step S101, an area including all subjects corresponding to the tracking subject type is set as a tracking subject mode.

[0201] As in Figure 7 As illustrated in the image of (step S211), when multiple subjects corresponding to the tracking subject type ("birds" in this example) are detected in the captured image, the area including all of these subjects is set as the tracking subject.

[0202] (Step S212)

[0203] If the judgment in step S201 is “Yes” and the judgment in step S202 is “Yes”, that is, when in step S201 the user selects the setting to limit the number of tracked subjects to 1, and in step S202 the user selects the setting that the tracked subject is a subject close to the center of the image, the processing enters step S212.

[0204] In this case, the data processing unit of the image processing device (the automatic tracking imaging device 10 or the communication terminal 30 ) sets the tracking mode to be executed to the (B) center subject tracking mode.

[0205] (B) The center subject tracking mode is a mode in which a center subject tracking image is detected from a captured image. Figure 6 In the case of a plurality of subjects corresponding to the tracking subject type specified in step S101, one subject closest to the center of the captured image among the plurality of detected subjects is set as the tracking subject mode.

[0206] As in Figure 7 As shown in the diagram in the image of (step S212), when multiple subjects corresponding to the tracking subject type ("birds" in this case) are detected in the captured image, one of the detected subjects that is closest to the center of the captured image is selected and set as the tracking subject.

[0207] (Step S213)

[0208] In a case where the determination in step S201 is “Yes” and the determination in step S202 is “No”, that is, when in step S201 the user selects the setting to limit the number of tracked subjects to 1, and in step S202 the user does not select the setting in which the tracked subject is a subject close to the center of the image, the processing proceeds to step S213.

[0209] In this case, the data processing unit of the image processing device (the automatic tracking imaging device 10 or the communication terminal 30 ) sets the tracking mode to be executed to (C) the maximum subject tracking mode.

[0210] (C) The maximum subject tracking mode is a mode in which the subject is detected from the captured image and the subject is Figure 6 In the case of a plurality of subjects corresponding to the tracking subject type specified in step S101, a subject having the largest image area among the plurality of detected subjects is set as the tracking subject mode.

[0211] As in Figure 7 As shown in the diagram in the image of (step S213), when multiple subjects corresponding to the tracking subject type ("birds" in this case) are detected in the captured image, one subject with the largest image area in the captured image is selected from these detected subjects and set as the tracking subject.

[0212] according to Figure 7 The tracking mode information set by the processing flow illustrated in FIG. 1 is stored in a storage unit within the image processing device (the automatic tracking imaging device 10 or the communication terminal 30 ).

[0213] According to the set tracking mode, execute Figure 6 The above-mentioned success determination processing of the processing of detecting the tracking subject from within the captured image in steps S104 to S105 of the process.

[0214] Note that although Figure 7 In the example shown in the diagram, three mode classification examples are described, namely (A) multi-subject tracking mode, (B) center subject tracking mode, and (C) maximum subject tracking mode, but other different modes can also be set. For example, a mode setting such as a leading subject selection mode can be selected, in which a leading car or person in a race or the like is set as a tracking subject.

[0215] Will refer to Figure 8 A specific example of the process of detecting a tracking subject from a captured image will be described.

[0216] Figure 8 illustrate Figure 6 2 is a diagram illustrating an example of the processing in steps S104 to S105 of the flow shown in FIG. 4 , that is, the processing of detecting a tracking subject from within a captured image.

[0217] Figure 8 Illustrate a captured image on which tracking subject detection processing is performed.

[0218] Notice, Figure 8The example illustrated in FIG. 2 is an example of a case where the tracking subject type is set to “bird”.

[0219] first, Figure 8 The (state A) illustrated in the figure is a state in which no subject corresponding to the tracking subject type = bird is detected in the captured image. In this case, the determination in step S105 is "No", so the processing returns to step S104 to perform processing to detect the bird as the tracking subject from the continuously captured images.

[0220] Figure 8 (State B) in is a state in which a subject corresponding to tracking subject type=bird is detected in the captured image. In this case, the determination in step S105 is YES, and the processing proceeds to step S106.

[0221] Fig. 9 This is a processing example of a case where a plurality of subjects corresponding to tracking subject type=bird are detected from a captured image.

[0222] In step S104, if Fig. 9 As illustrated in (State 1), first, based on the tracking subject type=bird specified by the user (photographer) in step S101, an image region of a bird as a candidate for a tracking subject is detected from a captured image.

[0223] In this example, two birds are detected and are candidate tracking subjects.

[0224] Next, as the second half of the processing in step S104 , a process of determining a tracking object is performed according to a preset tracking mode.

[0225] Here, it is assumed that the above reference Figure 7 (B) Center subject tracking mode shown.

[0226] According to the setting of (B) center subject tracking mode, the data processing unit of the image processing device determines the bird closer to the center of the image among the two birds detected from the captured image as the tracking subject. In this way, the process of determining the tracking subject is performed.

[0227] return Figure 6 , explaining the processing of step S106 and subsequent steps.

[0228] (Step S106)

[0229] When it is determined in step S105 that the process of detecting the tracking subject from within the captured image in step S104 has succeeded, the process proceeds to step S106 .

[0230] In step S106 , the image processing apparatus performs processing of calculating a predicted motion vector corresponding to the type and posture of the tracked object.

[0231] The type of the tracking subject is, for example, the type of subject designated as the tracking subject by the user, such as a bird, a person, or a dog. The posture is the posture of a bird perched on a tree, the posture of a bird flying, the posture of a person walking, the posture of a person running, etc. In addition, the posture also includes the size of the tracking subject in the image, and when calculating the predicted motion vector, the size of the tracking subject in the image is taken into consideration for processing.

[0232] Note that the process of calculating the predicted motion vector corresponding to the type and posture of the tracking subject is performed in a data processing unit in the image processing device (the automatic tracking imaging device 10 or the communication terminal 30).

[0233] The details of the predicted motion vector calculation process will be described later.

[0234] refer to Fig.10 and 11 A specific example of a predicted motion vector corresponding to the type and posture of the tracking object will be described.

[0235] Fig.10 1 is a diagram illustrating a specific example of a predicted motion vector corresponding to one posture of a “bird” in a case where the type of the tracked subject is a “bird”.

[0236] exist Fig.10 In the example shown in the diagram, a vector extending from the center position of a bird as a tracking object to the upper right direction is illustrated. This vector is a predicted motion vector of the bird as a tracking object.

[0237] The direction of the predicted motion vector is set to a direction corresponding to the predicted moving direction of the bird as the tracking object, and the length of the predicted motion vector is set to a length corresponding to the predicted moving speed (80 km / h) of the bird as the tracking object.

[0238] A specific example of the predicted motion vector calculation process will be described later.

[0239] Fig.11 1 is a diagram illustrating a specific example of a predicted motion vector corresponding to one posture of a “person” in a case where the type of the tracking subject is a “person”.

[0240] exist Fig.11 In the example shown in the diagram, a vector extending horizontally to the left from the center position of a person who is a tracking subject is illustrated. This vector is a predicted motion vector of the person who is a tracking subject.

[0241] The direction of the predicted motion vector is set to a direction corresponding to the predicted moving direction of the person as the tracking subject, and the length of the predicted motion vector is set to a length corresponding to the predicted moving speed (38 km / h) of the person as the tracking subject.

[0242] Thus, in Figure 6 In step S106 illustrated in the figure, a process of calculating a predicted motion vector corresponding to the type and posture of the tracked object is executed.

[0243] As described above, the process of calculating the predicted motion vector corresponding to the type and posture of the tracking object is performed in the data processing unit in the image processing device (the automatic tracking imaging device 10 or the communication terminal 30).

[0244] The process of calculating the predicted motion vector corresponding to the type and posture of the tracked object can be executed as a process using training data.

[0245] Alternatively, a table that records in advance predicted motion vectors corresponding to the types and postures of the tracked objects may be stored in the storage unit, and the predicted motion vector may be acquired with reference to the table.

[0246] Note that although the process of calculating the predicted motion vector corresponding to the type and posture of the tracked object is performed in units of image frames captured by, for example, the camera 11, when calculating the predicted motion vector corresponding to a new image frame, the process of calculating a new predicted motion vector may be performed with reference to the predicted motion vector that has been calculated based on past frames such as the previous frame, or the measured motion vector that reflects the actual movement of the tracked object obtained from the past captured images.

[0247] For example, for the initial frame at the start of tracking, a predicted motion vector is calculated based on the result of analyzing the initial frame because the initial frame has no information corresponding to the past frame. However, for the next frame and subsequent frames, a process may be performed in which the predicted motion vector calculated from the latest captured image frame and the measured motion vector reflecting the actual motion of the tracked subject obtained from the past captured images are compared, correction processing is performed so that the difference is reduced, and a final predicted motion vector is calculated.

[0248] In addition, a process of setting weights for the predicted motion vector and the measured motion vector and calculating the final predicted motion vector by weight addition can be performed. As an example, a process of weighting the measured motion vector more may be performed in which, in the case of irregular motion of the subject, the accuracy will deteriorate if the predicted motion vector is weighted.

[0249] (Step S107)

[0250] Next, in step S107, the image capturing direction of the camera is controlled according to the predicted motion vector corresponding to the type and posture of the tracking object calculated in step S106. That is, assuming that the tracking object moves according to the predicted motion vector corresponding to the type and posture of the tracking object calculated in step S106, a camera direction control signal for changing the image capturing direction of the camera is generated so that the image capturing direction of the camera is consistent with the moving direction.

[0251] Furthermore, the camera direction is changed according to the generated camera direction control signal.

[0252] Note that based on the predicted motion vector corresponding to the type and posture of the tracked object calculated in step S106, the data processing unit in the image processing device (automatic tracking imaging device 10 or communication terminal 30) calculates the camera image capture direction movement vector used to make the moving position of the tracked object consistent with the image capture direction of the camera 11.

[0253] In addition, the data processing unit in the image processing device (automatic tracking imaging device 10 or communication terminal 30) determines the driving mode (driving direction and driving amount) of the camera pan head 12 required to change the camera image capturing direction according to the calculated camera image capturing direction movement vector.

[0254] In addition, the data processing unit in the image processing device (automatic tracking imaging device 10 or communication terminal 30) generates a camera pan tilt drive control signal for driving the camera pan tilt 12 according to the determined drive mode (drive direction and drive amount) of the camera pan tilt 12, and outputs the generated camera pan tilt drive control signal to the camera pan tilt control unit 13. The camera pan tilt control unit 13 drives the camera pan tilt 12 by applying the input camera pan tilt drive control signal.

[0255] Through these processes, when the tracking object moves according to the predicted motion vector corresponding to the type and posture of the tracking object calculated in step S106, the image capturing direction of the camera is controlled so that the image capturing direction of the camera coincides with the moving direction.

[0256] As a result, image capturing of a captured image (moving image) in which the tracking subject is tracked can be performed.

[0257] Note that not only the image capturing direction of the camera can be calculated, but also the camera setting information such as pan, tilt or push-pull for capturing the best image of the tracked subject can be calculated, and the calculated information can also be output to the camera 11 or the camera pan control unit 13 for control.

[0258] (Step S108)

[0259] Finally, in step S108 , it is determined whether the automatic tracking mode is set to off.

[0260] In a case where the automatic tracking mode is set to off, the image capturing process in the automatic tracking mode ends.

[0261] When the automatic tracking mode is not set to off, the process returns to step S104, and the processes of steps S104 to S107 are repeated for a new captured image frame.

[0262] Through this repetitive processing, the image capturing process in the automatic tracking mode is continuously performed.

[0263] [3. Details of the Image Processing Device Configuration and Subject Prediction Motion Vector Generation Process]

[0264] Next, the configuration of the image processing device and details of the subject prediction motion vector generation process are described.

[0265] Fig.12 A configuration example of the image processing apparatus 100 according to the present disclosure is illustrated.

[0266] Fig.12 The image processing device 100 illustrated in FIG. 1 corresponds to the image processing device 100 illustrated in FIG. 1 by combining the communication terminal 30 and the reference Figures 1 to 3 The automatic tracking imaging device 10 of the present invention or the reference Figures 4-5 The automatic tracking imaging device 10 is obtained by the device described above.

[0267] Notice, Fig.12 The configuration diagram illustrated in the figure is a configuration diagram obtained by extracting only the part related to the process of controlling the image capturing direction of the camera performed by the image processing device according to the present disclosure. The general configuration of the camera, the communication terminal and the camera pan / tilt control unit is omitted.

[0268] like Fig.12 As shown in the diagram, the image processing apparatus 100 has an input unit 101 , an imaging unit 102 , a subject predicted motion vector generation unit 110 , a camera control signal generation unit 121 , and a camera (camera pan / tilt) drive unit 122 .

[0269] The subject predicted motion vector generation unit 110 includes a tracking subject recognition unit 111 , subject estimation training data 112 , a tracking subject type and posture correspondence predicted motion vector calculation unit 113 , and a tracking subject type and posture correspondence predicted motion vector estimation training data 114 .

[0270] The input unit 101 inputs, for example, tracking subject type information.

[0271] Specifically, as mentioned above Figure 2 and 4 As described above, by using the tracking subject type selection UI displayed on the display unit of the camera 11 or the display unit of the communication terminal 30 , the user selects and inputs the type of the subject to be tracked.

[0272] The tracking subject type designation information 201 is input from the input unit 101 to the subject prediction motion vector generation unit 110 .

[0273] The imaging unit 102 corresponds to the reference Figures 1 to 5 The imaging unit of the camera 11 in the described configuration.

[0274] The imaging unit 102 acquires a captured image such as a current image before the start of video recording (a so-called live view image (LV image)) or an image for video recording after the start of video recording.

[0275] The captured image 202 acquired by the imaging unit 102 is input to the subject prediction motion vector generation unit 110 .

[0276] The subject predicted motion vector generation unit 110 inputs the tracking subject type designation information 201 from the input unit 101 , inputs the captured image 202 from the imaging unit 102 , and generates and outputs the tracking subject type and posture correspondence predicted motion vector 204 by using these input information.

[0277] The tracking subject type and posture corresponding predicted motion vector 204 is a predicted motion vector corresponding to the type of the tracking subject, for example, the type of the tracking subject such as a bird, a dog, a person, or a ball, and the posture of the tracking subject.

[0278] For example, when the tracking subject type is a bird, the length of the predicted motion vector of a bird perched on a tree is basically 0, while when the bird is flying in the sky, the predicted motion vector is a vector whose direction is roughly the same as the orientation of the bird's head and whose length corresponds to the bird's flying speed.

[0279] In the subject prediction motion vector generation unit 110, first, the tracking subject recognition unit 111 detects the tracking subject from the captured image 202 by using the tracking subject type designation information 201 input from the input unit 101 and the captured image 202 input from the imaging unit 102. Specifically, a tracking subject that is consistent with the tracking subject type designated by the user is detected from the captured image 202.

[0280] For example, in a case where the tracking subject type designated by the user is “bird”, a process of detecting “bird” from the captured image 202 or the like is performed.

[0281] For example, in a case where the tracking subject type designated by the user is “person”, a process of detecting the “person” from the captured image 202 or the like is performed.

[0282] When performing the process of detecting a tracking subject, the tracking subject recognition unit 111 executes the subject recognition process by using the subject estimation training data 112 .

[0283] The subject estimation training data 112 is data accumulated through a learning process performed in advance, and is training data capable of accumulating image feature information of various moving subjects such as humans, birds, dogs, cats, and balls, and estimating the type of the subject from the image.

[0284] By utilizing, for example, a neural network constructed using the subject estimation training data 112 , the tracking subject recognition unit 111 detects a tracking object consistent with the tracking subject type designated by the user, such as “bird”, from the captured image 202 .

[0285] The tracking subject recognition unit 111 detects an image area consistent with a tracking subject type specified by a user (photographer), for example, “bird”, and outputs a captured image in which “label=bird” is set for the detected image area as tracking subject detection information 203 .

[0286] The detection information of the tracking subject recognition unit 111 is output to the tracking subject type and posture correspondence predicted motion vector calculation unit 113 as the tracking subject detection information 203 .

[0287] The tracking subject type and posture correspondence predicted motion vector calculation unit 113 inputs the tracking subject detection information 203 from the tracking subject recognition unit 111. The tracking subject detection information 203 is a captured image in which "label = bird" is set for an image region that coincides with a tracking subject specified by a user, such as "bird". The image includes an image of a bird in a certain posture, such as an image of a bird flying in the sky.

[0288] Based on the input information, the tracking subject type and posture correspondence predicted motion vector calculation unit 113 calculates a predicted motion vector corresponding to the type and posture of the tracking subject included in the captured image 202 .

[0289] The tracking subject type and posture corresponding predicted motion vector calculation unit 113 calculates the predicted motion vector corresponding to the type of the tracking subject, for example, the type of the tracking subject such as a bird, dog, person or ball, and the posture of the tracking subject, that is, the tracking subject type and posture corresponding predicted motion vector 204.

[0290] When performing the process of calculating the tracking subject type and posture correspondence predicted motion vector 204 , the tracking subject type and posture correspondence predicted motion vector calculation unit 113 performs vector calculation processing by using the tracking subject type and posture correspondence predicted motion vector estimation training data 114 .

[0291] The tracking subject type and posture corresponding prediction motion vector estimation training data 114 is data accumulated through a learning process performed in advance, and can accumulate motion vectors corresponding to the types and postures of various moving subjects such as people, birds, dogs, cats and balls, and can estimate the motion vector of the subject based on the type and posture of the subject detected from the image.

[0292] The tracking subject type and posture corresponding predicted motion vector calculation unit 113 calculates a predicted motion vector corresponding to the type and posture of the tracking subject contained in the captured image 202 by using, for example, a neural network constructed using the tracking subject type and posture corresponding predicted motion vector estimation training data 114.

[0293] Note that in addition to the process using the above-described training data, the process in which the tracking subject type and posture correspondence predicted motion vector calculation unit 113 calculates a predicted motion vector corresponding to the type and posture of the tracking subject may also be acquired from a pre-generated table.

[0294] That is, a table in which predicted motion vectors corresponding to the types and postures of the tracked object are recorded in advance may be stored in the storage unit, and the predicted motion vector may be acquired with reference to the table.

[0295] The tracking subject type and posture correspondence predicted motion vector 204 calculated by the tracking subject type and posture correspondence predicted motion vector calculation unit 113 is output to the camera control signal generation unit 121 .

[0296] The camera control signal generating unit 121 generates a control signal for controlling the image capturing direction of the camera 11 by using the tracking subject type and posture correspondence predicted motion vector 204 calculated by the tracking subject type and posture correspondence predicted motion vector calculating unit 113 .

[0297] That is, a camera direction control signal is generated for setting the image capture direction of the camera to the position where the tracking object moves according to the predicted motion vector 204 corresponding to the tracking object type and posture. Note that the camera direction control signal specifically corresponds to a drive control signal of the camera pan / tilt 12 that controls the image capture direction of the camera 11.

[0298] The camera direction control signal generated by the camera control signal generating unit 121 is input to the camera (camera pan / tilt head) driving unit 122 .

[0299] Note that the camera (camera pan / tilt) drive unit 122 is Figure 1 Components of the camera pan / tilt control unit 13 are shown in the diagram.

[0300] The camera (camera platform) driving unit 122 drives the camera platform based on the camera direction control signal generated by the camera control signal generating unit 121 .

[0301] That is, the camera pan / tilt is driven so that the image capturing direction of the camera corresponds to the predicted motion vector 204 according to the type and posture of the tracked object, and is consistent with the position to which the tracked object moves.

[0302] Through this processing, the image capturing direction of the camera moves in accordance with the movement of the tracking object designated by the user (photographer), so that a tracking image (moving image) of the tracking object can be captured.

[0303] Next, refer to Fig.13 and 14 A specific processing example of the processing performed by the subject prediction motion vector generation unit 110, namely, generating the type of the tracking subject, for example, the type of the tracking subject such as a bird, a dog, a person or a ball, and the tracking subject type and posture corresponding prediction motion vector 204 as a prediction motion vector corresponding to the posture of the tracking subject is described.

[0304] Fig.13 Illustrated is a processing example of a case in which the type of tracking subject designated by the user (photographer) is “bird”.

[0305] Fig.14 Illustrated is a processing example of a case in which the type of tracking subject designated by the user (photographer) is “person”.

[0306] First, refer to Fig.13 A processing example is described for a case where the type of tracking subject designated by the user (photographer) is “bird”.

[0307] Fig.13 The tracking subject recognition unit 111 and the tracking subject type and posture correspondence prediction motion vector calculation unit 113 are illustrated as components of the subject prediction motion vector generation unit 110. They both have a neural network generated from training data.

[0308] The tracking subject recognition unit 111 has a designated subject type detection neural network generated from training data.

[0309] Note that the designated subject type detection neural network is a neural network that inputs the tracking subject type designation information 201 from the input unit 101 and the captured image 202 from the imaging unit 102 , and outputs the tracking subject detection information 203 .

[0310] exist Fig.13 In the example shown in the diagram, the tracking subject recognition unit 111 uses a neural network to detect an image area consistent with the tracking subject type “bird” specified by the user (photographer) from the captured image 202, and outputs the captured image in which “label = bird” is set for the detected image area as the tracking subject detection information 203.

[0311] The tracking subject detection information 203 generated by the tracking subject recognition unit 111 is output to the tracking subject type and posture correspondence predicted motion vector calculation unit 113 .

[0312] The tracking subject type and posture correspondence predicted motion vector calculation unit 113 has a bird posture correspondence predicted motion vector estimation neural network generated from training data.

[0313] This neural network is a neural network that estimates a predicted motion vector corresponding to the posture of the bird when the subject type = bird.

[0314] Note that the neural network can be set in units of bird types, for example. For example, a neural network in units of bird types such as pigeons, sparrows, and swans can be applied. For example, in the case of subject type = pigeon, a process of estimating a predicted motion vector corresponding to the posture of the pigeon can be performed.

[0315] Note that the tracking subject type and posture corresponding predicted motion vector calculation unit 113 has a predicted motion vector estimation neural network corresponding to the subject type (human, bird, dog...), and utilizes the neural network while switching the neural network according to the subject type (human, bird, dog...) specified by the user as the tracking object.

[0316] Note that although in the present embodiment, the tracking subject type and posture corresponding predicted motion vector calculation unit 113 performs processing for identifying the tracking subject type and processing for calculating the posture corresponding predicted motion vector, the processing for identifying the tracking subject type and processing for calculating the posture corresponding predicted motion vector may be performed by separate processing units. In addition, these processes may be performed by different devices. For example, the type of subject (whether the subject is a subject designated to be tracked) may be identified on the camera side, and motion prediction may be performed in other information processing devices such as an external communication terminal or a server on the cloud side by using this information.

[0317] The tracking subject type and posture corresponding predicted motion vector calculation unit 113 inputs the tracking subject detection information 203 input from the tracking subject recognition unit 111, that is, for the image area consistent with "bird", the captured image of "label = bird" is set, and by utilizing a neural network, calculates the predicted motion vector corresponding to the type of tracking subject (bird) and its posture, that is, the tracking subject type and posture corresponding predicted motion vector 204.

[0318] like Fig.13 As shown in the diagram, the tracking subject type and posture correspondence prediction motion vector calculation unit 113 calculates a vector whose direction is the front direction of a bird flying in the sky as the tracking subject and whose length corresponds to the flying speed of the bird as the tracking subject type and posture correspondence prediction motion vector 204.

[0319] The tracking object type and posture corresponding predicted motion vector 204 is output to the camera control signal generating unit 121. The camera control signal generating unit 121 uses the predicted motion vector to generate a camera direction control signal for setting the image capturing direction of the camera to the position to which the tracking object moves according to the predicted motion vector.

[0320] Next, refer to Fig.14 A processing example of a case in which the type of tracking subject designated by the user (photographer) is “person” is described.

[0321] Similar to Fig.13 , Fig.14 The tracking subject recognition unit 111 and the tracking subject type and posture correspondence prediction motion vector calculation unit 113 are illustrated as components of the subject prediction motion vector generation unit 110. They both have a neural network generated from training data.

[0322] exist Fig.14 In the example shown in the diagram, the tracking subject recognition unit 111 uses a neural network to detect an image area consistent with the tracking subject type “person” specified by the user (photographer) from the captured image 202, and outputs the captured image in which “label = person” is set for the detected image area as the tracking subject detection information 203.

[0323] The tracking subject detection information 203 generated by the tracking subject recognition unit 111 is output to the tracking subject type and posture correspondence predicted motion vector calculation unit 113 .

[0324] The tracking subject type and posture correspondence predicted motion vector calculation unit 113 has a “human” posture correspondence predicted motion vector estimation neural network generated from training data.

[0325] The neural network estimates predicted motion vectors corresponding to various types of people and postures of people, such as adults, children, men and women, and postures of people such as walking postures and running postures.

[0326] Note that a neural network may be generated and utilized for each specific person as a type of person. A neural network may be generated that learns motions corresponding to various postures of a certain person A and estimates predicted motion vectors corresponding to various postures of a certain person A based on training data.

[0327] In the case where a certain person A is detected from a captured image, by applying a neural network unique to person A, a highly accurate predicted motion vector based on data unique to person A can be estimated.

[0328] The tracking subject type and posture corresponding predicted motion vector calculation unit 113 inputs the tracking subject detection information 203 input from the tracking subject recognition unit 111, that is, for the image area of ​​the tracking subject that is consistent with "person" and is selected according to a pre-specified subject tracking mode, for example, the leading subject tracking mode, a captured image of "label = person" is set, and by utilizing a neural network, a predicted motion vector corresponding to the type of the tracking subject (person) and its posture, that is, a predicted motion vector 204 corresponding to the tracking subject type and posture is calculated.

[0329] like Fig.14 As shown in the diagram, the tracking subject type and posture correspondence predicted motion vector calculation unit 113 calculates a vector that faces in the front direction of a person in a running posture as the tracking subject and has a length corresponding to the person's running speed as the tracking subject type and posture correspondence predicted motion vector 204.

[0330] The tracking object type and posture corresponding predicted motion vector 204 is output to the camera control signal generating unit 121. The camera control signal generating unit 121 uses the predicted motion vector to generate a camera direction control signal for setting the image capturing direction of the camera to the position to which the tracking object moves according to the predicted motion vector.

[0331] Next, refer to Fig.15 A specific example of the processing performed by the camera control signal generating unit 121 will be described.

[0332] As above reference Fig.12 As described above, the camera control signal generating unit 121 generates a control signal for controlling the image capturing direction of the camera 11 by using the tracking subject type and posture correspondence predicted motion vector 204 calculated by the tracking subject type and posture correspondence predicted motion vector calculating unit 113 .

[0333] That is, a camera direction control signal is generated for setting the image capture direction of the camera to the position where the tracking object moves according to the predicted motion vector 204 corresponding to the tracking object type and posture. Note that the camera direction control signal specifically corresponds to a drive control signal of the camera pan / tilt 12 that controls the image capture direction of the camera 11.

[0334] Fig.15 2 is a diagram illustrating a processing example of the camera control signal generating unit 121 . Fig.15 An example of processing for a case where the tracking subject is a bird is illustrated.

[0335] The figure illustrates a current frame (n) and a next frame (n+1) to be captured by moving the camera 11 in accordance with the movement of the tracking subject.

[0336] “Tracking subject type and posture correspondence predicted motion vector” is indicated by a solid line going forward and upward from the bird as the tracking subject detected in the current frame (n).

[0337] Fig.15 The “tracking subject type and posture corresponding predicted motion vector” shown in the diagram is a vector calculated based on the tracking subject type and posture corresponding predicted motion vector 204 calculated by the subject type and posture corresponding predicted motion vector calculation unit 113, and corresponds to a vector indicating the moving destination of the subject within a frame between the image capture frames (n) and (n+1) of the camera 11.

[0338] In the next frame (n+1), it is predicted that the subject will move to the position of the bird as indicated by the dotted line as shown in the diagram. Therefore, in order to capture the image of the bird at the position of the bird as shown by the dotted line in the next frame (n+1), it is necessary to move the image capture direction of the camera so as to capture the next frame (n+1) indicated by the dotted box in the figure.

[0339] To this end, it is necessary to change the image capturing direction of the camera 11 from the center of the current frame (n) image indicated by the black circle in the figure to the center of the next frame (n+1) image indicated by the dotted circle in the figure.

[0340] The amount of directional control is defined by the dashed arrow illustrated in the figure, ie, the camera image capture direction movement vector.

[0341] Notice, Fig.15 The processing example shown in the diagram is an example. According to the image processing device of the present disclosure, the motion vector of the tracking object can be predicted and the object can be controlled at the target position (for example, the center position of the image) including the information. Fig.15In addition to the position of the bird shown by the dashed line in frame (n+1) shown in the diagram. Fig.15 In addition to the setting of the center of the next frame (n+1) indicated by the dotted circle in FIG. 1 , the image capturing direction of the camera 11 may be set differently.

[0342] The camera control signal generating unit 121 calculates the camera image capturing direction movement vector and moves the image capturing direction of the camera according to the vector based on the tracking subject type and posture corresponding predicted motion vector 204 calculated by the subject type and posture corresponding predicted motion vector calculating unit 113. Through this processing, a tracking image in which the tracking subject designated by the user is captured in each captured image frame can be captured.

[0343] [4. Conclusion of the present disclosure]

[0344] The embodiments according to the present disclosure are described in detail with reference to specific embodiments. However, it is obvious that those skilled in the art may make various modifications or substitutions to the embodiments without departing from the scope of the present disclosure. That is, the present invention is disclosed in the form of examples and should not be construed as being limited. In order to determine the scope of the present disclosure, reference should be made to the claims.

[0345] Note that the technology disclosed in this specification can have the following configurations.

[0346] (1) An image processing device comprising:

[0347] a subject prediction motion vector calculation unit that detects a tracking subject corresponding to a pre-specified type from a captured image and calculates a prediction motion vector corresponding to the type and posture of the detected tracking subject, and

[0348] A camera control signal generating unit generates a camera control signal for tracking the tracking object based on the predicted motion vector.

[0349] (2) The image processing device according to (1),

[0350] The object prediction motion vector calculation unit has

[0351] a tracking subject recognition unit that detects a tracking subject corresponding to a specified type from a captured image, and

[0352] A tracking subject type and posture corresponding subject predicted motion vector calculation unit is configured to calculate a predicted motion vector corresponding to the type and posture of the tracking subject detected by the tracking subject recognition unit.

[0353] (3) The image processing device according to (1) or (2),

[0354] The subject predicted motion vector calculation unit performs tracking subject recognition processing and predicted motion vector calculation processing using training data.

[0355] (4) The image processing device according to any one of (1) to (3),

[0356] wherein the image processing apparatus has a display unit that displays a user interface (UI) for allowing a user to select a type of a tracking object, and

[0357] The subject predicted motion vector calculation unit detects a subject of a type corresponding to a type of a tracking subject designated by a user using the UI from a captured image, and calculates a predicted motion vector corresponding to the type and posture of the detected tracking subject.

[0358] (5) The image processing device according to any one of (1) to (4),

[0359] In the automatic tracking mode started according to the user input, the image processing device starts the process of detecting a tracking object of a pre-specified type from a captured image.

[0360] (6) The image processing device according to any one of (1) to (5),

[0361] The object prediction motion vector calculation unit has

[0362] A tracking subject recognition unit detects a tracking subject of a type specified by a user from a captured image by using a neural network generated based on the training data.

[0363] (7) The image processing device according to any one of (1) to (6),

[0364] The object prediction motion vector calculation unit has

[0365] The tracking subject type and posture corresponding subject prediction motion vector calculation unit calculates the prediction motion vector corresponding to the type and posture of the tracking subject by using a neural network generated based on training data.

[0366] (8) The image processing device according to any one of (1) to (7),

[0367] When a plurality of tracking objects of a type specified by the user are detected in the captured image,

[0368] The object prediction motion vector calculation unit determines the tracking object according to preset object tracking mode setting information.

[0369] (9) The image processing device according to (8),

[0370] The object prediction motion vector calculation unit determines the tracking object according to any one of the following:

[0371] (A) Multi-subject tracking mode,

[0372] (B) Center subject tracking mode,

[0373] (C) Maximum subject tracking mode.

[0374] (10) The image processing device according to any one of (1) to (9),

[0375] The object prediction motion vector calculation unit refers to a table recording prediction motion vectors corresponding to the type and posture of the tracked object to obtain the prediction motion vector corresponding to the type and posture of the tracked object.

[0376] (11) The image processing device according to any one of (1) to (10),

[0377] The object prediction motion vector calculation unit performs calculation of a prediction motion vector taking into account the size of the tracking object within the image.

[0378] (12) The image processing device according to any one of (1) to (11),

[0379] The subject prediction motion vector calculation unit calculates the prediction motion vector by using a measured motion vector reflecting the motion of the tracking subject obtained from a past captured image.

[0380] (13) The image processing device according to any one of (1) to (12),

[0381] The subject prediction motion vector calculation unit compares the prediction motion vector calculated based on the latest captured image with the measured motion vector reflecting the motion of the tracking subject obtained from the past captured images, and corrects the prediction motion vector.

[0382] (14) The image processing device according to any one of (1) to (13),

[0383] The camera control signal generating unit generates a control signal for the camera direction for capturing a tracking image of the tracking object based on the predicted motion vector calculated by the object predicted motion vector calculating unit.

[0384] (15) The image processing device according to any one of (1) to (14),

[0385] Based on the predicted motion vector calculated by the object predicted motion vector calculation unit, the camera control signal generation unit generates a camera control signal for at least any one of pan, tilt or push of a camera for capturing a tracking image of a tracking object.

[0386] (16) An image processing system comprising

[0387] A camera mounted on a camera platform; and a camera platform control unit for controlling the camera platform, wherein

[0388] The camera has

[0389] a subject prediction motion vector calculation unit that detects a tracking subject corresponding to a pre-specified type from a captured image and calculates a prediction motion vector corresponding to the type and posture of the detected tracking subject, and

[0390] a camera control signal generating unit, the camera control signal generating unit generating a camera control signal for tracking the tracking object based on the predicted motion vector, and

[0391] The camera pan / tilt control unit performs camera pan / tilt control for capturing a tracking image of a tracking object based on a camera control signal.

[0392] (17) The image processing system according to (16),

[0393] The object prediction motion vector calculation unit of the camera has

[0394] a tracking subject recognition unit that detects a tracking subject corresponding to a specified type from a captured image, and

[0395] A tracking subject type and posture corresponding subject predicted motion vector calculation unit is configured to calculate a predicted motion vector corresponding to the type and posture of the tracking subject detected by the tracking subject recognition unit.

[0396] (18) An image processing method executed in an image processing device,

[0397] wherein the subject prediction motion vector calculation unit performs a subject prediction motion vector calculation step of detecting a tracking subject corresponding to a pre-specified type from a captured image and calculating a prediction motion vector corresponding to the type and posture of the detected tracking subject, and

[0398] The camera control signal generating unit performs a camera control signal generating step of generating a camera control signal for tracking the tracking object based on the predicted motion vector.

[0399] In addition, the series of processes described in this specification can be performed by hardware, software or a combination of the two. In the case of performing the process by software, the program in which the process sequence is recorded can be installed on a memory in a computer incorporated into the dedicated hardware and the program can be executed, or the program can be installed on a general-purpose personal computer capable of performing various processes and the program can be executed. For example, the program can be recorded in advance on a recording medium. In addition to being installed to a computer from a recording medium, the program can be received via a network such as a local area network (LAN) or the Internet and installed on a recording medium such as a built-in hard disk.

[0400] Note that the various processes described in this specification can be executed not only in the order in which they are described, but also in parallel or individually according to the processing capabilities of the device that executes the processes or when necessary. In addition, in this specification, a system is a logical collection of multiple devices, and is not limited to a system in which the various devices are in the same housing.

[0401] Industrial Applicability

[0402] As described above, according to the configuration of the embodiment of the present disclosure, an apparatus and method for calculating a predicted motion vector corresponding to the type and posture of a tracking object and generating a camera control signal required to capture an image of the tracking object are realized.

[0403] Specifically, for example, it includes a subject prediction motion vector calculation unit that detects a tracking subject of a pre-specified type in a captured image input from an imaging unit and calculates a prediction motion vector corresponding to the type and posture of the detected tracking subject, and a camera control signal generation unit that generates a camera control signal for capturing a tracking image of the tracking subject based on the prediction motion vector calculated by the subject prediction motion vector calculation unit. The subject prediction motion vector calculation unit performs processing for detecting a tracking subject of a type specified by a user from a captured image and prediction motion vector calculation processing by using a neural network or the like.

[0404] With this configuration, an apparatus and method are realized that calculates a predicted motion vector corresponding to the type and posture of a tracking object and generates a camera control signal required to capture an image of the tracking object.

[0405] Reference numerals list

[0406] 10 Automatic tracking imaging equipment

[0407] 11 Camera

[0408] 12 Camera PTZ

[0409] 13 Camera PTZ control unit

[0410] 15 Camera display unit

[0411] 16 Tracking subject type selection UI

[0412] 17 Auto-tracking mode setting icon

[0413] 18 Image capture (video recording) start command icon

[0414] 30 Communication Terminal

[0415] 31 Tracking subject type selection UI

[0416] 32 Auto tracking mode setting icon

[0417] 338 Image capture (video recording) start command icon

[0418] 100 Image processing equipment

[0419] 101 Input Unit

[0420] 102 Imaging Unit

[0421] 110 Subject prediction motion vector generation unit

[0422] 111 Tracking subject recognition unit

[0423] 112 Subject Estimation Training Data

[0424] 113 Tracking subject type and posture corresponding prediction motion vector calculation unit

[0425] 114 Tracking subject type and posture correspondence prediction motion vector estimation training data

[0426] 121 Camera control signal generation unit

[0427] 122 Camera (Camera pan / tilt) drive unit

[0428] 201 Tracking subject type specification information

[0429] 202 Capturing Images

[0430] 203 Tracking subject detection information

[0431] 204 Tracking the subject type and posture corresponding to the predicted motion vector

Claims

1. An image processing device, comprising: A subject prediction motion vector calculation unit having a tracking subject recognition unit that detects a tracking subject corresponding to a specified type from a captured image, and a tracking subject type and posture corresponding subject prediction motion vector calculation unit, the tracking subject type and posture corresponding subject prediction motion vector calculation unit calculating the prediction motion vector corresponding to the type and posture of the tracking subject detected by the tracking subject recognition unit by using a neural network generated based on the tracking subject type and posture corresponding subject prediction motion vector training data, wherein the tracking subject type and posture corresponding subject prediction motion vector training data is data accumulated through a learning process performed in advance and can accumulate motion vectors corresponding to various types and postures of moving subjects and can estimate the motion vector of the subject according to the type and posture of the subject detected from the image, and a camera control signal generating unit, the camera control signal generating unit generating a camera control signal for tracking the tracking object based on the predicted motion vector; The subject prediction motion vector calculation unit compares the prediction motion vector calculated based on the latest captured image with the measured motion vector reflecting the motion of the tracking subject obtained from the past captured images, and corrects the prediction motion vector.

2. The image processing device according to claim 1, The image processing device includes a display unit that displays a user interface UI for allowing a user to select a type of a tracking object, and The subject predicted motion vector calculation unit detects a subject of a type corresponding to a type of a tracking subject designated by a user using the UI from a captured image, and calculates a predicted motion vector corresponding to the type and posture of the detected tracking subject.

3. The image processing device according to claim 1, In the automatic tracking mode started according to the user input, the image processing device starts the process of detecting a tracking object of a pre-specified type from a captured image.

4. The image processing device according to claim 1, The tracking subject recognition unit detects a tracking subject of a type specified by a user from a captured image by using a neural network generated based on training data.

5. The image processing device according to claim 1, in, When a plurality of tracking subjects of a type specified by a user are detected in a captured image, the subject prediction motion vector calculation unit determines the tracking subject according to preset subject tracking mode setting information.

6. The image processing device according to claim 5, The object prediction motion vector calculation unit determines the tracking object according to any one of the following: (A) Multi-subject tracking mode, (B) Center subject tracking mode, (C) Maximum subject tracking mode.

7. The image processing device according to claim 1, The object prediction motion vector calculation unit performs calculation of a prediction motion vector taking into account the size of the tracking object within the image.

8. The image processing device according to claim 1, The camera control signal generating unit generates a control signal for a camera direction of an image for capturing a tracking image of a tracking object based on the predicted motion vector calculated by the object predicted motion vector calculating unit.

9. The image processing device according to claim 1, Based on the predicted motion vector calculated by the subject predicted motion vector calculation unit, the camera control signal generation unit generates a camera control signal for at least any one of pan, tilt or zoom of a camera for capturing an image of a tracking image of a tracking subject.

10. An image processing system, comprising: A camera mounted on a camera platform; and a camera platform control unit for controlling the camera platform, wherein The camera has A subject prediction motion vector calculation unit having a tracking subject recognition unit that detects a tracking subject corresponding to a specified type from a captured image, and a tracking subject type and posture corresponding subject prediction motion vector calculation unit, the tracking subject type and posture corresponding subject prediction motion vector calculation unit calculating the prediction motion vector corresponding to the type and posture of the tracking subject detected by the tracking subject recognition unit by using a neural network generated based on the tracking subject type and posture corresponding subject prediction motion vector training data, wherein the tracking subject type and posture corresponding subject prediction motion vector training data is data accumulated through a learning process performed in advance and can accumulate motion vectors corresponding to various types and postures of moving subjects and can estimate the motion vector of the subject according to the type and posture of the subject detected from the image, and a camera control signal generating unit, the camera control signal generating unit generating a camera control signal for tracking the tracking object based on the predicted motion vector, and wherein the subject prediction motion vector calculation unit compares the prediction motion vector calculated based on the latest captured image with the measured motion vector reflecting the motion of the tracking subject obtained from the past captured images, and corrects the prediction motion vector; The camera pan / tilt control unit performs camera pan / tilt control for capturing a tracking image of a tracking object based on a camera control signal.

11. An image processing method performed in an image processing device, wherein the subject prediction motion vector calculation unit performs a subject prediction motion vector calculation step of detecting a tracking subject corresponding to a pre-specified type from a captured image, and calculating a prediction motion vector corresponding to the detected type and posture of the tracking subject by using a neural network generated based on training data of subject prediction motion vectors corresponding to the type and posture of the tracking subject, wherein the training data of subject prediction motion vectors corresponding to the type and posture of the tracking subject is data accumulated through a learning process performed in advance, and is capable of accumulating motion vectors corresponding to various types and postures of moving subjects and is capable of estimating the motion vector of the subject according to the type and posture of the subject detected from the image, wherein the subject prediction motion vector calculation unit compares the prediction motion vector calculated based on the latest captured image with the measured motion vector reflecting the motion of the tracking subject obtained from the past captured images, and corrects the prediction motion vector, and The camera control signal generating unit performs a camera control signal generating step of generating a camera control signal for tracking the tracking object based on the predicted motion vector.

Images