IMAGE PROCESSING APPARATUS, IMAGING APPARATUS, IMAGE PROCESSING METHOD, AND PROGRAM

Abstract

To enable the whole region and a local region of a specific subject to be detected with good accuracy.SOLUTION: The present invention comprises acquisition means for acquiring image data, first detection means for detecting a first region of a subject from the image data, second detection means for detecting a second region of the subject from the image data, third detection means for detecting a third region from the image data, output means or outputting the detection results by the first, second and third detection means, and tracking means for tracking on the basis of the detection results by the third detection means. The second detection means detects the second region in a region corresponding to the first region when the first detection means detects the first region in first image data, and the tracking means tracks the third region when the third detection means detects the third region.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an imaging device, an image processing method, and a program that can execute subject detection processing to detect a specific subject. [Background technology]

[0002] There is a technology for performing image processing for detecting a subject based on a trained model that has undergone machine learning to detect a specific subject from image data captured by an imaging device such as a digital camera. In Patent Document 1, a specific region of the subject is automatically detected, and then a narrower region contained in that region is detected, thereby making it possible to detect a local region of the specific subject. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-197795 [Non-patent literature]

[0004] [Non-Patent Document 1] “Fully-Convolutional Siamese Networks for Object Tracking”,arXiv 2016 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Document 1, if the entire area of ​​the specific subject cannot be detected, then the local area cannot be detected.

[0006] In view of the above-mentioned problems, an object of the present invention is to provide an image processing device, an imaging device, an image processing method, and a program that are capable of detecting the overall area and local areas of a specific subject with high accuracy. [Means for solving the problem]

[0007] In order to achieve the above object, an image processing device of the present invention comprises an acquisition means for acquiring image data, a first detection means for detecting a first region of a subject from the image data, a second detection means for detecting a second region of the subject from the image data, and a tracking means for tracking the subject; The second detection means When the first detection means detects the first region in the first image data, the second region is detected in a region corresponding to the first region, and when the first region is detected in the first image data and the first region is not detected in the second image data captured after the first image data, a region wider than the first region detected in the first image data is detected. image data processed based on the position and size of the first region detected in the first image data The second area is detected by the first detecting means, and the tracking means tracks the subject using at least one of the detection results from the first and second detecting means. [Effects of the Invention]

[0008] According to the present invention, it is possible to accurately detect the entire area and local areas of a subject. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. [Figure 2] FIG. 2 is a diagram showing the relationship between a system control unit and each unit. [Figure 3] FIG. 10 is a diagram showing an example of a method for setting a detection target by a user. [Figure 4] 3 is a flowchart showing the flow of processing according to the present embodiment. [Figure 5] FIG. 10 is a diagram showing an example of the positional relationship between a local region and an entire region. [Figure 6] 4A and 4B are diagrams showing time-series changes in captured images when photographing a subject. [Figure 7] 10A and 10B are diagrams showing an example of a method for trimming image data used for subject detection. [Figure 8]FIG. 10 is a diagram showing an example of a switching sequence of a plurality of dictionary data for one frame. [Figure 9] 10 is a flowchart showing a flow in which a tracking process is added to the process of the present embodiment. [Figure 10] 10 is a flowchart showing the flow of a tracking process. [Figure 11] FIG. 10 is a diagram showing an example of adding tracking to a dictionary data switching sequence. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0011] 1(a) and 1(b) show external views of an imaging device 100 as an example of an apparatus to which the present invention can be applied. Fig. 1(a) is a front perspective view of the imaging device 100, and Fig. 1(b) is a rear perspective view of the imaging device 100.

[0012] In FIG. 1, the display unit 28 is a display unit provided on the back of the camera that displays images and various information. The touch panel 70a can detect touch operations on the display surface (operation surface) of the display unit 28. The outside-finder display unit 43 is a display unit provided on the top surface of the camera, and displays various camera setting values ​​such as shutter speed and aperture. The shutter button 61 is an operation unit for issuing shooting instructions. The mode change switch 60 is an operation unit for switching between various modes. The terminal cover 40 is an operation unit for connecting to an external device. Totake It is a cover for protecting a connector (not shown) such as a connection cable that connects to the imaging device 100.

[0013] The main electronic dial 71 is a rotary operation member included in the operation unit 70, and by turning this main electronic dial 71, settings such as shutter speed and aperture can be changed. The power switch 72 is an operation member that switches the power of the imaging device 100 on and off. The sub electronic dial 73 is a rotary operation member included in the operation unit 70, and is used to move the selection frame, advance images, etc. The cross key 74 is included in the operation unit 70, and is a cross key (four-way key) that can be pressed up, down, left, and right. Operations can be performed according to the part of the cross key 74 that is pressed. The SET button 75 is included in the operation unit 70, and is a push button that is mainly used to confirm selections, etc.

[0014] The video button 76 is used to start and stop video shooting (recording). The AE lock button 77 is included in the operation unit 70 and can be pressed to lock the exposure state while the camera is ready to shoot. The enlarge button 78 is included in the operation unit 70 and is an operation button for turning the enlargement mode on and off in the live view display in shooting mode. Turning the enlargement mode on and operating the main electronic dial 71 allows the user to enlarge or reduce the LV image. In playback mode, the button functions as an enlargement button for enlarging the playback image and increasing the magnification. The playback button 79 is included in the operation unit 70 and is an operation button for switching between shooting mode and playback mode. Pressing the playback button 79 in shooting mode switches to playback mode, allowing the most recent image recorded on the recording medium 200 to be displayed on the display unit 28. The menu button 81 is included in the operation unit 70 and, when pressed, displays a menu screen on the display unit 28 that allows various settings to be made. The user can intuitively configure various settings using the menu screen displayed on the display unit 28, the cross key 74, and the SET button 75.

[0015] The touch bar 82 is a line-shaped touch operation member (line touch sensor) that can receive touch operations, and is located in a position that can be operated with the thumb of the right hand that is gripping the grip portion 90. The touch bar 82 can receive tap operations (operations in which the user touches and then releases the touch bar without moving it within a predetermined period of time), left and right slide operations (operations in which the user touches and then moves the touched position while still touching the bar), and the like. The touch bar 82 is an operation member that is different from the touch panel 70a, and does not have a display function.

[0016] The communication terminal 10 is a communication terminal through which the imaging device 100 communicates with the lens side (detachable). The eyepiece 16 is the eyepiece of an eyepiece finder (a peer-type finder), and the user can view the image displayed on the internal EVF 29 through the eyepiece 16. The eyepiece detection unit 57 is an eyepiece detection sensor that detects whether the photographer has placed their eye on the eyepiece 16. The lid 83 is a lid for a slot that stores a recording medium 200. The grip unit 90 is a holding unit shaped to be easily held in the right hand when the user holds the imaging device 100. When the digital camera is held by gripping the grip unit 90 with the little finger, ring finger, and middle finger of the right hand, the shutter button 61 and main electronic dial 71 are located in positions that can be operated with the index finger of the right hand. In the same position, the sub electronic dial 73 and touch bar 82 are located in positions that can be operated with the thumb of the right hand.

[0017] FIG. 2A is a block diagram showing an example of the configuration of an image capture device 100 according to this embodiment. In FIG. 2, a lens unit 150 is a lens unit equipped with an interchangeable photographic lens. The lens 103 is usually composed of multiple lenses, but for simplicity, only a single lens is shown here. A communication terminal 6 is a communication terminal through which the lens unit 150 communicates with the image capture device 100, and a communication terminal 10 is a communication terminal through which the image capture device 100 communicates with the lens unit 150. The lens unit 150 communicates with the system control unit 50 via these communication terminals 6 and 10, and the lens system control circuit 4 controls the aperture 1 via the aperture drive circuit 2, and adjusts the focus by displacing the position of the lens 103 via the AF drive circuit 3.

[0018] The shutter 101 is a focal plane shutter that can freely control the exposure time of the imaging unit 22 under the control of the system control unit 50.

[0019] The imaging unit 22 is an imaging element configured with a CCD, CMOS element, or the like that converts an optical image into an electrical signal. The imaging unit 22 may have an imaging surface phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 23 converts an analog signal into a digital signal. The A / D converter 23 is used to convert the analog signal output from the imaging unit 22 into a digital signal.

[0020] The image processing unit 24 performs predetermined pixel interpolation, resizing such as reduction, and color conversion processing on the data from the A / D converter 23 or the data from the memory control unit 15. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data. The system control unit 50 performs exposure control and distance measurement control based on the arithmetic results obtained by the image processing unit 24. This allows TTL (through-the-lens) type AF (autofocus) processing, AE (autoexposure) processing, and EF (flash pre-flash) processing to be performed. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data, and performs TTL type AWB (auto white balance) processing based on the arithmetic results obtained.

[0021] The output data from the A / D converter 23 is written into the memory 32 via the image processing unit 24 and the memory control unit 15, or directly via the memory control unit 15. The memory 32 stores image data obtained by the imaging unit 22 and converted into digital data by the A / D converter 23, as well as image data to be displayed on the display unit 28 and the EVF 29. The memory 32 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio.

[0022] The memory 32 also serves as a memory (video memory) for image display. The D / A converter 19 converts the image display data stored in the memory 32 into an analog signal and supplies it to the display unit 28 and the EVF 29. In this way, the display image data written to the memory 32 is displayed on the display unit 28 and the EVF 29 via the D / A converter 19. The display unit 28 and the EVF 29 perform display according to the analog signal from the D / A converter 19 on a display device such as an LCD or an organic EL. The digital signal that has been A / D converted once by the A / D converter 23 and stored in the memory 32 is converted to analog in the D / A converter 19 and then sequentially transferred and displayed on the display unit 28 or the EVF 29, thereby performing a live view display (LV display). Hereinafter, an image displayed in live view will be referred to as a live view image (LV image).

[0023] The outside-finder liquid crystal display 43 displays various camera settings such as shutter speed and aperture via an outside-finder display drive circuit 44 .

[0024] The nonvolatile memory 56 is an electrically erasable and recordable memory, such as an EEPROM. The nonvolatile memory 56 stores constants, programs, etc. for the operation of the system control unit 50. The programs referred to here are programs for executing various flowcharts described later in this embodiment.

[0025] The system control unit 50 is a control unit made up of at least one processor or circuit, and controls the entire imaging device 100. Each process of this embodiment, which will be described later, is realized by executing a program recorded in the nonvolatile memory 56 described above. The system memory 52, for example, is a RAM, and constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 56, and the like are loaded into the system memory 52. ​​The system control unit 50 also performs display control by controlling the memory 32, D / A converter 19, display unit 28, etc.

[0026] The system timer 53 is a timekeeping unit that measures the time used for various controls and the time of a built-in clock.

[0027] The operation unit 70 is an operating means for inputting various operational instructions to the system control unit 50. The mode selector switch 60 is an operating member included in the operation unit 70 and switches the operation mode of the system control unit 50 to one of still image capture mode, video capture mode, playback mode, etc. Modes included in the still image capture mode include auto capture mode, auto scene determination mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program AE mode (P mode). There are also various scene modes and custom modes that provide capture settings for specific shooting scenes. The mode selector switch 60 allows the user to directly switch to one of these modes. Alternatively, the user may first switch to a list screen of shooting modes using the mode selector switch 60, then select one of the displayed modes and switch using other operating members. Similarly, the video capture mode may also include multiple modes.

[0028] The first shutter switch 62 is turned on and generates a first shutter switch signal SW1 when the shutter button 61 provided on the imaging device 100 is pressed halfway (a shooting preparation instruction) during operation. The first shutter switch signal SW1 starts shooting preparation operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing.

[0029] The second shutter switch 64 is turned on when the shutter button 61 is fully pressed (photographing instruction) and generates a second shutter switch signal SW2. The system control unit 50 starts a series of photographing processing operations, from reading out a signal from the imaging unit 22 to writing the captured image to the recording medium 200 as an image file, in response to the second shutter switch signal SW2.

[0030] The operation unit 70 is a variety of operation members that serve as an input unit for accepting operations from the user. The operation unit 70 includes at least the following operation members: the shutter button 61, the main electronic dial 71, the power switch 72, the sub electronic dial 73, the cross key 74, the SET button 75, the movie button 76, the AF lock button 77, the magnification button 78, the playback button 79, the menu button 81, and the touch bar 82. Other operation members 70b collectively represent operation members that are not individually shown in the block diagram.

[0031] The power supply control unit 80 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between powered blocks, etc., and detects whether a battery is installed, the battery type, and the remaining battery charge. The power supply control unit 80 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the required voltage for the required period to each unit, including the recording medium 200. The power supply unit 30 is composed of primary batteries such as alkaline batteries or lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, or Li batteries, an AC adapter, etc.

[0032] The recording medium I / F 18 is an interface with a recording medium 200 such as a memory card or a hard disk. The recording medium 200 is a recording medium such as a memory card for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like.

[0033] The communication unit 54 is connected wirelessly or via a wired cable, and transmits and receives video signals and audio signals. The communication unit 54 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 54 can also communicate with external devices via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 54 can transmit images (including LV images) captured by the imaging unit 22 and images recorded on the recording medium 200, and can also receive images and various other information from external devices.

[0034] The orientation detection unit 55 detects the orientation of the image capture device 100 with respect to the direction of gravity. Based on the orientation detected by the orientation detection unit 55, it is possible to determine whether an image captured by the image capture unit 22 was captured with the image capture device 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the orientation detected by the orientation detection unit 55 to the image file of the image captured by the image capture unit 22, or rotate and record the image. An acceleration sensor, a gyro sensor, or the like can be used as the orientation detection unit 55. The acceleration sensor or gyro sensor of the orientation detection unit 55 can also be used to detect movement of the image capture device 100 (panning, tilting, lifting, whether it is stationary, etc.).

[0035] 2(b) illustrates a configuration characteristic of this embodiment of the image processing unit 24. The image processing unit 24 has an image data generation unit 201, an object detection unit 202, a detection history storage unit 203, a dictionary data storage unit 204, and a dictionary data selection unit 205. The image data generation unit 201, the object detection unit 202, the detection history storage unit 203, the dictionary data storage unit 204, and the dictionary data selection unit 205 may be part of the system control unit 50, or may be provided separately from the image processing unit 24 and the system control unit 50. The image data generation, detection history storage, and image processing device may be installed in, for example, a smartphone, a tablet terminal, or the like.

[0036] The image processing unit 24 generates image data based on the data output from the A / D converter 23 and sends the generated image data to an image data generating unit 201 within the image processing unit 24 .

[0037] When dictionary data for whole area detection is selected by dictionary data selection unit 205 (described later), image data generation unit 201 uses input image data to generate image data to be used in subject detection unit 202 for whole area detection, and sends the image data to subject detection unit 202. On the other hand, when dictionary data for local area detection is selected by dictionary data selection unit 205, image data to be used in subject detection unit 202 for local area detection is generated based on the detection history in detection history storage unit 203 (described later), and sends the image data to subject detection unit 202. Specific methods for generating each type of detection image data will be described later.

[0038] The subject detection unit 202 receives dictionary data determined by the dictionary data selection unit 205 from dictionary data generated by machine learning and stored in the dictionary data storage unit 204, and performs subject detection on the image data input from the image data generation unit 201. The subject detection unit 202 estimates the position of the subject included in the image data as a detection result, and outputs the estimation result to the detection history storage unit 203 and to a location outside the configuration shown in FIG. 2(b).

[0039] In this embodiment, the object detection unit 202 is configured with a machine-learned CNN, and performs overall area detection and local area detection of a specific object. Objects for which overall area detection and local area detection are possible are based on dictionary data stored in the dictionary data storage unit 204. In this embodiment, the object detection unit 202 is configured with different CNNs (convolutional neural networks) for overall area detection and local area detection. Furthermore, different CNNs may be configured depending on the objects that can be detected. The object detection unit 202 may be realized by a GPU (graphics processing unit) or a circuit specialized for estimation processing using CNN.

[0040] The machine learning of the CNN can be performed by any method. For example, a predetermined computer such as a server may perform the machine learning of the CNN, and the imaging device 100 may acquire the trained CNN from the predetermined computer. In this embodiment, the predetermined computer inputs image data for training and performs supervised learning using position information and the like of the subject corresponding to the image data for training as training data (annotation), thereby training the CNN of the subject detection unit 202. In this manner, a trained CNN is generated. The training of the CNN may be performed by the imaging device 100 or the image processing device described above.

[0041] As described above, the object detection unit 202 includes a CNN (trained model) trained by machine learning. The object detection unit 202 receives image data as input, estimates the position, size, reliability, etc. of the object, and outputs the estimated information. The CNN may be, for example, a network in which a fully connected layer and an output layer are connected to a layer structure in which convolutional layers and pooling layers are alternately stacked. In this case, for example, backpropagation may be applied to train the CNN. The CNN may also be a neocognitron CNN that includes a feature detection layer (S layer) and a feature integration layer (C layer). In this case, a learning method called "Add-if Silent" may be applied to train the CNN.

[0042] The object detection unit 202 may use any trained model other than a trained CNN. For example, a trained model generated by machine learning such as a support vector machine or a decision tree may be applied to the object detection unit 202. Furthermore, the object detection unit 202 does not need to use a trained model generated by machine learning. For example, the object detection unit 202 may use any object detection method that does not use machine learning.

[0043] The detection history storage unit 203 stores the subject detection history detected by the subject detection unit 202 in the image data, and the system control unit 50 sends the history to the image data generation unit 201 and the dictionary data selection unit 205. In this embodiment, the subject detection history includes the dictionary data used for detection, the number of detections, the position of the detected subject, and the identifier of the image data containing the detected subject, but it may also be configured to store at least one of these pieces of data.

[0044] The dictionary data storage unit 204 stores dictionary data for detecting specific subjects, and the system control unit 50 reads dictionary data selected by the dictionary data selection unit 205 from the dictionary data storage unit 204 and sends it to the subject detection unit 202. Each dictionary data is, for example, data in which the characteristics of each area of ​​a specific subject are registered. Furthermore, to detect multiple types of subjects, dictionary data for each subject and each area of ​​the subject may be used. Therefore, multiple dictionary data are stored in the dictionary data storage unit 204. The dictionary data storage unit 204 stores multiple types of subject detection dictionary data, such as dictionary data for detecting "people," dictionary data for detecting "animals," and dictionary data for detecting "vehicles." In addition to "animals," the unit also stores dictionary data for detecting "birds," which have unique shapes and are in high demand for detection.

[0045] Furthermore, dictionary data for detecting "vehicles" can be further subdivided into "cars," "motorcycles," "trains," "airplanes," etc., and stored individually.

[0046] In this embodiment, dictionary data for detecting the entire region and dictionary data for detecting the local region are further prepared for each of the specific subjects. The entire region of a specific subject may be defined as the region literally including the entire subject, or as the region including the central portion of the subject, i.e., the main body. For example, the entire region of a "vehicle" subject can be defined for each type of subject, such as the "body" of a car or motorcycle, the "lead car" of a train, or the "fuselage" of an airplane. A local region is defined as a portion of the subject identified in the entire region. For example, a local region is defined as being included in the entire region, such as the "eyes of a person" as a local region for the "entire face of a person" as the entire region, or the "eyes of an animal" as a local region for the "entire face of an animal" as the entire region. Similarly, for a "bird," the "entire region of the bird" or the "entire face of the bird" is defined as the entire region, and the "eyes of the bird" are detected as a local region, and dedicated dictionary data is stored. Furthermore, a local region may not be included in the entire region, such as the "entire body of the motorcycle" as the entire region and the "driver's helmet" extending beyond the motorcycle body as a local region.

[0047] Furthermore, the relationship may be one in which a local area does not necessarily exist relative to the entire area of ​​the subject, such as "the entire body of a car" and "the driver's helmet" limited to a "formula car" among cars.

[0048] As described above, the dictionary data for detecting local regions is assumed to be a partial region of the subject detected in the entire region. Therefore, in this embodiment, the dictionary data used for detecting local regions is generated by learning an input image with the subject detected as the entire region as the background, and the positions or sizes of the local regions in the input image as annotations.

[0049] The entire area of ​​the subject detected using multiple dictionary data stored in the dictionary data storage unit 204 can be used as the focus detection area. For example, in a composition where an obstacle is present in the foreground and the subject is present in the background, it is possible to focus on the subject by focusing within the detected entire area.

[0050] However, for example, when there is a large difference in depth within the overall region, the problem of where to focus within the overall region arises. To address the aforementioned problem, local region detection can be used to limit the range, making it possible to focus on more specific positions that cannot be determined from the overall region and depth information within the region alone, such as the "driver's seat" of a train or the "cockpit" of an airplane. Furthermore, in the case of a "vehicle" such as a motorcycle, the desired focal position may differ depending on whether or not a person is riding. By performing overall region detection and local region detection using dictionary data in which the overall region is the "entire motorcycle body" and the local region is the "driver's helmet," it becomes possible to switch the focal position for the same subject depending on whether or not the driver is present.

[0051] Furthermore, in this embodiment, the multiple dictionary data used by the object detection unit 202 for detection are generated by machine learning, but dictionary data generated by a rule base may also be used. Dictionary data generated by a rule base is, for example, data that stores images of objects to be detected or feature quantities specific to the objects, as determined by a designer. The objects can be detected by comparing the images or feature quantities of the dictionary data with the images or feature quantities of captured image data. Rule-based dictionary data is less complicated than models set by machine learning trained models, and therefore requires less data volume. Object detection using rule-based dictionary data also has faster processing speeds (and a lower processing load) than trained models.

[0052] The dictionary data selection unit 205 selects the dictionary data to be used next based on the detection history stored in the detection history storage unit 203 and notifies the dictionary data storage unit 204 of the selected dictionary data.

[0053] In this embodiment, dictionary data for each of multiple types of objects and each area of ​​the object is stored separately in the dictionary data storage unit 204, and object detection is performed multiple times for the same image data by switching between multiple dictionary data. The dictionary data selection unit 205 determines a dictionary data switching sequence based on the detection history stored in the detection history storage unit 203 and a user selection (described later), and determines the dictionary data to be used in accordance with the determined sequence. Examples of dictionary data switching sequences will be described later.

[0054] The dictionary data storage unit 204 stores dictionary data for detecting multiple types of subjects and subject regions individually. The dictionary data selected by the dictionary data selection unit 205 is switched depending on whether or not any subjects have been detected so far, the type of dictionary data used at that time, the type of subject to be preferentially detected, and a combination thereof. The type of subject to be preferentially detected may be selected in advance by the user. Alternatively, the user may specify a subject within the live view screen displayed on the display unit 28. Furthermore, the user may select in advance whether to perform local region detection for each type of dictionary data for overall region detection or all at once. In this case, the system control unit 20 may display information about the user's selection or the dictionary data selected by the dictionary data selection unit 205 on the display unit 28.

[0055] As an example, FIG. 3 shows an example in which the user selects the type of subject to be detected with priority and whether or not to perform local area detection from a menu screen displayed on the display unit 28. FIG. 3(a) shows a setting screen for selecting a detection subject displayed on the display unit 28, and the user operates the operation unit 70 to select the subject to be detected with priority from specific detectable subjects (e.g., vehicles, animals, and people) (in this embodiment, "vehicle" is selected). FIG. 3(b) shows a setting screen for whether or not to perform local area detection displayed on the display unit 28, and the user operates the operation unit 70 to select local area detection ON / OFF (in this embodiment, local area detection ON is selected). FIG. 3(c) shows a live view screen displayed on the display unit 28 after the priority subject and local area detection settings have been made. The result of the selection of the priority subject is displayed on the display unit 28 as a subject icon 301, and the result of the selection of whether or not to perform local area detection (or whether or not to display the detection results) is displayed as a local area detection ON / OFF icon 302. This allows the user to confirm the settings they made on the live view screen (in this embodiment, the operation when a vehicle is selected as the subject to be detected with priority and local area detection ON is selected will be described later).

[0056] 4 is a flowchart showing the flow of characteristic processing of the present invention performed by the imaging device 100 of this embodiment. Each step of this flowchart is executed by the system control unit 50 or by each unit in response to an instruction from the system control unit 50. When this flowchart starts, it is assumed that the imaging device 100 is powered on and in live view imaging mode, and that an instruction to start capturing (recording) still images or videos can be given by operating the operation unit 70.

[0057] 4 is assumed to be processing performed when one frame (one piece of image data) is captured by the imaging unit 22 of the imaging device 100. However, this is not limiting, and the processing from step S401 to step S413 may be performed over multiple frames. In other words, the result of subject detection in the first frame may be reflected in any of the frames from the second frame onwards.

[0058] In step S401, the system control unit 50 acquires the captured image data captured by the imaging unit 22 and output by the A / D conversion unit .

[0059] In step S 402 , the image processing unit 24 resizes the image data to an image size that is easy to process (for example, QVGA), and sends the resized image data to the image data generation unit 201 .

[0060] In step S403, the dictionary data selection unit 205 selects dictionary data to be used for subject detection, and sends selection information indicating which dictionary data has been selected to the dictionary data storage unit 204 and the image data generation unit 201.

[0061] In step S404, the system control unit 50 determines whether or not dictionary data for whole area detection has been selected in step S403 by the dictionary data selection unit 205. If the determination in step S404 is Yes, the process proceeds to step S405, and if the determination is No, the process proceeds to step S408.

[0062] In step S405, in response to the selection of dictionary data for detecting the entire area, the image data generation unit 201 performs a process of resizing the entire image area to a size to be input to the subject detection unit 202, and sends the resized image data to the subject detection unit 202.

[0063] In step S406, subject detection unit 202 outputs information such as the position, size, and reliability of the subject detected using dictionary data for entire area detection, which is stored in dictionary data storage unit 204 and is used to detect the entire area of ​​a specific subject (object), to system control unit 50. At this time, the output of subject detection unit 202, the type of dictionary data used, and the identifier of the image data used for detection are stored in detection history storage unit 203. Furthermore, at this time, system control unit 50 may cause part or all of the information output by subject detection unit 202 to be displayed on display unit 28. The display method described above may also be changed depending on the dictionary data used.

[0064] In step S407, it is determined whether detection has been performed using all necessary dictionary data for image data having the same identifier (image data of the same frame) from the detection history stored in the detection history storage unit 203. If the determination result is Yes, the process ends, and if No, the process returns to step S403 and the next dictionary data to be used is selected.

[0065] In step S408, it is determined whether the local area detection function is ON. If Yes, the process proceeds to step S409, and if No, the process proceeds to step S407.

[0066] In step S409, it is checked whether the entire area was detected in the past detection of image data having the same identifier as the image data to be detected from the detection history stored in the detection history storage unit 203. If Yes, the process proceeds to step S410, and if No, the process proceeds to step S412.

[0067] In step S410, the image data generation unit 201 performs a cropping process on the image data for subject detection based on the detection results of the entire area of ​​image data (of the same frame) that has the same identifier as the image data to be detected, which is stored in the detection history storage unit 203. Specifically, an area that is wider by a predetermined angle of view (number of pixels) from the area where the entire area of ​​the subject was detected is set as the cropping area. The size and proportion of how wide the cropping area is set from the detected entire area may be set depending on the subject (object) to be detected. In step S410, the cropped image data is further resized to match the input size of the subject detection unit 202, and sent to the subject detection unit 202.

[0068] In step S411, subject detection unit 202 outputs information such as the position, size, and reliability of the subject detected using dictionary data for local area detection stored in dictionary data storage unit 204 to system control unit 50. At this time, the output of subject detection unit 202, the type of dictionary data used, and the identifier of the image data used for detection are stored in detection history storage unit 203. Furthermore, at this time, system control unit 50 may cause part or all of the information output by subject detection unit 202 to be displayed on display unit 28. The display method described above may also be changed depending on the dictionary data used.

[0069] In step S412, it is checked whether there is a detection result of the entire area or local area of ​​the same subject in a past frame from the detection history stored in the detection history storage unit 203. In this embodiment, a past frame refers to a frame corresponding to image data captured chronologically immediately before a frame having the same identifier and output to the image processing unit 24, but the number of frames going back chronologically does not matter as long as it is an earlier frame. If the determination in step S412 is Yes, the process proceeds to step S413. If No, the process proceeds to step S407.

[0070] In step S413, if necessary, a trimming process is performed corresponding to the angle of view to be input to the subject detection unit 202. In this embodiment, the image data generation unit 201 performs a trimming process on the image data based on the detection results of past image data stored in the detection history storage unit 203. Furthermore, in step S413, the trimmed image data is resized to match the input size of the subject detection unit 202, and sent to the subject detection unit 202.

[0071] Steps S409, S410, S412, and S413 are expected to improve the performance of local region detection using dictionary data generated by machine learning.

[0072] The relationship between the aforementioned improvement in performance of local region detection and steps S412 and S413 will now be described. In this embodiment, in step S412, it is checked whether the entire region of the subject has been detected in the past in image data having the same identifier as the image data to be detected from the detection history stored in the detection history storage unit 203. If the entire region of the subject has been detected in the past, step S41 3 As described above, the image data is resized in an area corresponding to the angle of view of the entire image, or based on the position and size of the entire area of ​​the subject detected in the past.

[0073] Here, dictionary data generated by machine learning can be generated by extracting common features of specific subjects from a large amount of image data containing those subjects. Common features include, for example, the size, position, and color of the subject, as well as areas outside the specific subject, such as the background. Therefore, the more limited the background in which the subject is detected, the easier it is to improve detection performance (detection accuracy) with less training. On the other hand, training to detect specific subjects regardless of the background provides high versatility across shooting scenes but is less likely to improve detection accuracy. The more image data and diversity are used to generate the dictionary data, the higher the detection performance tends to be. On the other hand, by limiting the size and position of the detection area of ​​the subject to be detected in the image data used for detection, it is possible to improve detection performance even if the number and diversity of image data required to generate the dictionary data is reduced. Furthermore, if part of the subject is outside the image data, some of the subject's features are lost, resulting in lower detection performance.

[0074] In general, the larger the area of ​​a subject, the more features it contains. In detection using the aforementioned machine-learned dictionary data, there is a possibility that an object with similar features other than the specific subject to be detected using the dictionary data may be mistakenly detected as the specific subject. The area defined as a local area is narrower than the entire area. The narrower the area, the fewer features it contains, and the fewer features it contains, the more objects with similar features there will be, leading to an increase in false detections.

[0075] FIG. 5 is a diagram showing an example of the positional relationship of local regions with respect to the overall region, where the overall region is defined as the "entire body of the car" and the local region is defined as the "driver's helmet." In FIG. 5, local region 502 is contained within overall region 501. FIG. 5(a) shows two pieces of image data in which the same car body exists at different positions, distances, and backgrounds. FIG. 5(b) shows images obtained by cropping the two aforementioned image data based on the overall region of the car. Compared to FIG. 5(a), cropping based on the overall region, as in the two images in FIG. 5(b), makes it possible to generate image data with narrower variations in the position, size, and background of the local region.

[0076] When detecting local regions from the entire image data (entire angle of view) regardless of whether or not there is a history of overall region detection, it is necessary to prepare image data of various captured scenes (both subject and background) and generate dictionary data for detecting local regions while preventing erroneous detection. On the other hand, in this embodiment, local regions are also detected from the entire image in scenes where the entire region of the subject was detected in previous frames but is no longer detected, i.e., when the entire region of the subject becomes so large that it is cut off from the captured angle of view. This is because the subject is sufficiently large in this case, and even if local regions are detected from the entire image, it is assumed that most of the background region is the subject region. Therefore, sufficiently high performance detection is possible even using dictionary data trained on the assumption that the background is the subject. Alternatively, in another embodiment, image data obtained by trimming the current processing frame based on the position and size of the entire region of the subject detected in previous frames can be used for local region detection, thereby reducing erroneous detection and improving detection performance. Furthermore, the trained model for local regions in this embodiment can reduce the variation in image data used for detection, thereby reducing the amount of image data and annotation data used for training.

[0077] Next, we will explain the relationship between the above-mentioned improvement in performance of local area detection and step S412. In this embodiment, if the detection history stored in the detection history storage unit 203 does not contain a history of detection of the entire area in image data having the same identifier as the image data to be detected, the system refers to past history to determine whether to perform local area detection.

[0078] FIG. 6 is a diagram showing an example of the positional relationship between the overall region and the local region when the overall region is defined as the "entire body" of the motorcycle and the local region as the "driver's helmet." In FIG. 6, local region 602 is not included in overall region 601. FIG. 6 shows the time series changes in captured image data when a subject approaches from FIG. 6(a) to FIG. 6(c). In FIG. 6(a), the local region cannot be detected because the subject is small, and in FIG. 6(c), the overall region cannot be detected because part of the subject is cut off. In a situation where a subject approaches from the back to the front as in FIG. 6(b), the image quality changes at nearby times as in FIG. 6(c). oh If the entire area or a local area of ​​the subject is detected in the previous frame, the subject is likely to be present in the next image data. Therefore, the system control unit 50 reads the detection history of image data of past frames stored in the detection history storage unit 203 and determines whether the entire area or a local area was detected in image data at a nearby time. Only if either area was detected, a local area is detected even if the entire area of ​​the subject cannot be detected in specific image data. A condition for detecting a local area may be a case in which the subject is approaching, as shown in FIG. 6, and it may be determined whether the entire area of ​​the subject is increasing with time (over time). Furthermore, a scene change may be determined by, for example, calculating the amount of change in the entire image or the entire area of ​​the subject from previous image data. As a result, if a major scene change occurs, local area detection may not be performed even if the entire area or a local area was detected in image data at a nearby time. Here, image data at a nearby time is compared with the current image data, and a major scene change is determined if the amount of change in the image data is equal to or greater than a certain threshold.

[0079] By increasing the chances of detecting local areas only for image data that is likely to contain local areas, the chances of incorrectly determining an area as a local area when no local area exists or when other objects similar to a local area exist can be reduced, thereby improving the performance of local area detection.

[0080] Next, the relationship between the above-mentioned improvement in performance of local area detection and step S413 will be described. In this embodiment, it is determined whether there is a history of detection of the entire area in image data having the same identifier as the image data to be processed (second image data) in the detection history stored in the detection history storage unit 203. If there is no history, the detection process is performed based on the detection history of the entire area or local area in the previous image (first image data).

[0081] 7(a), 7(b), and 7(c) are diagrams illustrating an example of the positional relationship between the overall region 701 and the local region 702, similar to FIGS. 6(a), 6(b), and 6(c), when the overall region is defined as the entire motorcycle body and the local region is defined as the driver's helmet. Both diagrams show time-series changes in captured image data. Also, FIG. 7(c) illustrates an example in which the overall region cannot be detected because part of the motorcycle body is cut off. If the overall region or local region of the subject was detected at a time close to the previous time, as in FIGS. 7(a) and 7(b) compared to FIG. 7(c), it is highly likely that the subject will also be present in the image data at the next time. Furthermore, by calculating the amount of movement from the positional relationship between the overall region and local region detected previously for a specific subject stored in the detection history storage unit 203 and the size of the detected region, it is possible to estimate the likely location of the subject in FIG. 7(c). FIG. 7(d) illustrates an example in which the image data of FIG. 7(c) has been cropped based on the previously detected detection history so that the local region is within a fixed size and position. By trimming the image data based on past history, as shown in Figure 7(d), the possibility of objects similar to the subject's features being present in the image data is reduced, image data that is easier to detect can be generated, and detection performance can be improved.

[0082] The trimming range may be determined by predicting the position of the next subject based on the positional relationship and movement amount of the whole area and local area detected in the past, as in the example shown in FIG. oh In cases where local detection has not been performed in the image capturing apparatus, or in cases where the positional relationship between the global region and the local region changes over time, the trimming range may be determined using only the detection result of the global region. Furthermore, depending on the movement of the subject, such as when the subject approaches the image capturing apparatus as in the example shown in FIG. 6(c), it is possible that the local region will not fit within the trimmed image even if trimming is performed based on the position of the global region in a previous frame. Therefore, the detection history for past image data (first image data) stored in the detection history storage unit 203 is checked. As a result, if it is determined that the global region does not fit within the image data based on the transitions in the positions and sizes of the global region and local region detected in image data at nearby times, local detection may be performed without trimming.

[0083] Next, a description will be given of the sequence of switching between multiple dictionary data sets for one frame (one piece of image data) by the imaging device 100, which is performed in step S403 of Fig. 4. When multiple dictionary data sets are stored in the dictionary data storage unit 204, it is possible to perform detection using multiple dictionaries for one frame. However, due to issues with imaging speed and processing speed, images captured sequentially are output and processed in live view mode, and video data during video recording may have a limited number of subject detections that can be performed for one frame.

[0084] In this case, the type and order of dictionary data to be used can be determined based on, for example, whether or not a subject has been detected in the past, the type of dictionary data used at that time, the type of subject to be detected with priority, and whether or not local area detection is performed. Depending on the dictionary data switching sequence, when a specific subject is included in a frame, the corresponding dictionary data for subject detection may not be selected, resulting in a missed detection opportunity. For this reason, the dictionary data switching sequence must also be changed according to the settings and scene.

[0085] For example, in a structure that can perform subject detection up to three times per frame (or has three detectors capable of processing in parallel), an example of a dictionary data switching sequence when a vehicle is selected as the subject to be detected first is shown in Figure 8. V0 and V1 each indicate the vertical synchronization period for one frame, and the rectangular blocks such as a person's head, vehicle 1 (motorcycle), and vehicle 2 (car) indicate that subject detection can be performed using three dictionary data (trained models) in chronological order within one vertical synchronization period.

[0086] FIG. 8(a) shows an example of dictionary data switching when no subject has already been detected in a frame. The dictionary data is switched in the following order: a person's head, vehicle 1 (motorcycle), and vehicle 2 (car). For example, if a switching sequence is not provided and dictionary data capable of detecting a subject selected by the user from the menu screen is always used, as in FIG. 3(a), the priority detection subject setting is switched for each scene, with vehicles being selected when a vehicle is present and people being selected otherwise, which requires time and effort. Furthermore, if it is unclear when a vehicle will arrive, switching the priority detection subject setting after noticing the vehicle may result in missing a shot. In contrast, in this embodiment, as in FIG. 8(a), during periods when no specific subject is detected, the "person's head," which is a detection target that is relatively likely to be a subject, is always detected, and dictionary data is switched for other subjects according to the priority detection subject setting. This reduces the number of times the priority detection subject setting needs to be switched.

[0087] Figure 8(b) shows an example of dictionary data switching in the next frame when a motorcycle is detected in the previous frame. The dictionary data is switched in the following order: Vehicle 1 (motorcycle) as a whole, the person's entire head, and then localized Vehicle 1 (motorcycle). The order of dictionary data switching does not necessarily need to be as described above. For example, the dictionary data for the person's entire head in the example of dictionary data switching described above may be switched to dictionary data that is more likely to be selected as a subject other than a motorcycle in a scene capturing a motorcycle, depending on the scene. The example in Figure 8(b) can be considered a sequence when the "vehicle priority" shooting mode is set. Furthermore, at this time, control may be exercised to avoid detection using dictionary data for "animals" and "birds," as well as their localized regions, which are unlikely to be detected alongside vehicles. This ultimately improves the detection accuracy of the desired subject.

[0088] Figure 8(c) shows an example of dictionary data switching when a person's head is detected in the previous frame, with dictionary data switching in the following order: the entire person's head, the entire vehicle 1 (motorcycle), and the person's eyes. Here, the person's eyes are a local area relative to the entire person's head. By using the entire vehicle 1 (motorcycle) as the second dictionary data, even if a local area of ​​the person is being detected, the entire vehicle area can be detected when the vehicle set as the priority detection subject enters the frame.

[0089] 8, multiple types of dictionary data for whole area detection are used in combination until a subject is detected using one of the multiple dictionary data stored in dictionary data storage unit 204. If there is a high possibility that the whole area will be detected after that, the subject to be detected can be narrowed down and dictionary data for local area detection can be used in combination to increase the detection rate.

[0090] Figure 8(d) shows an example of dictionary data switching when detecting an object by switching between multiple dictionary data sets for detecting the entire area over two frames. The same dictionary switching sequence is not used for each frame as in the examples shown in Figures 8(a), (b), and (c). For example, if there is no past detection history and no object is set as a priority for detection, a dictionary switching sequence spanning multiple frames as in the example shown in Figure 8(d) is used. Here, after the entire area has been detected using one of the dictionary data sets, the dictionary switching sequence may be switched to as in the example shown in Figure 8.

[0091] In addition, when a user specifies an area within the live view screen displayed on the display unit 28, a dictionary switching sequence spanning multiple frames, such as the example shown in Figure 8(d), may be used to identify the subject from the specified area.

[0092] FIG. 8(e) shows an example of dictionary data switching when local area detection OFF is selected, and is a sequence performed only on the entire vehicle 1 (motorcycle).

[0093] User-selected priority detection body If a vehicle is selected and local area detection is turned off, it can be assumed that the subject the user wants to capture is not the driver but the vehicle itself. The dictionary data generated by machine learning may mistakenly detect an object similar to the target object as the target object. The dictionary data switching sequence shown in Figure 8(b) is used when a vehicle is selected as the target object to be detected first, and is modified to not perform local area detection alone. In this case, because a helmet has a similar shape to a person's head, there is a possibility that it will be detected as the entire head area using dictionary data for detecting the entire head area. As shown in Figure 8(e), depending on the target object to be detected, a decrease in detection accuracy can be prevented by narrowing down the types of dictionary data used.

[0094] The above describes a case where the overall region and local regions of a specific subject are detected. Next, an embodiment will be described in which a tracking means is provided in addition to the detection of the overall region and local region as a method for estimating the overall region and local region of a subject. There are techniques for tracking a specific subject in an image, such as those that use brightness or color information or template matching, but in recent years, a technique that uses CNN has been attracting attention as a highly accurate tracking technique.

[0095] Non-Patent Document 1 describes a method for tracking a specific subject in an image. An image containing the target to be tracked and an image serving as a search range are input to a CNN with the same weights, and the position of the target to be tracked within the image in the search range is identified by calculating the cross-correlation between the features obtained from the CNN.

[0096] The tracking process is performed by the subject detection unit 202 in Figure 2, and the dictionary data storage unit 204 is provided with dictionary data for "tracking" in addition to the dictionary data for "people" and "vehicles." Note that "tracking" can be applied to any subject, and the initial value for tracking is based on the subject detection result using the dictionary data for "people" and "vehicles." For the first frame of tracking, the feature amount obtained from the CNN is registered as a template in the detection history storage unit 203 based on the detection area of ​​the subject detection. For subsequent frames, the subject area is estimated by template matching using the registered template and the input image.

[0097] Fig. 9 adds tracking processing to the flowchart of Fig. 4. As in Fig. 4, each step of this flowchart is executed by the system control unit 50 or by each unit in response to an instruction from the system control unit 50. When this flowchart starts, it is assumed that the imaging device 100 is powered on and in live view imaging mode, and that an instruction to start capturing (recording) a still image or a video can be given by operating the operation unit 70.

[0098] Steps S901 to S907 are similar to steps S401 to S407, and therefore description thereof will be omitted. In step S904, the system control unit 50 determines whether or not the dictionary data selection unit 205 selected dictionary data for entire area detection in step S903. If the determination in step S904 is Yes, the process proceeds to step S905; if the determination is No, the process proceeds to step S908. In step S908, the system control unit 50 determines whether or not the dictionary data selection unit 205 selected dictionary data for tracking. If the determination in step S908 is Yes, the process proceeds to step S909; if the determination is No, the process proceeds to step S910. Here, the tracking process in the object detection unit 202 in this embodiment requires an object detection result based on the dictionary data for "person" and "vehicle." Therefore, if the detection result of the entire area in step S906 or the detection result of the local area in step S913, which will be described later, has not been obtained in advance, the tracking dictionary data will not be selected. In this case, the detection flow without using the dictionary data for "tracking" is repeated until the entire area and local areas of the subject are detected using the dictionary data for "person" and "vehicle."

[0099] In step S909, in response to the selection of tracking dictionary data, subject detection unit 202 performs tracking processing. Subject detection unit 202 outputs information such as the position, size, and reliability of the subject detected using the tracking dictionary data stored in dictionary data storage unit 204 to system control unit 50. At this time, the output of subject detection unit 202, the type of dictionary data used, and the identifier of the image data used for detection are stored in detection history storage unit 203. Furthermore, at this time, system control unit 50 may cause display unit 28 to display some or all of the information output by subject detection unit 202. In addition, the display method described above may be changed depending on the dictionary data used.

[0100] Step S90 in FIG. 9 is executed by the subject detection unit 202. 9 Regarding the tracking process flow This will be explained using FIG. 10.

[0101] In step S1001, the system control unit 50 determines whether the frame is a tracking initial frame based on the detection history stored in the detection history storage unit 203. Here, the tracking initial frame refers to a frame at the timing when a subject appears or when the subject changes. For example, this is a frame in which the entire area or local areas of the subject have been successfully detected using new dictionary data for "people" or "vehicles." If the frame is a tracking initial frame (YES in step S1001), the process proceeds to step S1002.

[0102] In step S1002, the image data generation unit 201 performs a trimming process on the tracking image data based on the detection results of the entire area or local area of ​​image data (of the same frame) that has the same identifier as the image data to be tracked, which is stored in the detection history storage unit 203. Specifically, the entire area of ​​the subject or the area where the local area is detected is trimmed to a predetermined size.

[0103] In step S 1003 , the image data that has been further trimmed by the image data generation unit 201 is resized to match the input size of the subject detection unit 202 and sent to the subject detection unit 202 .

[0104] In step S1004, feature extraction is performed by CNN in the subject detection unit 202. This CNN is trained in advance to extract feature amounts suitable for tracking, and its weighting coefficients and operation parameters are stored as dictionary data for tracking in the dictionary data storage unit 204. In step S1005, a partial region of the feature amounts extracted in step S1004 is registered as a template in the detection history storage unit 203. In the case of the initial tracking frame, this marks the end of the tracking process.

[0105] If in step S1001 the tracking process is for the second frame or later (not the initial tracking frame), the process proceeds to step S1006. In step S1006, image data generation unit 201 performs a trimming process on the tracking image data based on the tracking area of ​​the immediately previous frame stored in detection history storage unit 203. Here, as with the initial frame, trimming is performed so that the tracking area has a predetermined size. In step S1007, image data generation unit 201 further resizes the trimmed image data to match the input size of subject detection unit 202, and sends it to subject detection unit 202.

[0106] In step S1008, the subject detection unit 202 extracts features from the trimmed image data using CNN, based on the dictionary data for tracking stored in the dictionary data storage unit 204.

[0107] In step S1009, a tracking area is estimated by performing a correlation calculation between the feature amount extracted in step S1008 and the feature amount of a template registered in detection history storage unit 203. This estimated result is used as the tracking result, and the process ends. In this tracking process, trimming is performed based on the subject area in step S1006, so changes in the size of the subject over time are reduced, thereby improving tracking performance.

[0108] When the tracking process in step S909 is completed, the process proceeds to step S907. In step S907, the system control unit 50 determines whether detection has been performed using all of the necessary dictionary data for image data having the same identifier (image data of the same frame) from the detection history stored in the detection history storage unit 203. If the determination result is Yes, the process ends, and if No, the process returns to step S903 and the next dictionary data to be used is selected.

[0109] In step S910, the system control unit 50 determines whether the local area detection function is ON. If Yes, the process proceeds to step S911, and if No, the process proceeds to step S907.

[0110] In step S911, the system control unit 50 checks whether there is a detection result or a tracking result for the entire area for image data having the same identifier as the image data to be detected from the detection history stored in the detection history storage unit 203. If Yes, the process proceeds to step S912, and if No, the process proceeds to step S914. Note that if the reliability of the entire area detection in step S906 is low, or if the reliability of tracking in step S909 is low, it is determined that there is no result.

[0111] In step S912, the image data generation unit 201 performs a trimming process on the image data for subject detection based on the detection result or tracking result of the entire area of ​​image data (of the same frame) that has the same identifier as the image data to be detected, which is stored in the detection history storage unit 203. Specifically, an area that is wider by a predetermined angle of view (number of pixels) than the area where the entire area of ​​the subject was detected or the tracking area is set as the trimming area. The size and ratio of how wide the trimming area is set from the detected entire area and tracking area may be set depending on the target subject (object). Additionally, in step S912, the image data generation unit 201 resizes the image data that has been further trimmed to match the input size of the subject detection unit 202 and sends it to the subject detection unit 202. Note that if both the entire detection result and the tracking result are available, the entire detection result is treated as having priority.

[0112] In step S913, the subject detection unit 202 outputs information such as the position, size, and reliability of the subject detected using the dictionary data for local area detection stored in the dictionary data storage unit 204 to the system control unit 50. At this time, the output of the subject detection unit 202, the type of dictionary data used, and the identifier of the image data used for detection are stored in the detection history storage unit 203. Furthermore, at this time, the system control unit 50 may cause the display unit 28 to display some or all of the information output by the subject detection unit 202. The display method described above may also be changed depending on the dictionary data used.

[0113] Next, the process proceeds to step S907. In step S907, if the whole area detection, tracking, and local area detection have been completed in the same frame, the determination is YES, and the process ends.

[0114] The processing from step S914 to step S915 is the same as the processing from step S412 to step S413, and therefore a description thereof will be omitted.

[0115] Compared to the processing flow in Figure 4, the processing flow in Figure 9 adds tracking processing of the entire area. This improves the ability to estimate the entire area. As a result, the probability of determining YES in step S910 increases. In other words, there are more cases where cropping and resizing are performed based on the detection or tracking results of the same frame, rather than cropping and resizing based on the detection history of past frames. When the subject is moving, the accuracy of local detection is improved by generating an image based on detection and tracking information of the same frame rather than based on past information.

[0116] Next, a description will be given of a switching sequence of multiple dictionary data for one frame (one piece of image data) by the imaging device 100, which is performed in step S903 in Fig. 9. As an example, Fig. 11 shows an example of a dictionary data switching sequence when a vehicle is selected as the subject to be detected with priority in a structure that can perform subject detection up to three times per frame (or has three detectors that can process in parallel), similar to the description in Fig. 8. In Fig. 11, a tracking block is added compared to Fig. 8. Figs. 11(a), (c), and (d) are similar to Fig. 8, and therefore description will be omitted.

[0117] Figure 11(b) shows an example of dictionary data switching in the next frame when a motorcycle is detected in the previous frame, with dictionary data switching in the following order: vehicle 1 (motorcycle) as a whole, tracking, and then local vehicle 1 (motorcycle). For tracking, in the initial frame, a template is registered based on the detection area of ​​vehicle 1 (motorcycle) as a whole, and from the second frame onwards, the area to be tracked is estimated by template matching. This means that there are two methods for estimating vehicle 1 (motorcycle) as a whole: subject detection and tracking, improving the detection rate.

[0118] Figure 11(e) is an example of dictionary data switching when local area detection is OFF, switching dictionary data in the order of vehicle 1 (motorcycle) as a whole and tracking. If the user selects the vehicle as the subject to be detected first and then selects local area detection OFF, it can be assumed that the subject the user wants to photograph is the vehicle body, not the driver. By adding tracking processing to Figure 8(e), the detection rate can be improved.

[0119] In the above explanation, we have described a method in which the subject detection unit 202 detects objects of a specific category, such as "bike" or "person," and tracks them based on the detected area, but there may also be a method in which any object specified by the user is tracked.

[0120] In object detection for a specific category, object detection is achieved by learning objects in a specific category to be detected, such as "bikes" or "people." Dictionary data is provided for each specific category, and the dictionary data selection unit 205 switches the dictionary to be used depending on the shooting conditions, enabling the detection of specific subjects. On the other hand, in object detection for any specific object, learning is performed using information on objects in various categories, enabling simultaneous detection of objects in various categories without depending on the object category. This object candidate detection can accommodate various categories with a single dictionary data. However, while object candidate detection can accommodate objects in various categories, it can be difficult to determine a single tracking target from among multiple object-like regions detected in an image. Therefore, as a solution, a method is described in which an object candidate region corresponding to a position specified by the user is used as the tracking region. The user's specification can be input, for example, via the touch panel 70a of the imaging device 100.

[0121] FIG. 11(f) shows an example of dictionary data switching when a user inputs a position.

[0122] Timing V0 in FIG. 11(f) is the timing at which a position is specified by the user, and shows an example in which the target determined at V0 is tracked at V1 and thereafter. At timing V0, the user's position input triggers the dictionary data selection unit 205 to select an object candidate dictionary stored in the dictionary data storage unit 204, and the object detection unit 202 uses that dictionary to detect object candidates. The object detection unit 202 also sets the detected object candidate closest to the specified position as the tracking target. Next, the dictionary data selection unit 205 selects a tracking dictionary from the dictionary data storage unit 204, and the object detection unit 202 registers a template. From V1 onward, the dictionary data selection unit 205 selects a tracking dictionary and performs tracking processing based on the registered template. Switching of dictionary data when the user inputs a position is common regardless of the setting of the object to be detected. Furthermore, because the object specified by the user is being tracked, automatic switching of the tracking target is unnecessary, and subject detection using dictionary data for other specific objects is not performed.

[0123] The process flow is the same as that explained in FIG. 9. Step S90 6 In step S903, if a position is input by the user, an object candidate dictionary is selected, a determination of YES is made in step S904, and object candidate detection processing is performed using the image of the entire region. 7 In step S904, the result is determined to be NO, and the process returns to step S903, where the tracking dictionary is selected. in If the determination is NO, step S90 8 The answer is YES in step S90. 9 Then, in step S90, the tracking process is performed. 7 So YES This flow is the processing at timing V0 shown in FIG. 11(f). In the processing after V1 in FIG. 11(f), steps S901, S902, S903, S904, S905, S906, S907, S908, S909, S910, S911, S912, S913, S914, S915, S916, S917, S918, S919, S920, S921, S922, S923, S924, S925, S926, S927, S928, S929, S930, S931, S932, S933, S934, S935, S936, S937, S938, S939, S940, S941, S942, S943, S944, 8 , step S90 9 , step S90 7 The process proceeds in this order and ends.

[0124] The sequence for switching between a plurality of dictionary data for one frame (one piece of image data) by the imaging device 100 has been described above.

[0125] [Other embodiments] The embodiments described above are merely representative examples, and various modifications and alterations are possible to each embodiment when implementing the present invention.

[0126] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0127] 100 Imaging device 28 Display section 50 System control section 201 Image data generation unit 202 Subject detection unit 203 Detection history memory unit 204 Dictionary data storage unit 205 Dictionary Data Selection Unit

Claims

1. An acquisition means for acquiring image data; a first detection means for detecting a first region of the object from the image data; a second detection means for detecting a second region of the subject from the image data; A tracking means for tracking the subject. The second detection means is When the first detection means detects the first region in the first image data, the second region is detected in a region corresponding to the first region; when the first area is detected in the first image data and the first area is not detected in the second image data captured after the first image data, the second area is detected in image data that is larger than the first area detected in the first image data and that has been processed based on the position and size of the first area detected in the first image data; The image processing device according to claim 1, wherein the tracking means tracks the subject using at least one of the detection results from the first and second detection means.

2. An image processing device as described in Claim 1, characterized in that the image data processed based on the position and size of the first area detected in the first image data is image data trimmed to an area based on the position and size of the first area detected in the first image data.

3. 3. The image processing device according to claim 1, wherein the first and second detection means respectively use a first trained model that has been machine-learned in advance and a second trained model that has been machine-learned in advance.

4. The image processing device according to claim 3, characterized in that the second trained model is a model trained using an image having a subject to be detected as the first region as a background as an input image, and using position information of the second region in the input image as an annotation.

5. The image processing device according to claim 4, characterized in that the first trained model is a model that is trained by using an image including a subject and a background to be detected as the first region as an input image, and using position information of the first region in the input image as an annotation.

6. The image processing device according to claim 1 , wherein the tracking means tracks the subject using a third trained model that has been machine-learned in advance.

7. The image processing device according to claim 6 , wherein the third trained model is a model trained to extract features suitable for tracking.

8. 8. The image processing device according to claim 1, wherein the first area is larger than the second area.

9. third and fourth detection means for detecting at least a partial area of ​​a subject different from the subjects detected by the first and second detection means, respectively; A setting means for setting an imaging mode; 9. The image processing apparatus according to claim 1, further comprising: a control unit which switches among the first, second, third and fourth detection units in accordance with a photography mode set by the setting unit.

10. 10. The image processing apparatus according to claim 9, wherein the first and second regions correspond to at least a portion of a vehicle, and the second region is smaller than the first region.

11. The third detection means detects a third region corresponding to at least a part of a human, and the fourth detection means detects a fourth region corresponding to at least a part of an animal other than a human; 11. The image processing apparatus according to claim 10, wherein said control means does not cause said fourth detection means to detect image data that is to be detected by said first detection means.

12. An acquisition means for acquiring image data; a first detection means for detecting a first region of the object from the image data; a second detection means for detecting a second region of the subject from the image data; A tracking means for tracking the subject. The second detection means is When the first detection means detects the first region in the first image data, the second region is detected in a region corresponding to the first region; When the first region is detected in the first image data and the first region is not detected in the second image data captured after the first image data, the second region is detected in an area larger than the first region detected in the first image data; when the first area is not detected by the first detection means in the first image data and the second area is detected by the second detection means, the second area is detected in the second image data based on the position and size of the second area detected in the first image data; The image processing device according to claim 1, wherein the tracking means tracks the subject using at least one of the detection results from the first and second detection means.

13. An acquisition means for acquiring image data; a first detection means for detecting a first region of the object from the image data; a second detection means for detecting a second region of the subject from the image data; A tracking means for tracking the subject. The second detection means is When the first detection means detects the first region in the first image data, the second region is detected in a region corresponding to the first region; When the first region is detected in the first image data and the first region is not detected in the second image data captured after the first image data, the second region is detected in an area larger than the first region detected in the first image data; when at least one of the first area and the second area is detected in the first image data and it is determined that the first area does not fit within the current image data, detect the second area from the acquired image data without performing cropping; The image processing device according to claim 1, wherein the tracking means tracks the subject using at least one of the detection results from the first and second detection means.

14. An acquisition means for acquiring image data; a first detection means for detecting a first region of the object from the image data; a second detection means for detecting a second region of the subject from the image data; A tracking means for tracking the subject. The second detection means is When the first detection means detects the first region in the first image data, the second region is detected in a region corresponding to the first region; When the first region is detected in the first image data and the first region is not detected in the second image data captured after the first image data, the second region is detected in an area larger than the first region detected in the first image data; when the second area is detected in the first image data and the first area is detected in the detection result in the second image data, cropping the image data so that a size relationship between the first area and the second area in the second image data becomes similar to a size relationship between the first area and the past second area in the first image data, and detecting the second area; The image processing device according to claim 1, wherein the tracking means tracks the subject using at least one of the detection results from the first and second detection means.

15. An acquisition means for acquiring image data; a first detection means for detecting a first region of the object from the image data; a second detection means for detecting a second region of the subject from the image data; A tracking means for tracking the subject. The second detection means is When the first detection means detects the first region in the first image data, the second region is detected in a region corresponding to the first region; When the first region is detected in the first image data and the first region is not detected in the second image data captured after the first image data, the second region is detected in an area larger than the first region detected in the first image data; calculating an amount of movement of the first area and the second area by using the first image data in which the first area or the second area is detected, generating image data by trimming an area in the second image data in which the first area or the second area is predicted to exist from the image data acquired by the acquisition means, and detecting the second area; The image processing device according to claim 1, wherein the tracking means tracks the subject using at least one of the detection results from the first and second detection means.

16. An acquisition means for acquiring image data; a first detection means for detecting a first region of the object from the image data; a second detection means for detecting a second region of the subject from the image data; A tracking means for tracking the subject. The second detection means is When the first detection means detects the first region in the first image data, the second region is detected in a region corresponding to the first region; When the first region is detected in the first image data and the first region is not detected in the second image data captured after the first image data, the second region is detected in an area larger than the first region detected in the first image data; comparing a plurality of image data including the first image data in which the first region or the second region is detected, and when it is determined that the first region or the second region has become larger over time, detecting the second region in the second image data; The image processing device according to claim 1, wherein the tracking means tracks the subject using at least one of the detection results from the first and second detection means.

17. An acquisition means for acquiring image data; a first detection means for detecting a first region of the object from the image data; a second detection means for detecting a second region of the subject from the image data; A tracking means for tracking the subject. The second detection means is When the first detection means detects the first region in the first image data, the second region is detected in a region corresponding to the first region; When the first region is detected in the first image data and the first region is not detected in the second image data captured after the first image data, the second region is detected in an area larger than the first region detected in the first image data; comparing a plurality of past image data including the first image data in which the first region and the second region are detected, estimating a relationship in size between the first region and the second region in the current image from a transition in size between the first region and the second region, trimming the image data acquired by the acquisition means so that the relationship in size between the first region and the second region in the past is similar to that between the first region and the second region in the past, and detecting the second region; The image processing device according to claim 1, wherein the tracking means tracks the subject using at least one of the detection results from the first and second detection means.

18. An acquisition means for acquiring image data; a first detection means for detecting a first region of the object from the image data; a second detection means for detecting a second region of the subject from the image data; A tracking means for tracking the subject. The second detection means is When the first detection means detects the first region in the first image data, the second region is detected in a region corresponding to the first region; When the first region is detected in the first image data and the first region is not detected in the second image data captured after the first image data, the second region is detected in an area larger than the first region detected in the first image data; Even if the first region is not detected in the second image data and the first region is detected in the first image data, if an amount of change in image data calculated from the first image data and the second image data is equal to or greater than a certain threshold value, the second region is not detected; The image processing device according to claim 1, wherein the tracking means tracks the subject using at least one of the detection results from the first and second detection means.

19. An imaging element; An image processing device according to any one of claims 1 to 18, An imaging device comprising:

20. An acquisition step of acquiring captured image data; a first detection step of detecting a first region of the object from the image data; a second detection step of detecting a second region of the object from the image data; a tracking step of tracking the subject, In the second detection step, when the first region is detected in the first image data in the first detection step, the second region is detected in a region corresponding to the first region; when the first area is detected in the first image data and the first area is not detected in the second image data captured after the first image data, the second area is detected in image data that is larger than the first area detected in the first image data and that has been processed based on the position and size of the first area detected in the first image data; a control method for an image processing device, wherein in the tracking step, the subject is tracked using at least one of the detection results in the first and second detection steps.

21. A program for causing a computer to execute each step of the control method according to claim 20.

22. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to claim 20.

Images