IMAGE PROCESSING APPARATUS, IMAGING APPARATUS, IMAGE PROCESSING METHOD, AND PROGRAM

Abstract

To make it possible to accurately detect of an entire region and a local region of a specific subject.SOLUTION: Included are acquisition means configured to acquire captured image data; first detection means configured to detect a first region of a subject from the image data; second detection means configured to detect a second region of the subject from the image data; and output means configured to output a result of detection by the first detection means and a result of detection by the second detection means. In a case where the first detection means detects the first region in first image data, the second detection means detects the second region in a region corresponding to the first region. In a case where the first region is detected in the first image data and the first region is not detected in second image data captured after the first image data, the second detection means detects the second region in a region wider than the first region detected in the first image data.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing device, an imaging device, an image processing method, and a program capable of executing subject detection processing for detecting a specific subject. [Background technology]

[0002] There is a technology for performing image processing for detecting a subject based on a trained model that has undergone machine learning to detect a specific subject from image data captured by an imaging device such as a digital camera. In Patent Document 1, a specific area of ​​the subject is automatically detected, and then a narrower area included in the area is detected, thereby making it possible to detect a local area of ​​the specific subject. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-197795 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in Patent Document 1, if the entire area of ​​a specific subject cannot be detected, then a local area cannot be detected.

[0005] In view of the above problems, an object of the present invention is to provide an image processing device, an imaging device, an image processing method, and a program that are capable of detecting a global area and local areas of a specific subject with high accuracy. [Means for solving the problem]

[0006] In order to achieve the above object, the image processing device of the present invention comprises: The imaging device includes an acquisition means for acquiring captured image data, a first detection means for detecting a first region of a subject from the image data, and a second detection means for detecting a second region of the subject from the image data, and when the first detection means detects the first region in the first image data, the second detection means detects the second region in a region corresponding to the first region, and when the first region is detected in the first image data and the first region is not detected in second image data captured after the first image data, the second detection means detects the second region in image data that is a region larger than the first region detected in the first image data and that has been processed based on the position and size of the first region detected in the first image data. . Effect of the Invention

[0007] According to the present invention, the entire area and local areas of a subject can be detected with high accuracy. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an external view of an imaging device. [Diagram 2] FIG. 2 is a diagram illustrating the relationship between a system control unit and each unit. [Diagram 3] FIG. 11 is a diagram showing an example of a detection target setting method by a user. [Figure 4] 3 is a flowchart showing a process flow of the present embodiment. [Diagram 5] FIG. 13 is a diagram illustrating an example of the positional relationship of a local region with an entire region. [Figure 6] FIG. 2 is a diagram showing a time series change in a captured image when a subject is photographed. [Figure 7] 11A and 11B are diagrams illustrating an example of a method for trimming image data used for subject detection. [Figure 8] FIG. 13 is a diagram showing an example of a switching sequence of a plurality of dictionary data for one frame. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0010] 1(a) and (b) show external views of an image capturing device 100 as an example of an apparatus to which the present invention can be applied. Fig. 1(a) is a front perspective view of the image capturing device 100, and Fig. 1(b) is a rear perspective view of the image capturing device 100.

[0011] In FIG. 1, display unit 28 is a display unit provided on the rear surface of the camera that displays images and various information. Touch panel 70a can detect touch operations on the display surface (operation surface) of display unit 28. Outside-finder display unit 43 is a display unit provided on the top surface of the camera, and displays various settings of the camera, including shutter speed and aperture. Shutter button 61 is an operation unit for issuing shooting instructions. Mode change switch 60 is an operation unit for switching between various modes. Terminal cover 40 is a display unit that can be used to connect to external devices. To shoot It is a cover for protecting a connector (not shown) such as a connection cable that connects to the imaging device 100.

[0012] The main electronic dial 71 is a rotary operation member included in the operation unit 70, and by rotating the main electronic dial 71, the settings such as the shutter speed and the aperture can be changed. The power switch 72 is an operation member for switching the power of the imaging device 100 ON and OFF. The sub electronic dial 73 teeth, A rotary operation member included in the operation unit 70, which can be used to move the selection frame, advance images, etc. The cross key 74 is included in the operation unit 70, and is a cross key (four-way key) that can be pressed up, down, left, or right. An operation can be performed according to which part of the cross key 74 is pressed. The SET button 75 is included in the operation unit 70, and is a push button that is mainly used to confirm a selection item, etc.

[0013] The video button 76 is used to start and stop video shooting (recording). The AE lock button 77 is included in the operation unit 70, and by pressing it in the shooting standby state, the exposure state can be fixed. The enlargement button 78 is included in the operation unit 70, and is an operation button for turning the enlargement mode ON and OFF in the live view display in the shooting mode. By turning the enlargement mode ON and operating the main electronic dial 71, the LV image can be enlarged or reduced. In the playback mode, it functions as an enlargement button for enlarging the playback image and increasing the magnification ratio. The playback button 79 is included in the operation unit 70, and is an operation button for switching between the shooting mode and the playback mode. By pressing the playback button 79 during the shooting mode, the mode is switched to the playback mode, and the latest image among the images recorded in the recording medium 200 can be displayed on the display unit 28. The menu button 81 is included in the operation unit 70, and when the menu button 81 is pressed, a menu screen in which various settings can be made is displayed on the display unit 28. The user can intuitively make various settings using the menu screen displayed on the display unit 28, the cross key 74, and the SET button 75.

[0014] The touch bar 82 is a line-shaped touch operation member (line touch sensor) capable of receiving touch operations, and is disposed at a position operable by the thumb of the right hand gripping the grip portion 90. The touch bar 82 can receive tap operations (operations in which the user touches and then releases the touch bar without moving it within a predetermined period of time), left and right slide operations (operations in which the user touches and then moves the touched position while still touching), and the like. The touch bar 82 is an operation member different from the touch panel 70a, and does not have a display function.

[0015] The communication terminal 10 is a communication terminal for the imaging device 100 to communicate with the lens side (detachable). The eyepiece 16 is an eyepiece of an eyepiece finder (a peer-in type finder), and the user can view an image displayed on the EVF 29 inside the eyepiece 16 through the eyepiece 16. The eyepiece detection unit 57 is an eyepiece detection sensor that detects whether the photographer places his / her eye on the eyepiece 16. The cover 202 is a cover for a slot that stores the recording medium 200. The grip unit 90 is a holding unit that is shaped so that the user can easily hold it with the right hand when holding the imaging device 100. When the user holds the digital camera by holding the grip unit 90 with the little finger, ring finger, and middle finger of the right hand, the shutter button 61 and the main electronic dial 71 are arranged at positions that can be operated with the index finger of the right hand. In the same state, the sub electronic dial 73 and the touch bar 82 are arranged at positions that can be operated with the thumb of the right hand.

[0016] FIG. 2(a) is a block diagram showing an example of the configuration of an image pickup device 100 according to this embodiment. In FIG. 2, a lens unit 150 is a lens unit equipped with an interchangeable photographing lens. The lens 103 is usually composed of multiple lenses, but is shown here as a single lens for simplicity. The communication terminal 6 is a communication terminal for the lens unit 150 to communicate with the image pickup device 100, and the communication terminal 10 is a communication terminal for the image pickup device 100 to communicate with the lens unit 150. The lens unit 150 communicates with the system control unit 50 via the communication terminals 6 and 10, controls the aperture 1 via the aperture drive circuit 2 by the internal lens system control circuit 4, and adjusts the focus by displacing the position of the lens 103 via the AF drive circuit 3.

[0017] The shutter 101 is a focal plane shutter that can freely control the exposure time of the imaging unit 22 under the control of the system control unit 50 .

[0018] The imaging unit 22 is an imaging element configured with a CCD, CMOS element, or the like that converts an optical image into an electrical signal. The imaging unit 22 may have an imaging surface phase difference sensor that outputs defocus amount information to the system control unit 50. The A / D converter 23 converts an analog signal into a digital signal. The A / D converter 23 is used to convert the analog signal output from the imaging unit 22 into a digital signal.

[0019] The image processing unit 24 performs predetermined pixel interpolation, resizing such as reduction, and color conversion processing on the data from the A / D converter 23 or the data from the memory control unit 15. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data. The system control unit 50 performs exposure control and distance measurement control based on the arithmetic results obtained by the image processing unit 24. This allows TTL (through-the-lens) type AF (autofocus) processing, AE (autoexposure) processing, and EF (flash pre-flash) processing to be performed. The image processing unit 24 also performs predetermined arithmetic processing using the captured image data, and performs TTL type AWB (auto white balance) processing based on the arithmetic results obtained.

[0020] The output data from the A / D converter 23 is written into the memory 32 via the image processing unit 24 and the memory control unit 15, or directly via the memory control unit 15. The memory 32 stores image data obtained by the imaging unit 22 and converted into digital data by the A / D converter 23, and image data to be displayed on the display unit 28 and EVF 29. The memory 32 has a storage capacity sufficient to store a predetermined number of still images and a predetermined period of moving images and audio.

[0021] The memory 32 also serves as a memory (video memory) for image display. The D / A converter 19 converts the image display data stored in the memory 32 into an analog signal and supplies it to the display unit 28 and the EVF 29. In this way, the image data for display written in the memory 32 is displayed by the display unit 28 and the EVF 29 via the D / A converter 19. The display unit 28 and the EVF 29 perform display according to the analog signal from the D / A converter 19 on a display such as an LCD or an organic EL. The digital signal once A / D converted by the A / D converter 23 and stored in the memory 32 is converted to analog in the D / A converter 19 and sequentially transferred to and displayed on the display unit 28 or the EVF 29, thereby performing live view display (LV display). Hereinafter, an image displayed in live view is referred to as a live view image (LV image).

[0022] Various camera settings such as shutter speed and aperture are displayed on the outside viewfinder liquid crystal display 43 via an outside viewfinder display drive circuit 44 .

[0023] The non-volatile memory 56 is an electrically erasable and recordable memory, and may be, for example, an EEPROM. Constants, programs, and the like for operating the system control unit 50 are stored in the non-volatile memory 56. The programs referred to here are programs for executing various flowcharts described later in this embodiment.

[0024] The system control unit 50 is a control unit composed of at least one processor or circuit, and controls the entire imaging device 100. Each process of this embodiment, which will be described later, is realized by executing a program recorded in the nonvolatile memory 56 described above. The system memory 52 uses, for example, a RAM, and constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 56, and the like are loaded therein. The system control unit 50 also performs display control by controlling the memory 32, the D / A converter 19, the display unit 28, and the like.

[0025] The system timer 53 is a timing unit that measures the time used for various controls and the time of a built-in clock.

[0026] The operation unit 70 is an operation means for inputting various operation instructions to the system control unit 50. The mode changeover switch 60 is an operation member included in the operation unit 70, and switches the operation mode of the system control unit 50 to one of a still image shooting mode, a video shooting mode, a playback mode, etc. Modes included in the still image shooting mode include an auto shooting mode, an auto scene discrimination mode, a manual mode, an aperture priority mode (Av mode), a shutter speed priority mode (Tv mode), and a program AE mode (P mode). In addition, there are various scene modes and a custom mode that are shooting settings according to shooting scenes. The user can directly switch to one of these modes using the mode changeover switch 60. Alternatively, after switching to a list screen of shooting modes using the mode changeover switch 60, one of the displayed modes may be selected and switched using other operation members. Similarly, the video shooting mode may also include multiple modes.

[0027] The first shutter switch 62 is turned on and generates a first shutter switch signal SW1 when the shutter button 61 provided on the imaging device 100 is pressed halfway (instruction to prepare for shooting) during operation of the shutter button 61. The first shutter switch signal SW1 starts shooting preparation operations such as AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, and EF (pre-flash) processing.

[0028] The second shutter switch 64 is turned on when the shutter button 61 is fully pressed (photographing instruction) and generates a second shutter switch signal SW2. The system control unit 50 starts a series of photographing processing operations from reading out a signal from the imaging unit 22 to writing the captured image to the recording medium 200 as an image file in response to the second shutter switch signal SW2.

[0029] The operation unit 70 is a variety of operation members that serve as an input unit that accepts operations from the user. The operation unit 70 includes at least the following operation members: the shutter button 61, the main electronic dial 71, the power switch 72, the sub electronic dial 73, the cross key 74, the SET button 75, the movie button 76, the AF lock button 77, the enlargement button 78, the playback button 79, the menu button 81, and the touch bar 82. Other operation members 70b are a collective representation of operation members that are not individually shown in the block diagram.

[0030] The power supply control unit 80 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to which electricity is applied, etc., and detects whether a battery is attached, the type of battery, and the remaining battery power. The power supply control unit 80 also controls the DC-DC converter based on the detection results and instructions from the system control unit 50, and supplies the necessary voltage for the necessary period to each unit including the recording medium 200. The power supply unit 30 is composed of primary batteries such as alkaline batteries and lithium batteries, secondary batteries such as NiCd batteries, NiMH batteries, and Li batteries, an AC adapter, etc.

[0031] The recording medium I / F 18 is an interface with a recording medium 200 such as a memory card or a hard disk. The recording medium 200 is a recording medium such as a memory card for recording captured images, and is composed of a semiconductor memory, a magnetic disk, or the like.

[0032] The communication unit 54 is connected wirelessly or via a wired cable, and transmits and receives video signals and audio signals. The communication unit 54 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 54 can also communicate with external devices via Bluetooth (registered trademark) or Bluetooth Low Energy. The communication unit 54 can transmit images (including LV images) captured by the imaging unit 22 and images recorded on the recording medium 200, and can also receive images and various other information from external devices.

[0033] The attitude detection unit 55 detects the attitude of the imaging device 100 with respect to the direction of gravity. Based on the attitude detected by the attitude detection unit 55, it is possible to determine whether the image captured by the imaging unit 22 is an image captured with the imaging device 100 held horizontally or vertically. The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 55 to the image file of the image captured by the imaging unit 22, or rotate and record the image. An acceleration sensor, a gyro sensor, or the like can be used as the attitude detection unit 55. It is also possible to detect the movement of the imaging device 100 (panning, tilting, lifting, whether it is stationary, etc.) using the acceleration sensor or gyro sensor that is the attitude detection unit 55.

[0034] 2(b) illustrates a configuration characteristic of this embodiment of the image processing unit 24. The image processing unit 24 has an image data generation unit 201, an object detection unit 202, a detection history storage unit 203, a dictionary data storage unit 204, and a dictionary data selection unit 205. The image data generation unit 201, the object detection unit 202, the detection history storage unit 203, the dictionary data storage unit 204, and the dictionary data selection unit 205 may be part of the system control unit 50, or may be provided separately from the image processing unit 24 and the system control unit 50. picture Image Processing Part 24 may be installed in, for example, a smartphone or a tablet terminal.

[0035] The image processing unit 24 sends image data generated based on the data output from the A / D converter 23 to an image data generating unit 201 within the image processing unit 24 .

[0036] When dictionary data for entire area detection is selected by dictionary data selection unit 205 described later, image data generation unit 201 uses input image data to generate image data to be used in entire area detection by subject detection unit 202, and sends the image data to subject detection unit 202. On the other hand, when dictionary data for local area detection is selected by dictionary data selection unit 205, image data to be used in local area detection by subject detection unit 202 is generated based on the detection history in detection history storage unit 203 described later, and sends the image data to subject detection unit 202. A specific method for generating each piece of detection image data will be described later.

[0037] The subject detection unit 202 receives dictionary data determined by a dictionary data selection unit 205 from dictionary data generated by machine learning stored in a dictionary data storage unit 204 and performs subject detection on the image data input from the image data generation unit 201. The subject detection unit 202 estimates the position of the subject included in the image data as a detection result, and outputs the estimation result to a detection history storage unit 203 and to a configuration outside the configuration shown in FIG. 2(b).

[0038] In this embodiment, the subject detection unit 202 is configured with a machine-learned CNN, and performs whole area detection and local area detection of a specific subject. Subjects for which whole area detection and local area detection are possible are based on dictionary data stored in the dictionary data storage unit 204. In this embodiment, the subject detection unit 202 is configured with different CNNs (Convolutional Neural Networks) for whole area detection and local area detection. Furthermore, the subject detection unit 202 may be configured with different CNNs depending on the subjects that can be detected. The subject detection unit 202 may be realized by a GPU (Graphics Processing Unit) or a circuit specialized for estimation processing by CNN.

[0039] The machine learning of CNN may be performed by any method. For example, a predetermined computer such as a server may perform the machine learning of CNN, and the imaging device 100 may acquire the trained CNN from the predetermined computer. In this embodiment, the predetermined computer inputs image data for learning, and performs supervised learning using position information of the subject corresponding to the image data for learning as teacher data (annotation), thereby training the CNN of the subject detection unit 202. As a result, a trained CNN is generated. The training of CNN may be performed by the imaging device 100 or the image processing device described above.

[0040] As described above, the object detection unit 202 includes a CNN (trained model) trained by machine learning. The object detection unit 202 estimates the position, size, reliability, etc. of the object using image data as input, and outputs the estimated information. The CNN may be, for example, a network in which a fully connected layer and an output layer are connected to a layer structure in which convolutional layers and pooling layers are alternately stacked. In this case, for example, an error backpropagation method or the like may be applied for training the CNN. The CNN may also be a neocognitron CNN that includes a feature detection layer (S layer) and a feature integration layer (C layer) as a set. In this case, a training method called "Add-if Silent" may be applied for training the CNN.

[0041] Any trained model other than trained CNN may be used in the subject detection unit 202. For example, a trained model generated by machine learning such as a support vector machine or a decision tree may be applied to the subject detection unit 202. In addition, the subject detection unit 202 does not need to use a trained model generated by machine learning. For example, any subject detection method that does not use machine learning may be applied to the subject detection unit 202.

[0042] The detection history storage unit 203 stores the subject detection history detected by the subject detection unit 202 in the image data, and the system control unit 50 sends the history to the image data generation unit 201 and the dictionary data selection unit 205. In this embodiment, the subject detection history includes the dictionary data used for detection, the number of detections, the position of the detected subject, and an identifier of the image data including the detected subject, but a configuration in which at least one of these data is stored may also be used.

[0043] The dictionary data storage unit 204 stores dictionary data for detecting a specific object, and the system control unit 50 reads the dictionary data selected by the dictionary data selection unit 205 from the dictionary data storage unit 204 and sends it to the object detection unit 202. Each dictionary data is, for example, data in which the characteristics of each area of ​​a specific object are registered. Furthermore, in order to detect multiple types of objects, dictionary data for each object and each area of ​​the object may be used. Therefore, multiple dictionary data are stored in the dictionary data storage unit 204. The dictionary data storage unit 204 stores multiple types of object detection dictionary data, such as dictionary data for detecting "people", dictionary data for detecting "animals", and dictionary data for detecting "vehicles". Furthermore, in addition to "animals", dictionary data for detecting "birds", which have a unique shape among animals and are in high demand for detection, is also stored.

[0044] Furthermore, dictionary data for detecting "vehicles" can be further subdivided into "cars," "motorcycles," "trains," "airplanes," and the like, and stored individually.

[0045] In this embodiment, furthermore, dictionary data for detecting whole area and dictionary data for detecting local area are prepared for each of the above-mentioned specific subjects. The whole area of ​​a specific subject may literally be set as an area including the whole subject, or may be set as an area including the central part of the subject, i.e., the main body. For example, the whole area of ​​a subject related to a "vehicle" can be defined for each type of subject, such as the "body" of a car or motorcycle, the "leading car" of a train, or the "fuselage" of an airplane. The definition of a local area indicates a part of the subject specified in the whole area. For example, it is set as being included in the whole area, such as "eyes of a person" as a local area for "whole face of a person" as the whole area, or "eyes" as a local area for "whole face of an animal" as the whole area. Similarly, for "birds," the whole area is set as the "whole area of ​​the bird" or "whole face of the bird," and the "eyes of the bird" are detected as a local area, and dedicated dictionary data is stored. In addition, the positional relationship may be such that the local area is not included in the whole area, such as "whole body of the motorcycle" as the whole area and "driver's helmet" protruding from the motorcycle body as the local area.

[0046] Furthermore, the relationship may be one in which a local region does not necessarily exist for the entire region of the subject, such as "the entire body of an automobile" and "the driver's helmet" limited to a "formula car" within an automobile.

[0047] As described above, the dictionary data for detecting a local region is assumed to be a partial region of a subject detected in the entire region. Therefore, in this embodiment, the dictionary data used for detecting a local region is generated by learning an image having the subject detected as the entire region as a background as an input image, and learning the positions or sizes of the local regions in the input image as annotations.

[0048] The entire area of ​​the subject detected by the multiple dictionary data stored in the dictionary data storage unit 204 can be used as the focus detection area. For example, in a composition in which an obstacle is present in the foreground and the subject is present in the background, it is possible to focus on the subject by focusing within the detected entire area.

[0049] However, for example, when the depth difference within the entire region is large, the problem of where to focus within the entire region arises. In response to the above problem, by limiting the range using local region detection, it becomes possible to focus on more detailed positions that are unclear only from the overall region and the depth information within the region, such as the "driver's seat" of a train or the "cockpit" of an airplane. Furthermore, in the case of a "vehicle" such as a motorcycle, it is possible that the desired focal position differs depending on whether or not a person is riding on the vehicle. By performing overall region detection and local region detection using dictionary data in which the overall region is the "whole body of the motorcycle" and the local region is the "driver's helmet," it becomes possible to switch the focal position for the same subject depending on whether or not the driver is present.

[0050] In addition, in this embodiment, the multiple dictionary data used by the object detection unit 202 for detection are generated by machine learning, but dictionary data generated by a rule base may be used in combination. The dictionary data generated by a rule base is, for example, data in which an image of an object to be detected or a feature amount specific to the object is stored, as determined by a designer. The object can be detected by comparing the image or feature amount of the dictionary data with the image or feature amount of the captured image data. The rule-based dictionary data is less complicated than a model set by a trained model by machine learning, and therefore has a smaller data capacity, and object detection using the rule-based dictionary data has a faster processing speed (lower processing load) than that of a trained model.

[0051] The dictionary data selection unit 205 selects the dictionary data to be used next based on the detection history stored in the detection history storage unit 203 and notifies the dictionary data storage unit 204 of the selection.

[0052] In this embodiment, dictionary data is stored individually for each of a plurality of types of objects and for each region of the object in the dictionary data storage unit 204, and object detection is performed a plurality of times for the same image data by switching between the plurality of dictionary data. The dictionary data selection unit 205 determines a dictionary data switching sequence based on the detection history stored in the detection history storage unit 203 and a user selection (to be described later), and determines the dictionary data to be used according to the determined sequence. An example of the dictionary data switching sequence will be described later.

[0053] The dictionary data storage unit 204 stores multiple types of subjects and dictionary data for detecting subject regions individually. The dictionary data selected by the dictionary data selection unit 205 is switched depending on the presence or absence of subjects detected so far, the type of dictionary data used at that time, the type of subject to be preferentially detected, and a combination thereof. The type of subject to be preferentially detected may be selected in advance by the user. Alternatively, a method in which the user specifies a subject in a live view screen displayed on the display unit 28 may be used. Furthermore, the user may select in advance whether to perform local region detection for each type of dictionary data for detecting the entire region or all at once. At this time, the system control unit 20 may cause the display unit 28 to display information on the dictionary data selected by the user or selected by the dictionary data selection unit 205.

[0054] As an example, FIG. 3 shows an example in which the user selects the type of subject to be detected with priority and whether or not to perform local area detection from a menu screen displayed on the display unit 28. FIG. 3(a) shows a setting screen for selecting a detection subject displayed on the display unit 28, and the user selects a subject to be detected with priority from among specific subjects (e.g., vehicles, animals, and people) that can be detected by operating the operation unit 70 (in this embodiment, "vehicle" is selected). FIG. 3(b) shows a setting screen for whether or not to perform local area detection displayed on the display unit 28, and the user selects local area detection ON / OFF by operating the operation unit 70 (in this embodiment, local area detection ON is selected). FIG. 3(c) shows a live view screen displayed on the display unit 28 in a state in which the priority subject and local area detection have been set. The result of the selection of the priority subject is displayed on the display unit 28 as a subject icon 301, and the result of the selection of whether or not to perform local area detection (or whether or not to display the detection result) is displayed as a local area detection ON / OFF icon 302. This allows the user to confirm the settings that he or she has made on the live view screen (in this embodiment, the operation in the case where a vehicle is selected as the subject to be detected with priority and local area detection is selected as ON will be described later).

[0055] 4 is a flowchart showing the flow of characteristic processing of the present invention performed by the imaging device 100 of this embodiment. Each step of this flowchart is executed by the system control unit 50 or each unit in response to an instruction from the system control unit 50. At the start of this flowchart, the imaging device 100 is powered on and in live view imaging mode, and is in a state in which the start of imaging (recording) of a still image or video can be instructed by operating the operation unit 70.

[0056] 4 is assumed to be processing performed when one frame (one piece of image data) is captured by the imaging unit 22 of the imaging device 100. However, the present invention is not limited to this, and the processing from step S401 to step S414 may be performed between multiple frames. In other words, the result of subject detection in the first frame may be reflected in any of the frames after the second frame.

[0057] In step S401, the system control unit 50 acquires captured image data captured by the imaging unit 22 and output by the A / D conversion unit .

[0058] In step S 402 , the image processing unit 24 resizes the image data to an image size that is easy to process (for example, QVGA), and sends the resized image data to the image data generating unit 201 .

[0059] In step S403, the dictionary data selection unit 205 selects dictionary data to be used for subject detection, and sends selection information indicating which dictionary data has been selected to the dictionary data storage unit 204 and the image data generation unit 201.

[0060] In step S404, the system control unit 50 judges whether or not the dictionary data for whole area detection has been selected by the dictionary data selection unit 205 in step S403. If the judgment in step S404 is Yes, the process proceeds to step S405. If the judgment is No, the process proceeds to step S405. 7 Proceed to.

[0061] In step S405, in response to the selection of dictionary data for entire area detection, the image data generation unit 201 performs a process of resizing the entire image area to a size to be input to the subject detection unit 202, and sends the resized image data to the subject detection unit 202.

[0062] Also, Step S40 5In this example, subject detection unit 202 uses dictionary data for entire area detection for detecting the entire area of ​​a specific subject (object) stored in dictionary data storage unit 204 to output information such as the position, size, and reliability of the detected subject to system control unit 50. At this time, the output of subject detection unit 202, the type of dictionary data used, and an identifier of the image data used for detection are stored in detection history storage unit 203. Furthermore, at this time, system control unit 50 may cause display unit 28 to display part or all of the information output by subject detection unit 202. Also, the above-mentioned display method may be changed depending on the dictionary data used.

[0063] Step S40 6 Then, it is determined whether detection has been performed using all necessary dictionary data for image data having the same identifier (image data of the same frame) from the detection history stored in the detection history storage unit 203. If the determination result is Yes, the process ends, and if No, the process returns to step S403 and the next dictionary data to be used is selected.

[0064] Step S40 7 Then, it is determined whether the local area detection function is ON. If Yes, the process proceeds to step S40. 8 If No, proceed to step S40 6 Proceed to.

[0065] Step S40 8 Then, it is checked from the detection history stored in the detection history storage unit 203 whether the entire area was detected in the past detection of the image data having the same identifier as the image data to be detected. If Yes, the process returns to step S4. 09 If the answer is No, proceed to step S412.

[0066] Step S4 09Then, based on the detection result of the entire area of ​​image data (of the same frame) having the same identifier as the image data to be detected, stored in detection history storage unit 203, image data generation unit 201 performs a trimming process on the image data for subject detection. Specifically, an area that is wider by a predetermined angle of view (number of pixels) from the area where the entire area of ​​the subject was detected is set as the trimming area. The size or ratio of how wide the trimming area is set from the detected entire area may be set depending on the subject (object) to be detected. In step S410, the image data that has been further trimmed is resized to match the input size of subject detection unit 202, and sent to subject detection unit 202.

[0067] In step S411, subject detection unit 202 outputs information such as the position, size, and reliability of a subject detected using dictionary data for local area detection stored in dictionary data storage unit 204 to system control unit 50. At this time, the output of subject detection unit 202, the type of dictionary data used, and an identifier of the image data used for detection are stored in detection history storage unit 203. Furthermore, at this time, system control unit 50 may cause display unit 28 to display part or all of the information output by subject detection unit 202. Also, the above-mentioned display method may be changed depending on the dictionary data used.

[0068] In step S412, it is checked whether there is a detection result of the whole area or local area of ​​the same subject in a past frame from the detection history stored in the detection history storage unit 203. In this embodiment, a past frame refers to a frame corresponding to image data captured one frame before in time series with respect to a frame having the same identifier and output to the image processing unit 24, but the number of frames going back in time series does not matter as long as it is a frame before in time series. If the determination in step S412 is Yes, the process proceeds to step S413. If No, the process proceeds to step S40. 6 Proceed to next step.

[0069] In step S413, a trimming process is performed as necessary to correspond to the angle of view input to the subject detection unit 202. In this embodiment, the image data is trimmed in the image data generation unit 201 based on the detection results in the past image data stored in the detection history storage unit 203. 4 Then, the image data that has been subjected to the trimming process is resized to match the input size of the subject detection unit 202 and sent to the subject detection unit 202 .

[0070] Step 409, Step S410, Step S41 3 and step S41 4 This is expected to improve the performance of local region detection using dictionary data generated by machine learning.

[0071] The above-mentioned improvement in performance of local region detection and step S412 and step S413 and step S41 4 In this embodiment, in step S412, it is checked from the detection history stored in the detection history storage unit 203 whether the entire area of ​​the subject has been detected in the image data having the same identifier as the image data to be detected. If the entire area of ​​the subject has been detected in the past, the process returns to step S41. 3 or step S414 As mentioned above, resizing is performed in the area that corresponds to the angle of view of the entire image. stomach The image data is resized based on the position and size of the entire area of ​​the subject detected previously.

[0072] Here, the dictionary data by machine learning can be generated by extracting common features of a specific subject from a large amount of image data in which the specific subject exists. The common features include, for example, the size, position, color, etc. of the subject, as well as areas outside the specific subject, such as the background. Therefore, the more limited the background in which the subject to be detected exists, the easier it is to improve detection performance (detection accuracy) with less learning. On the other hand, if a system is trained to detect a specific subject regardless of the background, it is highly versatile for shooting scenes, but the detection accuracy is unlikely to be high. The more image data used to generate the dictionary data and the more diverse it is, the higher the detection performance tends to be. On the other hand, by limiting the size and position of the detection area of ​​the subject to be detected in the image data used for detection to a certain value, it is possible to improve detection performance even if the number and diversity of image data required to generate the dictionary data is narrowed. In addition, if a part of the subject is cut off outside the image data, part of the features of the subject are lost, and therefore the detection performance is low.

[0073] In general, the larger the region of the subject, the more features it contains. In detection using the aforementioned machine-learned dictionary data, there is a possibility that an object with similar features other than the specific subject to be detected using the dictionary data may be mistakenly detected as a specific subject. The region defined as a local region is a narrower region compared to the entire region. The narrower the region, the fewer the features it contains, and the fewer the features, the more objects with similar features there are, leading to an increase in false positives.

[0074] FIG. 5 is a diagram showing an example of the positional relationship between the overall region and the local region when the overall region is defined as the "entire body" of the car and the local region is defined as the "driver's helmet." In FIG. 5, local region 502 is included in overall region 501. FIG. 5(a) shows two pieces of image data in which the same car body exists at different positions, distances, and backgrounds. FIG. 5(b) shows images obtained by cropping the above-mentioned two pieces of image data based on the overall region of the car. Compared to FIG. 5(a), cropping based on the overall region as in the two images in FIG. 5(b) makes it possible to generate image data with narrower variations in the position, size, and background of the local region.

[0075] When detecting a local area from the entire image data (entire angle of view) regardless of the presence or absence of a history of detecting the entire area, it is necessary to prepare image data of various shooting scenes (both the subject and the background) and generate dictionary data for detecting a local area while preventing erroneous detection. On the other hand, in this embodiment, in a scene where the entire area of ​​the subject has been detected in a past frame, but is no longer detected, that is, the entire area of ​​the subject has become so large that it can be cut off from the imaging angle of view, a local area is also detected from the entire image. This is because the subject is sufficiently large at this time, and even if a local area is detected from the entire image, it is assumed that most of the background area is the subject area, and detection is possible with sufficiently high performance even if dictionary data learned on the assumption that the background is the subject is used. Alternatively, as another embodiment, image data obtained by trimming the current processing target frame based on the position and size of the entire area of ​​the subject detected in the past frame is used for local area detection, thereby reducing erroneous detection and improving detection performance. In addition, in the learned model for local areas in this embodiment, the variation of image data used for detection can be reduced, so that the amount of image data and annotation data used for learning can be reduced.

[0076] Next, a description will be given of the relationship between the above-mentioned improvement in performance of local area detection and step S412. In this embodiment, if there is no detection history stored in the detection history storage unit 203 in which the entire area has been detected in image data having the same identifier as the image data to be detected, the detection history storage unit 203 determines whether or not to perform local area detection by referring to past history.

[0077] FIG. 6 is a diagram showing an example of the positional relationship between the entire region and the local region when the entire region is defined as the "whole body" of the motorcycle and the local region is defined as the "driver's helmet". In FIG. 6, the local region 602 is not included in the entire region 601. FIG. 6 shows a time series change in captured image data when a subject approaches from FIG. 6(a) to FIG. 6(c). FIG. 6(a) shows an example in which a local region cannot be detected because the subject is small, and FIG. 6(c) shows an example in which the entire region cannot be detected because a part of the subject is cut off. In a situation in which a subject approaches from the back to the front as in FIG. 6(b) with respect to FIG. 6(c), if the entire region or local region of the subject is detected at a nearby time, as in FIG. 6(b), the subject is highly likely to be present in the next image data. Therefore, the system control unit 50 reads out the detection history of the image data of the past frames stored in the detection history storage unit 203, and judges whether the entire region or local region was detected in the image data at the nearby time. Only when either region was detected, the local region is detected even when the entire region of the subject cannot be detected in the specific image data. As a condition for detecting a local region, it may be assumed that the subject is approaching as shown in FIG. 6, and it may be determined whether the entire area of ​​the subject has increased with time (as time passes). Furthermore, a scene change determination may be performed, for example, by calculating the amount of change in the entire image or the entire area of ​​the subject from past image data. As a result, if the scene has changed significantly, local area detection may not be performed even if the entire area or local area has been detected in image data at a nearby time. Here, image data at a nearby time is compared with the current image data, and if the amount of change in image data is equal to or greater than a certain threshold, it is determined that the scene has changed significantly.

[0078] By increasing the chances of local area detection only for image data where there is a high possibility of a local area being present, the chances of incorrectly determining that a local area is a local area when no local area exists or when another object similar to a local area is present can be reduced, thereby improving the performance of local area detection.

[0079] Next, the relationship between the above-mentioned improvement in performance of local area detection and step S413 will be described. In this embodiment, it is determined whether there is a history of detection of the entire area in image data having the same identifier as the image data to be processed (second image data) in the detection history stored in the detection history storage unit 203. If there is no history, the detection process is performed based on the detection history of the entire area or local area of ​​the previous image (first image data).

[0080] 7(a), 7(b), and 7(c) are diagrams showing an example of the positional relationship between the whole region 701 and the local region 702 when the whole region is defined as the "whole body" of the motorcycle and the local region is defined as the "driver's helmet" as in FIG. 6(a), 6(b), and 7(c). Both diagrams show the time series change of the captured image data. FIG. 7(c) shows an example in which the whole region cannot be detected because part of the body is cut off. If the whole region or local region of the subject was detected at a time close to the past as in FIG. 7(a) and FIG. 7(b) for FIG. 7(c), it is highly likely that the subject will be present in the image data at the next time. Furthermore, by calculating the amount of movement from the positional relationship between the whole region and the local region detected in the past for a specific subject stored in the detection history storage unit 203 and the size of the detection region, it is possible to estimate the position where the subject is likely to be present in FIG. 7(c). FIG. 7(d) is an example in which the image data of FIG. 7(c) is trimmed based on the above-mentioned past detection history so that the local region fits in a certain position with a certain size. By cropping the image data based on past history, as shown in Figure 7(d), the possibility of objects similar to the subject's features being present in the image data can be reduced, image data that is easier to detect can be generated, and detection performance can be improved.

[0081] The trimming range may be determined by predicting the position of the next subject from the positional relationship and movement amount between the whole region and the local region detected in the past as in the example shown in FIG. 7. In addition, in the case where local detection has not been performed at a nearby time, or in consideration of the fact that the positional relationship between the whole region and the local region changes with time, the trimming range may be determined using only the detection result of the whole region. Furthermore, depending on the movement of the subject, such as when the subject approaches the imaging device as in the example shown in FIG. 6(c), it is possible that the local region will not fit into the trimmed image by performing trimming based on the position of the whole region in the past frame. Therefore, the detection history for the past image data (first image data) stored in the detection history storage unit 203 is checked. As a result, when it is determined that the whole region does not fit into the image data from the transition of the positions and sizes of the whole region and the local region detected in the image data at a nearby time, local detection may be performed without performing trimming.

[0082] Next, a switching sequence of multiple dictionary data for one frame (one piece of image data) by the imaging device 100 performed in step S403 in Fig. 4 will be described. When multiple dictionary data are stored in the dictionary data storage unit 204, it is possible to perform detection using multiple dictionaries for one frame. However, due to issues with imaging speed and processing speed, images captured sequentially are output and processed in live view mode, and the number of times subject detection can be performed for one frame in video data during video recording is considered to be limited.

[0083] In this case, the type and order of dictionary data to be used may be determined based on, for example, whether or not a subject has been detected in the past, the type of dictionary data used at that time, the type of subject to be detected with priority, and whether or not local area detection is performed. Depending on the dictionary data switching sequence, when a specific subject is included in a frame, the corresponding dictionary data for subject detection may not be selected, and the detection opportunity may be missed. For this reason, the dictionary data switching sequence must be switched according to the settings and scene.

[0084] For example, in a structure that can perform subject detection up to three times per frame (or has three detectors capable of processing in parallel), an example of a dictionary data switching sequence when a vehicle is selected as the subject to be detected with priority is shown in Figure 8. V0 and V1 each indicate a vertical synchronization period for one frame, and the blocks enclosed by rectangles such as a person's head, vehicle 1 (motorcycle), and vehicle 2 (car) indicate that subject detection can be performed using three dictionary data (trained models) in time series within one vertical synchronization period.

[0085] FIG. 8(a) is an example of dictionary data switching when there is no subject already detected in one frame, and the dictionary data is switched in the order of the entire head of a person, the entire vehicle 1 (motorcycle), and the entire vehicle 2 (car). For example, if there is no switching sequence and dictionary data capable of detecting a subject selected by the user from the menu screen is always used as in FIG. 3(a), it is troublesome to switch the priority detection subject setting for each scene, that is, when a vehicle is photographed, the vehicle, and otherwise the person. Also, when it is unclear when a vehicle will arrive, if the priority detection subject setting is switched after noticing the vehicle, there is a risk that the image will not be captured in time. In contrast, in this embodiment, as in FIG. 8(a), during a period when a specific subject is not detected, the "head of a person," which is a detection target that is relatively likely to become a subject, is always detected, and the dictionary data is switched according to the priority detection subject setting for the rest. This makes it possible to reduce the number of times the priority detection subject setting is switched.

[0086] FIG. 8(b) is an example of dictionary data switching in the next frame when a motorcycle is detected in the previous frame, and the dictionary data is switched in the order of vehicle 1 (motorcycle) as a whole, the person's head as a whole, and vehicle 1 (motorcycle) local area. The order of detection for switching dictionary data does not necessarily have to be as described above. For example, the dictionary data for the person's head as a whole in the above dictionary data switching example may be switched to dictionary data that is likely to be selected as a subject other than a motorcycle in a scene where a motorcycle is being photographed, depending on the scene. The example in FIG. 8(b) may be considered as a sequence when the shooting mode is set to "vehicle priority". In addition, at this time, it may be controlled so that detection is not performed using dictionary data for "animals" and "birds," which are unlikely to be detected in parallel with vehicles, and their local areas. This ultimately improves the detection accuracy for the desired subject.

[0087] Figure 8(c) shows an example of dictionary data switching when a person's head is detected in the previous frame, where the dictionary data is switched in the following order: the entire person's head, the entire vehicle 1 (bike), and the person's eyes. Here, the person's eyes are a local area relative to the entire person's head. By using the entire vehicle 1 (bike) as the second dictionary data, the entire vehicle area can be detected when the vehicle set as the priority detection subject enters the frame, even if a local area of ​​the person is detected.

[0088] 8, multiple types of dictionary data for whole area detection are used in combination until a subject is detected using any of the multiple dictionary data stored in dictionary data storage unit 204. If there is a high possibility that the whole area will be detected thereafter, it is possible to increase the detection rate by narrowing down the subjects to be detected and using dictionary data for local area detection in combination.

[0089] Fig. 8(d) is an example of dictionary data switching when detecting an object by switching between multiple dictionary data for detecting the entire area over two frames. The same dictionary switching sequence is not used for each frame as in the examples shown in Figs. 8(a), (b), and (c). For example, when there is no past detection history and an object to be detected with priority is not set, a dictionary switching sequence across multiple frames as in the example shown in Fig. 8(d) is used. Here, after the entire area is detected using one of the dictionary data, it is also possible to switch to a dictionary switching sequence as in the example shown in Fig. 8.

[0090] In addition, when a user specifies an area within the live view screen displayed on the display unit 28, a dictionary switching sequence spanning multiple frames such as the example shown in Figure 8(d) may be used to identify the subject from the specified area.

[0091] FIG. 8(e) shows an example of dictionary data switching when local area detection OFF is selected, and is a sequence performed only on the entire vehicle 1 (bike).

[0092] When the user selects a vehicle as the object to be detected with priority and also selects local area detection OFF, it can be assumed that the object the user wants to capture is not the driver but the vehicle body. The dictionary data generated by machine learning may erroneously detect an object similar to the object to be detected as the object to be detected. Based on the dictionary data switching sequence when the vehicle is selected as the object to be detected with priority shown in Figure 8(b), the sequence is changed so that only local area detection is not performed. In this case, since the helmet has a similar shape to the human head, it may be detected as the entire head area by the dictionary data for detecting the entire head area of ​​the human. As shown in Figure 8(e), depending on the object to be detected, it is possible to prevent a decrease in detection accuracy by narrowing down the types of dictionary data to be used.

[0093] The object of the present invention can also be achieved as follows: A storage medium on which software program code describing procedures for implementing the functions of each of the above-described embodiments is recorded is supplied to a system or device. Then, a computer (or a CPU, MPU, etc.) of the system or device reads out and executes the program code stored in the storage medium.

[0094] In this case, the program code itself read out from the storage medium realizes the novel functions of the present invention, and the storage medium storing the program code and the program constitute the present invention.

[0095] Furthermore, examples of storage media for supplying the program code include flexible disks, hard disks, optical disks, magneto-optical disks, etc. Also usable are CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0096] The functions of the above-mentioned embodiments are realized by making the computer executable the program code that has been read out. Furthermore, the functions of the above-mentioned embodiments are also realized by the processing of an operating system (OS) or the like running on the computer, which performs part or all of the actual processing based on the instructions of the program code.

[0097] In addition, the following cases are also included: First, the program code is read from a storage medium and written into a memory in an expansion board inserted into a computer or in an expansion unit connected to the computer, and then, based on the instructions of the program code, a CPU or the like in the expansion board or expansion unit performs all or part of the actual processing. [Explanation of symbols]

[0098] 100 Imaging device 28 Display section 50 System control section 201 Image data generation unit 202 Subject detection unit 203 Detection history memory unit 204 Dictionary data storage unit 205 Dictionary Data Selection Section

Claims

1. An acquisition means for acquiring captured image data; a first detection means for detecting a first region of the object from the image data; a second detection means for detecting a second region of the subject from the image data; having The second detection means is When the first detection means detects the first region in the first image data, the second region is detected in a region corresponding to the first region; An image processing device characterized in that, when the first area is detected in the first image data and the first area is not detected in second image data captured after the first image data, the second area is detected in image data that is larger than the first area detected in the first image data and that has been processed based on the position and size of the first area detected in the first image data.

2. An image processing device as described in Claim 1, characterized in that the image data processed based on the position and size of the first area detected in the first image data is image data in the second image data that has been cropped to a region based on the position and size of the first area detected in the first image data.

3. 3. The image processing device according to claim 1, wherein the first detection means and the second detection means respectively use a first trained model that has been machine-learned in advance and a second trained model that has been machine-learned in advance.

4. The image processing device according to claim 3, characterized in that the second trained model is a model trained using an image having a subject to be detected as the first region as a background as an input image, and using position information of the second region in the input image as an annotation.

5. The image processing device according to claim 4, characterized in that the first trained model is a model that is trained by using an image including a subject and a background to be detected as the first region as an input image, and using position information of the first region in the input image as an annotation.

6. 6. The image processing device according to claim 1, wherein the first area is larger than the second area.

7. third and fourth detection means for detecting at least a partial area of ​​a subject different from the subjects detected by the first and second detection means, respectively; A setting means for setting an imaging mode; 7. The image processing apparatus according to claim 1, further comprising: a control unit which switches between said first to fourth detection units in accordance with a photography mode set by said setting unit.

8. 8. The image processing apparatus according to claim 7, wherein the first and second regions correspond to at least a part of a vehicle, and the second region is smaller than the first region.

9. The third detection means detects a third region corresponding to at least a part of a human, and the fourth detection means detects a fourth region corresponding to at least a part of an animal other than a human; 9. The image processing apparatus according to claim 8, wherein said control means prevents said fourth detection means from detecting image data that is to be detected by said first detection means.

10. An image processing device as described in any one of claims 1 to 9, characterized in that when the first area is not detected in the first image data by the first detection means but the second area is detected by the second detection means, the second detection means detects the second area in the second image data in an area based on the position and size of the second area detected in the first image data.

11. An acquisition means for acquiring captured image data; a first detection means for detecting a first region of the object from the image data; a second detection means for detecting a second region of the subject from the image data; having The second detection means is When the first detection means detects the first region in the first image data, the second region is detected in a region corresponding to the first region; When the first region is detected in the first image data and the first region is not detected in the second image data captured after the first image data, the second region is detected in an area larger than the first region detected in the first image data; An image processing device characterized in that, when the first area is not detected by the first detection means in the first image data and the second area is detected by the second detection means, the second area is detected in the second image data in an area based on the position and size of the second area detected in the first image data.

12. An acquisition means for acquiring captured image data; a first detection means for detecting a first region of the object from the image data; a second detection means for detecting a second region of the subject from the image data; having The second detection means is When the first detection means detects the first region in the first image data, the second region is detected in a region corresponding to the first region; When the first region is detected in the first image data and the first region is not detected in the second image data captured after the first image data, the second region is detected in an area larger than the first region detected in the first image data; An image processing device characterized in that, when at least one of the first area and the second area is detected in the first image data and it is determined that the first area does not fit within the current image data, the image processing device detects the second area from the acquired image data without cropping.

13. An acquisition means for acquiring captured image data; a first detection means for detecting a first region of the object from the image data; a second detection means for detecting a second region of the subject from the image data; having The second detection means is When the first detection means detects the first region in the first image data, the second region is detected in a region corresponding to the first region; When the first region is detected in the first image data and the first region is not detected in the second image data captured after the first image data, the second region is detected in an area larger than the first region detected in the first image data; An image processing device characterized in that, when the second area is detected in the first image data and the first area is detected in the detection result in the second image data, the image data is trimmed so that the relationship in size between the first area and the second area in the second image data is similar to the relationship in size between the first area and the second area in the past, and the second area is detected.

14. An acquisition means for acquiring captured image data; a first detection means for detecting a first region of the object from the image data; a second detection means for detecting a second region of the subject from the image data; having The second detection means is When the first detection means detects the first region in the first image data, the second region is detected in a region corresponding to the first region; When the first region is detected in the first image data and the first region is not detected in the second image data captured after the first image data, the second region is detected in an area larger than the first region detected in the first image data; An image processing device characterized in that it calculates the amount of movement of the first area and the second area using the first image data in which the first area or the second area is detected, generates image data by cropping an area in the second image data in which the first area or the second area is predicted to exist from the image data acquired by the acquisition means, and detects the second area.

15. An acquisition means for acquiring captured image data; a first detection means for detecting a first region of the object from the image data; a second detection means for detecting a second region of the subject from the image data; having The second detection means is When the first detection means detects the first region in the first image data, the second region is detected in a region corresponding to the first region; When the first region is detected in the first image data and the first region is not detected in the second image data captured after the first image data, the second region is detected in an area larger than the first region detected in the first image data; An image processing device characterized by comparing multiple image data including the first image data in which the first area or the second area is detected, and if it is determined that the first area or the second area is growing over time, detecting the second area in the second image data.

16. An acquisition means for acquiring captured image data; a first detection means for detecting a first region of the object from the image data; a second detection means for detecting a second region of the subject from the image data; having The second detection means is When the first detection means detects the first region in the first image data, the second region is detected in a region corresponding to the first region; When the first region is detected in the first image data and the first region is not detected in the second image data captured after the first image data, the second region is detected in an area larger than the first region detected in the first image data; An image processing device characterized by comparing a plurality of past image data including the first image data in which the first area and the second area are detected, inferring the relationship in size between the first area and the second area in the current image from the transition in size between the first area and the second area, trimming the image data acquired by the acquisition means so that the relationship in size between the first area and the second area in the past is similar to the relationship in size between the first area and the second area in the past, and detecting the second area.

17. An acquisition means for acquiring captured image data; a first detection means for detecting a first region of the object from the image data; a second detection means for detecting a second region of the subject from the image data; having The second detection means is When the first detection means detects the first region in the first image data, the second region is detected in a region corresponding to the first region; When the first region is detected in the first image data and the first region is not detected in the second image data captured after the first image data, the second region is detected in an area larger than the first region detected in the first image data; An image processing device characterized in that, even if the first area is not detected in the second image data and the first area is detected in the first image data, if the amount of change in image data calculated from the first image data and the second image data becomes equal to or greater than a certain threshold value, the image processing device does not detect the second area.

18. An imaging element; An image processing device according to any one of claims 1 to 17; An imaging device comprising:

19. An acquisition step of acquiring captured image data; a first detection step of detecting a first region of the object from the image data; a second detection step of detecting a second region of the object from the image data; In the second detection step, when the first region is detected in the first image data in the first detection step, the second region is detected in a region corresponding to the first region; A control method for an image processing device, characterized in that when the first area is detected in the first image data and the first area is not detected in second image data captured after the first image data, the second area is detected in image data that is larger than the first area detected in the first image data and that has been processed based on the position and size of the first area detected in the first image data.

20. A program for causing a computer to execute each step of the control method according to claim 19.

21. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to claim 19.

Images