Image processing device and method for controlling this

Abstract

Image processing device (24) with a recording device (201) for recording a variety of types of motifs for an input image, a setting device (205) for setting a type of motif as a priority motif, a main motive determination device (206) for determining a recording result as the main motive based on the multitude of types of motives recorded by the recording device (201) and a control device (204), wherein, in a case where recording results of a large number of types of motifs are available in the same region, the main motif determination device (206) determines one motif type in the same region based on the set priority motif and the types of motifs recorded, wherein in a case where any region of the input image is specified, the control device (204) selects a switching sequence that toggles between all detectable description directories, wherein the recordable descriptive directories contain descriptive directory data for the multitude of motif types and for each motif region, and where the capture of the multitude of motif types is performed many times for the same entered image, while switching between the captureable description directories.

Description

BACKGROUND OF THE INVENTION Area of ​​the invention

[0001] The present invention relates to an image processing device with a motif detection function and a method for controlling the image processing device. Description of the related technique

[0002] To capture a multitude of subject types based on image data acquired by an imaging device, such as a digital camera, a known method captures a multitude of subject types based on a learned model that has completed machine learning for each subject type. To perform image capture in which the focal point, brightness, and color are adapted to suitable conditions with respect to the captured subjects, it is necessary to determine a principal subject from the multitude of captured subjects. Japanese patent application JP 2017-5738 A discloses a method for determining a principal subject from a multitude of captured subjects based on the stable presence factor, which indicates whether subject capture is performed stably across a multitude of frames.

[0003] WO 2019 / 212644 discloses an image capture device that receives initial image data from a front-facing camera located on the front of the device, which includes a display. The initial image data represents a subject within the image capture device's field of view. Face recognition is performed in response to the initial image data from the front camera. The front-facing image data is used to determine whether a subject is a priority subject, for which at least one priority subject image is accessible through the image capture device. A region of interest is selected that corresponds to the subject classified as a priority subject. Autofocus, autoexposure, or autowhite balance are then performed using the selected region of interest.Second image data from the front camera is captured based on autofocus, auto exposure, or auto white balance using the selected region of interest.

[0004] JP 4 669 150 B2 discloses an image recording device which is equipped with a video sensor which generates the image signal of an image input from an image-generating optical system, a speech input device for generating a predetermined speech signal by inputting speech, a control device for generating the control signal of an image recording condition and a processing device for estimating the direction or position of a main subject based on the image signal or the speech signal.The processing unit is equipped with a visual information processing unit for searching the category of a given main subject and its direction or position based on the image signal; a secondary sensory information processing unit for searching the category of the main subject and its direction or position by processing the speech signal; and a main subject estimation unit for estimating the direction or position of the main subject by unifying the respective outputs of the two processing units. The control unit is equipped with an image acquisition condition control unit that sets an image acquisition condition suitable for capturing the image of the main subject estimated by the main subject estimation unit. SUMMARY OF THE INVENTION

[0005] The present invention is directed to the provision of an image processing device that can suitably capture a motif even when a multitude of capture results exist for the same motif through a multitude of descriptive directories, and a method for controlling the image processing device.

[0006] According to the invention, an image processing device, a method for controlling an image processing device, a computer-executable program and a non-volatile, computer-readable storage medium as specified in the accompanying patent claims are provided.

[0007] Further features of the present invention will become apparent from the following description of the exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1A and Fig.Figure 1B shows external views of an imaging device with an image processing device. Fig. 2A and Fig. Figure 2B shows block diagrams of a configuration of an imaging system with the image processing device. Fig. Figure 3 illustrates an example of a procedure for setting a target motive that is to be captured primarily by a user. Fig. Figure 4 shows a flowchart of the entire processing process. Fig. 5A and Fig. Section 5B illustrates examples of sequences for switching between a variety of types of descriptive directory data. Fig. Figure 6 shows a flowchart of a determination process for determining motif types in the same region. Fig. 7A to Fig. 7F illustrates an example of species identification processing for determining motif types in the same region. Fig.Figure 8 shows a flowchart of a main motive determination processing. Fig. 9A to Fig. 9C illustrates an example of main motive determination processing. Fig. Figure 10 illustrates an example of a sequence for switching between a variety of types of descriptive directory data in any setting by the user. DESCRIPTION OF THE EXAMPLES OF EXECUTION

[0008] Fig. 1A and Fig. Figure 1B illustrates external views of an imaging device 100 with an image processing device as an example of a device to which the present invention is applicable. Fig. Figure 1A shows a perspective view of the front of the imaging device 100, and Fig. Figure 1B shows a perspective view of the rear of the imaging device 100.

[0009] According to Fig. 1A and Fig.1B displays an image and various types of information on a display unit 28 located on the rear of the camera. An interactive control panel 70a can detect touch operation on the display surface (control surface) of the display unit 28. An extra viewfinder display unit 43, a display unit located on the top of the camera, displays the shutter speed, aperture, and various other camera settings. A shutter release button 61 is a control section for issuing an imaging instruction. A mode selector switch 60 is a control section for switching between different modes. A connector cover 40 is a cover for protecting (not shown) connection devices for connecting cables to an external device and the imaging device 100.

[0010] An electronic main dial 71 is a rotary control contained in a control unit 70. Rotating the electronic main dial 71 allows changes to settings such as shutter speed and aperture. An on / off switch 72 is a control for turning the imaging device 100 ON and OFF. An electronic secondary dial 73, a rotary control contained in the control unit 70, allows movement of a selection frame and feeding of images. A cross button 74 contained in the control unit 70 is a four-way switch whose upper, lower, right, and left sections can be pressed. An operation corresponding to a pressed section on the cross button 74 is activated. A SET button 75, a push button contained in the control unit 70, is primarily used to determine a selection element.

[0011] A motion capture button 76 is used to issue commands to start and stop motion capture (recording). An Auto Exposure (AE) lock button 77, located in the control unit 70, is pressed in the ready-to-shoot state to lock the exposure condition. A magnification button 78, located in the control unit 70, toggles the magnification mode on or off when using the live view display in image capture mode. When magnification mode is turned on, the live view image can be enlarged and reduced by operating the main electronic dial 41. In playback mode, the magnification button 78 enlarges the playback image to increase the magnification. A playback button 79, located in the control unit 70, toggles between image capture mode and playback mode.When the user presses the playback button 79 in image acquisition mode, the imaging device 100 enters playback mode, enabling the display of the most recent image from among those recorded on a recording medium 200 on the display unit 28. A menu button 81 located on the control unit 70 is pressed to display a menu screen on the display unit 28, allowing the user to make various settings. The user can intuitively adjust various settings using the menu screen displayed on the display unit 28, the cross button 74, and the SET button 75.

[0012] A touch bar 82 is a linear touch control element (line touch sensor) that accepts touch operation. The touch bar 82 is positioned so that the user can operate it with the thumb of their right hand, which grasps a handle section 90. The touch bar 82 accepts tap operation (touching the touch bar 82 and then removing the finger without moving it within a predetermined time period) and right / left sliding operation (touching the touch bar 82 and then moving the touch position while maintaining contact with the touch bar 82). The touch bar 82 is a separate control element from the interactive control panel 70a and does not have a display function.

[0013] A communication port 10 is used by the imaging device 100 to communicate with the lens side, which can be attached to and removed from the device. An eyepiece section 16 of the eyepiece viewfinder (in-view viewfinder) allows the user to visually detect the image displayed in an electric viewfinder (EVF) 29 within the viewfinder. The eye contact detection unit 57 is an eye contact detection sensor that detects whether the photographer's eye is in contact with the eyepiece section 16. A cover 207 covers the slot that accommodates the recording medium 100. The grip section 90 is shaped for a secure grip with the right hand when the user is holding the imaging device 100.

[0014] The shutter release button 61 and the electronic main dial 71 are positioned so that these controls can be operated with the index finger of the right hand while the digital camera is held by gripping the grip section 90 with the little finger, ring finger, and middle finger of the right hand. The electronic secondary dial 73 and the touch bar 82 are positioned so that these controls can be operated with the thumb of the right hand in the same situation. (Imaging device configuration)

[0015] The Fig. 2A and Fig. Figure 2B shows block diagrams of an example configuration of the imaging device 100 according to the present embodiment. Fig. 2A and Fig. 2B comprises a lens unit 150, an interchangeable lens. Although a lens 103 normally comprises a variety of lenses, this illustrates Fig.2A For simplification, a single lens is designated as lens 103. A communication port 6 is used by the lens unit 150 to communicate with the imaging device 100. A communication port 10 is used by the imaging device 100 to communicate with the lens unit 150. The lens unit 150 communicates with a system control unit 50 via communication ports 6 and 10. An internal lens system control circuit 4 controls an aperture 1 via an aperture control circuit 2 and focuses on the subject by moving the position of lens 103 via an automatic focus (AF) control circuit 3.

[0016] A shutter 101 is a focal plane shutter that allows arbitrary control of the exposure time of an imaging unit 22 under the control of the system control circuit 50.

[0017] The imaging unit 22 is an image sensor containing a charge-coupled device (CCD) or complementary metal oxide semiconductor (CMOS) sensor that converts an optical image into an electrical signal. The imaging unit 22 can be equipped with an imaging plane phase-difference sensor that outputs defocus amount information to the system control unit 50. An analog-to-digital (A / D) converter 23 converts an analog signal into a digital signal. The A / D converter 23 converts the analog signal output from the imaging unit 22 into a digital signal.

[0018] An image processing unit 24 subjects the data from the A / D converter 23 or the data from a storage control unit 15 to predetermined pixel interpolation, resizing (such as reduction), and color conversion processing. The image processing unit 24 also subjects the acquired image data to predetermined computational processing. The system control unit 50 performs exposure control and distance measurement control based on the computational result obtained by the image processing unit 24. This enables AF processing, Automatic Exposure (AE) processing, and Electronic Flash Preliminary Emission (EF) processing based on the Through-The-Lens (TTL) method.The image processing unit 24 also subjects the captured image data to a predetermined calculation processing and performs a TTL-based Automatic White Balance (AWB) processing based on the obtained calculation result.

[0019] The data output from the A / D converter 23 is written to memory 32 via the image processing unit 24 and the memory control unit 15, or directly to memory 32 via the memory control unit 15. Memory 32 stores image data acquired by the imaging unit 22 and then converted into digital data by the A / D converter 23, as well as image data to be displayed on the display unit 28 and the EVF 29. Memory 32 has sufficient storage capacity to hold a predetermined number of still images, moving images, and sound for a predetermined period of time.

[0020] Memory 32 also serves as image display memory (video memory). A digital-to-analog (D / A) converter 19 converts the image display data stored in memory 32 into an analog signal and then feeds the signal to the display unit 28 and the EVF 29. The image data stored in memory 32 is displayed on the display unit 28 and the EVF 29 via the D / A converter 19. The display unit 28 and the EVF 29 display data on a liquid crystal display (LCD) or organic electroluminescent (EL) display according to the analog signal from the D / A converter 19. The digital signal is converted once by the analog-to-digital converter 23, stored in memory 32, and then converted into an analog signal by the D / A converter 19. The analog signal is then successively transmitted to the display unit 28 or the EVF 29 to be displayed on it, in order to enable a live view (LV) display.Below, an image displayed in the live view is referred to as the live view (LV) image.

[0021] The shutter speed, aperture and various other camera settings are displayed on the extra viewfinder display unit 43 via an extra viewfinder display unit control circuit 44.

[0022] A non-volatile memory 56 is an electrically erasable recordable memory, such as an electrically erasable programmable read-only memory (EEPROM). Constants and programs used for the operation of the system control unit 50 are stored in the non-volatile memory 56. Programs stored in the non-volatile memory 56 relate to programs for executing various flowcharts (described below) according to the present embodiment.

[0023] The system control unit 50, which contains at least one processor or at least one circuit, controls the entire imaging device 100. Each part of the processing according to the present embodiment (described below) is implemented when the system control unit 50 executes the programs described above, which are stored in the non-volatile memory 56. A system memory 52 is, for example, random access memory (RAM). Constants and variables used for the operation of the system control unit 50 and programs read from the non-volatile memory 56 are loaded into the system memory 52. ​​The system control unit 50 also controls the memory 32, the D / A converter 19, and the display unit 28 for display control.

[0024] A system timer 53 is a time measuring unit that measures a time used for various types of controls and a time of a built-in clock.

[0025] The operating unit 70 is a control element that inputs various operating instructions into the system control unit 50.

[0026] The mode selector switch 60, a control element integrated into the control unit 70, switches the operating mode of the system control unit 50 between still image capture mode, motion image capture mode, and playback mode. Still image capture mode includes automatic image capture mode, automatic scene selection mode, manual mode, aperture priority mode (AV mode), shutter speed priority mode (TV mode), and program auto exposure (AE) mode (P mode). Still image capture mode also includes various scene modes as imaging settings for each captured scene and a custom mode. The mode selector switch 60 allows the user to directly select one of these modes.Alternatively, the user can select an image capture mode list screen once using the mode selector switch 60, choose one of a variety of displayed modes, and then change the mode using other controls. Similarly, the motion capture mode can contain a variety of modes.

[0027] The first shutter switch 62 is activated midway through the operation of the release button 61, which is provided on the imaging device 100. This is called a half-press (imaging preparation instruction) to generate a first shutter switching signal SW1. The first shutter switching signal SW1 causes the system control unit 50 to start imaging preparation operations, such as autofocus (AF) processing, auto exposure (AE) processing, auto white balance (AWB) processing, and electronic pre-flash (EF) processing.

[0028] The second shutter switch 64 is activated when the operation of the release button 61 is completed, which is called full press (image capture instruction), in order to generate a second shutter switching signal SW2. In response to the second shutter switching signal SW2, the system control unit 50 starts a series of operations in the image processing, from reading a signal from the imaging unit 22 to writing a captured image (as an image file) to the recording medium 200.

[0029] The control unit 70 contains various control elements as input elements that receive operations from the user.

[0030] The control unit 70 contains at least the following controls: the shutter release button 61, the electronic main dial 71, the on / off switch 72, the electronic secondary dial 73, the cross button 74, the SET button 75, the motion picture button 76, an AF lock button 77, the magnification button 78, the playback button 79, the menu button 61, and the touch bar 82. Other controls 70b indicate any controls not individually described in the block diagram.

[0031] A power source control unit 80 contains a battery detection circuit, a direct current-to-direct current (DC-DC) converter, and a switching circuit that selects a power block to be supplied. The power source control unit 80 detects the presence or absence of a battery, the battery type, and the remaining battery charge. Based on the detection result and an instruction from the system control unit 50, the power source control unit 80 also controls the DC-DC converter to apply the required voltages to the recording medium 200 and other components for the required time intervals. A power source unit 30 contains a primary battery (such as an alkaline or lithium battery), a secondary battery (such as a nickel-cadmium battery, a nickel-metal hydride battery, or a lithium battery), and an alternating current (AC) adapter.

[0032] A recording medium interface (-I / F) 18 is an interface to the recording medium 200, such as a memory card or hard disk. The recording medium 200 is, for example, a memory card for recording captured images, containing semiconductor memory, or a magnetic disk.

[0033] A communication unit 54 establishes a wireless or wired connection for sending and receiving video and audio signals. The communication unit 54 can also be connected to a wireless local area network (LAN) and the internet. The communication unit 54 can also communicate with an external device via Bluetooth® and Bluetooth Low Energy. The communication unit 54 can transmit images (including the LV image) captured by the imaging unit 22 and images recorded on the recording medium 200, and receive images and various other types of information from an external device.

[0034] An orientation detection unit 55 detects the orientation of the imaging device 100 in the direction of gravity. Based on the orientation detected by the orientation detection unit 55, the system control unit 50 can determine whether the image captured by the imaging device 22 is an image captured with the imaging device 100 held horizontally or with the imaging device 100 held vertically. The system control unit 50 can add directional information corresponding to the orientation detected by the orientation detection unit 55 to the image file of the image captured by the imaging device 22 or rotate the image before recording. An accelerometer or gyroscope sensor can be used as the orientation detection unit 55.Using an accelerometer or gyroscope sensor as an orientation detection unit 55, movements of the imaging device 100 (swivel, tilt, lift and stand still) can also be detected. (Configuration of the image processing unit)

[0035] Fig.Figure 2B illustrates a characteristic configuration of the image processing unit 24 according to the present embodiment. The image processing unit 24 includes a subject acquisition unit 201, an acquisition history storage unit 202, a description directory data storage unit 203, a description directory data selection unit 204, a species identification unit 205, and a main subject identification unit 206. Although these units are described as part of the image processing unit 24 in the present embodiment, they can be provided as part of the system control unit 50 or separately from the image processing unit 24 and the system control unit 50. The image processing unit 24 can, for example, be located on a smartphone or a tablet device.

[0036] The image processing unit 24 sends image data, which is generated based on data output from the A / D converter 23, to the subject detection unit 201 in the image processing unit 24.

[0037] According to the present embodiment, the motif detection unit 201 contains a convolutional neural network (CNN) that has completed machine learning (deep learning) and detects a specific motif. The types of detectable motifs are based on description directory data stored in the description directory data storage unit 203. According to the present embodiment, the motif detection unit 201 contains a different CNN (different network parameters) depending on the types of detectable motifs. The motif detection unit 201 can be implemented by a graphics processing unit (GPU) or a circuit specialized for CNN-based estimation processing.

[0038] Machine learning of the CNN can be performed using any method. For example, a predetermined computer, such as a server, can perform the machine learning of the CNN, and the imaging device 100 can obtain the learned CNN from the predetermined computer. According to the present embodiment, the predetermined computer inputs image data for learning and performs supervised learning using subject position information corresponding to the image data for learning (note), thereby enabling CNN learning for the subject acquisition unit 201. This completes the generation of a learned CNN. The CNN learning can be performed by the imaging device 100 or the image processing device described above.

[0039] As described above, the subject detection unit 201 contains a CNN (trained model) that has completed machine learning. The subject detection unit 201 inputs image data, estimates the subject's position, size, and reliability, and outputs the estimated information. The CNN can be, for example, a layered network (consisting of convolution and union layers stacked alternately), a fully connected layer, and an output layer, with the fully connected and output layers linked to the layered structure. In this case, backpropagation, for example, is applicable for CNN learning. The CNN can also be a Neocognitron CNN, which contains a set of a feature detection layer (S layer) and a feature integration layer (C layer). In this case, a learning technique called "add-if silent," for example, is applicable for CNN learning.

[0040] Any model other than a learned CNN can be used for the motivation detection unit 201. For example, a learned model generated via machine learning, such as a support vector machine or a decision tree, can be applied to the motivation detection unit 201. The motivation detection unit 201 does not necessarily have to be a learned model generated via machine learning. For example, any motivation detection method without using machine learning can be applied to the motivation detection unit 201.

[0041] The acquisition history storage unit 202 stores a motif acquisition history for image data acquired by the motif acquisition unit 201. The system control unit 50 transmits the motif acquisition history to the description directory data selection unit 204. According to the present embodiment, the acquisition history storage unit 202 stores the description directory data used for motif acquisition and the positions, sizes, and reliabilities of captured motifs as the motif acquisition history. The acquisition history storage unit 202 can additionally store data such as image data identifiers containing the object acquisition times and captured motifs.

[0042] The description directory data storage unit 203 stores the description directory data for capturing specific motifs. The system control unit 50 reads the description directory data selected by the description directory data selection unit 204 from the description directory data storage unit 203 and then sends the data to the motif capture unit 201. The description directory data for capturing each motif, for example, records features of each region of the specific motif. To capture a large number of motif types, description directory data can also be used for each motif and for each motif region.The Descriptive Directory Data Storage Unit 203 stores descriptive directory data for capturing a variety of motif types, including descriptive directory data for capturing "person," descriptive directory data for capturing "animal," and descriptive directory data for capturing "vehicle." In addition to descriptive directory data for capturing "animal," the Descriptive Directory Data Storage Unit 203 can also store descriptive directory data for capturing "bird," which has specific forms and is subject to high demand for motif capture among animals. The Descriptive Directory Data Storage Unit 203 can also store descriptive directory data for "automobile," "motorcycle," "train," "airplane," etc., as subdivisions of descriptive directory data for capturing "vehicle."

[0043] Subject regions captured by a variety of types of description directory data stored in the description directory data storage unit 203 can be used as focal point capture regions. In a composition containing, for example, an obstacle in the foreground and a subject in the background, a target subject can be brought into focus by focusing on the interior of a captured region.

[0044] Although in the present embodiment the multitude of types of descriptive directory data used in subject acquisition by the subject acquisition unit 201 are generated via machine learning, rule-based descriptive directory data can be used or combined with it. The rule-based descriptive directory data, for example, relates to data that stores images of a subject to be acquired or stores subject-specific feature sizes predetermined by the developer. The subject can be acquired by comparing the images or feature sizes of the descriptive directory data with the images or feature sizes of acquired image data. The rule-based descriptive directory data is less complex and therefore has a smaller data size than the model set by the learned model via machine learning.Motive capture using the rule-based description directory data therefore provides a processing speed that is greater than that provided by motive capture using the learned model (and a processing load that is lower than that provided by motive capture using the learned model).

[0045] The description directory data selection unit 204 selects the description directory data to be used next based on the motif capture history stored in the capture history storage unit 202, the predetermined sequence and rules or instructions from the user, and then communicates the selected description directory data to the description directory data storage unit 203.

[0046] According to the present embodiment, the description directory data storage unit 203 individually stores description directory data for a multitude of subject types and for each subject region. Subject acquisition is performed many times using the same image data, while switching between a multitude of description directory data types. The description directory data selection unit 204 determines a description directory data switching sequence and then determines the description directory data to be used according to the determined sequence. An example of a description directory data switching sequence is described below.

[0047] When a large number of motifs are captured in the same region, the type identification unit 205 determines the types of motifs for the region. The type identification unit 205 determines a capture result based on a preferred motif setting, which is set by the user via the control unit 70, from a large number of capture histories stored in the capture history storage unit 202. The identification procedure is described below.

[0048] Fig. Figure 3 illustrates an example in which, with regard to a procedure for setting a preferred subject to capture, the user selects the type of preferred subject to capture from the menu screen displayed on the display unit 28. Fig.Figure 3 illustrates a settings screen for selecting a subject to be captured, which is displayed on the display unit 28. The user selects a preferred subject to be captured from specific detectable subjects (such as vehicles, animals, and people) via the control unit 70. Fig. Figure 3 illustrates a state in which "Vehicle" is selected. Fig. 3 indicates “None” as a mode in which no motive is captured, and “Automatic” as a mode in which a motive is captured in which no priority is assigned to any of the specific detectable motives.

[0049] The main motif determination unit 206 determines the main motif based on the multitude of acquisition histories stored in the acquisition history storage unit 202, the setting of the preferred motif to be acquired, which is set by the user via the control unit 70, and the motif determined by the species identification unit 205. A procedure for determining the main motif is described below. (Processing sequence of the imaging device)

[0050] Fig.Figure 4 shows a flowchart of the sequence of a characteristic processing operation of the present invention, which is carried out by the imaging device 100 according to the present embodiment. Each step of this flowchart is executed by the system control unit 50 or by a respective unit following an instruction from the system control unit 50. At the start of this flowchart, the imaging device 100 is switched ON and the device is in live view image acquisition mode, in which the device is ready to issue an instruction to start still or moving image acquisition (recording) via operation via the control unit 70.

[0051] It is assumed that a series of processes from step S401 to S409 in Fig.4 is performed when the imaging unit 22 of the imaging device 100 performs image acquisition for one frame (one piece of image data). However, the present invention is not limited to this. A series of processes from step S401 to step S409 can be performed over a plurality of frames. In particular, the result of the image acquisition in the first frame can be reflected on any of the second and subsequent frames.

[0052] In step S401, the system control unit 50 obtains image data acquired by the imaging unit 22 and then output by the A / D converter 23.

[0053] In step S402, the image processing unit 24 changes the size of the image data so that it fits into an easily processable image size (for example, Quarter Video Graphics Array (QVGA)) and then sends the resized image data to the image generation unit 201.

[0054] In step S403, the description directory data selection unit 204 selects the description directory data generated via machine learning to be used for motif detection and then sends selection information to the description directory data storage unit 203 to identify the selected description directory data.

[0055] The descriptive directory data generated via machine learning can be created by extracting common features of a specific subject from a large amount of image data containing that subject. Examples of common features include the background and other regions outside the specific subject, in addition to the subject's size, position, and color. Therefore, if the subject to be captured exists within a more restrictive background, the capture performance (capture accuracy) can be improved with less training effort. Conversely, if the training is performed with the intention of capturing a specific subject regardless of the background, the flexibility regarding captured scenes increases, but it becomes difficult to improve capture accuracy.Capture performance tends to increase with the quantity and variety of image data used for descriptive directory data generation. Conversely, even if the number and variety of image data pieces required for descriptive directory data generation are reduced, capture performance can be improved by limiting the size and position of the capture region for the subject to predetermined values ​​within the image data used for subject capture. If a subject partially protrudes from the image data, some of its features are lost, which degrades capture performance.

[0056] In general, a larger motif region contains a greater number of features. When capturing using descriptive directory data, after machine learning has completed, an object with features similar to those of the specific motif being captured using the descriptive directory data may be mistakenly identified as the specific motif. A region defined as a local region is small compared to the entire region. The feature set contained in a region decreases as the region's area decreases, and the number of objects with similar features increases as the feature set decreases, thus increasing the error rate.

[0057] The following is made with reference to Fig. 5A and Fig.5B describes a sequence for switching between a variety of types of description directory data for a frame (a piece of image data) in step S403. If a variety of types of description directory data are stored in the description directory data storage unit 203, subject acquisition based on a variety of description directories for a frame can be performed. On the other hand, in still and moving image data during live view recording, where sequentially captured images are output and processed, the number of subject acquisitions that can be performed for a frame is assumed to be limited due to image acquisition and processing speed issues.

[0058] In this case, the type and order of the descriptive directory data to be used can be determined, for example, according to the presence or absence of motifs captured in the past, the types of descriptive directory data used in past capture, and the types of motifs to be captured preferentially. If a frame contains a specific motif, the descriptive directory data for capturing that specific motif cannot be selected based on the descriptive directory data switching sequence, potentially resulting in a missed motif capture opportunity.

[0059] Therefore, it is also necessary to change the description directory data switching sequence according to settings and scenes.

[0060] Fig. 5A and Fig.Figure 5B shows examples of descriptive directory data switching sequences when a vehicle is selected as a preferred subject for capture, in a structure where subject capture can be performed up to three times (or there are three different capture devices capable of processing in parallel) for a frame. V0 and V1 each indicate the vertical synchronization interval for a frame. Framed blocks, such as Head of a Person, Vehicle 1 (Motorcycle), and Vehicle 2 (Automobile), indicate that subject capture based on three different types of descriptive directory data (trained models) can be performed sequentially within a vertical synchronization interval.

[0061] Fig.Figure 5A illustrates an example of a description directory data switch when no subject is being acquired. In the first frame, the description directory data switch is performed in the sequence: person's head, vehicle 1 (motorcycle), and vehicle 2 (automobile). In the second frame, the description directory data switch is performed in the sequence: animal (dog / cat), vehicle 1 (motorcycle), and vehicle 2 (automobile). For example, the Imaging Device 100 uses constant description directory data that allows the acquisition of a subject selected by the user from the menu screen, as described in Fig.Figure 3 illustrates this without a switching sequence. This case causes a problem when changing the priority capture subject setting for each scene, for example, selecting Vehicle when a vehicle is being captured and selecting Person and Animal when other objects are being captured. If the time when a vehicle will appear is unknown, selecting the priority capture subject setting after noticing an approaching vehicle might cause the image capture time to elapse. On the other hand, the present embodiment allows the user to capture an image without regard to the priority capture subject setting. In particular, the present embodiment switches between all types of descriptive directory data across a multitude of frames, as shown in Figure 3. Fig.Figure 5A illustrates this during the time period when no specific subject is being acquired. By selecting the descriptive directory data according to the priority acquisition subject setting in the first or second frame, while switching between all types of descriptive directory data, the acquisition accuracy of the priority acquisition subject can be improved even while acquiring all acquirable subjects. This makes it possible to reduce the number of times the priority acquisition subject setting needs to be changed. The Imaging Device 100 can be separately equipped with a mode in which only specific descriptive directories (groups) are constantly accessed in order of precedence of events according to a user-defined setting.

[0062] Fig.Figure 5B illustrates an example of a description directory data switch to the next frame when a motorcycle is captured in the preceding frame. The description directory data switch is performed in the sequence Vehicle 1 (Motorcycle), Head of a Person, and Vehicle 1 (Motorcycle). The description directory data switch need not necessarily be performed in the order described above. In the description directory data switch example described above, the description directory data "Head of a Person" can be changed according to a scene, for example, to description directory data where different subjects are likely to be selected from a motorcycle in a motorcycle imaging scene.In this case, too, an exclusive control can be applied, simultaneously omitting subject detection using the "Animal" description directory data, which has a low detection probability. Depending on the texture (design) and color of the vehicle, it might be incorrectly detected as an animal. Consequently, implementing an exclusive control in this way improves the detection accuracy for the desired subject.

[0063] In step S404, the subject acquisition unit 201 acquires a subject (or the region where the subject is located) based on image data acquired by the imaging unit 22 and input into the image processing unit 24, using the descriptive directory data for acquiring a specific subject (object) stored in the descriptive directory data storage unit 203. The position and size of the acquired subject, information such as the calculated reliability, the type of descriptive directory data used, and the identifier of the image data used for subject acquisition are stored in the acquisition history storage unit 202.

[0064] In step S405, the image processing unit 24 determines, based on the image acquisition history stored in the acquisition history storage unit 202, whether image acquisition was performed with all required description directory data for image data with the same identifier (image data in the same frame). If image acquisition was performed with all required description directory data (YES in step S405), processing proceeds to step S406. If, on the other hand, image acquisition was not performed with all required description directory data (NO in step S405), processing returns to step S403. In step S403, the image processing unit 24 selects the description directory data to be used next.

[0065] In step S406, the image processing unit 24 determines, based on the subject acquisition history stored in the acquisition history storage unit 202, whether subject acquisition was performed with all types of description directory data. If subject acquisition was performed with all types of description directory data (YES in step S406), processing proceeds to step S407. If, on the other hand, subject acquisition was not performed with all types of description directory data (NO in step S406), the image processing unit 24 proceeds to processing for the next frame. Fig.In step 5A, the image processing unit 24 requires, for example, two frames to perform image acquisition with all necessary description directory data. Therefore, it skips the processing of the subsequent stage in the first frame and then proceeds to the next frame. Thus, processing continues to step S407 in the second frame. According to the present embodiment, the image processing unit 24 skips the processing of the subsequent stage until image acquisition with all necessary description directory data has been completed. However, the present invention is not limited to this. For processing that requires a fast response, such as automatic focusing, the image processing unit 24 can perform the processing of the subsequent stage using only one image acquired for each frame, without waiting for image acquisition with all types of description directory data.If all types of currently set description directory data can be accessed according to precedence in two frames as in the present embodiment, the image processing unit 24 can, for example, perform the processing of the subsequent stage in step S407 and subsequent steps based on the acquisition result for two frames that contain the last of the previous frames.

[0066] In step S407, the image processing unit 24 reads a setting for selecting a subject to be preferentially captured from specific detectable subjects that are preset by the user via the control unit 70.

[0067] In step S408, the image processing unit 24 determines, based on the subject acquisition history for acquisition results of image data with the same identifier stored in the acquisition history storage unit 202, whether a large number of acquisition results are present in the same region.

[0068] If multiple capture results are present in the same region (YES in step S408), processing proceeds to step S409. Conversely, if no multiple capture results are present (NO in step S408), processing proceeds to step S410. For example, image processing unit 24 can determine that multiple capture results are present in the same region if capture center coordinates are present in a different capture result region. Image processing unit 24 can also determine that multiple capture results are present in the same region if the capture regions overlap by a predetermined amount (e.g., a threshold ratio) or more.

[0069] In step S409, the species identification unit 205 determines a region recording result based on the priority motivation setting configured in step S407, the recording results stored in step S405, and the result of the determination in step S408 that a large number of recording results exist in the same region. The determination procedure is described below.

[0070] In step S410, the main subject determination unit 206 determines the main subject from the multitude of image data acquisition results with the same identifier, using the priority subject setting established in step S407, based on the subject acquisition history stored in the acquisition history storage unit 202. If, in this case, the image processing unit 24 determines in step S408 that a multitude of acquisition results exist in the same region, the image processing unit 24 also uses the result from step S409. In this case, the system control unit 50 can display some or all of the information output by the main subject determination unit 206 on the display unit 28. The determination procedure is described below. (Process of species identification processing to determine the type of motive based on a large number of motive recording results in the same region)

[0071] The species identification processing in step S409 is described below with reference to the flowchart in Fig. 6. Species identification processing in the Fig. 7A to 7F and Table 1 are described. Each step of this flowchart is executed by the system control unit 50 or by a respective unit following an instruction from the system control unit 50.

[0072] Fig. 7A to Fig. 7F illustrates examples of species identification processing. Fig. 7A illustrates an entered image in which a motorcycle 701 is captured as the subject. Fig. Figure 7B illustrates a state in which the Person Description Directory is selected in step S403 and a Person 702 is recorded. Fig. Figure 7C illustrates a state in which the motorcycle description directory is selected in step S403 and a motorcycle 703 is recorded. Fig.Figure 7D illustrates a state in which the Automobile Description Directory is selected in step S403 and an Automobile 704 fault is recorded. Fig. Figure 7E illustrates a state in which the dog description directory is selected in step S403 and a dog 705 error is recorded. Fig. Figure 7F illustrates a state in which the cat description directory is selected in step S403 and no capture result is obtained as a result of the processing.

[0073] In step S601, the image processing unit 24 assigns a priority to each type of subject to be captured, according to the priority setting set in step S407.

[0074] Table 1 illustrates an example of priority classification by priority settings and motif types. In Table 1, the vertically arranged priority settings are "Person", "Animal", "Vehicle", "None", and "Automatic", according to the setting procedure in Fig. 3. The horizontally arranged motif types to be captured include "person", "cat", "dog", "automobile" and "motorcycle" according to the species identification processing in Fig. 7A to Fig. 7F. In Table 1, a lower priority number indicates a higher priority, and “No priority” indicates that the motif is not used.

[0075] Although in the present embodiment motifs are classified into three different motifs (values): priority motif (priority 1 in Table 1), non-priority motif (priority 2 in Table 1), and unused motif (no priority in Table 1), the present invention is not limited thereto. For example, motifs can be classified into two different motifs (values): used motif and unused motif. Alternatively, motifs can be classified into four different motifs (values): top priority motif, priority motif, non-priority motif, and unused motif. The number of motif types can be changed according to the number of detectable motif types and the possible priority settings.If, with reference to Table 1, "Vehicle" is selected as the priority motive, "Automobile" and "Motorcycle" are classified as priority motives, "Person" is classified as a non-priority motive, and "Dog" and "Cat" are classified as unused motives. However, the classification procedure is not limited to this. For example, motive types that differ from motive types with the priority setting (also referred to as priority motive types) should not be recorded; "Person" can also be classified as an unused motive. If motive types that differ from the priority motive types are to be recorded, "Dog" and "Cat" can be classified as non-priority motives. [Table 1] Motif to be captured person Dog Cat automobile motorcycle Priority setting person Priority 1 Priority 2 Priority 2 Priority 2 Priority 2 animal Priority 2 Priority 1 Priority 1 Not a priority Not a priority vehicle Priority 2 Not a priority Not a priority Priority 1 Priority 1 None Not a priority Not a priority Not a priority Not a priority Not a priority Automatically Priority 1 Priority 1 Priority 1 Priority 1 Priority 1

[0076] In step S602, the image processing unit 24 performs the priority-based motif type determination processing for the same region according to the priority determined in step S601.

[0077] The following describes a specific procedure with reference to species identification processing in the Fig. 7A to 7F are described. If a priority setting is assigned to the person with reference to Table 1, priority 1 is given to a person motif type, and thus the image processing unit 24 confirms whether a capture result for person exists. Since in Fig. If a person 702 is present, image processing unit 24 applies person 702 as the subject type in the region and then terminates the type determination processing. If a priority setting is assigned to vehicle, automobile and motorcycle subject types are given priority 1, as illustrated in Table 1. Therefore, image processing unit 24 confirms whether there are capture results for automobile and motorcycle with priority 1. Since both a motorcycle 703 ( Fig. 7C) as well as a 704 automobile ( Fig.7D) are present, processing proceeds to step S603. If, in this case, neither the motorcycle 703 ( Fig. 7C) nor the 704 automobile ( Fig. If 7D) is present, the image processing unit 24 confirms whether a capture result of person with priority 2 is present. If no capture result for person is present, the image processing unit 24 determines that no subject is present in the same region, since dog and cat are given "No Priority", as illustrated in Table 1, and then terminates the species identification processing.

[0078] In step S603, the image processing unit 24 subjects the reliabilities of the acquisition results stored in step S405 to normalization processing for each subject. Normalization is performed because the maximum reliability value of an acquisition result and the reliability threshold for a subject differ for each individually applied description directory. Normalization enables reliability comparison between subjects with different description directories in the processing of the following stage. According to the present embodiment, the minimum and maximum reliability values ​​that can be used for each description directory are normalized to 0 and 1, respectively. This normalization limits the reliability to values ​​between 0 and 1, thus enabling subject comparison based on reliability.The normalization procedure is not limited to this. For example, the reliability threshold as a motive can be set to 1, and the minimum reliability value that can be assumed can be set to 0.

[0079] If the image processing unit 24 confirms in step S602 that a multitude of motif types with the same priority exist, the image processing unit 24 then determines, in step S604, a motif with high reliability as a result of the normalization in step S603 as the motif in the region and then terminates the motif identification processing. Although in the present embodiment a motif in the region is determined based on reliability, the identification procedure is not limited to this. For example, the image processing unit 24 can refer to the acquisition results of past frames to determine the motif type that is most frequently acquired in a multitude of frames as the motif in the region.

[0080] According to the Fig. 7A to 7F determine the image processing unit 24 in step S602, the motorcycle 703 in Fig. 7C and the 704 automobile in Fig.7D as priority subjects in the same region and then compares the two subjects. Since, according to the present embodiment, the input subject is the motorcycle 701, the image processing unit 24 determines the motorcycle 703 as the subject in the region, assuming that the motorcycle 703 is in Fig. 7C has the highest reliability.

[0081] Before the reliability comparison in step S604, the image processing unit selects 24 subjects based on the priority in step S602. A case involving a dog and a cat is assumed as subjects with similar common characteristics, such as quadrupedal locomotion. If, in this case, a cat image is entered into the dog description directory, it is very likely that the cat will be incorrectly captured as a dog. However, a case involving a dog and a motorcycle is assumed as subjects with dissimilar common characteristics. If, in this case, a motorcycle image is entered into the dog description directory, it is unlikely that the motorcycle will be incorrectly captured as a dog. In the case of an incorrect capture of the dog 705 in Fig.However, with unit 7E it is difficult to determine which feature of the input image was perceived, which may result in high reliability. In such a case, it can be difficult to prevent the final output from being incorrectly captured as a dog. Therefore, the image processing unit 24 first performs a subject selection according to the set priority to eliminate incorrect capture of unwanted subjects. (Process of main motive determination)

[0082] The main motive determination processing in step S410 is described below with reference to the flowchart in Fig. 8 and the pictures in the Fig. 9A to 9C are described. Each step of this flowchart is executed by the system control unit 50 or by a respective unit according to an instruction from the system control unit 50.

[0083] Fig. 9A to Fig.9C illustrates an example of determining the main motive when a large number of motives are captured in the same framework. Fig. 9A illustrates a state in which a person's face 901 and cats 902 and 903 are captured.

[0084] Fig. 9B illustrates a state in which a person's face 904 is selected as the main motif from the person's face 901 and the cats 902 and 903. Fig. 9C illustrates a state in which a cat 905 is selected as the main motif from the person's face 901 and the cats 902 and 903.

[0085] In step S801, image processing unit 24 selects 24 principal subject candidates according to the priority setting established in step S407. If the principal subject candidate is uniquely determined, image processing unit 24 selects the principal subject candidate as the principal subject and then terminates the principal subject determination processing. If no candidate is found, image processing unit 24 determines that no principal subject exists and then terminates the principal subject determination processing. If multiple subject candidates are found (A MULTIPLE OF CANDIDATES in step S801), processing proceeds to step S802.

[0086] The following is with reference to the Fig. Sections 9A to 9C describe a specific example of determining the main motive.

[0087] If “person” in Fig. When step S407 is set, the image processing unit 24 selects the person's face 904. Fig.9B as the main motif from the person's face 901 and the cats 902 and 903 in Fig. 9A according to the priority setting and then finishes determining the main motive.

[0088] When “animal” in Fig. When step S407 is set to 3, there are a variety of capture results for cat under person face 901 and cats 902 and 903 in Fig. 9A. Then the processing proceeds to step S802.

[0089] When “Automatic” in Fig. When step S407 is set to 3, there is no preferred subject for capture, and therefore a variety of capture results are available for both the person and the cat. The processing then proceeds to step S802.

[0090] If “vehicle” in Fig. 3 in step S407 is set, none of the person face 901 and the cats 902 and 903 are in Fig.9A is selected as the subject. The image processing unit 24 therefore determines that no main subject is present and then terminates the main subject determination processing.

[0091] In step S802, the image processing unit 24 selects the main subject from the multitude of subject candidates determined in step S801 based on the positions, sizes, and reliabilities of the subjects captured in step S404. For example, a case is assumed in which the image processing unit 24 selects a subject near the center of the field of view as the main subject. If, in this case, the person's face 901 and the cats 902 and 903 remain as subject candidates in step S801, the image processing unit 24 selects the person's face 904 in step S802. Fig. 9B is chosen as the main motif, since the person's face is closest to the center in 901.

[0092] If cats 902 and 903 remain as possible subjects, the image processing unit 24 selects cat 905. Fig.9C is chosen as the main motif, since the cat is closest to the center in 902.

[0093] Although in the present embodiment the image processing unit 24 selects a subject near the center of the field of view from candidate subjects as the main subject, the present invention is not limited thereto. The image processing unit 24 can, for example, select the subject closest to the center of the region subjected to automatic focusing as the main subject, select the largest subject as the main subject, select the subject with the highest detection reliability as the main subject, and determine the main subject by a combination of these factors. (Example of implementation when the user performs a setting operation on the screen)

[0094] The embodiment described above is based on an example in which the imaging device automatically detects 100 subjects, determines subject types in the same region, and identifies the main subject. The present embodiment is described below in a focused manner, focusing on an example in which, when the user specifies a particular region in the live view screen displayed on the display unit 28, the image processing unit 24 changes the description directory switching sequence, identifies the subject types in the same region, and identifies the main subject.

[0095] The description directory switching sequence performed by the description directory data selection unit 204 in step S403 when the user specifies any region in the live view screen is described below with reference to Fig. 10 described.

[0096] According to the Fig. 5A and Fig.5B The image processing unit 24 changes the description directory switching sequence according to the previously captured subjects and the priority capture subject setting. However, if the user defines a region in the live view screen according to the present embodiment, the image processing unit 24 changes all captureable description directories regardless of the previously captured subjects and the priority capture subject setting. This processing is intended to accurately capture subjects in the defined region by switching between all captureable description directories to precisely reflect the region definition made by the user, irrespective of the previously captured subjects.

[0097] An example of switching description directory data is given below with reference to Fig.As described in section 10, the image processing unit 24 switches between the description directory data in the order person's head, vehicle 1 (motorcycle), and vehicle 2 (automobile) in the first frame; switches between the description directory data in the order person's head, animal (dog / cat), and animal (bird) in the second frame; and switches the description directory data across a plurality of frames. Although, in the present embodiment, the image processing unit 24 switches the person's head description directory in both the first and second frames, it can change the person's head description directory in one frame to a different description directory according to the priority capture motif setting. For example, if vehicle is assigned priority, the image processing unit 24 can use any of the vehicle description directories in the second frame.If Animal priority is assigned, the image processing unit 24 can use any of the animal description directories in the second frame.

[0098] The species identification process in step S409 is described below in focus on a characteristic process according to the present embodiment.

[0099] In the present embodiment, the species identification processing is carried out when a large number of motif types are detected in a region defined by the user.

[0100] The main motif determination processing in step S410 is described below, focusing on a characteristic processing method according to the present embodiment. In this embodiment, a motif present in the user-defined region is determined as the main motif.

[0101] If no subject is detected in the specified region, the image processing unit 24 designates the specified region as the primary subject. However, during the description directory data switching sequence in step S403, the image processing unit 24 switches between all description directories in the next frame until a detectable subject is detected in the specified region.

[0102] The image processing unit 24 can restrict the types of subjects in the defined region that are to be designated as the main subject according to the priority subject capture setting. Examples of possible restrictions are as follows: if person is assigned priority, all subjects can be selected as the main subject. If animal is assigned priority, a vehicle captured in the defined region will not be selected as the main subject. If vehicle is assigned priority, an animal captured in the defined region will not be selected as the main subject. By restricting the type of main subject, the image processing unit 24 can select the defined region as the main subject, as in the case described above where no subject is captured in the defined region, or it can simply apply the positions and sizes of subjects from capture results.

[0103] If it is determined that a restricted subject is to be set, the image processing unit 24 can use the description directories with the priority setting without selecting the description directory of the restricted subject in the next and subsequent frames. An example case is assumed in which animal is assigned priority. If a vehicle subject is set in this case, the image processing unit 24 does not select a vehicle description directory to avoid capturing a vehicle, but frequently switches between animal description directories in the subsequent frames, thus facilitating the capture of an animal. Implementing the control in this way facilitates the transition to a subject with the priority setting.

[0104] The present embodiment has been described above with a focus on region setting on the display screen of the display unit 28 during live view image acquisition, where the display unit 28 sequentially displays images sequentially input from the image sensor. However, the user can set a region on the screen displayed in the viewfinder using the line of sight, or set a region on the screen displayed in the live view or the viewfinder by operating a displayed pointer. The method for setting a region is not limited.

[0105] While the present invention has been specifically described based on the embodiments described above, the present invention is not limited thereto, but can be modified and altered in various ways within the scope of the accompanying patent claims.

[0106] The present invention enables the selection of a correct acquisition method even in a case where a large number of acquisition results are available based on a large number of descriptive directories for the same subject. Further examples of implementation

[0107] Embodiments of the present invention can also be implemented by a computer of a system or device which reads and executes computer-executable instructions (for example, one or more programs) recorded on a recording medium (which can also be more fully described as a "non-volatile, computer-readable storage medium") for performing the functions of one or more of the embodiments described above, and / or which contains one or more circuits (for example, an application-specific integrated circuit (ASIC)) for performing the functions of one or more of the embodiments described above, and by means of a computer of the system or device which, for example, reads and executes the computer-executable instructions from the storage medium for performing the functions of one or more of the embodiments described above.and / or control of one or more circuits to perform the functions of one or more of the embodiments described above, a method is implemented. The computer may comprise one or more processors (for example, a central processing unit (CPU), microprocessing unit (MPU)) and may include a network of separate computers or separate processors for reading and executing the computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may be, for example, a hard disk and / or random access memory (RAM) and / or read-only memory (ROM) and / or memory of distributed computing systems and / or an optical disc (such as a compact disc (CD),Digital Versatile Disc (DVD) or Blu-ray Disc (BD™) and / or a flash memory device and / or a memory card or the like.

[0108] Although the present invention has been described with reference to exemplary embodiments, it is evident that the invention is not limited to the disclosed embodiments. The scope of protection of the following patent claims is to be interpreted in the broadest possible way.

Claims

[1] Image processing device (24) with a recording device (201) for recording a variety of types of motifs for an input image, a setting device (205) for setting a type of motif as a priority motif, a main motive determination device (206) for determining a recording result as the main motive based on the multitude of types of motives recorded by the recording device (201) and a control device (204), wherein, in a case where recording results of a large number of types of motifs are available in the same region, the main motif determination device (206) determines one motif type in the same region based on the set priority motif and the types of motifs recorded, wherein in a case where any region of the input image is specified, the control device (204) selects a switching sequence that toggles between all detectable description directories, wherein the recordable descriptive directories contain descriptive directory data for the multitude of motif types and for each motif region, and where the capture of the multitude of motif types is performed many times for the same entered image, while switching between the captureable description directories. [2] Image processing device (24) according to claim 1, further comprising a calculation device for calculating a detection reliability for the motifs detected by the detection device (201), wherein the main motif determination device (206) determines a motif type in the same region based on the reliability calculated by the calculation device. [3] Image processing device (24) according to claim 1 or 2, wherein the acquisition device (201) contains descriptive directory data that have completed a neural network-based learning for each motif type, and where the description directory data contains various network parameters. [4] Image processing device (24) according to one of claims 1 to 3, wherein, after obtaining acquisition results of a plurality of preset types of motifs, the main motif determination device (206) performs processing to determine the main motif. [5] Image processing device (24) according to one of claims 1 to 4, wherein a priority is set for each type of subject. [6] Image processing device (24) according to any one of claims 1 to 5, wherein in a case where detection results of the plurality of types of motifs are available in the same region, the main motif determination device (206) determines the motif with the highest priority as the main motif. [7] Image processing device (24) according to any one of claims 1 to 6, wherein in a case where detection results of a plurality of types of subjects with the same priority are available in the same region, the main subject determination device (206) determines the subject with the greatest reliability as the main subject. [8] Image processing device (24) according to any one of claims 1 to 7, wherein the main subject determination device (206) normalizes a reliability according to the subject types and determines the main subject using the normalized reliability. [9] Method for controlling an image processing device (24), the method comprising Capturing a wide variety of motif types for an entered image, Setting one type of motif as the priority motif and Determining a data collection result as the main motive based on the multitude of motive types captured by the data collection process, in determining the main motive, where, in a case where recording results of a large number of motive types are available in the same region, the main motive determination is made to determine one motive type in the same region based on the set priority motive and the types of motives recorded. where, in a case where any region of the input image is specified, a toggle sequence is selected that switches between all detectable description directories, wherein the recordable descriptive directories contain descriptive directory data for the multitude of motif types and for each motif region, and where the capture of the multitude of motif types is performed many times for the same input image, while switching between the captureable description directories. [10] Method according to claim 9, further comprising calculating a detection reliability for the motifs detected by the detection, wherein the main motif determination determines a motif type in the same region based on the reliability. [11] Computer-executable program for causing a computer to execute each procedure of the method for controlling an image processing device (24) according to claim 9 or claim 10. [12] Non-volatile computer-readable storage medium that stores a program for causing a computer to execute each process of the method for controlling an image processing device (24) according to claim 9 or claim 10.

Images