Image processing apparatus, image processing method, storage medium, and computer program product

Abstract

This disclosure relates to image processing apparatus, image processing method, storage medium, and computer program products. When multiple detection results exist for the same subject from multiple dictionaries, it may be impossible to correctly select the subject type. The image processing apparatus includes: a subject detection unit configured to detect multiple types of subjects for an input image; a detection reliability calculation unit configured to calculate the detection reliability of detected subjects; a priority subject setting unit configured to set the subject type as a priority subject; and a main subject determination unit configured to determine the detection result as the main subject from the detected subjects based on the set priority subject and detection reliability. The main subject determination unit determines one subject type within the same region based on the set priority subject, detection reliability, and the type of the detected subject.

Description

[0001] (This application is a divisional application of the application filed on April 6, 2022, with application number 2022103549547, entitled "Image Processing Apparatus and Control Method Thereof and Storage Medium Thereof".) Technical Field

[0002] The present invention relates to an image processing device with a subject detection function and a control method for the image processing device. Background Technology

[0003] To detect multiple types of subjects based on image data captured by a camera device (such as a digital camera), a known technique relies on a learned model that has undergone machine learning for each subject type. To capture an image with focus, brightness, and color adjusted to suitable conditions based on the detected subjects, a primary subject needs to be determined from the acquired multiple subjects. Japanese Patent Application Laid-Open No. 2017-5738 discusses a method for determining the primary subject based on a stability factor among the detected subjects, where the stability factor indicates whether subject detection is stable across multiple frames. Summary of the Invention

[0004] The present invention aims to provide an image processing device and a control method for the image processing device that can properly detect a subject even when there are multiple detection results from multiple dictionaries for the same subject.

[0005] According to an aspect of the present invention, an image processing apparatus includes: a subject detection unit configured to detect multiple types of subjects in an input image; a detection reliability calculation unit configured to calculate the detection reliability of detected subjects; a priority subject setting unit configured to set the type of a subject as a priority subject; and a main subject determination unit configured to determine a main subject from the detected subjects based on the set priority subject and the detection reliability. When multiple types of subject detection results exist in the same area, the main subject determination unit determines one subject type in the same area based on the set priority subject, the detection reliability, and the type of the detected subject.

[0006] According to one aspect of the present invention, an image processing apparatus includes: a detection unit for detecting multiple types of subjects in an input image; a setting unit for setting the type of a subject as a priority subject; and a main subject determination unit for determining a main subject based on the multiple types of subjects detected by the detection unit, wherein, when there are detection results for multiple types of subjects in the same area, the main subject determination unit determines one subject type in the same area based on the set priority subject and the types of the detected subjects.

[0007] According to another aspect of the present invention, a control method for an image processing apparatus includes: detecting multiple types of subjects for an input image; setting the type of the subject as a priority subject; and in determining a primary subject, determining a detection result as a primary subject based on the multiple types of subjects detected by the detection, wherein, in the case where there are detection results for multiple types of subjects in the same area, the primary subject determination determines one subject type in the same area based on the set priority subject and the types of the detected subjects.

[0008] According to another aspect of the present invention, a non-transitory computer-readable storage medium stores a program for causing a computer to perform various processes of the control method of the image processing apparatus described above.

[0009] Further features of the invention will become clear from the following description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0010] Figure 1A and 1B The appearance of the camera equipment, including image processing devices, is shown.

[0011] Figure 2A and 2B This is a block diagram illustrating the configuration of a camera system, including image processing equipment.

[0012] Figure 3 This shows an example of a method for setting the objects that the user should prioritize detecting.

[0013] Figure 4 This is a flowchart illustrating the overall process.

[0014] Figure 5A and 5B An example of a sequence for switching between multiple types of dictionary data is shown.

[0015] Figure 6 This is a flowchart illustrating the determination process used to identify the subject type in the same area.

[0016] Figures 7A to 7F An example of a type determination process for determining the type of a subject in the same area is shown.

[0017] Figure 8 This is a flowchart illustrating the process of determining the main subject.

[0018] Figures 9A to 9C This shows an example of the main subject determination process.

[0019] Figure 10 This example shows a sequence of switching between multiple types of dictionary data when arbitrarily specified by the user. Detailed Implementation

[0020] Figure 1A and 1B The appearance of a camera device 100, which includes an image processing device as an example of a device to which the present invention is applicable, is shown. Figure 1A This is a perspective view showing the front of the camera device 100. Figure 1B This is a perspective view showing the back of the camera device 100.

[0021] refer to Figure 1A and 1B The display unit 28, located on the back of the camera, displays images and various types of information. The touch panel 70a can detect touch operations on the display surface (operation surface) of the display unit 28. The viewfinder external display unit 43, located on the top surface of the camera, displays shutter speed, aperture, and various other camera settings. The shutter button 61 is an operating unit for issuing recording instructions. The mode selection switch 60 is an operating unit for switching between various modes. The terminal cover 40 is a cover for protecting the connector (not shown) of the connection cable used to connect external devices and the camera equipment 100.

[0022] The main electronic dial 71 is a rotary operating component included in the operation unit 70. Rotating the main electronic dial 71 allows for changing settings (such as shutter speed and aperture). The power switch 72 is an operating component used to turn the power of the camera device 100 on and off. The secondary electronic dial 73 (a rotary operating component included in the operation unit 70) allows for moving the selection box and sending images. The cross key 74 included in the operation unit 70 is a four-way key (up, down, right, and left) that can be pressed. This allows for operations corresponding to the pressed portions of the cross key 74. The SET button 75 (a push button included in the operation unit 70) is mainly used to determine the selection item.

[0023] The motion image button 76 is used to indicate the start and stop of motion image capture (recording). In the shooting standby state, pressing the auto exposure (AE) lock button 77 included in the operation unit 70 fixes the exposure conditions. The magnification button 78 included in the operation unit 70 turns the magnification mode on or off in the live view display of the image capture mode. After activating the magnification mode, the live view image can be magnified and reduced by operating the main electronic dial 71. In playback mode, the magnification button 78 magnifies the playback image to increase the magnification. The playback button 79 included in the operation unit 70 switches between image capture mode and playback mode. When the user presses the playback button 79 in image capture mode, the camera device 100 enters playback mode, enabling the display unit 28 to display the latest image from the recording medium 200. Pressing the menu button 81 included in the operation unit 70 displays a menu screen on the display unit 28, allowing the user to make various settings. The user can intuitively make various settings using the menu screen displayed on the display unit 28, the cross key 74, and the SET button 75.

[0024] The touch bar 82 is a linear touch operation component (line touch sensor) for receiving touch operations. The touch bar 82 is positioned where the user can operate it with the thumb of their right hand while holding the grip portion 90. The touch bar 82 accepts light touch operations (touching the touch bar 82 and then separating the finger without moving it for a predetermined time period) and right / left swipe operations (touching the touch bar 82 and then moving the touch position while in contact with the touch bar 82). The touch bar 82 is an operation component distinct from the touch panel 70a and does not have a display function.

[0025] The communication terminal 10 is used by the imaging device 100 to communicate with the lens side, which can be attached to and detached from the device. The eyepiece section 16 of the eyepiece viewfinder (internal viewfinder) allows the user to visually identify the image displayed in the electronic viewfinder (EVF) 29 within the viewfinder. The eyepiece detection unit 57 is an eye contact detection sensor that detects whether the photographer's eye is in contact with the eyepiece section 16. A cover 207 covers the slot for storing the recording medium 200. The grip section 90 has a shape that facilitates right-handed grip when the user holds the imaging device 100.

[0026] The shutter button 61 and the main electronic dial 71 are positioned such that, while holding the digital camera by gripping the grip 90 with the little, ring, and middle fingers of the right hand, the index finger of the right hand can operate these operating components. The secondary electronic dial 73 and the touch bar 82 are positioned such that, in the same state, the thumb of the right hand can operate these operating components.

[0027] (Camera equipment configuration)

[0028] Figure 2A and 2B This is a block diagram illustrating a configuration example of a camera device 100 according to this exemplary embodiment. (See reference...) Figure 2A and 2B The lens unit 150 is equipped with a replaceable camera lens. Although the lens 103 typically includes multiple lenses, for simplicity... Figure 2A A single lens is shown as lens 103. Lens unit 150 communicates with camera device 100 via communication terminal 6. Camera device 100 communicates with lens unit 150 via communication terminal 10. Lens unit 150 communicates with system control unit 50 via communication terminals 6 and 10. Internal lens system control circuit 4 controls aperture 1 via aperture drive circuit 2 and focuses on the subject by shifting the position of lens 103 via autofocus (AF) drive circuit 3.

[0029] Shutter 101 is a focal plane shutter that allows the exposure time of the imaging unit 22 to be arbitrarily controlled under the control of the system control unit 50.

[0030] The camera unit 22 is an image sensor that converts optical images into electrical signals, including a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) sensor. The camera unit 22 may be equipped with a camera surface phase difference sensor for outputting defocus information to the system control unit 50. An analog-to-digital (A / D) converter 23 converts analog signals into digital signals. The A / D converter 23 converts the analog signals output from the camera unit 22 into digital signals.

[0031] The image processing unit 24 performs predetermined pixel interpolation, size adjustment processing (such as reduction), and color conversion processing on data from the A / D converter 23 or the memory control unit 15. The image processing unit 24 also performs predetermined calculations on the captured image data. The system control unit 50 performs exposure control and range control based on the calculation results obtained by the image processing unit 24. This enables AF processing, automatic exposure (AE) processing, and electronic flash pre-emission (EF) processing based on the through-the-lens (TTL) method. The image processing unit 24 also performs predetermined calculations on the captured image data and performs TTL-based automatic white balance (AWB) processing based on the obtained calculation results.

[0032] Data output from A / D converter 23 is written to memory 32 via image processing unit 24 and memory control unit 15, or directly to memory 32 via memory control unit 15. Memory 32 stores image data captured by camera unit 22 and then converted into digital data by A / D converter 23, as well as image data to be displayed on display unit 28 and EVF 29. Memory 32 is provided with sufficient storage capacity to store a predetermined number of still images and moving images and sounds for a predetermined time period.

[0033] The memory 32 also serves as an image display memory (video memory). The digital-to-analog (D / A) converter 19 converts the image display data stored in the memory 32 into an analog signal, which is then supplied to the display unit 28 and the EVF 29. The display image data stored in the memory 32 is displayed on the display unit 28 and the EVF 29 via the D / A converter 19. The display unit 28 and the EVF 29 display data on a liquid crystal display (LCD) or an organic electroluminescent (EL) display based on the analog signal from the D / A converter 19. A digital signal undergoes an A / D conversion once by the A / D converter 23, is stored in the memory 32, and is then converted into an analog signal by the D / A converter 19. The analog signal is then sequentially transmitted to the display unit 28 or the EVF 29 for display, thereby enabling live view (LV) display. Hereinafter, the image displayed in live view is referred to as a live view (LV) image.

[0034] The camera's shutter speed, aperture, and other various settings are displayed on the external display unit 43 via the external display unit drive circuit 44.

[0035] The non-volatile memory 56 is an electrically erasable and recordable memory, such as an electrically erasable programmable read-only memory (EEPROM). Constants and programs for the operation of the system control unit 50 are stored in the non-volatile memory 56. The programs stored in the non-volatile memory 56 refer to programs used to execute (as described below) the various flowcharts according to this exemplary embodiment.

[0036] A system control unit 50, including at least one processor or circuitry, controls the entire camera device 100. When the system control unit 50 executes the program recorded in the non-volatile memory 56, it implements various processes according to this exemplary embodiment (described below). The system memory 52 is, for example, random access memory (RAM). Constants and variables for the operation of the system control unit 50, as well as the program read from the non-volatile memory 56, are loaded into the system memory 52. ​​The system control unit 50 also controls the memory 32, the D / A converter 19, and the display unit 28 for display control.

[0037] System timer 53 is a time measurement unit that measures the time used for various types of control and the time of the built-in clock.

[0038] The operation unit 70 is an operation component that inputs various operation instructions to the system control unit 50.

[0039] The mode selection switch 60 (including the operating element in the operation unit 70) switches the operating mode of the system control unit 50 between still image shooting mode, moving image shooting mode, and playback mode. Still image shooting modes include automatic image shooting mode, automatic scene determination mode, manual mode, aperture priority mode (Av mode), shutter speed priority mode (Tv mode), and program automatic exposure (AE) mode (P mode). Still image shooting modes also include various scene modes as shooting settings for various shooting scenarios, and include a custom mode. The mode selection switch 60 allows the user to directly select any of these modes. Alternatively, once the user selects the image shooting mode list screen using the mode selection switch 60, they can select any of the displayed modes and then change the mode using other operating elements. Similarly, moving image shooting modes may also include multiple modes.

[0040] During the operation of the shutter button 61 on the camera device 100 (i.e., half-pressed (camera preparation indication)), the first shutter switch 62 is turned on to generate a first shutter switch signal SW1. The first shutter switch signal SW1 causes the system control unit 50 to start camera preparation operations (such as autofocus (AF) processing, auto exposure (AE) processing, auto white balance (AWB) processing, and electronic flash pre-illumination (EF) processing, etc.).

[0041] When the shutter button 61 is fully pressed (i.e., the so-called full press (image capture indication)), the second shutter switch 64 is activated to generate a second shutter switch signal SW 2. In response to the second shutter switch signal SW 2, the system control unit 50 begins a series of shooting processes from reading signals from the imaging unit 22 to writing the captured image (as an image file) into the recording medium 200.

[0042] The operation unit 70 includes various operation components that serve as input components for receiving operations from the user.

[0043] The operating unit 70 includes at least the following operating components: shutter button 61, main electronic dial 71, power switch 72, secondary electronic dial 73, cross keys 74, SET button 75, moving image button 76, AF lock button 77, zoom button 78, playback button 79, menu button 81, and touch bar 82. Other operating components 70b are also indicated together with the operating components not described separately in the block diagram.

[0044] The power control unit 80 includes a battery detection circuit, a DC-DC converter, and a switching circuit for selecting the block to be powered. The power control unit 80 detects the presence or absence of a battery, the battery type, and the remaining battery capacity. Based on the detection results and instructions from the system control unit 50, the power control unit 80 also controls the DC-DC converter to supply the required voltage to the recording medium 200 and other components for the desired time period. The power unit 30 includes a primary battery (such as an alkaline battery or a lithium battery), a secondary battery (such as a NiCd battery, NiMH battery, or lithium-ion battery), and an AC adapter.

[0045] The recording medium interface (I / F) 18 is an interface with the recording medium 200, such as a memory card or a hard disk. The recording medium 200 is, for example, a memory card used to record captured images, including semiconductor memory or a hard disk.

[0046] Communication unit 54 establishes a wireless or wired connection for transmitting and receiving video and audio signals. Communication unit 54 can also connect to a wireless local area network (LAN) and the Internet. Communication unit 54 can also communicate with external devices via Bluetooth® and Bluetooth Low Energy. Communication unit 54 can transmit images captured by camera unit 22 (including LV images) and images recorded on recording medium 200, and receive images and other various types of information from external devices.

[0047] The attitude detection unit 55 detects the attitude of the camera device 100 in the direction of gravity. Based on the attitude detected by the attitude detection unit 55, the system control unit 50 can determine whether the image captured by the camera unit 22 is an image captured by the horizontally held camera device 100 or an image captured by the vertically held camera device 100. The system control unit 50 can add orientation information corresponding to the attitude detected by the attitude detection unit 55 to the image file of the image captured by the camera unit 22, or rotate the image before recording. An accelerometer or gyroscope sensor can be used as the attitude detection unit 55. The movement (pan, pitch, rise, and stationary) of the camera device 100 can also be detected by using an accelerometer or gyroscope sensor as the attitude detection unit 55.

[0048] (Configuration of the image processing unit)

[0049] Figure 2BThe image processing unit 24 according to this exemplary embodiment is illustrated. The image processing unit 24 includes a subject detection unit 201, a detection history storage unit 202, a dictionary data storage unit 203, a dictionary data selection unit 204, a type determination unit 205, and a main subject determination unit 206. In this exemplary embodiment, although these units are described as part of the image processing unit 24, these units may be configured as part of the system control unit 50, or separately from the image processing unit 24 and the system control unit 50. For example, the image processing unit 24 may be located on a smartphone or tablet terminal.

[0050] The image processing unit 24 sends the image data generated based on the data output from the A / D converter 23 to the subject detection unit 201 in the image processing unit 24.

[0051] According to this exemplary embodiment, the subject detection unit 201 includes a convolutional neural network (CNN) that has undergone machine learning (deep learning) and detects specific subjects. The type of detectable subject is based on dictionary data stored in the dictionary data storage unit 203. According to this exemplary embodiment, the subject detection unit 201 includes different CNNs (different network parameters) depending on the type of detectable subject. The subject detection unit 201 may be implemented by a graphics processing unit (GPU) or circuitry dedicated to CNN-based estimation processing.

[0052] CNN machine learning can be performed using any method. For example, a predetermined computer, such as a server, can perform CNN machine learning, and the camera device 100 can acquire the learned CNN from the predetermined computer. According to this exemplary embodiment, the predetermined computer inputs image data for learning and performs supervised learning by using subject position information corresponding to the image data for learning as teacher data (note), thereby enabling CNN learning for the subject detection unit 201. This completes the generation of the learned CNN. CNN learning can be performed by the camera device 100 or the image processing device described above.

[0053] As described above, the subject detection unit 201 includes a CNN (learned model) that has been learned through machine learning. The subject detection unit 201 takes image data as input, estimates the position, size, and reliability of the subject, and outputs the estimated information. The CNN can be, for example, a network with a layered structure (consisting of alternating stacked convolutional and pooling layers), fully connected layers, and an output layer, wherein the fully connected layers and the output layer are connected to the layered structure. For example, in this case, backpropagation is used for CNN learning. The CNN can also be a Neocognitron CNN, which includes a set of feature detection layers (S layers) and feature integration layers (C layers). For example, in this case, a learning technique called "Add-if Silent" is used for CNN learning.

[0054] Any model other than a learned CNN can be used for the subject detection unit 201. For example, a learned model generated through machine learning (such as a support vector machine or decision tree) can be applied to the subject detection unit 201. The subject detection unit 201 does not necessarily need to be a learned model generated through machine learning. For example, any subject detection method that does not use machine learning can be applied to the subject detection unit 201.

[0055] The detection history storage unit 202 stores the subject detection history from the image data detected by the subject detection unit 201. The system control unit 50 sends the subject detection history to the dictionary data selection unit 204. According to this exemplary embodiment, the detection history storage unit 202 stores dictionary data for subject detection, as well as the location, size, and reliability of the detected subjects, as subject detection history. The detection history storage unit 202 may additionally store data including the number of times the subject was detected and identifiers such as image data of the detected subjects.

[0056] The dictionary data storage unit 203 stores dictionary data for detecting specific subjects. The system control unit 50 reads the dictionary data selected by the dictionary data selection unit 204 from the dictionary data storage unit 203 and then sends the data to the subject detection unit 201. For example, the dictionary data for detecting various subjects registers the features of each region of a specific subject. To detect multiple types of subjects, dictionary data for each subject and each subject region can also be used. The dictionary data storage unit 203 stores dictionary data for detecting multiple types of subjects (including dictionary data for detecting "people", dictionary data for detecting "animals", and dictionary data for detecting "vehicles"). In addition to the dictionary data for detecting "animals", the dictionary data storage unit 203 can also store dictionary data for detecting "birds", which have a special shape and are in high demand for subject detection. The dictionary data storage unit 203 can also store dictionary data for "cars", "motorcycles", "trains", "airplanes", etc., as a subdivision of the dictionary data for detecting "vehicles".

[0057] The subject region detected by multiple types of dictionary data stored in the dictionary data storage unit 203 can be used as a focus detection region. For example, in a composition that includes an obstacle on the front side and a subject on the back side, the target subject can be focused by focusing on the interior of the detected region.

[0058] In this exemplary embodiment, although the subject detection unit 201 uses multiple types of dictionary data in subject detection generated through machine learning, rule-based dictionary data can be used or used together. Rule-based dictionary data refers to data that, for example, stores images of subjects to be detected or feature quantities specific to those subjects, predetermined by the designer. Subjects can be detected by comparing the images or feature quantities of the dictionary data with the images or feature quantities of the captured image data. Rule-based dictionary data is less complex and therefore has a smaller data size compared to models set by learned models through machine learning. Therefore, subject detection using rule-based dictionary data provides higher processing speed (and lower processing load) compared to subject detection using learned models.

[0059] The dictionary data selection unit 204 selects the dictionary data to be used next based on the subject detection history, predetermined order and rules stored in the detection history storage unit 202, or instructions from the user, and then notifies the dictionary data storage unit 203 of the selected dictionary data.

[0060] According to this exemplary embodiment, the dictionary data storage unit 203 stores dictionary data for each type of subject and each subject region among multiple types of subjects. When switching between multiple types of dictionary data, subject detection is performed multiple times on the same image data. The dictionary data selection unit 204 determines a dictionary data switching sequence, and then determines the dictionary data to be used based on the determined sequence. An example of the dictionary data switching sequence will be described below.

[0061] When multiple subjects are detected in the same area, the type determination unit 205 determines the type of the subjects in that area. The type determination unit 205 determines a detection result based on the subject priority settings set by the user via the operation unit 70 from multiple detection histories stored in the detection history storage unit 202. The determination method will be described below.

[0062] Figure 3 The following example illustrates how, regarding the method for setting the subject to be detected first, the user selects the type of subject to be detected first from the menu screen displayed on the display unit 28. Figure 3 A settings screen is shown for selecting the subject to be detected displayed on the display unit 28. The user selects the subject to be detected first from a list of detectable subjects (such as vehicles, animals, and people) through operations on the operation unit 70. Figure 3 This indicates the status of the selected "Vehicle". (Reference) Figure 3 "None" means that no subject pattern was detected, and "Automatic" means that the subject pattern was detected by not prioritizing specific detectable subjects.

[0063] The primary subject determination unit 206 determines the primary subject based on the following: multiple detection histories stored in the detection history storage unit 202, the user's settings for subjects to be prioritized for detection via the operation unit 70, and the subjects determined by the type determination unit 205. The method for determining the primary subject will be described below.

[0064] (Processing flow of camera equipment)

[0065] Figure 4 This is a flowchart illustrating the feature processing flow of the present invention performed by the camera device 100 according to this exemplary embodiment. Each step of this flowchart is executed by the system control unit 50, or by each unit according to the instructions of the system control unit 50. When this flowchart is started, the camera device 100 is powered on and is in a live view image capture mode, in which the device is prepared to issue an instruction to begin capturing (recording) still or moving images via operation of the operation unit 70.

[0066] Assuming that when the camera unit 22 of the camera device 100 captures one frame (one image data), the process begins from... Figure 4 The process described in steps S401 to S409 is a series of processes. However, the invention is not limited thereto. The series of processes from steps S401 to S409 can be performed in multiple frames. More specifically, the result of subject detection in the first frame can be reflected in any of the second and subsequent frames.

[0067] In step S401, the system control unit 50 acquires image data captured by the camera unit 22 and then output by the A / D converter 23.

[0068] In step S402, the image processing unit 24 adjusts the size of the image data to adapt it to an image size that is easy to process (e.g., quarter video graphics array (QVGA)), and then sends the resized image data to the subject detection unit 201.

[0069] In step S403, the dictionary data selection unit 204 selects the dictionary data generated by machine learning to be used for subject detection, and then sends the selection information for identifying the selected dictionary data to the dictionary data storage unit 203.

[0070] Dictionary data generated through machine learning can be produced by extracting common features of a specific subject from a large amount of image data containing that subject. Examples of common features include not only the size, position, and color of the subject, but also the background and other areas outside the specific subject. Therefore, if the subject to be detected exists in a more constrained background, detection performance (detection accuracy) can be improved with a smaller amount of learning. On the other hand, if learning is performed to detect a specific subject regardless of the background, the generality of the shooting scene increases, but detection accuracy becomes difficult to improve. Detection performance tends to improve as the amount and variety of image data used for dictionary data generation increases. Conversely, even if the amount and variety of image data required for dictionary data generation are reduced, detection performance can be improved by limiting the size and position of the detection region of the subject to be detected in the image data used for subject detection to predetermined values. If the subject partially extends beyond the image data, some features of the subject are lost, resulting in reduced detection performance.

[0071] Typically, larger subject regions contain a larger number of features. In detection using dictionary data with machine learning capabilities, objects with features similar to those of a specific subject to be detected using the dictionary data may be falsely detected as that specific subject. Regions defined as local areas are small compared to the entire region. The number of features included in a region decreases as the area of ​​the region decreases, and the number of objects with similar features increases as the number of features decreases, leading to an increase in false detections.

[0072] The following will refer to Figure 5A and 5B The switching sequence between multiple types of dictionary data for a single frame (one image data) is described in step S403. When multiple types of dictionary data are stored in the dictionary data storage unit 203, subject detection can be performed for a single frame based on multiple dictionaries. On the other hand, in outputting and processing images in live view mode that are captured sequentially, as well as in motion image data during motion image recording, due to issues with image capture speed and processing speed, it is assumed that the number of subject detections that can be performed for a single frame will be limited.

[0073] In this case, the type and order of the dictionary data to be used can be determined based on, for example, the presence or absence of previously detected subjects, the type of dictionary data used in previous detections, and the type of subject to be detected first. When a specific subject is included in a frame, the dictionary data switching sequence may not select dictionary data for detecting that specific subject, which may result in missed subject detection opportunities.

[0074] Therefore, the dictionary data switching sequence needs to be changed according to the settings and scenarios.

[0075] Figure 5A and 5B This diagram illustrates an example of a dictionary data switching sequence when selecting a vehicle as the primary subject to be detected in a structure where up to three subject detections can be performed for a single frame (or where three different detectors can be processed in parallel). V0 and V1 each represent a vertical synchronization time interval for a frame. Objects such as a person's head, vehicle 1 (motorcycle), and vehicle 2 (car) are represented by square boxes indicating that subject detection based on three different types of dictionary data (learned models) can be performed in a time series within a vertical synchronization time interval.

[0076] Figure 5AThis illustrates an example of dictionary data switching when no subject is detected. In the first frame, dictionary data switching occurs in the order of human head, vehicle 1 (motorcycle), and vehicle 2 (car). In the second frame, dictionary data switching occurs in the order of animal (dog / cat), vehicle 1 (motorcycle), and vehicle 2 (car). For example, the camera device 100 always uses dictionary data, enabling the detection of the user from any subject, such as... Figure 3 The menu screen shown indicates the selected subject. This situation can cause inconvenience in changing the priority detection subject settings for different scenes, such as selecting a vehicle when shooting a vehicle and selecting people and animals when shooting other objects. If the timing of the vehicle's appearance is unknown, selecting the priority detection subject setting after noticing the approaching vehicle may result in losing the timing of image capture. On the other hand, this exemplary embodiment allows the user to capture images without considering the priority detection subject settings. More specifically, during periods when no specific subject is detected, such as... Figure 5A As shown, this exemplary embodiment switches between all types of dictionary data across multiple frames. By selecting dictionary data corresponding to the priority detection subject setting in the first or second frame during switching between all types of dictionary data, the detection accuracy of the priority detection subject can be improved even while detecting all detectable subjects. This makes it possible to reduce the number of times the priority detection subject setting is changed. The camera device 100 can be individually configured with a mode that always accesses only specific dictionaries (groups) according to the priority order based on user-specified settings.

[0077] Figure 5B This example illustrates a dictionary data switching process in the next frame when a motorcycle was detected in a previous frame. The dictionary data switching proceeds in the order of vehicle 1 (motorcycle), person's head, and vehicle 1 (motorcycle). However, the dictionary data switching does not necessarily have to follow this order. For example, in the example above, the "person's head" dictionary data can be changed depending on the scene, such as changing to dictionary data for subjects other than motorcycles in a motorcycle camera scene. Furthermore, in this case, exclusion control can be applied to prevent the detection of subjects using the "animal" dictionary data, which has a low probability of being detected in parallel with the "vehicle" dictionary data. Based on the vehicle's texture (design) and color, the vehicle might be falsely detected as an animal. As a result, this exclusion control improves the detection accuracy of the desired subject.

[0078] In step S404, the subject detection unit 201 detects a subject (or the area where the subject exists) based on image data captured by the camera unit 22 and input to the image processing unit 24, using dictionary data for detecting a specific subject (object) stored in the dictionary data storage unit 203. The location and size of the detected subject, information such as calculated reliability, the type of dictionary data used, and the identifier of the image data used for subject detection are stored in the detection history storage unit 202.

[0079] In step S405, the image processing unit 24 determines, based on the subject detection history stored in the detection history storage unit 202, whether subject detection using all required dictionary data has been performed on image data (image data in the same frame) with the same identifier. If subject detection using all required dictionary data has been performed (yes in step S405), the process proceeds to step S406. On the other hand, if subject detection using all required dictionary data has not been performed (no in step S405), the process returns to step S403. In step S403, the image processing unit 24 selects the dictionary data to be used next.

[0080] In step S406, the image processing unit 24 determines whether subject detection using all types of dictionary data has been performed based on the subject detection history stored in the detection history storage unit 202. If subject detection using all types of dictionary data has been performed (yes in step S406), the process proceeds to step S407. On the other hand, if subject detection using all types of dictionary data has not been performed (no in step S406), the image processing unit 24 continues processing the next frame. For example, refer to... Figure 5A To perform subject detection using all the necessary dictionary data, the image processing unit 24 requires two frames and therefore skips subsequent processing stages in the first frame before proceeding to the next frame. Thus, processing proceeds to step S407 in the second frame. According to this exemplary embodiment, the image processing unit 24 skips subsequent processing stages until subject detection using all the necessary dictionary data has been performed. However, the invention is not limited to this. For processing requiring a fast response (such as autofocus), the image processing unit 24 can perform subsequent processing stages only on subjects detected for each frame without waiting for subject detection using all types of dictionary data. For example, as in this exemplary embodiment, if all types of dictionary data currently set can be accessed in priority within two frames, the image processing unit 24 can always perform subsequent processing stages in step S407 and subsequent steps based on the detection results of the two frames, including the last frame from the past frames.

[0081] In step S407, the image processing unit 24 reads the settings for selecting the subject to be detected first from the specific detectable subjects preset by the user via the operation unit 70.

[0082] In step S408, the image processing unit 24 determines whether there are multiple detection results in the same area based on the subject detection history of the detection results of image data with the same identifier stored in the detection history storage unit 202.

[0083] When multiple detection results exist in the same area (yes in step S408), the process proceeds to step S409. Conversely, when multiple detection results do not exist (no in step S408), the process proceeds to step S410. For example, when the detection center coordinates exist in other detection result areas, the image processing unit 24 can determine that multiple detection results exist in the same area. When the detection areas overlap by a predetermined amount (e.g., a threshold ratio) or more, the image processing unit 24 can also determine that multiple detection results exist in the same area.

[0084] In step S409, the type determination unit 205 determines a region detection result based on the following: the priority subject setting set in step S407, the detection result stored in step S405, and the result determined in step S408 that multiple detection results exist in the same region. The determination method will be described below.

[0085] In step S410, the main subject determination unit 206 determines the main subject from multiple detection results of image data with the same identifier based on the subject detection history stored in the detection history storage unit 202, using the priority subject setting set in step S407. In this case, when the image processing unit 24 determines in step S408 that multiple detection results exist in the same area, the image processing unit 24 also uses the result from step S409. In this case, the system control unit 50 can display some or all of the information output by the main subject determination unit 206 on the display unit 28. The determination method will be described below.

[0086] (A process for determining the type of a subject based on the detection results of multiple subjects in the same area)

[0087] The following will refer to Figure 6 Flowchart in Figures 7A to 7F The type determination process in step S409 is described in the flowchart and Table 1. Each step of this flowchart is executed by the system control unit 50, or by individual units following the instructions of the system control unit 50.

[0088] Figures 7A to 7F An example of type determination processing is shown. Figure 7A The input image shown is of a motorcycle 701 as the subject. Figure 7B The state of selecting the character dictionary and detecting character 702 is shown in step S403. Figure 7C The text shows that in step S403, the motorcycle dictionary is selected and the status of motorcycle 703 is detected. Figure 7D This shows the state of selecting the car dictionary in step S403 and falsely detecting car 704. Figure 7E The state of selecting the dog dictionary and falsely detecting dog 705 is shown in step S403. Figure 7F This shows the state where the cat dictionary was selected in step S403 and no detection result was obtained as a result of the processing.

[0089] In step S601, the image processing unit 24 assigns a priority to each subject type to be detected according to the priority settings set in step S407.

[0090] Table 1 shows examples of priority classification based on priority settings and subject type. Referring to Table 1, vertically arranged priority settings include those related to... Figure 3 The settings correspond to "People," "Animal," "Vehicle," "None," and "Auto." The horizontally arranged subject types to be detected include those related to... Figures 7A to 7F The type in the table determines the corresponding "people", "dogs", "cats", "cars", and "motorcycles". Referring to Table 1, a smaller priority number indicates a higher priority, and "no priority" means that the subject is not used.

[0091] In this exemplary embodiment, although the subjects are classified into three different subjects (values): priority subjects (priority 1 in Table 1), non-priority subjects (priority 2 in Table 1), and unused subjects (no priority in Table 1), the invention is not limited thereto. For example, subjects may be classified into two different subjects (values): used subjects and unused subjects. Subjects may be classified into four different subjects (values): highest priority subjects, priority subjects, non-priority subjects, and unused subjects. The number of subject types can be varied according to the number of detectable subject types and possible priority settings. Referring to Table 1, when vehicles are selected as priority subjects, cars and motorcycles are classified as priority subjects, people are classified as non-priority subjects, and dogs and cats are classified as unused subjects. However, the classification method is not limited to this. For example, people can be classified as unused subjects if subject types other than those with priority settings (also known as priority subject types) are not detected. If subject types other than priority subject types are to be detected, dogs and cats can be classified as non-priority subjects.

[0092] [Table 1]

[0093]

[0094] In step S602, the image processing unit 24 performs priority-based subject type determination processing for the same region according to the priority determined in step S601.

[0095] The following will refer to Figures 7A to 7F The specific method is described in the type determination process. Referring to Table 1, when a person is assigned a priority setting, the person subject type is assigned priority 1, so the image processing unit 24 confirms whether a person is detected. Because of the presence of... Figure 7B The image processing unit 24 uses person 702 as the subject type in the area and then terminates the type determination process. When a vehicle is assigned a priority setting, as shown in Table 1, cars and motorcycles are assigned priority 1. Therefore, the image processing unit 24 confirms whether there are detection results for cars and motorcycles with priority 1. Since motorcycle 703 ( Figure 7C ) and car 704 ( Figure 7D Both of these exist, therefore the process continues to step S603. In this case, when motorcycle 703 ( Figure 7C ) and car 704 ( Figure 7D If none of the following are found, the image processing unit 24 checks whether there is a detection result for a person with priority 2. When there is no detection result for a person, as shown in Table 1, since dogs and cats are assigned "no priority", the image processing unit 24 determines that there is no subject in the same area, and then terminates the type determination process.

[0096] In step S603, the image processing unit 24 normalizes the reliability of the detection results stored in step S405 for each subject. Normalization is performed because the maximum value of the reliability of the detection results and the threshold for the reliability of the subject are different for each individually used dictionary. Normalization enables reliability comparisons between subjects with different dictionaries in subsequent processing stages. According to this exemplary embodiment, the minimum and maximum values ​​of reliability that can be adopted by each dictionary are normalized to 0 and 1, respectively. This normalization limits the reliability to a value between 0 and 1, thereby enabling reliability-based subject comparisons. The normalization method is not limited to this. For example, the threshold for the reliability of the subject can be set to 1, and the minimum value of reliability that can be adopted can be set to 0.

[0097] When the image processing unit 24 identifies multiple subject types with the same priority in step S602, then in step S604, the image processing unit 24 determines the subject with high reliability as the result of normalization in step S603 as the subject in the region; then the type determination process terminates. Although this exemplary embodiment determines the subject in the region based on reliability, the determination method is not limited to this. For example, the image processing unit 24 may refer to the detection results of past frames to determine the subject type that has been detected most frequently in multiple frames as the subject in the region.

[0098] refer to Figures 7A to 7F In step S602, the image processing unit 24 will... Figure 7C Motorcycle 703 and Figure 7D The car 704 is identified as the preferred subject in the same area, and then the two subjects are compared. According to this exemplary embodiment, since the input subject is the motorcycle 701, the image processing unit 24 assumes... Figure 7C Motorcycle 703 was identified as the subject in the area under the condition of highest reliability.

[0099] Before the reliability comparison in step S604, in step S602, the image processing unit 24 selects a subject based on priority. Let's assume dogs and cats are subjects with similar common characteristics (such as quadrupedal walking). In this case, if a cat image is input into the dog dictionary, the cat is likely to be misdetected as a dog. However, let's assume dogs and motorcycles are subjects with different common characteristics. In this case, if a motorcycle image is input into the dog dictionary, the motorcycle is unlikely to be misdetected as a dog. However, in... Figure 7E In the case of a false detection of a dog 705, it is difficult to determine which feature of the input image is being perceived, thus hindering the attainment of high reliability. In this situation, it may be difficult to prevent the final output from being falsely detected as a dog. Therefore, the image processing unit 24 first selects the subject according to a set priority to eliminate false detections of unwanted subjects.

[0100] (Process for determining the main subject)

[0101] The following will refer to Figure 8 Flowcharts and Figures 9A to 9C The images in the flowchart are used to describe the main subject determination process in step S410. Each step of this flowchart is executed by the system control unit 50, or by individual units according to instructions from the system control unit 50.

[0102] Figures 9A to 9C This illustrates an example of main subject determination when multiple subjects are detected in the same frame. Figure 9AThe status shows that a human face (901) and cats (902 and 903) were detected.

[0103] Figure 9B This shows the state in which face 904 is selected as the main subject from face 901 and cats 902 and 903. Figure 9C This shows the state where cat 905 is selected as the main subject from human face 901 and cats 902 and 903.

[0104] In step S801, the image processing unit 24 selects a primary subject candidate according to the priority settings set in step S407. In this case, when a primary subject candidate is uniquely determined, the image processing unit 24 selects that candidate as the primary subject and then terminates the primary subject determination process. When no candidate exists, the image processing unit 24 determines that a primary subject does not exist and then terminates the primary subject determination process. When multiple subject candidates exist (the multiple candidates in step S801), the process proceeds to step S802.

[0105] The following will refer to Figures 9A to 9C Describe a specific example of how the main subject is determined.

[0106] When set in step S407 Figure 3 When selecting a "person" from the image processing unit 24, the image processing unit 24 sets the priority based on the selected person. Figure 9A Choose from faces (901) and cats (902 and 903) in the image. Figure 9B The face 904 in the image is selected as the main subject, and then the determination of the main subject is terminated.

[0107] When set in step S407 Figure 3 When referring to "animals", in Figure 9A Multiple detection results for cats are found in face 901 and cats 902 and 903. Then, the processing proceeds to step S802.

[0108] When set in step S407 Figure 3 When the process is set to "automatic," there is no subject to be prioritized for detection, resulting in multiple detection results for people and cats. The process then proceeds to step S802.

[0109] When set in step S407 Figure 3 When referring to "vehicles", Figure 9A None of the faces 901 and the cats 902 and 903 in the image were selected as subjects. Therefore, the image processing unit 24 determined that the main subject did not exist and then terminated the main subject determination process.

[0110] In step S802, the image processing unit 24 selects the main subject from the multiple subject candidates determined in step S801 based on the position, size, and reliability of the subject detected in step S404. For example, suppose the image processing unit 24 selects the subject closest to the center of the field of view. In this case, when the face 901 and cats 902 and 903 remain as subject candidates in step S801, the image processing unit 24 selects the face 901 because it is closest to the center. Figure 9B The face in the image is 904, which is the main subject.

[0111] When cats 902 and 903 remain as subject candidates, image processing unit 24 selects cat 902 because it is closest to the center. Figure 9C The cat 905 is the main subject in the photograph.

[0112] In this exemplary embodiment, although the image processing unit 24 selects the subject closest to the center of the field of view among the candidate subjects as the main subject, the present invention is not limited thereto. For example, the image processing unit 24 may select the subject closest to the center of the area that has passed through autofocus as the main subject, select the subject with the largest size as the main subject, select the subject with the highest detection reliability as the main subject, and determine the main subject by comprehensively determining these factors.

[0113] (Exemplary embodiment when the user performs a specified operation on the screen)

[0114] The exemplary embodiments described above are examples of camera device 100 automatically detecting a subject, determining the type of a subject in the same area, and determining the main subject. The exemplary embodiments will now be described with reference to an example in which, when a user specifies a region in the live view displayed on display unit 28, image processing unit 24 changes the dictionary switching sequence, determines the type of a subject in the same area, and determines the main subject.

[0115] The following will refer to Figure 10 This describes the dictionary switching sequence performed by the dictionary data selection unit 204 in step S403 when the user specifies any area in the live view.

[0116] refer to Figure 5A and 5BThe image processing unit 24 changes the dictionary switching sequence based on previously detected subjects and priority detection subject settings. However, according to this exemplary embodiment, when the user specifies a region in the live view, the image processing unit 24 changes all detectable dictionaries regardless of previously detected subjects and priority detection subject settings. This processing aims to correctly detect subjects in the specified region by switching between all detectable dictionaries, so as to correctly reflect the user's region specification regardless of previously detected subjects.

[0117] The following will refer to Figure 10 An example describing dictionary data switching is provided. Image processing unit 24 switches between dictionary data in the order of human head, vehicle 1 (motorcycle), and vehicle 2 (car) in the first frame, and in the order of human head, animal (dog / cat), and animal (bird) in the second frame, and switches between dictionary data across multiple frames. In this exemplary embodiment, although image processing unit 24 switches the human head dictionary in both the first and second frames, it can change the human head dictionary to another dictionary in any frame based on the priority detection subject setting. For example, when a vehicle is given priority, image processing unit 24 can use any one of the vehicle dictionaries in the second frame. When an animal is given priority, image processing unit 24 can use any one of the animal dictionaries in the second frame.

[0118] The type determination process in step S409 will now be described in relation to the feature processing according to this exemplary embodiment.

[0119] When multiple types of subjects are detected in a user-specified area, this exemplary embodiment performs type determination processing.

[0120] The main subject determination process in step S410 will now be described in relation to the feature processing according to this exemplary embodiment. This exemplary embodiment determines a subject existing in a region specified by the user as the main subject.

[0121] When no subject is detected in the designated area, the image processing unit 24 identifies the designated area as the main subject. However, in the dictionary data switching sequence in step S403 of the next frame, the image processing unit 24 then switches between all dictionaries until a detectable subject is detected in the designated area.

[0122] The image processing unit 24 can restrict the types of subjects to be identified as primary subjects in a designated area based on the priority detection subject settings. Examples of possible restrictions are as follows: When people are given priority, all subjects can be selected as primary subjects. When animals are given priority, vehicles detected in the designated area will not be selected as primary subjects. When vehicles are given priority, animals detected in the designated area will not be selected as primary subjects. When restricting the type of primary subject, the image processing unit 24 can select the designated area as the primary subject as described above when no subject is detected in the designated area, or it can simply use the position and size of the subject in the detection results.

[0123] When a restricted subject is determined to be specified, the image processing unit 24 can use a dictionary with priority settings in the next frame and subsequent frames without selecting the dictionary for the restricted subject. Consider the example case where animals are given priority. In this case, when a vehicle is specified as the subject, the image processing unit 24 does not select the vehicle dictionary to avoid detecting the vehicle, but instead frequently switches between animal dictionaries in subsequent frames, making it easier to detect animals. Controlling in this way makes it easier to transfer to the subject with priority settings.

[0124] This exemplary embodiment has been described in terms of the area specified in the display screen of the display unit 28 during live view image capture, wherein the display unit 28 sequentially displays images input from the image sensor. However, a user can specify an area on the screen displayed in the viewfinder by using their gaze, or by operating a pointer on the display to specify an area on the live view screen or the screen displayed in the viewfinder. The method for specifying an area is not limited.

[0125] Although the present invention has been specifically described based on the exemplary embodiments described above, the present invention is not limited thereto, but can be modified and changed in various ways within the scope of the appended claims.

[0126] This invention enables the selection of the correct detection type even when multiple detection results using multiple dictionaries exist for the same subject.

[0127] Other embodiments

[0128] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads and executes the program.

[0129] Although the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be interpreted in the broadest possible sense.

Claims

1. An image processing apparatus, comprising: A detection component for detecting multiple types of subjects in an input image; Setting component for setting a priority subject based on user input, wherein one of a plurality of subjects is set among the priority subjects, wherein each subject includes at least one type of subject among the plurality of subjects; as well as The subject determination component is used to determine a subject of a certain type corresponding to the same area when the detection results of the multiple types of subjects overlap in the same area, based on the set priority subjects and the detected types of subjects.

2. The image processing apparatus according to claim 1, further comprising a computing unit, the computing unit being configured to calculate the detection reliability of the plurality of subjects detected by the detection unit, wherein, The subject determination component determines the type of subject in the same region based on the reliability calculated by the calculation component.

3. The image processing apparatus according to claim 1, in, The detection component has dictionary data learned based on a neural network for various types of subjects, and The dictionary data includes different network parameters.

4. The image processing apparatus according to claim 1, wherein, After obtaining the detection results of multiple preset types of subjects, the subject determination component performs processing to determine a subject of a type corresponding to the same region.

5. The image processing apparatus according to any one of claims 1 to 4, wherein, The setting component sets priorities for each type of subject.

6. The image processing apparatus according to any one of claims 1 to 4, wherein, When the detection results of multiple types of subjects overlap in the same area, the subject determination component will identify the subject with the highest priority as the subject.

7. The image processing apparatus according to any one of claims 1 to 4, wherein, When the detection results of multiple types of subjects with the same priority overlap in the same area, the subject determination component will determine the subject with the highest reliability as the subject.

8. The image processing apparatus according to any one of claims 1 to 4, wherein, The subject determination component normalizes the reliability based on the type of the subject, and determines the subject by using the normalized reliability.

9. An image processing method, comprising: Detect multiple types of subjects from the input image; The priority subject is set according to the user's input, and one of a plurality of subjects is set in the priority subject, wherein each subject includes at least one of the plurality of subjects of the subject type; as well as When the detection results of multiple types of subjects overlap in the same area, a subject of one type corresponding to the same area is determined based on the set priority subject and the detected subject type.

10. The image processing method according to claim 9, further comprising: The detection reliability of the plurality of subjects detected by the detection is calculated, wherein the subject determination is based on the reliability to determine the type of subject in the same region.

11. A non-transitory computer-readable storage medium storing a program for causing a computer to perform various processes of the image processing method according to claim 9 or 10.

12. A computer program product comprising a program for causing a computer to perform various processes of the image processing method according to claim 9 or 10.

Images