Image processing apparatus, method for processing image, program, and storage medium

Abstract

To provide an image processing apparatus capable of determining an object desired by a user more accurately.SOLUTION: The image processing apparatus includes: a detection unit for detecting an object from an image; a classification unit for classifying the object detected by the detection unit into a category; and a determination unit for determining whether to set the object as a main object on the basis of the category of the object, the determination unit changing a condition for determining whether to set the object as the main object in accordance with the category of the object.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a technique for detecting a subject from an image captured by an imaging device such as a digital camera or a digital video camera. [Background technology]

[0002] Some digital cameras have been equipped with a function to support users in taking photographs by determining a subject to focus on and automatically maintaining focus on that subject. Because such cameras need to focus on the main subject intended by the user, technologies for determining the main subject have been proposed.

[0003] For example, Patent Document 1 discloses a technology for more appropriately selecting a subject as a main subject. Specifically, the reliability of the subject being the main subject is calculated based on posture information of the subject, and the main subject is determined based on the reliability. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-717944 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology disclosed in Patent Document 1 may not be able to accurately determine the main subject that matches the user's intention. For example, in soccer, the postures of field players, such as forwards and defensemen, who mainly kick the ball, and the postures of the goalkeeper, who mainly saves the ball, during the game are different. Therefore, it may be difficult to determine the subject that the user intends based on the posture of the player, and the main subject may not be able to be accurately determined.

[0006] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an image processing device that can more accurately determine a subject that a user intends to capture. [Means for solving the problem]

[0007] The image processing device according to the present invention comprises a detection means for detecting a subject from an image, a classification means for classifying the subjects detected by the detection means into categories, and a determination means for determining whether or not to set the subject as a main subject based on the category of the subject, and the determination means changes the conditions for determining whether or not to set the subject as the main subject depending on the category of the subject. [Effects of the Invention]

[0008] According to the present invention, it is possible to more accurately determine the subject that the user intends to photograph. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing the configuration of a digital camera as an embodiment of an image processing apparatus of the present invention. [Figure 2] FIG. 2 is a diagram illustrating the configuration of an imaging element. [Figure 3] 10 is a flowchart showing the operation in a still image shooting mode. [Figure 4] 10 is a flowchart showing the operation of a main subject determination process A. [Figure 5] 10 is a flowchart showing the operation of a main subject determination process B. [Figure 6] 10 is a flowchart showing the operation of a main subject determination process according to a soccer action. [Figure 7] FIG. 10 is a diagram showing an example of a screen for registering a person's role. [Figure 8] 10 is a flowchart showing an example of determining roles in a soccer game. [Figure 9] 10A and 10B are diagrams showing examples of postures that are determined to be postures for kicking a ball. [Figure 10]10A and 10B are diagrams showing examples of postures that are not determined as kicking a ball. [Figure 11] 10 is a flowchart showing the operation of a main subject determination process according to a volleyball action. [Figure 12] 10 is a flowchart showing an operation for determining roles in a volleyball game. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] <Camera configuration> FIG. 1 is a block diagram showing the configuration of a digital camera 100 which is an embodiment of an image processing device of the present invention.

[0012] 1, a photographic lens 10 collects light from a subject and forms an optical image. A mechanical shutter 12, which has an aperture function, controls the amount of light from the photographic lens 10 and also controls the exposure time of an image sensor 14. The image sensor 14 converts the optical image into an electrical signal and outputs an analog image signal. An A / D converter 16 converts the analog image signal output from the image sensor 14 into a digital image signal.

[0013] The timing generator 18 is a circuit that supplies clock signals and control signals to the image sensor 14 and A / D converter 16, and is controlled by the memory control circuit 22 and the system control unit 50. By controlling the reset timing of the image sensor 14 with the timing generator 18, it is possible to control the accumulation time using an electronic shutter without relying on the mechanical shutter 12, and this can be used for video shooting, etc.

[0014] The image processing circuit 20 performs pixel interpolation for image enlargement / reduction, color conversion, noise removal, edge enhancement, etc. on the data from the A / D converter 16 or the data from the memory control circuit 22 .

[0015] The image processing circuit 20 also has a face detection function that detects the facial areas of animals and people from an image, and an organ detection function that detects the pupils, nose, mouth, etc. of the face, and also performs a reliability calculation process that indicates the likelihood of face detection and organ detection.

[0016] In face detection by the image processing circuit 20, the shape of the facial contour is stored as feature data inside the image processing circuit 20, and an image area in the image to be detected that matches the feature data (template) is identified by pattern matching processing. The degree of match with the feature data is also calculated, and areas with a degree of match equal to or greater than a predetermined value are determined to be face areas.

[0017] In order to increase the number of face detection opportunities and improve detection accuracy, pattern matching may be performed using multiple feature data, or feature data representing only a portion of a face's shape may be prepared.Furthermore, pattern matching may be performed by varying the size of the feature data so that faces of different sizes can be detected.

[0018] In organ detection, image areas obtained by face detection are identified by pattern matching processing to match the organ feature data (template) stored in advance inside the image processing circuit 20. The reliability of the organ is calculated by calculating the degree of match with the feature data.

[0019] Another detection method is detection using deep learning. The image processing circuit 20 has multiple product-sum calculators and is also used as a processor that performs deep learning processing. The image processing circuit 20 executes subject detection processing on image data using one learning model selected by the system control unit 50 from multiple learning models stored in a non-volatile memory 31 (described later).

[0020] Furthermore, the image processing circuit 20 may switch the learning model stored in the nonvolatile memory 31 to execute a plurality of detection processes for one image data.

[0021] For example, the nonvolatile memory 31 stores three learning models: one that can detect dogs and cats, one that can detect birds, and one that can detect vehicles such as cars. The system control unit 50 then causes the image processing circuit 20 to execute subject detection processing using one or more of the three learning models.

[0022] By performing detection three times on one image using three learning models, it is possible to detect animals such as dogs, cats, and birds, and vehicles such as cars, from one image.

[0023] Furthermore, if a learning model capable of detecting the joint positions of the subject is stored in the nonvolatile memory 31, the image processing circuit 20 can also detect the joint positions of the subject.

[0024] Furthermore, if a learning model that receives the joint positions of the subject and outputs posture information is stored in the nonvolatile memory 31, the image processing circuit 20 can also output posture information of the subject.

[0025] The posture information may be determined using a learning model, or may be determined by calculating an evaluation value from the detected joint positions and determining whether the evaluation value satisfies a condition.

[0026] In this embodiment, face detection and organ detection are performed by pattern matching processing using feature data, and detection of animals such as dogs, cats, and birds, and vehicles such as cars is realized using a learning model. However, other methods may be used as long as they can detect the subject. Furthermore, the subject to be detected does not have to be limited to these.

[0027] Image processing circuit 20 also performs tracking processing between images. When image processing circuit 20 detects a subject, it creates a template of the detected subject and stores it in memory 30. Based on the template information stored in memory 30, image processing circuit 20 searches for an area that matches the template in the image generated at the next timing, and performs tracking processing on the matching area as the subject area.

[0028] The image processing circuit 20 can search for a region that matches the template by cutting out each region of the image, calculating the absolute value of the difference from the template, and determining the region with the smallest difference as the subject region. Another method is to search based on the degree of similarity between the template and histograms, color data, etc. However, other methods may be used as long as they are capable of tracking between images.

[0029] When multiple subjects are detected, the system control unit 50 sets the subject area or main subject area selected by the user as the focus detection area, which will be described later.

[0030] When the detected subject can no longer be detected, the system control unit 50 switches to tracking processing using the image processing circuit 20 and sets the subject area obtained by the tracking processing as the focus detection area. By performing tracking processing, it is possible to perform focus detection on the same subject for a longer period of time and achieve focus.

[0031] If the same subject is detected within a certain period of time, the system control unit 50 switches to detection processing, and if tracking becomes impossible, sets another subject as the subject for focus detection.

[0032] In addition, the image processing circuit 20 performs predetermined calculations using captured image data to perform auto white balance (hereinafter referred to as AWB) processing, and uses the calculation results as white balance (hereinafter referred to as WB) evaluation values. In addition, the color of the image data is converted based on the calculated WB evaluation value.

[0033] Furthermore, the image processing circuit 20 performs predetermined arithmetic processing using the captured image data to calculate an AE evaluation value and an EF evaluation value for performing automatic exposure control (hereinafter referred to as AE) processing and strobe exposure control (hereinafter referred to as EF) processing.

[0034] Based on the obtained AE evaluation value and EF evaluation value, the system control unit 50 controls the exposure control unit 40 according to an algorithm.

[0035] The memory control circuit 22 controls the A / D converter 16 , the timing generator 18 , the image processing circuit 20 , the memory 30 , and the compression / decompression circuit 32 .

[0036] The data from the A / D converter 16 is written into the memory 30 via the image processing circuit 20 and the memory control circuit 22, or the data from the A / D converter 16 is written directly into the memory control circuit 22.

[0037] Image display unit 28, which may be comprised of a TFT, LCD, or the like, acquires and displays the display image data written in memory 20 via memory control circuit 22. By sequentially displaying captured image data using image display unit 28, it is possible to realize an electronic viewfinder function. Furthermore, image display unit 28 can arbitrarily turn its display ON / OFF in response to instructions from system control unit 50, and turning the display OFF can significantly reduce the power consumption of digital camera 100.

[0038] The memory 30 for storing captured still images and moving images has a storage capacity sufficient to store a predetermined number of still images or a predetermined period of moving images. This allows for high-speed and large-volume image writing to the memory 30, even in the case of continuous shooting, in which multiple still images are taken in succession. The memory 30 can also be used as an area for temporarily storing feature data for authentication, and as a work area for the system control unit 50.

[0039] The nonvolatile memory 31 is configured with a flash ROM or the like. The program code executed by the system control unit 50 is written to the nonvolatile memory 31, and the system control unit 50 executes the program code while sequentially reading it out. The nonvolatile memory 31 also has an area for storing facial feature data for authentication as dictionary data, an area for storing system information, an area for storing user setting information, and the like, and various pieces of information and settings can be read and restored the next time the system is started up.

[0040] The compression / decompression circuit 32 is a circuit that compresses and decompresses image data using adaptive discrete cosine transform (ADCT) or the like. It reads images stored in the memory 30, performs compression or decompression processing, and writes the processed data into the memory 30.

[0041] The exposure control unit 40 controls the shutter 12 having an aperture function, and also controls flash light adjustment in conjunction with the flash 48 .

[0042] The focus control unit 42 controls the focusing of the photographic lens 10. The zoom control unit 44 controls the zooming of the photographic lens 10.

[0043] The flash 48 has a dimming function and illuminates the subject. The exposure control unit 40 and the focus control unit 42 perform exposure control and focus control using the TTL system. The system control unit 50 controls the exposure control unit 40 and the focus control unit 42 based on the results of processing the captured image data by the image processing circuit 20.

[0044] The system control unit 50 controls the entire digital camera 100. The system control unit 50 also performs autofocus (hereinafter referred to as AF) processing based on pixel data obtained from the image sensor 14. AF is an automatic focus detection method that detects the focus position by setting a subject area selected by the user or a subject area automatically set by the camera as the focus detection area.

[0045] The image sensor 14 is composed of a C-MOS sensor and its peripheral circuitry, and is configured such that unit pixels, each of which has one photoelectric conversion element, are arranged in a matrix of m pixels horizontally and n pixels vertically. The image sensor 14 is configured so that all pixels can output signals independently. In addition, some of the pixels are configured as focus detection pixels, enabling image sensor phase difference detection AF (image sensor phase difference AF).

[0046] Specifically, as shown in FIG. 2, the image sensor 14 has a plurality of imaging pixels 250 that generate an image of a subject by receiving light beams that pass through the entire area of ​​the exit pupil of the photographing optical system that forms the image of the subject. The image sensor 14 also has a plurality of focus detection pixels 251, each of which receives light beams that pass through a different area of ​​the exit pupil of the photographing optical system. The plurality of focus detection pixels as a whole can receive light beams that pass through the entire area of ​​the exit pupil of the photographing optical system. For example, the image sensor 14 can be configured by leaving a pair of diagonally arranged G pixels of the 2-row x 2-column pixel array as imaging pixels and replacing the R and B pixels with focus detection pixels.

[0047] The system controller 50 performs focus detection processing using the phase-difference AF method using image pickup signals from focus detection pixels embedded in the image sensor 14. More specifically, the system controller 50 performs focus detection based on the amount of misalignment between a pair of images formed on the focus detection pixels by light beams passing through a pair of pupil regions of the imaging optical system.

[0048] In this embodiment, image plane phase-difference AF is achieved by replacing some of the image capturing pixels arranged on the image capturing plane with focus detection pixels 251. However, the present invention is not limited to this method, and other methods may be used as long as they are capable of focus detection. For example, phase-difference focus detection using a known dedicated focus detection sensor or known contrast-based focus detection may be used.

[0049] The operation members 60, 62, 64, 66, 70, and 72 are used to input various operational instructions to the system control unit 50, and are configured by one or a combination of switches, dials, touch panels, pointing by line of sight detection, voice recognition devices, etc.

[0050] Here, these operating members will be specifically described.

[0051] The mode dial switch 60 can be used to switch between various function modes such as power off, automatic shooting mode, shooting mode, panoramic shooting mode, video shooting mode, playback mode, and PC connection mode.

[0052] The shutter switch 62, also called switch SW1, is turned on during operation of the shutter button to instruct the start of imaging preparation operations such as AF (autofocus) processing, AE (auto exposure) processing, and AWB (auto white balance) processing.

[0053] The shutter switch 64, also called switch SW2, is turned on when the shutter button is fully operated. The shutter switch 64 instructs the start of a series of imaging processes. The imaging process includes a read process in which a signal read from the image sensor 14 is written as image data into memory 30 via the A / D converter 16 and memory control circuit 22. Also included are a development process using calculations in the image processing circuit 20 and memory control circuit 22, a compression process in which image data is read from memory 30 and compressed in the compression / decompression circuit 32, and a recording process in which the compressed image data is written to recording medium 200.

[0054] In the imaging process for flash photography, after the flash pre-emission process for EF is performed, the image sensor 14 is exposed for the exposure time determined in the AE process. The flash 48 is then emitted during this exposure period. Furthermore, at the same time as the exposure period ends, the exposure control unit 40 blocks light, thereby ending the exposure of the image sensor 14.

[0055] The display changeover switch 66 can instruct to change the display on the image display unit 28. This function makes it possible to cut off the current supply to the image display unit 28, which is made up of a TFT, LCD, etc., when taking a picture using the optical viewfinder 104, thereby saving power.

[0056] The operation unit 70 is made up of various buttons, a touch panel, a rotary dial, etc. The operation unit 70 includes a menu button, a set button, a macro button, a multi-screen playback page break button, a flash setting button, a single / continuous shooting / self-timer switching button, etc. It also includes a menu movement + (plus) button, menu movement - (minus) button, playback image movement + (plus) button, playback image movement - (minus) button, a shooting image quality selection button, an exposure compensation button, a date / time setting button, etc.

[0057] The zoom switch 72 is a zoom operation means by which the user instructs a change in the magnification of the captured image. The zoom switch 72 consists of a telephoto switch that changes the imaging angle of view to the telephoto side, and a wide switch that changes it to the wide-angle side. Using the zoom switch 72, it is possible to instruct the zoom control unit 44 to change the imaging angle of view of the photographing lens 10. The zoom switch 72 also triggers the image processing circuit 20 to crop the image, or to electronically zoom the imaging angle of view through pixel interpolation processing, etc.

[0058] The power supply unit 86 is made up of a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a nickel-metal hydride battery or a Li-ion battery, an AC adapter, or the like.

[0059] The interface 90 is an interface with a recording medium such as a memory card or a hard disk, and the connector 92 is a connector for connecting to a recording medium such as a memory card or a hard disk.

[0060] The optical viewfinder 104 is a viewfinder that allows shooting without using the electronic viewfinder function of the image display unit 28.

[0061] The communication unit 110 has various communication functions such as USB, IEEE1394, LAN, and wireless communication.

[0062] The antenna 112 is a connector for connecting the digital camera 100 to other devices via the communication unit 110, or an antenna in the case of wireless communication.

[0063] The gyro sensor 115 detects the angular velocity of the digital camera 100. The gyro sensor 115 detects the change in angle per unit time, and the system control unit 50 can calculate the amount of movement of the digital camera 100 in the yawing direction and the amount of movement in the pitching direction based on the angular velocity.

[0064] Recording medium 200 is composed of a memory card, a hard disk, etc. Recording medium 200 is equipped with a recording unit 202 composed of a semiconductor memory, a magnetic disk, etc., an interface 204 for connecting with digital camera 100, and a connector 206 for connecting with digital camera 100. The operation of digital camera 100 configured as above will be described below.

[0065] <Still image shooting operation> Fig. 3 is a flowchart showing the operation when taking a still image with a digital camera. In Fig. 3 and subsequent figures, "S" stands for step. Each process in the flowchart of Fig. 3 is realized by the system control unit 50 executing a control program stored in the nonvolatile memory 31. The same applies to the operation of each flowchart from Fig. 3 onwards.

[0066] In S301, the system control unit 50 determines whether the digital camera 100 is set to still image shooting mode. If the digital camera 100 is set to still image shooting mode, the process proceeds to S302; if not, the still image shooting flow ends. After the still image shooting flow, processing according to the camera settings, such as playback mode or power off, is performed.

[0067] In S302, the system control unit 50 determines whether preparations for starting live view are complete. If it is determined that preparations for starting live view are complete, the process proceeds to S303; if not, the process remains in standby. Preparations for starting live view are a process in which the timing generator 18 supplies clock signals and control signals for live view to the image sensor 14 and the A / D converter 16.

[0068] In S303, the system control unit 50 starts live view processing. Live view processing is a process in which a signal captured by the image sensor 14 is processed by the image processing circuit 20 to generate a live view image, which is temporarily stored in the memory 30, and displayed on the image display unit 28. By continuously generating and displaying the live view image, the user can use the image display unit 28 to check the live view image captured in real time.

[0069] In S304, the system control unit 50 determines whether or not to continue the still image shooting mode. If it is determined that the still image shooting mode should be continued, the process proceeds to S305, and if it is determined that the still image shooting mode should not be continued, the process ends.

[0070] Whether or not to continue in still image shooting mode depends on the camera settings. If the camera is set to still image shooting mode, still image shooting mode continues, but if the setting is changed to a setting other than shooting mode, such as playback mode or power off, still image shooting mode ends.

[0071] In S305, the system control unit 50 determines whether or not the object tracking setting is ON as a camera setting. If it is determined that the object tracking setting is ON, the process proceeds to S307, and if not, the process proceeds to S306.

[0072] In S306, the system control unit 50 determines that specific subject information is not to be used for AF, and proceeds to S308 as there is no main subject.

[0073] In S307, a main subject to be subjected to AF is determined using main subject determination process A, and the process proceeds to S308. Main subject determination process A will be described later.

[0074] In S308, the system control unit 50 determines whether the shutter switch 62 (SW1) is turned on. If it is determined that the shutter switch 62 (SW1) is in the ON state, the process proceeds to S309; ​​otherwise, the process returns to S305.

[0075] In S309, the system control unit 50 determines whether or not there is a main subject. If it is determined that there is a main subject, the process proceeds to S310, and if not, the process proceeds to S311.

[0076] In S310, the system control unit 50 focuses on the main subject and proceeds to S312.

[0077] In S311, the system control unit 50 focuses on the center position of the focus frame (hereinafter referred to as the AF frame). Here, the method of focusing on the center position within the AF frame is one example, and the focusing position does not necessarily have to be the center of the AF frame as long as it is within the AF frame. For example, the focusing position may be an area within the AF frame that is determined to be closest in the depth direction from the image sensor 14, or a prominent area within the AF frame may be determined using color information and brightness information and used as the focusing position.

[0078] In S312, the system control unit 50 determines whether the shutter switch 64 (SW2) is turned on. If it is determined that the shutter switch 64 (SW2) is turned on, the process proceeds to S313; otherwise, the process proceeds to S314.

[0079] In S313, the system control unit 50 executes the photographing process and proceeds to S316.

[0080] In S314, the system control unit 50 determines whether the AF setting is set to servo AF. The servo AF setting is a setting that continuously keeps the focus on the main subject that is the AF target. If it is determined that the servo AF setting is set, proceed to S315; if not, return to S312.

[0081] In S315, the system control unit 50 executes main subject determination process B according to the action. Main subject determination process B is a process for determining the subject to focus on using the posture of the subject. Main subject determination process B according to the action will be described later. When main subject determination process B in S315 is completed, the process returns to S308.

[0082] In S316, the system control unit 50 determines whether the still image capture mode has ended. Whether the still image capture mode has ended is determined based on the user's camera settings, such as switching to playback mode or turning off the power. If it is determined that the still image capture mode has ended, this flow ends; if not, proceed to S317.

[0083] In S317, the system control unit 50 starts live view processing, and the process returns to S308.

[0084] <Main Subject Determination Process A> FIG. 4 is a flowchart showing the operation of the main subject determination process A.

[0085] Main subject determination process A in FIG. 4 is a method of determining a main subject that does not use posture information of the subject, and determines the main subject to be focused on from among multiple detected subjects according to the position and size of the subject.

[0086] In S401, the system control unit 50 determines whether or not a person has been detected within the angle of view. If it is determined that a person has been detected within the angle of view, the process proceeds to S402, and if not, the process proceeds to S405.

[0087] In S402, the system control unit 50 determines whether or not multiple people are detected within the angle of view. If it is determined that multiple people are detected within the angle of view, the process proceeds to S403; otherwise, the process proceeds to S404.

[0088] In S403, the system control unit 50 sets weights for determining the main subject according to the position and size of the person's face, and determines the person subject to be the main subject based on the weights. Once the main subject has been determined, this flow ends.

[0089] In S403, a position weight and a size weight are set for each person, and the subject with the largest weight when the position weight and the size weight are multiplied is determined to be the main subject. For position weight, a coefficient is set so that the closer the center of the person's face is to the center of the screen, the greater the weight. For size weight, a coefficient is set so that the larger the face size of the subject, the greater the weight.

[0090] In S403, it has been explained that the person to be the main subject is determined using position weight and size weight, but the main subject may also be determined using other criteria, such as using the orientation of the person's face, with a higher weight being assigned the more the person is facing forward.

[0091] In S404, the system control unit 50 sets the detected person as the main subject, and ends this flow.

[0092] In S405, the system control unit 50 determines whether or not an animal such as a dog or cat has been detected within the angle of view. If it is determined that an animal has been detected, the process proceeds to S406, and if not, the process proceeds to S411.

[0093] In S406, the system control unit 50 determines whether or not a vehicle such as an automobile has been detected within the angle of view. If it is determined that a vehicle has been detected, the process proceeds to S407, and if not, the process proceeds to S408.

[0094] In S407, the system control unit 50 determines the main subject within the angle of view from the position and subject size using the size normalized for each subject type, and ends this flow once the main subject has been determined.

[0095] The size normalization in S407 normalizes the size so that the subject regions have the same area when, for example, a dog, a cat, and a car are all at the same distance from the image sensor 14. The main subject is determined using the position weight and the normalized size weight.

[0096] In S408, the system control unit 50 determines whether or not multiple animals have been detected within the angle of view. If it is determined that multiple animals have been detected, the process proceeds to S409; otherwise, the process proceeds to S410.

[0097] In S409, the system control unit 50 determines an animal subject to be the main subject from the position weight and size weight for the animal subject within the angle of view, and once the main subject has been determined, this flow ends.

[0098] In S410, the system control unit 50 sets the animal detected within the angle of view as the main subject, and ends this flow.

[0099] In S411, the system control unit 50 determines whether or not a vehicle subject such as an automobile has been detected within the angle of view. If it is determined that a vehicle subject has been detected, the process proceeds to S412; otherwise, the process proceeds to S415.

[0100] In S412, the system control unit 50 determines whether or not multiple vehicles have been detected within the angle of view. If it is determined that multiple vehicles have been detected, the process proceeds to S413; otherwise, the process proceeds to S414.

[0101] In S413, the system control unit 50 determines the vehicle to be the main subject from the position weight and size weight of the vehicle subject within the angle of view, and ends this flow once the main subject has been determined.

[0102] In S414, the system control unit 50 sets the vehicle subject found within the angle of view as the main subject, and ends this flow.

[0103] In S415, the system control unit 50 determines that there is no main subject, and ends this flow.

[0104] <Main subject determination process B according to action> FIG. 5 is a flowchart showing the operation of the main subject determination process B according to the action.

[0105] In this embodiment, action-based main subject determination process B is intended for shooting sports events, particularly soccer or volleyball, and determines the main subject to focus on depending on whether the subject is taking a stance for an action that is an interesting highlight, such as a goal kick in soccer or an attack in volleyball.

[0106] In S501, the system control unit 50 determines whether or not a person has been detected. If it is determined that a person has been detected, the process proceeds to S502, and if not, the process proceeds to S507.

[0107] In S502, the system control unit 50 determines whether or not a soccer game is being filmed. If it is determined that a soccer game is being filmed, the process proceeds to S503, and if not, the process proceeds to S504.

[0108] The determination in S502 as to whether a soccer game is being filmed may be made based on whether the user has instructed the camera to film a soccer game, or based on whether a soccer ball has been detected. Alternatively, it may be determined that a soccer game is being filmed if it is determined that the subject is in a soccer playing pose, such as kicking the ball.

[0109] In S503, the system control unit 50 performs a main subject determination process based on soccer actions, which will be described later, and then proceeds to S506.

[0110] In S504, the system control unit 50 determines whether or not a volleyball game is being filmed. If it is determined that a volleyball game is being filmed, the process proceeds to S505, and if not, the process proceeds to S507.

[0111] The determination in S504 as to whether a volleyball game is being filmed may be made based on whether the user has instructed the camera to film a volleyball game, or based on whether a volleyball has been detected. Alternatively, it may be determined that a volleyball game is being filmed if it is determined that the subject is in a volleyball playing position, such as attacking the volleyball.

[0112] In S505, the system control unit 50 performs a main subject determination process based on a volleyball action, which will be described later, and then proceeds to S506.

[0113] In S506, the system control unit 50 determines whether or not a main subject was found in the subject determination process of S503 or S505. If it is determined that a main subject was found, the process proceeds to S508; otherwise, the process proceeds to S507.

[0114] In S507, the system control unit 50 performs main subject determination processing A, and then ends this flow.

[0115] In S508, the main subject determined in S503 or S505 is confirmed as the main subject, and then this flow ends.

[0116] <Problems to be solved by this embodiment> 5, it is desirable to determine the main subject so that the user can capture the decisive moment in the main subject determination process based on soccer action in S503 and the main subject determination process based on volleyball action in S505. Therefore, the main subject to be focused on is determined depending on whether the subject is taking a pose for an action that will be an interesting highlight, such as a shot in a soccer game or an attack in a volleyball game.

[0117] However, there are various postures that a goalkeeper in a soccer game takes when saving the ball, and there are cases where the main subject cannot be determined based on whether or not the subject is in a particular posture.

[0118] In this embodiment, subjects are classified into categories, and the method of determining whether or not a subject is suitable as a main subject is changed depending on the category, making it possible to select a subject that is suitable for the scene as the main subject.

[0119] <Main subject determination process according to soccer action> FIG. 6 is a flowchart showing an example of the operation of the main subject determination process according to a soccer action.

[0120] In S601, the system control unit 50 determines whether or not a goalkeeper is detected within the angle of view.

[0121] In S601, whether or not a goalkeeper has been detected can be determined by, for example, having the user register a person, and if the characteristics match those of the registered person, the person is deemed to be a goalkeeper.

[0122] For example, the user sets the person and the corresponding position on a setting screen for registering a person as shown in Fig. 7. In the example of screen 701 shown in Fig. 7, it is indicated that person 706 is a goalkeeper.

[0123] Another example of determining whether a goalkeeper has been detected in S601 is to identify subjects during filming and determine the role, including goalkeeper, of each identified subject based on their actions over a certain period of time going back in time.

[0124] An example of a method for determining the role from the state of the subject identified during shooting in S601 will be described with reference to the flowchart of FIG.

[0125] In S801, it is determined whether the range of movement of the identified subject during a certain period going back in time is equal to or greater than half of the court.

[0126] The movement range can be determined, for example, by recording the position history of the identified subject over a certain period going back in time, and determining the area that encompasses all of the position history within the soccer court as the movement range.

[0127] In S801, the system control unit 50 determines whether the range of movement of the subject in the past is equal to or greater than half of the court. If it is determined that the range of movement is equal to or greater than half of the court, the process proceeds to S802; otherwise, the process proceeds to S804.

[0128] In S802, it is determined whether the maximum speed of the identified subject during a certain period going back in time is equal to or greater than a predetermined speed.

[0129] The maximum speed is calculated, for example, by calculating the movement speed from the movement amount of the identified subject at a predetermined time during a certain period of time going back in time, and recording the maximum value of the calculated movement speed. The maximum value of the movement speed during the certain period of time going back in time is set as the maximum speed.

[0130] In S802, if it is determined that the maximum speed of the subject movement is equal to or greater than the predetermined speed, the process proceeds to S803, and if not, the process proceeds to S804.

[0131] In S803, the system control unit 50 sets the subject to be a field player, and ends this flow.

[0132] In S804, the system control unit 50 determines whether or not there is a history of the identified subject contacting a ball during a certain period going back in time. If it is determined that there is a history of contacting a ball, the system control unit 50 proceeds to S805; otherwise, the system control unit 50 proceeds to S806.

[0133] In S805, the system control unit 50 sets the identified subject as a keeper, and ends this flow.

[0134] In S806, the system control unit 50 sets the identified subject as a judge, and ends this flow.

[0135] In the flowchart of FIG. 8, the criterion for judgment is that the past movement range is equal to or greater than half of the court, but other criteria may be used, such as comparing the average amount of past movement with a threshold value.

[0136] In addition, information such as past movement range, maximum movement speed, ball contact history, etc. may be stored in non-volatile memory 31 together with the subject's identification information when the camera is turned off, and information for a certain period of time, such as the past 15 minutes or the past 30 days, may be accumulated and used to determine the subject.

[0137] Returning to the explanation of FIG. 6, if it is determined in S601 that a goalkeeper has been detected within the angle of view, the process proceeds to S602, and if not, the process proceeds to S604.

[0138] In S602, the system control unit 50 determines whether the distance between the goalkeeper and the detected ball is within a predetermined distance, regardless of the posture information of the subject. If it is determined that the distance between the goalkeeper and the detected ball is within the predetermined distance, the system control unit 50 proceeds to S603; if not, the system control unit 50 proceeds to S604.

[0139] The distance between the goalkeeper and the detected ball may be calculated as the distance between the center of gravity of the goalkeeper and the center of gravity of the ball, or the distance between the boundary of the detected position of the goalkeeper and the boundary of the ball.

[0140] In S603, the system control unit 50 sets the goalkeeper as the main subject and ends this flow.

[0141] In S604, the system control unit 50 determines whether or not there is a subject taking action within the angle of view. If it is determined that there is a subject taking action within the angle of view, the process proceeds to S605; otherwise, the process proceeds to S607.

[0142] Whether or not a soccer action is being performed is determined based on whether or not the subject is in a posture suitable for a shooting scene, such as kicking a ball.

[0143] More specifically, the following determination method is available. The image processing circuit 20 detects the joint positions of the subject using a learning model based on deep learning stored in the nonvolatile memory 31. Then, based on the detected relative positions of the joints, if the conditions for a posture for kicking a ball are met, it is determined that the subject is in a posture for kicking a ball. Alternatively, the image processing circuit 20 uses the joint positions stored in the nonvolatile memory 31 as input and a learning model based on deep learning to determine whether or not the subject is in a posture for kicking a ball.

[0144] 9 and 10 show examples of the results of the image processing circuit 20 detecting the joint positions of the subject.

[0145] 9, the joint positions of the subject 91 detected by the image processing circuit 20 are indicated by reference numerals 901 to 913. In addition, in FIG. 10, the joint positions of the subject 101 detected by the image processing circuit 20 are indicated by reference numerals 1001 to 1013.

[0146] The posture in FIG. 9 is determined to be a posture for kicking a ball, while the posture in FIG. 10 is not determined to be a posture for kicking a ball.

[0147] 9 and 10 may be applied to rules to determine whether or not the player is in a posture to kick a ball. Alternatively, the image processing circuit 20 may use the joint positions in Figures 9 and 10 as inputs for deep learning to determine whether or not the player is in a posture to kick a ball using a learning model of deep learning.

[0148] Furthermore, the position of the ball may be used as a rule in addition to the joint positions of the subject, or the position of the ball may be used as an input for deep learning in addition to the joint positions to determine whether or not the subject is in a posture to kick the ball. Note that in the above description, whether or not an action is being performed is determined based on whether or not the subject is in a posture to kick the ball. However, whether or not an action is being performed may also be determined based on a posture that is expected to kick the ball.

[0149] When multiple subjects that are thought to be taking an action are detected, the subject with the most ideal posture for a subject taking an action may be determined to be the subject taking the action.Alternatively, of the multiple detected subjects that are thought to be taking an action, the subject closest to the ball may be determined to be the subject taking the action.Furthermore, the subject taking the action may be determined based on the results of evaluations obtained from each of the posture and the proximity to the ball.

[0150] 6, in S605, the system control unit 50 determines whether the subject taking the action is other than the referee. If it is determined that the subject taking the action is other than the referee, the process proceeds to S606; otherwise, the process proceeds to S607.

[0151] In S606, the system control unit 50 sets the subject taking action as the main subject, and ends this flow.

[0152] In S607, the system control unit 50 determines that there is no main subject, and ends this flow.

[0153] <Main subject determination process according to volleyball action> FIG. 11 is a flowchart showing an example of the operation of the main subject determination process according to a volleyball action.

[0154] In S1101, the system control unit 50 determines whether or not the server who will serve is within the field of view. If it is determined that the server is within the field of view, the system control unit 50 proceeds to S1102, and if not, the system control unit 50 proceeds to S1104.

[0155] Whether or not a subject is a server can be determined by identifying the subject during shooting and judging from the relative position of the identified subject, etc. In addition, the role of each subject, including the server, can also be determined using a similar method.

[0156] Fig. 12 is a flowchart showing an example of a method for determining a role. By applying the process of the flowchart in Fig. 12 to the identified subjects, the role of each identified subject can be determined.

[0157] 12, the system control unit 50 determines whether the identified subject is close to other detected people. If it is determined that the identified subject is close to other detected people, the system control unit 50 proceeds to S1202; if not, the system control unit 50 proceeds to S1205.

[0158] In the case of volleyball servers, because they change frequently during a match, the determination of whether the identified subject is close to other detected people is made in real time or based on information from the past few seconds or tens of seconds.

[0159] In S1202, the system control unit 50 determines whether the identified subject is near the center of the group of detected people. If it is determined that the identified subject is near the center of the group of detected people, the system control unit 50 proceeds to S1203; if not, the system control unit 50 proceeds to S1204.

[0160] Whether an identified subject is near the center of a group of people can be determined, for example, as follows: First, people are detected and the area where people are gathered is determined. If the distance between the center of the area where people are gathered and the identified subject is within a predetermined range, it is determined that the subject is near the center of the group of people.

[0161] The area where the detected people are gathered may be updated in real time using the latest information on the detected people, or may be determined using information on people that has been accumulated over a predetermined period of time, such as the past few seconds or tens of seconds.

[0162] In S1203, the system control unit 50 sets the identified subject as the foreground subject, and ends the flow.

[0163] In S1204, the system control unit 50 sets the identified subject as the rear guard, and ends this flow.

[0164] In S1205, the system control unit 50 determines whether or not the identified subject has a history of contact with a ball. If it is determined that the subject has a history of contact with a ball, the system control unit 50 proceeds to S1206; otherwise, the system control unit 50 proceeds to S1207.

[0165] The history of whether or not the ball has been contacted may be stored in the nonvolatile memory 31 together with the subject's identification information, and information for a certain period such as the past 15 minutes or the past 30 days may be accumulated and used.

[0166] In S1206, the system control unit 50 sets the identified subject in the server, and ends this flow.

[0167] In S1207, the system control unit 50 sets the identified subject as a judge, and ends this flow.

[0168] In the flowchart of Figure 12, the role of the subject, such as forward or backcourt, is determined based on the position of the identified subject relative to the position of the group of people, but the role of forward or backcourt, server, etc. may also be determined based on the position of the net or court lines.

[0169] Returning to the explanation of FIG. 11, in S1101, if the system control unit 50 determines that there is a server serving within the field of view, the process proceeds to S1102, and if not, the process proceeds to S1104.

[0170] In S1102, the system control unit 50 determines whether or not an action has been detected within a predetermined time for the server within the field of view (information other than information about the posture of the subject). If it is determined that an action has been detected within the predetermined time for the server within the field of view, the system control unit 50 proceeds to S1104; otherwise, the system control unit 50 proceeds to S1103.

[0171] An action in volleyball is a posture for serving or attacking, and if no action from the server is detected within a predetermined time, it is assumed that the server has not yet served.

[0172] In S1103, the system control unit 50 predicts that the server is likely to serve soon after, and therefore ends this flow with the server as the main subject.

[0173] In S1104, the system control unit 50 determines whether or not a rear guard person is present within the angle of view. If it is determined that a rear guard person is present within the angle of view, the process proceeds to S1105, and if not, the process proceeds to S1107.

[0174] In S1105, the system control unit 50 determines whether or not there is a player in the rear guard position near the ball, regardless of the posture of the subject. If it is determined that there is a player in the rear guard position near the ball, the process proceeds to S1106; otherwise, the process proceeds to S1107.

[0175] The determination of whether or not there is a rear-court player near the ball is made, for example, as follows: If the center of gravity of the ball and the rear-court player are within a certain distance, it is determined that there is a rear-court player near the ball. Alternatively, the position of the rear-court player's hand is estimated, and if the boundary of the rear-court player's hand and the boundary of the ball are within a certain distance, it is determined that there is a rear-court player near the ball.

[0176] In S1106, the system control unit 50 sets the person in the back row who is closest to the ball used in the determination in S1105 as the main subject, and ends this flow.

[0177] If there are multiple rear guards near the ball, the person closest to the ball may be selected as the main subject. Alternatively, the distance between the rear guards and the ball and whether the rear guards are in a position that corresponds to the action of receiving the ball may be weighted based on the reliability, and the rear guards may be selected as the main subject based on the weighting.

[0178] In S1107, the system control unit 50 determines whether or not there is a subject performing a volleyball action. If it is determined that there is a subject performing a volleyball action, the process proceeds to S1108; if not, the process proceeds to S1110.

[0179] A volleyball action corresponds to actions such as serving and attacking. Whether or not the subject is taking a posture corresponding to an action such as serving or attacking can be determined by determining whether or not the relative positions of the subject's joints meet the conditions for serving or attacking. Alternatively, the image processing circuit 20 may input the joint positions stored in the nonvolatile memory 31 and use a deep learning model that can determine whether or not the subject is taking a posture corresponding to serving or attacking. Furthermore, the position of the ball may also be used in addition to the joint positions to make the determination.

[0180] In S1108, the system control unit 50 determines whether the subject taking an action is someone other than the referee. If it is determined that the subject taking an action is someone other than the referee, the system control unit 50 proceeds to S1109, and if not, the system control unit 50 proceeds to S1110.

[0181] In S1109, the system control unit 50 sets the subject taking action as the main subject, and ends this flow.

[0182] In S1110, the system control unit 50 determines that there is no main subject and ends this flow.

[0183] <Effects of this embodiment> As described above, in this embodiment, when determining a main subject based on the action of a subject, the subjects are classified into categories such as role, and the determination method for determining the main subject using posture information is changed depending on the category. This makes it possible to set the subject intended by the user as the main subject.

[0184] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0185] For example, in the above embodiment, still image capture by a camera has been described as an example, but the method of determining the main subject according to the action described in FIG. 5 may also be used when waiting for video recording or during video recording.

[0186] In addition, although the soccer action is the posture of kicking the ball, the target posture may be changed, for example, to a posture of heading the ball. Also, in the case of a defender, the target action may be changed depending on the category, for example, to include a posture of receiving the ball in addition to a posture of kicking the ball and a posture of heading the ball.

[0187] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more of the functions.

[0188] The disclosure of this specification includes the following image processing device, method, program, and storage medium.

[0189] (Item 1) a detection means for detecting a subject from an image; a classification means for classifying the subjects detected by the detection means into categories; a determination means for determining whether or not to set the subject as a main subject based on the category of the subject; The image processing device is characterized in that the determining means changes a condition for determining whether or not to set the subject as the main subject depending on the category of the subject.

[0190] (Item 2) 2. The image processing device according to item 1, wherein the category is a role of the subject.

[0191] (Item 3) 3. The image processing device according to item 1 or 2, wherein the condition is a posture of the subject used to determine whether or not to set the subject as the main subject.

[0192] (Item 4) 4. The image processing device according to item 3, wherein the determining means changes the posture used to determine whether or not to set the subject as the main subject depending on the category of the subject.

[0193] (Item 5) 5. The image processing device according to any one of items 1 to 4, wherein the determination means changes whether to refer to information about the posture of the subject in order to determine whether to set the subject as the main subject.

[0194] (Item 6) 6. The image processing device according to item 5, wherein the determination means does not refer to information about the posture of the subject when the subject is not a person in order to determine whether or not to set the subject as the main subject.

[0195] (Item 7) The image processing device described in any one of items 1 to 6, characterized in that the determination means changes whether to refer to information other than information about the posture of the subject in order to determine whether to set the subject as the main subject depending on the category.

[0196] (Item 8) The image processing device according to any one of items 1 to 7, characterized in that the determination means changes the weight of the posture information of the subject used to determine whether or not to set the subject as the main subject depending on the category.

[0197] (Item 9) 9. The image processing device according to any one of items 1 to 8, wherein the determining means determines that the subject belonging to a predetermined category is not the main subject, depending on the category.

[0198] (Item 10) 10. The image processing device according to item 9, wherein the image is a sports image and the predetermined category is a referee of the sports.

[0199] (Item 11) 11. The image processing device according to any one of items 1 to 10, wherein the image is an image of a sport using a ball, and the determination means determines that the subject closest to the ball should be set as the main subject.

[0200] (Item 12) 12. The image processing device according to any one of items 1 to 11, further comprising an imaging means for imaging the subject, wherein the imaging means focuses on the subject set as the main subject.

[0201] (Item 13) 13. The image processing device according to any one of items 1 to 12, further comprising a registration unit for registering the subject and the category of the subject in advance.

[0202] (Item 14) a detection step of detecting a subject from an image; a classification step of classifying the subjects detected in the detection step into categories; a determination step of determining whether or not to set the subject as a main subject based on the category of the subject; In the determining step, a condition for determining whether or not to set the subject as the main subject is changed depending on the category of the subject.

[0203] (Item 15) A program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 13.

[0204] (Item 16) A computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 13.

[0205] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0206] 10: photographing lens, 12: mechanical shutter, 14: image sensor, 16: A / D converter, 18: timing generator, 20: image processing circuit, 22: memory control circuit, 28: image display unit, 30: memory, 50: system control unit, 100: digital camera

Claims

1. a detection means for detecting a subject from an image; a classification means for classifying the subjects detected by the detection means into categories; a determination means for determining whether or not to set the subject as a main subject based on the category of the subject; The image processing device is characterized in that the determining means changes a condition for determining whether or not to set the subject as the main subject depending on the category of the subject.

2. 2. The image processing device according to claim 1, wherein the category is a role of the subject.

3. 2. The image processing device according to claim 1, wherein the condition is a posture of the subject used to determine whether or not to set the subject as the main subject.

4. 4. The image processing apparatus according to claim 3, wherein the determining means changes the posture used to determine whether or not to set the subject as the main subject depending on the category of the subject.

5. 2. The image processing apparatus according to claim 1, wherein the determining means changes whether or not to refer to information about the posture of the subject in order to determine whether or not to set the subject as the main subject.

6. 6. The image processing device according to claim 5, wherein the determining means, when the subject is not a person, does not refer to information about the posture of the subject in order to determine whether or not to set the subject as the main subject.

7. The image processing device according to claim 1, wherein the determining means changes whether to refer to information other than information about the posture of the subject in order to determine whether to set the subject as the main subject depending on the category.

8. 2. The image processing device according to claim 1, wherein the determining means changes a weight of information on the posture of the subject, which is used to determine whether or not to set the subject as the main subject, depending on the category.

9. 2. The image processing apparatus according to claim 1, wherein the determining means determines that the subject belonging to a predetermined category is not the main subject, depending on the category.

10. 10. The image processing device according to claim 9, wherein the image is a sports image, and the predetermined category is a referee of the sports.

11. 2. The image processing device according to claim 1, wherein the image is an image of a sport using a ball, and the determining means determines that the subject closest to the ball should be set as the main subject.

12. 2. The image processing apparatus according to claim 1, further comprising an image capturing means for capturing an image of the subject, wherein the image capturing means focuses on the subject set as the main subject.

13. 2. The image processing apparatus according to claim 1, further comprising a registration unit for registering the subject and the category of the subject in advance.

14. a detection step of detecting a subject from an image; a classification step of classifying the subjects detected in the detection step into categories; a determination step of determining whether or not to set the subject as a main subject based on the category of the subject; In the determining step, a condition for determining whether or not to set the subject as the main subject is changed depending on the category of the subject.

15. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 13.

16. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 13.

Images