Processor, processing method, and processing program

Abstract

To provide a processor, a processing method, and a processing program for improving a frequency of detecting a specific subject included in image data to make it possible to perform highly frequent processing depending on the detected specific subject.SOLUTION: A system control section includes a processor that performs: subject detection processing of detecting a first area of a detection target from image data acquired from an imaging element; tracking processing of detecting a second area based on the first area from the image data acquired from the imaging element; first processing using the first area detected by the subject detection processing; and second processing using the second area detected by the tracking processing, and further performs: the subject detection processing sequentially; and the tracking processing in each period between detection of the first area by the subject detection processing started at first timing and detection of the first area by the subject detection processing started next to the first timing.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a processing device, a processing method, and a processing program. [Background technology]

[0002] Patent document 1 describes an imaging device that includes a subject detection unit that uses image data that has been subjected to resolution conversion processing to detect a subject included in an image based on that image data in an initial frame at which a subject tracking operation begins, and a subject tracking unit that detects a subject from a frame following the initial frame by performing a template matching process using the subject image data generated by the subject detection unit in the initial frame and the image data that has been subjected to resolution conversion processing.

[0003] Patent Document 2 describes an image processing device that detects a subject to be tracked using different tracking methods depending on whether an input image is flat or not.

[0004] Patent document 3 describes an imaging device that includes an image dividing unit that divides image data of a subject stored in a memory into multiple regions, a region identifying unit that identifies corresponding regions corresponding to each of the multiple divided regions from image frames that are sequentially generated by an imaging unit based on feature information of each of the multiple divided regions, and a subject tracking unit that tracks an image region in the image frame that has the corresponding region as a tracking target based on the identified corresponding region.

[0005] Patent document 4 describes an image tracking device that tracks an object by repeatedly detecting a pattern corresponding to a specific object from multiple images obtained repeatedly, uses at least one of the color and brightness of the object as reference information, tracks the image of the object by repeatedly detecting the image of the object from multiple images based on this reference information, and changes control to tracking the image of the object if tracking of the object becomes impossible after starting to track the object.

[0006] Patent document 5 describes an optical device having a detection means for detecting an image area corresponding to the subject in each of a plurality of images generated sequentially by photoelectrically converting an optical image of the subject formed by an imaging optical system, and a control means for performing focus tracking control so as to maintain the in-focus state of the imaging optical system on the subject based on the image area, wherein the control means transitions from the focus tracking control on the first subject to the focus tracking control on the second subject in response to the subject changing from a first subject to a second subject different from the first subject in the image area, and performs the focus tracking control on the second subject only when the defocus state of the imaging optical system with respect to the first subject satisfies a predetermined condition. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2015-69064 [Patent Document 2] Japanese Patent Application Publication No. 2011-150594 [Patent Document 3] Japanese Patent Application Publication No. 2011-258180 [Patent Document 4] Japanese Patent Application Publication No. 2014-95907 [Patent Document 5] Japanese Patent Application Publication No. 2017-26914 Summary of the Invention [Problem to be solved by the invention]

[0008] An object of the present invention is to provide a processing device, processing method, and processing program that improve the detection frequency of specific subjects contained in image data and enable processing according to the detected specific subjects to be performed at a high frequency. [Means for solving the problem]

[0009] A processing device of one embodiment of the present invention is a processing device having a processor and a memory, wherein the processor performs a first detection process to detect a first area of ​​a detection object from image data acquired from an imaging element, a second detection process to detect a second area based on the first area from the image data acquired from the imaging element, a first process using the first area detected in the first detection process, and a second process using the second area detected in the second detection process, and further continues to perform the first detection process, and performs the second detection process in each period from when the first area is detected by the first detection process started at a first timing to when the first area is detected by the first detection process started next after the first timing, wherein the first process is a process to output an image based on the first area detected by the first detection process and the first and second areas detected before the first detection process, and the second process is a process to output an image based on the second area detected by the second detection process and the first and second areas detected before the second detection process.

[0010] A processing device of one embodiment of the present invention is a processing device having a processor and a memory, wherein the processor performs a first detection process to detect a first area of ​​a detection target object from image data generated by digital signal processing of an image signal output from an imaging element, a second detection process to detect a second area similar to an image of the first area as a reference image from image data of a captured image signal output from the imaging element after the image signal used in the first detection process, and starts the second detection process during the period from the start of the first detection process started at a first timing to the completion of detection of the first area by the first detection process started next after the first timing.

[0011] A processing method of one embodiment of the present invention performs a first detection process to detect a first area of ​​a detection object from image data acquired from an imaging element, a second detection process to detect a second area based on the first area from image data acquired from the imaging element, a first process using the first area detected in the first detection process, and a second process using the second area detected in the second detection process, and further performs the first detection process continuously, performing the second detection process and the second process during each period from when the first area is detected by the first detection process started at a first timing to when the first area is detected by the first detection process started next after the first timing, wherein the first process is a process to output an image based on the first area detected by the first detection process and the first area and the second area detected before the first detection process, and the second process is a process to output an image based on the second area detected by the second detection process and the first area and the second area detected before the second detection process.

[0012] A processing program of one embodiment of the present invention causes a processor to execute a first detection process to detect a first area of ​​a detection object from image data acquired from an imaging element, a second detection process to detect a second area based on the first area from image data acquired from the imaging element, a first process using the first area detected in the first detection process, and a second process using the second area detected in the second detection process, and further causes the processor to continuously perform the first detection process, and to perform the second detection process and the second process in each period from when the first area is detected by the first detection process started at a first timing to when the first area is detected by the first detection process started next after the first timing, wherein the first process is a process to output an image based on the first area detected by the first detection process and the first and second areas detected before the first detection process, and the second process is a process to output an image based on the second area detected by the second detection process and the first and second areas detected before the second detection process. [Effects of the Invention]

[0013] According to the present invention, it is possible to improve the frequency of detecting a specific subject included in image data, and to perform processing according to the detected specific subject with high frequency. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a diagram showing a schematic configuration of a digital camera 100 that is an embodiment of a processing device of the present invention. [Figure 2] 4 is a timing chart for explaining a first example of the operation of the digital camera 100 shown in FIG. 1 during display of a live view image. [Figure 3] 10 is a timing chart for explaining the operation of the digital camera of the reference example while displaying a live view image. [Figure 4] 10 is a timing chart for explaining a second example of the operation of the digital camera 100 shown in FIG. 1 during display of a live view image. [Figure 5] 10 is a timing chart for explaining a third example of the operation of the digital camera 100 shown in FIG. 1 during display of a live view image. [Figure 6] 10 is a timing chart for explaining a fourth example of the operation of the digital camera 100 shown in FIG. 1 during display of a live view image. [Figure 7] 2 shows the appearance of a smartphone 200 which is another embodiment of the processing device of the present invention. [Figure 8] FIG. 8 is a block diagram showing the configuration of the smartphone 200 shown in FIG. 7. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0016] FIG. 1 is a diagram showing a schematic configuration of a digital camera 100 that is an embodiment of a processing device of the present invention.

[0017] The digital camera 100 shown in FIG. 1 includes a lens device 40 having an imaging lens 1, an aperture 2, a lens control unit 4, a lens driving unit 8, and an aperture driving unit 9.

[0018] The lens device 40 may be detachable from the body of the digital camera 100, or may be integrated with the body of the digital camera 100.

[0019] The imaging lens 1 and aperture 2 constitute an imaging optical system, which includes a focus lens that can move in the optical axis direction. This focus lens is a lens for adjusting the focus of the imaging optical system, and is composed of a single lens or multiple lenses. As the focus lens moves in the optical axis direction, the position of the principal point of the focus lens changes along the optical axis direction, changing the focal position on the subject side. Note that the focus lens may be a liquid lens whose principal point position in the optical axis direction can be changed by electrical control.

[0020] The lens control unit 4 of the lens device 40 is configured to be able to communicate with the system control unit 11 of the body of the digital camera 100 via a wired or wireless connection. In accordance with commands from the system control unit 11, the lens control unit 4 drives the focus lens included in the imaging lens 1 via the lens drive unit 8 to change the position of the principal point of the focus lens (focus control), and controls the opening amount of the diaphragm 2 via the diaphragm drive unit 9.

[0021] The main body of the digital camera 100 comprises an image sensor 5 such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor that captures an image of a subject through an imaging optical system, an analog signal processing unit 6 connected to the image sensor 5 and performing analog signal processing such as correlated double sampling, an analog-to-digital conversion circuit 7 that converts the analog signal output from the analog signal processing unit 6 into a digital signal, an image sensor driving unit 10, a system control unit 11 that provides overall control of the entire system, and an operation unit 14.

[0022] The analog signal processing unit 6, the analog-to-digital conversion circuit 7, and the image sensor driving unit 10 are controlled by a system control unit 11. The analog signal processing unit 6 and the analog-to-digital conversion circuit 7 may be built into the image sensor 5 in some cases.

[0023] The image sensor 5 has an imaging surface on which a plurality of pixels are arranged two-dimensionally, and converts a subject image formed on this imaging surface by an imaging optical system into an electrical signal (pixel signal) by the plurality of pixels and outputs the signal.

[0024] The system control unit 11 drives the image sensor 5 via the image sensor drive unit 10, and outputs the subject image captured through the imaging optical system as a captured image signal (a collection of pixel signals). An instruction signal from the user is input to the system control unit 11 via the operation unit 14.

[0025] The system control unit 11 includes a processor, a RAM (Random Access Memory), and a ROM (Read-On Memory) memory such as a flash memory.

[0026] The system control unit 11 realizes each function described below by a processor executing programs, including a processing program stored in a built-in flash memory. Examples of processors include a central processing unit (CPU), which is a general-purpose processor that executes programs to perform various processes; a programmable logic device (PLD), such as a field programmable gate array (FPGA), whose circuit configuration can be changed after manufacture; and a dedicated electrical circuit, such as an application specific integrated circuit (ASIC), which is a processor having a circuit configuration specifically designed to perform specific processes. More specifically, the structure of a processor is an electrical circuit combining circuit elements such as semiconductor devices. The system control unit 11 may be configured with one of the above-mentioned processors, or may be configured with a combination of two or more processors of the same or different types (e.g., a combination of multiple FPGAs or a combination of a CPU and an FPGA).

[0027] The electronic control system of the digital camera 100 further includes a main memory 16, a memory control unit 15 that controls the storage of data in and the reading of data from the main memory 16, a digital signal processing unit 17 that performs digital signal processing on the captured image signal output from the analog-to-digital conversion circuit 7 to generate image data in various formats such as JPEG (Joint Photographic Experts Group), an external memory control unit 20 that controls the storage of data in and the reading of data from the storage medium 21, a display device 23 configured as an organic electroluminescence (EL) display, a liquid crystal display, or the like, and a display control unit 22 that controls the display on the display device 23. The display device 23 is mounted on the rear surface of the body of the digital camera 100, but may also be built into the body of the digital camera 100 and be viewable through an eyepiece window provided in the body of the digital camera 100. The digital camera 100 may be provided with a display device 23 mounted on the rear surface of the body and a display device 23 built into the body.

[0028] The storage medium 21 may be a semiconductor memory such as a flash memory built into the digital camera 100 or a portable semiconductor memory that can be attached to or detached from the digital camera 100, but is not limited to these, and may also be a storage device such as a PC (Personal Computer) connected via wired or wireless communication.

[0029] The memory control unit 15 , digital signal processing unit 17 , external memory control unit 20 , and display control unit 22 are interconnected by a control bus 24 and a data bus 25 , and are controlled by commands from the system control unit 11 .

[0030] Digital camera 100 has a still image mode in which still images are stored in storage medium 21, and a video mode in which video images are stored in storage medium 21. In either mode, digital camera 100 displays a live view image of the subject being captured on display device 23. When displaying a live view image on display device 23, digital camera 100 detects the area of ​​the main subject from image data generated by digital signal processing unit 17 and performs control to display a detection frame image (e.g., a rectangular frame image) indicating the detected area of ​​the main subject superimposed on the live view image. Digital camera 100 also performs focus control, exposure control, and white balance control based on the detected area of ​​the main subject. The main subject may be a human face, an animal face, a vehicle, or the like.

[0031] The main subject detection process is performed by the system control unit 11. As the main subject detection process, the system control unit 11 selectively performs a subject detection process that constitutes a first detection process and a tracking process that constitutes a second detection process.

[0032] The subject detection process is a process of detecting an area of ​​a detection target (first area) as an area of ​​a main subject from image data (hereinafter referred to as detection image data) obtained by digital signal processing output from the image sensor 5, based on an image recognition algorithm generated by machine learning, etc. The detection target is the same as the main subject described above, such as a human face, an animal face, or a vehicle.

[0033] The tracking process is a process of detecting a second area based on the first area detected by the subject detection process from the detection image data as the area of ​​the main subject. Specifically, the system control unit 11 stores the image of the first area detected by the subject detection process in memory as a reference image, searches for a similar area that is similar to this reference image from the detection image data, and detects the found similar area as the second area.

[0034] When a live view image is displayed, the system control unit 11 continues to perform subject detection processing, and performs tracking processing during each period from when a first area is detected by the subject detection processing started at a first timing until when the first area is detected by the subject detection processing started next after this first timing.

[0035] The frequency of performing the subject detection process is determined based on the time required for the subject detection process (the time from when the subject detection process starts until the detection result is obtained). The time required for the subject detection process may vary depending on conditions such as the content of the detection image data and the processing load of the system control unit 11, but a representative value (such as a median or average value) is determined. In this specification, this representative value is defined as the time required for the subject detection process. In the digital camera 100, the time required for the subject detection process is longer than the interval between images captured by the image sensor 5 when displaying a live view image (the period of the vertical synchronization signal (hereinafter referred to as the frame period)). Therefore, the period of the subject detection process is set to at least twice the frame period. In other words, the subject detection process is performed once for every multiple images captured.

[0036] It should be noted that the tracking process can obtain processing results faster than the subject detection process. In other words, the time required for the tracking process (the time from when the tracking process starts until the detection result is obtained) is shorter than the time required for the subject detection process. The time required for the tracking process may vary depending on conditions such as the processing load of the system control unit 11, but a representative value (such as a median or average value) is fixed. In this specification, this representative value is defined as the time required for the tracking process.

[0037] The operation of digital camera 100 when displaying a live view image will be described below with reference to a timing chart. Fig. 2 is a timing chart for explaining a first example of the operation of digital camera 100 shown in Fig. 1 when displaying a live view image.

[0038] 2, the period marked "F# exposure / capture (# is 1 to 7)" indicates the period during which the #th frame of the image sensor 5 is exposed and the pixel signals generated by this exposure are output from the image sensor 5 and captured by the digital signal processor 17. The length of this period corresponds to the frame interval.

[0039] In the "Display Device" chart in Figure 2, the period marked "F# Display" indicates the period during which the live view image obtained by processing the pixel signals output from the image sensor 5 during the "F# Exposure / Capture" period is displayed on the display device 23.

[0040] In the chart of "Generation of detection image data" in Figure 2, the period marked "f#" indicates the period during which the captured image signal output from the image sensor 5 is processed to generate detection image data f# during the "F# exposure / capture" period.

[0041] Exposure for the first frame begins, and when this exposure ends and readout of pixel signals begins, live view image data is sequentially generated by the digital signal processing unit 17, and the generated live view image data is displayed on the display device 23. When readout of all pixel signals by this exposure is complete, detection image data f1 is generated by the digital signal processing unit 17, and this detection image data f1 is acquired by the system control unit 11. This operation is repeated for the second frame and thereafter.

[0042] When the system control unit 11 acquires the detection image data f1, it starts the subject detection process to detect the detection target from the detection image data f1, as shown in period T11. As described above, the time required for this subject detection process is longer than the frame interval. Therefore, the system control unit 11 acquires the next detection image data f2 before the end of period T11.

[0043] When the area of ​​the detection object is detected from the detection image data f1, the system control unit 11 generates a detection frame image PD(f1) indicating that area in period T12, and then, in period T13, performs processing to display the generated detection frame image PD(f1) on the display device 23. Through this processing, the detection frame image PD(f1) is displayed superimposed on the live view image generated by the imaging processing for the fourth frame.

[0044] Furthermore, when the system control unit 11 detects an area of ​​the detection object from the detection image data f1, it stores an image Dr(f1) of that area in memory as a reference image during period T14. If a previous reference image is stored in memory, the system control unit 11 overwrites this previous reference image with the image Dr(f1) to update the reference image. Next, during period T21, the system control unit 11 performs a tracking process to search for an area similar to this reference image from the detection image data f2. Although not shown in the figure, after the end of period T11, the system control unit 11 performs focus control, exposure control, and white balance control based on the image Dr(f1) of the area of ​​the detection object detected from the detection image data f1.

[0045] When the system control unit 11 detects a similar area that is similar to the reference image from the detection image data f2 during period T21, it overwrites the image of that similar area in memory as the reference image, and then generates a detection frame image PT(f2) that indicates that similar area during period T22. Next, during period T23, the system control unit 11 performs processing to display the generated detection frame image PT(f2) on the display device 23. Through this processing, the detection frame image PT(f2) is displayed superimposed on the live view image generated by the imaging processing for the fifth frame.

[0046] Thereafter, the system control unit 11 repeatedly performs the same processing.

[0047] In other words, when the system control unit 11 acquires the detection image data f3, it starts a subject detection process to detect the detection object from the detection image data f3 (period T31), and when it detects the area of ​​the detection object from the detection image data f3, it generates a detection frame image PD(f3) showing that area (period T32), and performs a process to display the generated detection frame image PD(f3) on the display device 23 (period T33).

[0048] Furthermore, when the system control unit 11 detects an area of ​​the detection object from the detection image data f3 during period T31, it stores an image Dr(f3) of that area in memory as a reference image (period T34). Next, the system control unit 11 performs a tracking process to search for a similar area that is similar to this reference image from the detection image data f4 (period T41). When the system control unit 11 detects a similar area that is similar to the reference image from the detection image data f4, it overwrites the image of that similar area in memory as the reference image, and further generates a detection frame image PT(f4) that shows the similar area (period T42), and performs a process to display the generated detection frame image PT(f4) on the display device 23 (period T43).

[0049] When the system control unit 11 acquires the detection image data f5, it starts a subject detection process to detect the detection object from the detection image data f5 (period T51), and when it detects the area of ​​the detection object from the detection image data f5, it generates a detection frame image PD(f5) showing that area (period T52), and performs a process to display the generated detection frame image PD(f5) on the display device 23 (period T53). Furthermore, when the system control unit 11 detects the area of ​​the detection object from the detection image data f5, it stores an image Dr(f5) of that area in memory as a reference image (period T54).

[0050] Through the above operations, the detection frame image that indicates the position of the main subject and is superimposed on the live view image is updated for each display frame after the fourth frame of the live view image is displayed.

[0051] 3 is a timing chart for explaining the operation of a digital camera of a reference example. The configuration of the digital camera of this reference example is the same as that of digital camera 100, and only the function of system control unit 11 is different. System control unit 11 of the digital camera of the reference example does not perform tracking processing as a main subject detection process, but only performs subject detection processing. FIG. 3 is the same as FIG. 2 except that period T14, period T21 to period T23, period T34, period T41 to period T43, and period T54 have been deleted.

[0052] In the operation of the digital camera in the reference example, the time required for the subject detection process is longer than the frame interval, so the detection frame image is updated once every two frames of display. Furthermore, it is not possible to detect the main subject from the detection image data f2, f4, and f6, making it difficult to respond quickly to changes in the subject.

[0053] In contrast, digital camera 100 performs tracking processing between successive subject detection processes (the period from the end of period T11 to the end of period T31, and the period from the end of period T31 to the end of period T51), and updates the detection frame image using the results of this tracking processing. This allows for frequent updating of the detection frame image and focus control based on the main subject, thereby improving the quality of the live view image and the imaging quality. Furthermore, because the main subject can also be detected from detection image data f2, f4, and f6, it is possible to respond quickly to changes in the subject, thereby improving imaging quality.

[0054] Figure 4 is a timing chart for explaining a second example of the operation of digital camera 100 shown in Figure 1 while displaying a live view image. Figure 4 differs from Figure 2 in that the subject detection process performed in period T31 takes time, making period T31 longer. Also, a second difference from Figure 2 is that subject detection process is not performed on detection image data f5, and after the subject detection process in period T31, subject detection process is performed on detection image data f6 in period T61.

[0055] 4, because the reference image is updated in period T21, it is possible to perform tracking processing on detection image data f4 using this reference image and update the detection frame image based on the results of that tracking processing (updating the detection frame image during the "F6 display" period) before the end of period T31. However, digital camera 100 does not start the next tracking processing until the end of period T31. This prevents a detection frame image generated based on the main subject detected from detection image data f4 from being displayed on display device 23 before a detection frame image generated based on the main subject detected from detection image data f3.

[0056] After the end of period T31, the reference image is updated in period T34, and then, in the subsequent period T41, tracking processing is performed to search for a similar area similar to the reference image from the detection image data f4. A detection frame image based on the results of this tracking processing is then superimposed on the live view image of the eighth frame. Thereafter, a detection frame image based on the main subject detected in period T61 is displayed. In this way, even if the subject detection processing takes a long time, by performing tracking processing in the period from the end of one subject detection processing to the end of the next subject detection processing, it is possible to prevent the order of the detection image data that forms the basis of the detection frame images from being out of sync, and improve the quality of the live view image.

[0057] Fig. 5 is a timing chart for explaining a third example of the operation of digital camera 100 shown in Fig. 1 during live view image display. The example of Fig. 4 described above shows a case where period T31 is longer than in Fig. 2, but the example of Fig. 5 shows a case where period T31 is shorter than in Fig. 2.

[0058] 5, the subject detection process performed in period T31 ends before the tracking process performed in period T21 ends. In such a case, system control unit 11 ends (stops) the tracking process performed in period T21 midway. Therefore, processing such as display processing of the detection frame image and focus control based on the results of this tracking process is disabled. This prevents a detection frame image generated based on the main subject detected from detection image data f2 from being displayed on display device 23 after a detection frame image generated based on the main subject detected from detection image data f3.

[0059] Fig. 6 is a timing chart for explaining a fourth example of the operation during live view image display of digital camera 100 shown in Fig. 1. The example of Fig. 6 shows the operation when, in contrast to the operation of Fig. 2, the second and subsequent subject detection processes (the processes in periods T31 and T51 in Fig. 2) cannot be executed due to the processing load of system control unit 11 or the like.

[0060] 6, after the end of period T21, during period T31a, the system control unit 11 searches for a similar area from the detection image data f3 that is similar to the reference image (the image of the main subject detected in the tracking process during period T21), and if a similar area is detected, overwrites the image of that similar area in memory as the reference image. Next, during period T32a, the system control unit 11 generates a detection frame image PT(f3) that indicates that similar area. Next, during period T33a, the system control unit 11 performs processing to display the generated detection frame image PT(f3) on the display device 23. Through this processing, the detection frame image PT(f3) is displayed superimposed on the live view image generated by the imaging process for the sixth frame.

[0061] In period T41a after period T31a, the system control unit 11 searches for a similar area from the detection image data f4 that is similar to the reference image (the image of the main subject detected in the tracking process in period T31a), and if a similar area is detected, overwrites the image of that similar area in memory as the reference image. Next, in period T42a, the system control unit 11 generates a detection frame image PT(f4) that indicates that similar area. Next, in period T43a, the system control unit 11 performs processing to display the generated detection frame image PT(f4) on the display device 23. Through this processing, the detection frame image PT(f4) is displayed superimposed on the live view image generated by the imaging process for the seventh frame.

[0062] 6, after detection frame image PD(f1) based on the results of the subject detection processing is displayed, detection frame images PT(f2), PT(f3), PT(f4), ... based on the results of the tracking processing are displayed in sequence. In this way, even if the subject detection processing does not work for some reason, it is possible to frequently detect the main subject and update the detection frame images, etc.

[0063] 6, when the system control unit 11 cannot perform subject detection processing and performs tracking processing continuously, it is preferable that the system control unit 11 determines the reliability of the area of ​​the main subject detected in each tracking processing, and if the reliability is equal to or greater than a threshold, perform processing using that area (updating the detection frame image, focus control, exposure control, white balance control, etc.). The reliability can be, for example, the similarity with the reference image. This makes it possible to prevent the detection frame image from being superimposed on a subject other than the main subject detected in the subject detection processing.

[0064] In the explanation up to this point, it has been assumed that a detection frame image is generated from the area of ​​the most recent main subject detected by subject detection processing and tracking processing. However, it is preferable to generate a detection frame image using one generated in the past and one generated from the area of ​​the most recent main subject.

[0065] To explain using the example of FIG. 2, for example, in period T22, the system control unit 11 generates a detection frame image by averaging the detection frame image PT(f2) and the detection frame image PD(f1), and in period T23, performs processing to display this detection frame image.

[0066] Furthermore, in period T32, the system control unit 11 generates a detection frame image by averaging the detection frame image PD(f3), the detection frame image PT(f2), and the detection frame image PD(f1), and in period T33, performs processing to display this detection frame image.

[0067] Furthermore, in period T42, the system control unit 11 generates a detection frame image by averaging the detection frame image PT(f4), the detection frame image PD(f3), and the detection frame image PT(f2), and in period T43, performs processing to display this detection frame image.

[0068] In this way, by generating a detection frame image based on information about the area of ​​the main subject detected from multiple past frames of detection image data, it is possible to prevent small fluctuations in the display position of the detection frame image and improve the quality of the live view image. Compared to generating a detection frame image based solely on the results of the subject detection processing, or generating a detection frame image based solely on the results of the tracking processing, generating a detection frame image by handling the results of the subject detection processing and the results of the tracking processing in an integrated manner can simplify the processing.

[0069] So far, we have described an example in which subject detection processing and tracking processing are performed in combination when displaying a live view image. However, for example, when playing back video data stored in storage medium 21 on display device 23, by similarly performing subject detection processing and tracking processing in combination, it is possible to update the detection frame image and the white balance frequently. Furthermore, by similarly performing subject detection processing and tracking processing in combination not only when displaying a live view image but also when capturing video, it is possible to optimize the capturing conditions according to the main subject.

[0070] Next, the configuration of a smartphone as another embodiment of the processing device of the present invention will be described.

[0071] Fig. 7 shows the external appearance of smartphone 200. Smartphone 200 shown in Fig. 7 has a flat housing 201, and is provided on one surface of housing 201 with display panel 202 as a display unit and display input unit 204 which is an integrated unit of operation panel 203 as an input unit.

[0072] Such housing 201 also includes a speaker 205, a microphone 206, an operation unit 207, and a camera unit 208. The configuration of housing 201 is not limited to this, and for example, it is also possible to adopt a configuration in which the display unit and the input unit are independent, or a configuration having a foldable structure or a sliding mechanism.

[0073] FIG. 8 is a block diagram showing the configuration of the smartphone 200 shown in FIG.

[0074] As shown in Figure 8, the main components of the smartphone include a wireless communication unit 210, a display input unit 204, a call unit 211, an operation unit 207, a camera unit 208, a memory unit 212, an external input / output unit 213, a GNSS (Global Navigation Satellite System) receiving unit 214, a motion sensor unit 215, a power supply unit 216, and a main control unit 220.

[0075] The smartphone 200 also has, as its main function, a wireless communication function for performing mobile wireless communication via a base station device BS (not shown) and a mobile communication network NW (not shown).

[0076] The wireless communication unit 210 performs wireless communication with a base station device BS accommodated in the mobile communication network NW in accordance with instructions from the main control unit 220. Using this wireless communication, various file data such as audio data and image data, e-mail data, etc. are sent and received, and web data, streaming data, etc. are received.

[0077] The display input unit 204 is a so-called touch panel that displays images (still images and moving images) or text information, etc. under the control of the main control unit 220 to visually convey information to the user and detects user operations on the displayed information, and is equipped with a display panel 202 and an operation panel 203.

[0078] The display panel 202 uses an LCD (Liquid Crystal Display), an OLED (Organic Electro-Luminescence Display), or the like as a display device.

[0079] The operation panel 203 is placed so that an image displayed on the display surface of the display panel 202 can be seen, and is a device that detects one or more coordinates operated by a user's finger or a stylus. When this device is operated by the user's finger or a stylus, a detection signal generated by the operation is output to the main control unit 220. Next, the main control unit 220 detects the operation position (coordinates) on the display panel 202 based on the received detection signal.

[0080] As shown in FIG. 8, the display panel 202 and operation panel 203 of a smartphone 200, which is exemplified as one embodiment of the imaging device of the present invention, are integrated to form a display input unit 204, and the operation panel 203 is positioned so as to completely cover the display panel 202.

[0081] When such an arrangement is adopted, operation panel 203 may also have a function to detect user operations in areas outside display panel 202. In other words, operation panel 203 may have a detection area for the overlapping portion that overlaps display panel 202 (hereinafter referred to as a display area), and a detection area for the other outer edge portion that does not overlap display panel 202 (hereinafter referred to as a non-display area).

[0082] The size of the display area and the size of the display panel 202 may be completely the same, but they do not necessarily have to be the same. Also, the operation panel 203 may have two sensitive areas: an outer edge portion and an inner portion other than the outer edge portion. Furthermore, the width of the outer edge portion is designed appropriately depending on the size of the housing 201, etc.

[0083] Furthermore, the position detection method used in the operation panel 203 may be a matrix switch method, a resistive film method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, a capacitance method, or the like, and any method may be used.

[0084] The call unit 211 is equipped with a speaker 205 or a microphone 206, and converts the user's voice input through the microphone 206 into voice data that can be processed by the main control unit 220 and outputs it to the main control unit 220, or decodes voice data received by the wireless communication unit 210 or the external input / output unit 213 and outputs it from the speaker 205.

[0085] Also, as shown in FIG. 7, for example, speaker 205 can be mounted on the same surface as display input unit 204, and microphone 206 can be mounted on the side surface of housing 201.

[0086] The operation unit 207 is a hardware key using a key switch or the like, and receives instructions from a user. For example, as shown in Fig. 7, the operation unit 207 is a push-button switch mounted on the side of the housing 201 of the smartphone 200, which turns on when pressed with a finger or the like, and turns off when the finger is released by the restoring force of a spring or the like.

[0087] The storage unit 212 stores the control program and control data of the main control unit 220, application software, address data associated with names or telephone numbers of communication partners, data of sent and received e-mails, web data downloaded by web browsing, downloaded content data, and also temporarily stores streaming data, etc. The storage unit 212 is composed of an internal storage unit 217 built into the smartphone and an external storage unit 218 having a removable external memory slot.

[0088] The internal memory unit 217 and the external memory unit 218 constituting the memory unit 212 are realized using storage media such as a flash memory type, a hard disk type, a multimedia card micro type, a card-type memory (e.g., MicroSD (registered trademark) memory, etc.), a RAM (Random Access Memory), a ROM (Read Only Memory), etc.

[0089] The external input / output unit 213 serves as an interface with all external devices connected to the smartphone 200, and is used to directly or indirectly connect to other external devices via communication (e.g., Universal Serial Bus (USB), IEEE1394, Bluetooth (registered trademark), RFID (Radio Frequency Identification), Infrared Data Association (IrDA) (registered trademark), UWB (Ultra Wideband) (registered trademark), ZigBee (registered trademark), etc.) or a network (e.g., Ethernet (registered trademark), wireless LAN (Local Area Network), etc.).

[0090] Examples of external devices that can be connected to the smartphone 200 include wired / wireless headsets, wired / wireless external chargers, wired / wireless data ports, memory cards connected via card sockets, SIM (Subscriber Identity Module Card) / UIM (User Identity Module Card) cards, external audio / video devices connected via audio / video I / O (Input / Output) terminals, wirelessly connected external audio / video devices, wired / wirelessly connected smartphones, wired / wirelessly connected personal computers, wired / wirelessly connected personal computers, earphones, etc.

[0091] The external input / output unit 213 can transmit data received from such external devices to each component inside the smartphone 200, or transmit data inside the smartphone 200 to external devices.

[0092] The GNSS receiver 214 receives GNSS signals transmitted from GNSS satellites ST1 to STn in accordance with instructions from the main controller 220, executes positioning calculation processing based on the received multiple GNSS signals, and detects a position consisting of the latitude, longitude, and altitude of the smartphone 200. When the GNSS receiver 214 can acquire position information from the wireless communication unit 210 or the external input / output unit 213 (for example, a wireless LAN), it can also detect the position using the position information.

[0093] The motion sensor unit 215 includes, for example, a three-axis acceleration sensor, and detects the physical movement of the smartphone 200 in accordance with instructions from the main control unit 220. By detecting the physical movement of the smartphone 200, the direction of movement or acceleration of the smartphone 200 is detected. The detection result is output to the main control unit 220.

[0094] The power supply unit 216 supplies power stored in a battery (not shown) to each unit of the smartphone 200 in accordance with instructions from the main control unit 220.

[0095] The main control unit 220 includes a microprocessor, operates in accordance with the control program and control data stored in the storage unit 212, and controls all the components of the smartphone 200. The main control unit 220 also includes a mobile communication control function that controls all the components of the communication system to perform voice communication or data communication via the wireless communication unit 210, and an application processing function.

[0096] The application processing function is realized by the main control unit 220 operating in accordance with the application software stored in the storage unit 212. Examples of the application processing function include an infrared communication function that controls the external input / output unit 213 to perform data communication with a partner device, an email function that sends and receives emails, and a web browsing function that views web pages.

[0097] The main control unit 220 also has an image processing function for displaying video on the display input unit 204 based on image data (still image or moving image data) such as received data or downloaded streaming data.

[0098] The image processing function refers to a function in which the main control unit 220 decodes the image data, performs image processing on the decoded result, and displays the image on the display input unit 204.

[0099] Furthermore, the main control unit 220 executes display control for the display panel 202 and operation detection control for detecting user operations via the operation unit 207 and the operation panel 203 .

[0100] By executing display control, the main control unit 220 displays software keys such as icons or scroll bars for starting application software, or displays a window for creating an e-mail.

[0101] The scroll bar refers to a software key for receiving an instruction to move the displayed portion of an image, such as a large image that cannot fit in the display area of ​​the display panel 202.

[0102] In addition, by executing operation detection control, the main control unit 220 detects user operations through the operation unit 207, accepts operations on the above icons and input of character strings into the input field of the above window through the operation panel 203, or accepts requests to scroll the displayed image through the scroll bar.

[0103] Furthermore, by executing operation detection control, the main control unit 220 determines whether the operation position on the operation panel 203 is an overlapping portion (display area) that overlaps the display panel 202 or an outer edge portion (non-display area) that does not overlap the display panel 202, and is equipped with a touch panel control function that controls the sensitive area of ​​the operation panel 203 or the display position of the software key.

[0104] Furthermore, the main control unit 220 can also detect a gesture operation on the operation panel 203 and execute a preset function in response to the detected gesture operation.

[0105] Gesture operation is not a simple touch operation as in the past, but rather an operation in which a trajectory is drawn with a finger or the like, multiple positions are specified simultaneously, or a combination of these is used to draw a trajectory for at least one of multiple positions.

[0106] The camera unit 208 includes components other than the external memory control unit 20, storage medium 21, display control unit 22, display device 23, and operation unit 14 in the digital camera shown in FIG.

[0107] The captured image data generated by the camera unit 208 can be stored in the storage unit 212 or output via the external input / output unit 213 or the wireless communication unit 210 .

[0108] In the smartphone 200 shown in FIG. 7, the camera unit 208 is mounted on the same surface as the display input unit 204, but the mounting position of the camera unit 208 is not limited to this, and it may be mounted on the back surface of the display input unit 204.

[0109] Furthermore, the camera unit 208 can be used for various functions of the smartphone 200. For example, an image acquired by the camera unit 208 can be displayed on the display panel 202, or an image from the camera unit 208 can be used as one of the operation inputs for the operation panel 203.

[0110] Furthermore, when the GNSS receiver 214 detects a position, it can also detect the position by referring to an image from the camera unit 208. Furthermore, it can also refer to an image from the camera unit 208 to determine the optical axis direction of the camera unit 208 of the smartphone 200 or determine the current usage environment, without using a triaxial acceleration sensor or by using the image in combination with a triaxial acceleration sensor. Of course, the image from the camera unit 208 can also be used in application software.

[0111] In addition, image data of still or video images can be added with location information acquired by the GNSS receiving unit 214, audio information acquired by the microphone 206 (which may be converted to text information by the main control unit, etc.), posture information acquired by the motion sensor unit 215, etc., and stored in the memory unit 212, or output via the external input / output unit 213 or wireless communication unit 210.

[0112] The smartphone 200 configured as above can also detect the main subject with high frequency.

[0113] The embodiment of the processing device is not limited to an imaging device such as a digital camera or a smartphone, but may be any device having a processor capable of executing the processing program executed by the system control unit 11. For example, a processor of a personal computer connected to a surveillance camera via a network may acquire frames of video data captured by the surveillance camera, execute main subject detection processing that combines subject detection processing and tracking processing, and perform processing according to the detection results (for example, analysis and storage of the main subject area).

[0114] As explained above, this specification describes at least the following items. Note that the elements in parentheses correspond to those in the above-described embodiments, but are not limited to these.

[0115] (1) A processing device (digital camera 100) having a processor (processor of system control unit 11) and a memory (memory of system control unit 11), The processor is a first detection process (subject detection process) for detecting a first area of ​​the detection target object from image data (detection image data) acquired from the image sensor (image sensor 5); a second detection process (tracking process) for detecting a second area based on the first area from image data acquired from the imaging element; a first process (detection frame image display process, focus control, exposure control, white balance control) using the first area detected in the first detection process; performing second processing (detection frame image display processing, focus control, exposure control, white balance control) using the second area detected in the second detection processing; Furthermore, the processing device continues to perform the first detection process, and performs the second detection process during each period from when the first area is detected by the first detection process started at the first timing until when the first area is detected by the first detection process started next after the first timing.

[0116] (2) The processing device according to (1), The processor is a processing device that, after the first detection process is completed, performs the second detection process on image data acquired after the image data used in the first detection process.

[0117] (3) The processing device according to (1) or (2), The processor controls the number of times the second process is performed between two consecutive times of the first process, based on the end timing of the first detection process.

[0118] (4) (3) The processing device according to (3), The processor is a processing device that deactivates the second processing based on the results of the second detection processing if the end timing of the first detection processing is earlier than the end timing of the second detection processing for image data acquired before the image data used in the first detection processing.

[0119] (5) The processing device according to any one of (1) to (4), The processor is a processing device that, if the first detection process is deactivated after the first detection process is performed, continuously performs the second detection process.

[0120] (6) (5) The processing device according to the present invention, When the processor performs the second detection process continuously, if the reliability of the second area detected by the second detection process is greater than or equal to a threshold, the processor performs the second processing using the second area.

[0121] (7) The processing device according to any one of (1) to (6), the first processing is processing for outputting an image (detection frame image) based on the first area detected by the first detection processing and the first area and the second area detected before the first detection processing, The second processing is a processing device that outputs an image (detection frame image) based on the second area detected by the second detection processing and the first area and the second area detected before the second detection processing.

[0122] (8) (7) The processing device according to (7), A processing device in which the first process and the second process are processes for displaying the image on a display device.

[0123] (9) The processing device according to any one of (1) to (8), The processing device wherein the frequency of performing the first detection process is determined based on the time required for the first detection process.

[0124] (10) The processing device according to any one of (1) to (9), A processing device in which the time required for the first detection process is longer than the time required for the second detection process.

[0125] (11) a first detection process for detecting a first area of ​​the detection object from image data acquired from an imaging element; a second detection process for detecting a second area based on the first area from the image data acquired from the imaging element; a first process using the first area detected in the first detection process; and a second process using the second area detected in the second detection process, Furthermore, the processing method includes continuously performing the first detection process, and performing the second detection process during each period from when the first area is detected by the first detection process started at a first timing until when the first area is detected by the first detection process started next after the first timing.

[0126] (12) (11) The treatment method according to (11), A processing method in which, after the first detection process is completed, the second detection process is performed on image data acquired after the image data used in the first detection process.

[0127] (13) The treatment method according to (11) or (12), A processing method that controls the number of times the second process is performed between two consecutive times of the first process, based on the end timing of the first detection process.

[0128] (14) (13) The treatment method according to (13), A processing method in which, if the end timing of the first detection process is earlier than the end timing of the second detection process for image data acquired before the image data used in the first detection process, the second process based on the result of the second detection process is deactivated.

[0129] (15) The treatment method according to any one of (11) to (14), A processing method in which, if the first detection process is deactivated after the first detection process is performed, the second detection process is performed continuously.

[0130] (16) (15) The treatment method according to (15), A processing method in which, when the second detection process is performed continuously, the second process is performed using the second area if the reliability of the second area detected by the second detection process is above a threshold.

[0131] (17) The treatment method according to any one of (11) to (16), the first processing is processing for outputting an image based on the first area detected by the first detection processing and the first area and the second area detected before the first detection processing, The second processing is a processing method in which an image is output based on the second area detected by the second detection processing and the first area and the second area detected before the second detection processing.

[0132] (18) (17) The treatment method according to (17), The first process and the second process are processing methods in which the image is displayed on a display device.

[0133] (19) The treatment method according to any one of (11) to (18), A processing method in which the frequency of performing the first detection process is determined based on the time required for the first detection process.

[0134] (20) The treatment method according to any one of (11) to (19), A processing method in which the time required for the first detection process is longer than the time required for the second detection process.

[0135] (twenty one) a processor is caused to execute a first detection process for detecting a first area of ​​a detection object from image data acquired from an imaging element, a second detection process for detecting a second area based on the first area from the image data acquired from the imaging element, a first process using the first area detected in the first detection process, and a second process using the second area detected in the second detection process; Further, the processing program causes the processor to continue performing the first detection process, and to perform the second detection process during each period from when the first area is detected by the first detection process started at a first timing until when the first area is detected by the first detection process started next after the first timing.

[0136] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present invention is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present invention. Furthermore, the components of the above-described embodiments may be combined in any manner without departing from the spirit of the invention.

[0137] This application is based on a Japanese patent application (Patent Application No. 2021-005137) filed on January 15, 2021, the contents of which are incorporated by reference into this application. [Explanation of symbols]

[0138] 1 Imaging lens 2 apertures 4 Lens control unit 5. Image sensor 6 Analog signal processing section 7 Analog-to-digital conversion circuit 8 Lens drive unit 9 Aperture drive unit 10. Image sensor driver 11 System control section 14 Control section 15 Memory control unit 16 Main Memory 17 Digital Signal Processing Unit 20 External memory control unit 21 Storage medium 22 Display control unit 23 Display device 24 control bus 25 Data Bus 40 Lens device T11, T12, T13, T14 periods T21, T22, T23 periods T31, T32, T33, T34 period T31a, T32a, T33a period T41, T42, T43 period T41a, T42a, T43a period T51, T52, T53, T54 period T61 period 200 smartphones 201 Case 202 Display Panel 203 Operation Panel 204 Display and input section 205 Speaker 206 Microphone 207 Operation section 208 Camera Club 210 Radio Communication Department 211 Telephone section 212 Storage section 213 External input / output section 214 GNSS receiver 215 Motion sensor unit 216 Power supply section 217 Internal storage 218 External Memory Unit 220 Main control unit ST1~STn GNSS satellite

Claims

1. A processing device having a processor and a memory, The processor: a first detection process for detecting a first area of ​​the detection object from image data acquired from the imaging element; a second detection process for detecting a second area based on the first area from the image data acquired from the imaging element; a first process using the first area detected in the first detection process; a second process using the second area detected in the second detection process; Furthermore, the first detection process is continuously performed, and the second detection process is performed during each period from when the first area is detected by the first detection process started at a first timing until when the first area is detected by the first detection process started next after the first timing, the first processing is processing for outputting an image based on the first area detected by the first detection processing and the first area and the second area detected before the first detection processing, The second processing is a processing device that outputs an image based on the second area detected by the second detection processing and the first area and the second area detected before the second detection processing.

2. 2. The processing device according to claim 1, The processor controls the number of times the second process is performed between two consecutive first processes, that is, the first detection process and the first detection process before the first detection process, based on the end timing of the first detection process.

3. 3. The processing apparatus according to claim 1 or 2, The processor is a processing device that continuously performs the second detection process when the first detection process is deactivated after the first detection process is performed.

4. 4. The processing device according to claim 3, When the processor performs the second detection process continuously, the processor performs the second process of outputting the second area to a display device if the reliability of the second area detected by the second detection process is greater than or equal to a threshold.

5. The processing apparatus according to any one of claims 1 to 4, A processing device in which the frequency of performing the first detection process is determined based on the time required for the first detection process.

6. The processing apparatus according to any one of claims 1 to 5, A processing device in which the time required for the first detection process is longer than the time required for the second detection process.

7. a first detection process for detecting a first area of ​​the detection object from image data acquired from an imaging element; a second detection process for detecting a second area based on the first area from the image data acquired from the imaging element; a first process using the first area detected in the first detection process; and a second process using the second area detected in the second detection process, Furthermore, the first detection process is continuously performed, and the second detection process and the second process are performed during each period from when the first area is detected by the first detection process started at the first timing until when the first area is detected by the first detection process started next after the first timing, the first processing is processing for outputting an image based on the first area detected by the first detection processing and the first area and the second area detected before the first detection processing, The second processing is a processing method in which an image is output based on the second area detected by the second detection processing and the first area and the second area detected before the second detection processing.

8. 8. The processing method according to claim 7, A processing method that controls the number of times the second process is performed between two consecutive first processes, i.e., the first detection process and the first detection process before the first detection process, based on the end timing of the first detection process.

9. 9. The processing method according to claim 7 or 8, A processing method in which, if the first detection process is deactivated after the first detection process is performed, the second detection process is performed continuously.

10. 10. The processing method of claim 9, A processing method in which, when the second detection process is performed continuously, the second process is performed using the second area if the reliability of the second area detected by the second detection process is above a threshold.

11. 11. The processing method according to any one of claims 7 to 10, A processing method in which the frequency of performing the first detection process is determined based on the time required for the first detection process.

12. 12. The method of claim 7, further comprising: A processing method in which the time required for the first detection process is longer than the time required for the second detection process.

13. A processing device having a processor and a memory, The processor: a first detection process for detecting a first area of ​​the detection object from image data generated by digitally processing the captured image signal output from the image sensor; a second detection process for detecting a second area similar to the reference image, using an image of the first area as a reference image, from image data of a captured image signal output from the image sensor after the captured image signal used in the first detection process; A processing device that starts the second detection process during the period from when the first detection process starts at a first timing until when detection of the first area is completed by the first detection process that starts next after the first timing.

14. a processor is caused to execute a first detection process for detecting a first area of ​​a detection target from image data acquired from an imaging element, a second detection process for detecting a second area based on the first area from the image data acquired from the imaging element, a first process using the first area detected in the first detection process, and a second process using the second area detected in the second detection process; Furthermore, the processor is caused to continue the first detection process, and to perform the second detection process and the second process during each period from when the first area is detected by the first detection process started at a first timing until when the first area is detected by the first detection process started next after the first timing; the first processing is processing for outputting an image based on the first area detected by the first detection processing and the first area and the second area detected before the first detection processing, The second processing is a processing program that outputs an image based on the second area detected by the second detection processing and the first area and the second area detected before the second detection processing.

Images