Image generation method, processor, and program
By generating a first color image and a second monochrome image through distinct processing, the method enhances subject detection accuracy in monochrome modes, addressing the limitations of existing technologies by leveraging color information for precise identification.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- FUJIFILM CORP
- Filing Date
- 2022-07-15
- Publication Date
- 2026-07-07
AI Technical Summary
Existing image detection technologies face challenges in achieving high detection accuracy, particularly in monochrome or low-color images, due to the reliance on models trained primarily with color images, leading to reduced performance when color information is absent.
The method involves generating a first color image through initial processing, detecting subjects using a trained model, and then creating a second image, such as a monochrome or low-saturation image, through different processing to enhance detection accuracy by leveraging color information even in monochrome modes.
This approach improves subject detection accuracy by utilizing color information in monochrome images, ensuring precise subject identification even when color data is limited, and allows for efficient video capture without reducing frame rates or increasing processing load.
Smart Images
Figure 0007886341000001 
Figure 0007886341000002 
Figure 0007886341000003
Abstract
Description
Technical Field
[0001] The technology of the present disclosure relates to an image generation method, a processor, and a program.
Background Art
[0002] In Japanese Patent Application Laid-Open No. 2020-123174, in an image file generation device that generates an image file having image data and metadata, when creating an inference model with an image related to the image data as an input, information on whether to use the image data as teacher data for external commissioned learning or as confidential reference data is given as metadata, and an image file generation device having a file creation unit is disclosed.
[0003] In Japanese Patent Application Laid-Open No. 2020-166744, first learning request data including an image acquired by a first device and information on a first inference engine of the first device is given, and a first inference model creation unit that creates a first inference model that can be used in the first inference engine of the first device by learning using teacher data based on the image is provided. Second learning request data including information on a second inference engine of a second device is given, and a learning device including a second inference model creation unit that creates a second inference model adapted to the second inference engine of the second device using the first inference model is disclosed.
[0004] In Japanese Patent Application Laid-Open No. 2019-146022, an imaging unit that images a specific range to acquire an image signal, a storage unit that stores a plurality of object image dictionaries corresponding to a plurality of types of objects respectively, and an inference engine that determines the type of a specific object based on the image signal acquired by the imaging unit and the plurality of object image dictionaries stored in the storage unit, and selects an object image dictionary corresponding to the determined type of the specific object from the plurality of object image dictionaries, and an imaging control unit that performs imaging control based on the image signal acquired by the imaging unit and the object image dictionary selected by the inference engine are provided, and an imaging device is disclosed.
Summary of the Invention
Problems to be Solved by the Invention
[0005] One embodiment of the technology described herein provides an image generation method, an imaging device, and a program that enable improved detection accuracy of a subject. [Means for solving the problem]
[0006] To achieve the above objective, the image generation method disclosed herein includes an imaging step of acquiring an imaging signal output from an image sensor; a first generation step of generating a first image using the imaging signal by first image processing; a detection step of detecting a subject in the first image using the first image with a trained model that has undergone machine learning; and a second generation step of generating a second image using the imaging signal by second image processing different from the first image processing.
[0007] The process further includes a receiving step for receiving imaging instructions from the user, and in the second generation step, it is preferable to generate a second image when an imaging instruction is received in the receiving step.
[0008] Preferably, the process further includes a display step in which a live view image is created by changing the first image, and the live view image and the detection results of the subject detected in the detection step are displayed on the display unit.
[0009] The display step preferably involves generating a display signal for the live view image based on the image signals that constitute the first image, thereby displaying the live view image.
[0010] In the second generation step, it is preferable to make the color of the second image substantially identical to the color of the live view image.
[0011] The saturation or brightness of the first image is preferably higher than that of the second image and the live view image.
[0012] Preferably, the method further includes a recording step of recording the second image as a still image on a recording medium.
[0013] The first image preferably has a lower resolution than the imaging signal or the second image.
[0014] In the imaging process, an imaging signal is output from the image sensor for each frame period. In the first generation process and the second generation process, a first image and a second image are generated using the imaging signal for the same frame period. Preferably, the first image has a lower resolution than the imaging signal or the second image.
[0015] The second image preferably has a lower resolution than the captured signal.
[0016] In the imaging process, it is preferable that the image sensor outputs an imaging signal for each frame period, the first generation process generates a first image using the imaging signal for the first frame period, and the second generation process generates a second image using the imaging signal for a second frame period that is different from the first frame period.
[0017] The second image is preferably a moving image.
[0018] The saturation or brightness of the first image is preferably higher than that of the second image.
[0019] The trained model is a model that has been machine-learned using color images as training data, and it is preferable that the first image is a color image and the second image is a monochrome or sepia image.
[0020] The processor of this disclosure is a processor that acquires an imaging signal output from an imaging device, and is configured to perform a first generation process that generates a first image by first image processing using the imaging signal, a detection process that detects a subject in the first image using the first image with a trained model that has undergone machine learning, and a second generation process that generates a second image by second image processing different from the first image processing using the imaging signal.
[0021] The program of the present disclosure is a program used in a processor that acquires an imaging signal output from an imaging device. Using the imaging signal, the processor is made to execute a first generation process for generating a first image by first image processing, a detection process for detecting a subject in the first image using a learned model obtained by machine learning, and a second generation process for generating a second image by second image processing different from the first image processing using the imaging signal.
Brief Description of the Drawings
[0022] [Figure 1] It is a diagram showing an example of the internal configuration of an imaging device. [Figure 2] It is a block diagram showing an example of the functional configuration of a processor. [Figure 3] It is a diagram conceptually showing an example of a subject detection process and a display process in monochrome mode. [Figure 4] It is a diagram showing an example of a second image generated by a second image processing unit. [Figure 5] It is a flowchart showing an example of an image generation method by an imaging device. [Figure 6] It is a diagram showing an example of the generation timing of a first image and a second image in video imaging mode. [Figure 7] It is a flowchart showing an example of an image generation method in video imaging mode. [Figure 8] It is a diagram showing an example of the generation timing of a first image and a second image in video imaging mode according to a modification. [Figure 9] It is a flowchart showing an example of an image generation method in video imaging mode according to a modification. [Figure 10] It is a diagram showing an example of the generation timing of a first image and a second image in video imaging mode according to another modification.
Embodiments for Carrying Out the Invention
[0023] An example of an embodiment according to the technology of the present disclosure will be described with reference to the accompanying drawings.
[0024] First, let's explain the terminology used in the following explanation.
[0025] In the following explanation, "IC" is an abbreviation for "Integrated Circuit". "CPU" is an abbreviation for "Central Processing Unit". "ROM" is an abbreviation for "Read Only Memory". "RAM" is an abbreviation for "Random Access Memory". "CMOS" is an abbreviation for "Complementary Metal Oxide Semiconductor".
[0026] "FPGA" is an abbreviation for "Field Programmable Gate Array". "PLD" is an abbreviation for "Programmable Logic Device". "ASIC" is an abbreviation for "Application Specific Integrated Circuit". "OVF" is an abbreviation for "Optical View Finder". "EVF" is an abbreviation for "Electronic View Finder". "JPEG" is an abbreviation for "Joint Photographic Experts Group".
[0027] As one embodiment of the imaging device, the technology of this disclosure will be explained using a lens-interchangeable digital camera as an example. However, the technology of this disclosure is not limited to lens-interchangeable cameras, but can also be applied to lens-integrated digital cameras.
[0028] Figure 1 shows an example of the configuration of the imaging device 10. The imaging device 10 is a lens-interchangeable digital camera. The imaging device 10 consists of a main body 11 and an imaging lens 12 that is interchangeably attached to the main body 11. The imaging lens 12 is attached to the front side of the main body 11 via a camera-side mount 11A and a lens-side mount 12A.
[0029] The main unit 11 is provided with an operating section 13, which includes a dial, a shutter release button, and the like. The operating modes of the imaging device 10 include, for example, a still image capture mode, a video capture mode, and an image display mode. The operating section 13 is operated by the user when setting the operating mode. The operating section 13 is also operated by the user when starting still image capture or video capture.
[0030] Furthermore, the control panel 13 allows users to adjust settings such as image size, image quality mode, recording method, color tone (including film simulation), dynamic range, and white balance. Film simulation is a mode that allows users to set color reproduction and tonal expression as if changing film, according to their shooting intentions. Film simulation offers various modes that reproduce different films, such as vivid, soft, classic chrome, sepia, and monochrome, allowing users to adjust the color tone of the image.
[0031] Furthermore, the main body 11 is equipped with a viewfinder 14. Here, the viewfinder 14 is a hybrid viewfinder (registered trademark). A hybrid viewfinder refers to a viewfinder in which, for example, an optical viewfinder (hereinafter referred to as "OVF") and an electronic viewfinder (hereinafter referred to as "EVF") are selectively used. The user can observe the optical image or live view image of the subject projected by the viewfinder 14 through the viewfinder eyepiece (not shown).
[0032] Furthermore, a display 15 is provided on the back of the main unit 11. The display 15 displays images based on image signals obtained by imaging, as well as various menu screens, etc. The user can also observe the live view image displayed on the display 15 instead of the viewfinder 14. Note that the viewfinder 14 and the display 15 are examples of "display units" related to the technology disclosed herein.
[0033] The main unit 11 and the imaging lens 12 are electrically connected by contact between an electrical contact 11B provided on the camera-side mount 11A and an electrical contact 12B provided on the lens-side mount 12A.
[0034] The imaging lens 12 includes an objective lens 30, a focusing lens 31, a rear end lens 32, and an aperture 33. Each component is arranged along the optical axis A of the imaging lens 12, from the objective side, in the order of objective lens 30, aperture 33, focusing lens 31, and rear end lens 32. teeth This constitutes the imaging optical system. The type, number, and arrangement order of the lenses constituting the imaging optical system are not limited to the example shown in Figure 1.
[0035] Furthermore, the imaging lens 12 has a lens drive control unit 34. The lens drive control unit 34 is composed of, for example, a CPU, RAM, and ROM. The lens drive control unit 34 is electrically connected to the processor 40 in the main unit 11 via electrical contacts 12B and 11B.
[0036] The lens drive control unit 34 drives the focus lens 31 and aperture 33 based on control signals transmitted from the processor 40. The lens drive control unit 34 controls the drive of the focus lens 31 based on focus control control signals transmitted from the processor 40 in order to adjust the focus position of the imaging lens 12. The processor 40 may also perform focus control based on the detection result R detected by subject detection, which will be described later.
[0037] The aperture 33 has an aperture whose diameter is variable around the optical axis A. The lens drive control unit 34 controls the drive of the aperture 33 based on an aperture adjustment control signal transmitted from the processor 40 in order to adjust the amount of light incident on the light-receiving surface 20A of the image sensor 20.
[0038] Furthermore, the main unit 11 houses an image sensor 20, a processor 40, and a memory 42. The image sensor 20, memory 42, operation unit 13, viewfinder 14, and display 15 are all controlled by the processor 40.
[0039] The processor 40 is composed of, for example, a CPU, RAM, and ROM. In this case, the processor 40 performs various processes based on the program 43 stored in the memory 42. The processor 40 may also be composed of a collection of multiple IC chips. The memory 42 also stores a trained model LM that has been machine-learned for object detection.
[0040] The imaging sensor 20 is, for example, a CMOS type image sensor. The imaging sensor 20 is positioned such that the optical axis A is perpendicular to the light-receiving surface 20A and the optical axis A is located at the center of the light-receiving surface 20A. Light (subject image) that has passed through the imaging lens 12 is incident on the light-receiving surface 20A. Multiple pixels are formed on the light-receiving surface 20A, which generate an image signal by performing photoelectric conversion. The imaging sensor 20 generates and outputs an image signal by performing photoelectric conversion on the light incident on each pixel. Note that the imaging sensor 20 is an example of an "image sensor" related to the technology of this disclosure.
[0041] Furthermore, a Bayer-arranged color filter array is positioned on the light-receiving surface of the image sensor 20, with one of the R (red), G (green), or B (blue) color filters positioned opposite each pixel. Some of the multiple pixels arranged on the light-receiving surface of the image sensor 20 may be phase-difference pixels used for focus control.
[0042] Figure 2 shows an example of the functional configuration of the processor 40. The processor 40 implements various functional units by executing processing according to the program 43 stored in the memory 42. As shown in Figure 2, for example, the processor 40 implements a main control unit 50, an imaging control unit 51, a first image processing unit 52, a subject detection unit 53, a display control unit 54, a second image processing unit 55, and an image recording unit 56.
[0043] The main control unit 50 comprehensively controls the operation of the imaging device 10 based on instruction signals input from the operation unit 13. The imaging control unit 51 controls the imaging sensor 20 to perform imaging operations. The imaging control unit 51 drives the imaging sensor 20 in still image mode or video imaging mode. The imaging sensor 20 outputs an imaging signal RD generated by the imaging operation. The imaging signal RD is so-called RAW data.
[0044] The first image processing unit 52 acquires the imaging signal RD output from the imaging sensor 20 and performs a first generation process to generate the first image P1 by applying first image processing, including demosaicing, to the imaging signal RD. For example, the first image P1 is a color image in which each pixel is represented by the three primary colors R, G, and B. More specifically, for example, the first image P1 is a 24-bit color image in which each of the R, G, and B signals contained in one pixel is represented by 8 bits.
[0045] The subject detection unit 53 uses the trained model LM stored in the memory 42 to perform a detection process to detect a subject in the first image P1 generated by the first image processing unit 52. Specifically, the subject detection unit 53 inputs the first image P1 to the trained model LM and obtains the subject detection result R from the trained model LM. The subject detection unit 53 outputs the obtained subject detection result R to the display control unit 54. The subject detection result R is also used by the main control unit 50 to adjust the focus of the imaging lens 12 and the exposure of the subject.
[0046] The subjects detected by the subject detection unit 53 include not only specific objects such as people and cars, but also backgrounds such as the sky and the sea. Furthermore, the subject detection unit 53 may detect specific scenes such as weddings and festivals based on the detected subjects.
[0047] The trained model LM is, for example, composed of a neural network, and has undergone machine learning using multiple images containing a specific subject as training data. The trained model LM detects the region containing the specific subject within the first image P1 and outputs the detection result R. The trained model LM may also output the type of subject along with the region containing the subject.
[0048] The display control unit 54 creates a live view image PL by changing the first image P1, and performs display processing to display the created live view image PL and the detection result R input from the subject detection unit 53 on the display 15. Specifically, the display control unit 54 displays the live view image PL on the display 15 by generating a display signal for the live view image PL based on the image signals that constitute the first image P1.
[0049] The display control unit 54 is, for example, a display driver that adjusts the color of the display 15. The display control unit 54 adjusts the color of the display signal of the live view image PL to be displayed on the display 15 according to the selected mode. For example, if the monochrome mode in film simulation is selected, the display control unit 54 displays a monochrome live view image PL on the display 15 by setting the saturation of the display signal of the live view image PL to zero. For example, if the image signal is expressed in YCbCr format, the display control unit 54 makes the display signal monochrome by setting the color difference signals Cr and Cb to zero. In this disclosure, monochrome means substantially achromatic colors, including grayscale.
[0050] Furthermore, the display control unit 54 is not limited to the display 15, but also displays the live view image PL and the detection result R on the viewfinder 14 in response to the user's operation of the operation unit 13.
[0051] The second image processing unit 55 acquires the imaging signal RD output from the imaging sensor 20 and generates a second image P2 by a second image processing process that includes demosaicing and other processes, and is different from the first image processing process, on the imaging signal RD. 2nd grade Image processing is performed. Specifically, the second image processing unit 55 makes the color of the second image P2 substantially the same as the color of the live view image PL. For example, if the monochrome mode is selected in the film simulation, the second image processing unit 55 generates a grayscale second image P2 through second image processing. For example, the second image P2 is a monochrome image in which the signal of one pixel is represented by 8 bits. Note that the first image P1 and the second image P2 may be imaging signals output at different timings (i.e., different imaging frames).
[0052] The main control unit 50 performs reception processing to receive imaging instructions from the user via the operation unit 13. The second image processing unit 55 performs processing to generate the second image P2 when the main control unit 50 receives an imaging instruction from the user. Imaging instructions include instructions for capturing still images and instructions for capturing video.
[0053] The image recording unit 56 performs recording processing to record the second image P2 generated by the second image processing unit 55 as a recorded image PR in the memory 42. Specifically, when the image recording unit 56 receives a still image capture instruction from the main control unit 50, it records the recorded image PR in the memory 42 as a still image composed of one second image P2. Also, when the image recording unit 56 receives a video capture instruction from the main control unit 50, it records the recorded image PR in the memory 42 as a moving image composed of multiple second images P2. The image recording unit 56 also records the recorded image PR to a recording medium other than the memory 42 (for example, a memory card that can be attached to the main unit 11). Note You may record it.
[0054] Figure 3 conceptually illustrates an example of subject detection and display processing in monochrome mode. As shown in Figure 3, the trained model LM consists of a neural network having an input layer, a hidden layer, and an output layer. Each hidden layer is composed of multiple neurons. The number of hidden layers and the number of neurons in each hidden layer can be changed as appropriate.
[0055] The trained model LM is created by machine learning using color images containing specific subjects as training data, enabling it to detect specific subjects within images. For example, backpropagation is used as a machine learning technique. The trained model LM may also be created by machine learning performed on a computer outside the imaging device 10.
[0056] Since the trained model LM is primarily trained using color images, its accuracy in detecting subjects is low for monochrome images that do not contain color information. Therefore, if a monochrome image generated by image processing is directly input to the trained model LM in monochrome mode, the accuracy of subject detection will decrease. In the technology disclosed herein, the subject detection unit 53 detects subjects by inputting the first image P1, which is a color image generated by the first image processing unit 52, to the trained model LM, even in monochrome mode where the live view image PL and recorded image PR are monochrome.
[0057] For example, as shown in Figure 3, if the subject is a bird and there are trees in the background, in the case of a monochrome image, there is no color information, and the bird blends in with the trees and is difficult to distinguish, resulting in a decrease in detection accuracy by the trained model LM. Even in such cases, inputting a color image into the trained model LM improves the detection accuracy.
[0058] In the example shown in Figure 3, the trained model LM detects the region containing the bird as the subject within the first image P1 and outputs this region information as the detection result R to the display control unit 54. Based on the detection result R, the display control unit 54 displays a frame F in the live view image PL that corresponds to the region containing the detected subject. The display control unit 54 may also display the type of subject near the frame F. Note that the subject detection result R is not limited to the frame F, but may also be the name of the subject or the name of a scene based on the detection results of multiple subjects.
[0059] Figure 4 shows an example of the second image P2 generated by the second image processing unit 55. The color of the second image P2 generated by the second image processing unit 55 is substantially the same as the color of the live view image PL, and is monochrome in monochrome mode.
[0060] [Still image capture mode] Figure 5 is a flowchart showing an example of an image generation method by the imaging device 10. Figure 5 shows an example where still image acquisition mode is selected and the monochrome mode of film simulation is selected.
[0061] The main control unit 50 determines whether or not an instruction to start imaging preparation has been received by the user operating the operation unit 13 (step S10). If an instruction to start imaging preparation has been received (step S10: YES), the main control unit 50 controls the imaging control unit 51 to cause the imaging sensor 20 to perform an imaging operation (step S11).
[0062] The first image processing unit 52 acquires the imaging signal RD output from the imaging sensor 20 when the imaging sensor 20 performs an imaging operation, and generates a first color image P1 by applying first image processing to the imaging signal RD (step S12).
[0063] The subject detection unit 53 detects a subject by inputting the first image P1 generated by the first image processing unit 52 into the trained model LM (step S13). In step S13, the subject detection unit 53 outputs the subject detection result R output from the trained model LM to the display control unit 54.
[0064] The display control unit 54 modifies the first image P1 to create a monochrome live view image PL, and displays the created live view image PL and the detection result R on the display 15 (step S14).
[0065] The main control unit 50 determines whether or not a still image capture instruction has been given by the user operating the operation unit 13 (step S15). If there is no still image capture instruction (step S15: NO), the main control unit 50 returns to step S11 and causes the imaging sensor 20 to perform the imaging operation again. The processes in steps S11 to S14 are repeatedly executed until the main control unit 50 determines in step S15 that a still image capture instruction has been given.
[0066] If the main control unit 50 receives a still image acquisition instruction (step S15: YES), it causes the second image processing unit 55 to generate the second image P2 (step S16). In step S16, the second image processing unit 55 generates the second image P2, which is a monochrome image, by a second image processing process different from the first image processing.
[0067] The image recording unit 56 records the second image P2 generated by the second image processing unit 55 as the recorded image PR in the memory 42 (step S17).
[0068] In the flowchart above, step S11 corresponds to the "imaging process" related to the technology of this disclosure. Step S12 corresponds to the "first generation process" related to the technology of this disclosure. Step S13 corresponds to the "detection process" related to the technology of this disclosure. Step S14 corresponds to the "display process" related to the technology of this disclosure. Step S15 corresponds to the "reception process" related to the technology of this disclosure. Step S16 corresponds to the "second generation process" related to the technology of this disclosure. Step S17 corresponds to the "recording process" related to the technology of this disclosure.
[0069] As described above, with the imaging device 10 of this disclosure, even in monochrome mode, the subject is detected by inputting the first image P1, which is a color image, into the trained model LM, thereby improving the accuracy of subject detection.
[0070] Previously, subject detection primarily used the Viola-Jones algorithm, a classifier based on AdaBoost. The Viola-Jones algorithm detects subjects based on features derived from image brightness differences, meaning image color information was not considered important. However, when using a neural network as the trained model (LM), machine learning is typically performed using color images, allowing for the extraction of features based on both brightness and color information. Therefore, even in monochrome mode, generating color images and inputting them into the trained model (LM) improves subject detection accuracy.
[0071] [Video recording mode] Next, we will explain the video capture mode. Figure 6 shows an example of the generation timing of the first image P1 and the second image P2 in the video capture mode.
[0072] As shown in Figure 6, in video imaging mode, the imaging sensor 20 performs imaging operations at predetermined frame cycles (e.g., 1 / 60 second) and outputs an imaging signal RD for each frame cycle. If the first image processing unit 52 and the second image processing unit 55 attempt to generate the first image P1 and the second image P2 based on the same imaging signal RD within the same frame period, it may not be possible to generate the first image P1 and the second image P2 for each frame cycle due to limitations in image processing capability.
[0073] Therefore, in this example, the generation of the first image P1 by the first image processing unit 52 and the generation of the second image P2 by the second image processing unit 55 are performed alternately every 1 frame period. That is, the first image processing unit 52 generates the first image P1 using the imaging signal RD of the first frame period, and the second image processing unit 55 generates the second image P2 using the imaging signal RD of the second frame period, which is different from the first frame period. As a result, subject detection is performed every 2 frame periods. In addition, the frame rate of the moving image generated by multiple second images P2 is reduced to 1 / 2.
[0074] Figure 7 is a flowchart showing an example of an image generation method in video capture mode. Figure 7 shows an example where video capture mode is selected and the monochrome mode of film simulation is selected.
[0075] The main control unit 50 determines whether or not a video capture start command has been issued by the user operating the operation unit 13 (step S20). If a video capture start command has been issued (step S20: YES), the main control unit 50 controls the imaging control unit 51 to cause the imaging sensor 20 to perform an imaging operation (step S21).
[0076] The first image processing unit 52 acquires the imaging signal RD output from the imaging sensor 20 and generates a color first image P1 by performing first image processing on the imaging signal RD (step S22).
[0077] The subject detection unit 53 detects a subject by inputting the first image P1 generated by the first image processing unit 52 into the trained model LM (step S23). In step S23, the subject detection unit 53 outputs the subject detection result R output from the trained model LM to the main control unit 50. For example, the main control unit 50 controls the lens drive control unit 34 based on the detection result R to perform focusing control on the subject.
[0078] Next, the main control unit 50 controls the imaging control unit 51 to cause the imaging sensor 20 to perform an imaging operation (step S24). The second image processing unit 55 acquires the imaging signal RD output from the imaging sensor 20 and generates a monochrome second image P2 by applying second image processing to the imaging signal RD (step S25).
[0079] The main control unit 50 determines whether or not a user has issued an instruction to end video capture by operating the operation unit 13 (step S26). If no end instruction has been issued (step S26: NO), the main control unit 50 returns to step S21 and causes the imaging sensor 20 to perform the imaging operation again. The processes in steps S21 to S25 are repeatedly executed until the main control unit 50 determines in step S26 that an end instruction has been issued. Steps S21 to S23 are performed during the first frame period, and steps S24 to S25 are performed during the second frame period.
[0080] If a termination instruction is received (step S26: YES), the main control unit 50 causes the image recording unit 56 to generate a recorded image PR (step S27). In step S27, the image recording unit 56 generates a recorded image PR, which is a moving image, based on a plurality of second images P2 generated by repeatedly executing step S25. The image recording unit 56 then records the recorded image PR in the memory 42 (step S28).
[0081] As described above, by alternately generating the first image P1 and the second image P2 every frame period, it is possible to perform high-precision subject detection and video capture without being limited by image processing capabilities.
[0082] [Differentiation] Next, a modified version of the video capture mode will be described. Figure 8 shows an example of the generation timing of the first image P1 and the second image P2 in the modified video capture mode.
[0083] As described above, due to limitations in processing power, it may not be possible to generate the first image P1 and the second image P2 within the same frame period. Therefore, in this modified example, the image processing burden is reduced by lowering the resolution of the first image P1 to a lower resolution than the resolution of the imaging signal RD.
[0084] Specifically, the first image processing unit 52 reduces the resolution of the imaging signal RD acquired from the imaging sensor 20, and then generates a color first image P1 through first image processing. The first image processing unit 52 reduces the resolution of the imaging signal RD, for example, by downsampling pixels. As a result, a first image P1 with a lower resolution than the imaging signal RD is obtained.
[0085] In this modified example, the second image processing unit 55 generates the second image P2 without changing the resolution of the imaging signal RD acquired from the imaging sensor 20. Therefore, in this modified example, the machine learning model LM can detect the subject even when using an image with a lower resolution than the final recorded image, so the resolution of the first image P1 is lower than the resolution of the second image P2.
[0086] In this modified version, the image processing load is reduced by decreasing the resolution of the first image P1, so that the first image P1 and the second image P2 are generated within the same frame duration.
[0087] Figure 9 is a flowchart showing an example of an image generation method in the video capture mode according to the modified version. Figure 9 shows an example where the video capture mode according to the modified version is selected and the monochrome mode of the film simulation is selected.
[0088] The main control unit 50 determines whether or not a video capture start command has been issued by the user operating the operation unit 13 (step S30). If a video capture start command has been issued (step S30: YES), the main control unit 50 controls the imaging control unit 51 to cause the imaging sensor 20 to perform an imaging operation (step S31).
[0089] The first image processing unit 52 acquires the imaging signal RD output from the imaging sensor 20, reduces the resolution of the imaging signal RD, and then performs first image processing to generate a color first image P1 (step S32).
[0090] The subject detection unit 53 detects a subject by inputting the low-resolution first image P1 generated by the first image processing unit 52 into the trained model LM (step S33). In step S33, the subject detection unit 53 outputs the subject detection result R output from the trained model LM to the main control unit 50. For example, the main control unit 50 controls the lens drive control unit 34 based on the detection result R to perform focusing control on the subject.
[0091] In step S32, the second image processing unit 55 performs second image processing on the same imaging signal RD as the imaging signal RD acquired by the first image processing unit 52, thereby generating a monochrome second image P2 (step S34).
[0092] The main control unit 50 determines whether or not a user has issued a command to end video capture by operating the operation unit 13 (step S35). If no termination command has been issued (step S35: NO), the main control unit 50 returns to step S31 and causes the imaging sensor 20 to perform the imaging operation again. The processes in steps S31 to S34 are repeatedly executed until the main control unit 50 determines in step S35 that a termination command has been issued. Steps S31 to S34 are performed within one frame cycle.
[0093] If a termination instruction is received (step S35: YES), the main control unit 50 causes the image recording unit 56 to generate a recorded image PR (step S36). In step S36, the image recording unit 56 generates a recorded image PR, which is a moving image, based on a plurality of second images P2 generated by repeatedly executing step S34. The image recording unit 56 then records the recorded image PR in the memory 42 (step S37).
[0094] As described above, in this modified example, the image processing load is reduced by lowering the resolution of the imaging signal RD before generating the first image P1, so that the first image P1 and the second image P2 can be generated within the same frame period. This makes it possible to generate moving images without reducing the frame rate.
[0095] In the above modified example, the resolution of the first image P1 is reduced to that of the imaging signal RD, but the resolution of the second image P2 may also be reduced to that of the imaging signal RD. Specifically, as shown in Figure 10, the first image processing unit 52 and the second image processing unit 55 reduce the resolution of the imaging signal RD and then generate the first image P1 and the second image P2, respectively. This further reduces the burden on image processing, making it possible to generate the first image P1 and the second image P2 at a faster speed within the same frame period.
[0096] [Other variations] The above embodiments and modifications describe the case where a monochrome mode is selected in color adjustment such as film simulation. However, the technology of this disclosure is not limited to monochrome mode, and can also be applied when a mode that generates a low-saturation image, such as classic chrome mode, is selected. In other words, the technology of this disclosure can be applied when the second image P2 is a low-saturation image.
[0097] Furthermore, the technology of this disclosure can also be applied when the second image P2 is an image with low brightness. This is because the trained model LM, which has been trained using color images, experiences a decrease in subject detection accuracy even with images that have low brightness. Therefore, the technology of this disclosure is characterized in that the saturation or brightness of the first image P1 generated by the first image processing unit 52 is higher than that of the second image P2 and the live view image PL.
[0098] Furthermore, the technology of this disclosure can also be applied when a sepia mode for generating sepia images is selected. A sepia image is an image generated by multiplying the color difference signals Cr and Cb by 0 and then adding a fixed value, when the image signal of a color image is represented in YCbCr format. That is, the first image P1 may be a color image, and the second image P2 and the live view image PL may be sepia images. A trained model LM that has been machine-learned using color images will experience a decrease in subject detection accuracy even with sepia images; therefore, performing subject detection using color images improves the detection accuracy.
[0099] Furthermore, the technology disclosed herein is not limited to digital cameras, but can also be applied to electronic devices such as smartphones and tablet devices that have imaging capabilities.
[0100] In the above embodiment, the hardware structure of the control unit, with processor 40 as an example, can be any of the following types of processors. These types of processors include a CPU, which is a general-purpose processor that functions by executing software (programs), as well as processors whose circuit configuration can be changed after manufacturing, such as FPGAs. FPGAs include dedicated electrical circuits, which are processors with circuit configurations specifically designed to perform specific processing, such as PLDs or ASICs.
[0101] The control unit may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, multiple control units may consist of a single processor.
[0102] There are several possible examples of configuring multiple control units with a single processor. A first example is a configuration where a single processor is composed of one or more CPUs and software, as exemplified by client and server computers, and this processor functions as multiple control units. A second example is a configuration where a processor that realizes the functions of the entire system, including multiple control units, on a single IC chip is used, as exemplified by System-on-a-Chip (SOC) systems. Thus, a control unit can be configured as a hardware structure using one or more of the above-mentioned types of processors.
[0103] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices.
[0104] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.
[0105] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.
Claims
1. The imaging process involves acquiring the imaging signal output from the image sensor, A first generation step in which a first image is generated by first image processing using the aforementioned imaging signal, A detection step in which a trained model that has undergone machine learning is used to detect a subject in the first image, A display step of creating a live view image by changing the first image and displaying the live view image on a display unit, A second generation step in which a second image is generated using the aforementioned imaging signal by a second image processing process different from the first image processing, Includes, The second generation step makes the color of the second image substantially identical to the color of the live view image. Image generation method.
2. The display step involves displaying the live view image and the detection result of the subject detected in the detection step on the display unit. The image generation method according to claim 1.
3. This further includes a reception process for receiving imaging instructions from the user, In the second generation step, if the imaging instruction is received in the reception step, the second image is generated. The image generation method according to claim 1.
4. The display step involves generating a display signal for the live view image based on the image signals that constitute the first image, thereby displaying the live view image. The image generation method according to claim 1.
5. The saturation or brightness of the first image is higher than that of the second image and the live view image. The image generation method according to claim 1.
6. The recording step further includes recording the second image as a still image on a recording medium. The image generation method according to claim 1.
7. The first image has a lower resolution than the imaging signal or the second image. The image generation method according to claim 1.
8. In the imaging process, the imaging signal is output from the image sensor at each frame period, In the first generation step and the second generation step, the first image and the second image are generated using the imaging signal for the same frame period. The first image has a lower resolution than the imaging signal or the second image. The image generation method according to claim 1.
9. The second image has a lower resolution than the imaging signal. The image generation method according to claim 8.
10. In the imaging process, the imaging signal is output from the image sensor at each frame period, The first generation step generates the first image using the imaging signal during the first frame period, The second generation step generates the second image using the imaging signal for a second frame period that is different from the first frame period. The image generation method according to claim 1.
11. The second image is a moving image. The image generation method according to claim 8.
12. The saturation or brightness of the first image is higher than that of the second image. The image generation method according to claim 8.
13. The aforementioned trained model is a model that has undergone machine learning using color images as training data. The first image above is a color image. The second image is either a monochrome or sepia image. The image generation method according to any one of claims 1 to 12.
14. A processor that acquires imaging signals output from an imaging device, A first generation process generates a first image using the aforementioned imaging signal through a first image processing, A detection process is performed using a pre-trained model that has undergone machine learning to detect an object within the first image, A display process that involves changing the first image to create a live view image and displaying the live view image on the display unit, A second generation process that generates a second image using the aforementioned imaging signal by a second image processing process different from the first image processing, Execute, The second generation process makes the color of the second image substantially the same as the color of the live view image. A processor configured in such a way.
15. The display process displays the live view image and the detection result of the subject detected in the detection process on the display unit. The processor according to claim 14.
16. A program used in a processor that acquires imaging signals output from an imaging device, A first generation process generates a first image using the aforementioned imaging signal through a first image processing, A detection process is performed using a pre-trained model that has undergone machine learning to detect an object within the first image, A display process that involves changing the first image to create a live view image and displaying the live view image on the display unit, A second generation process that generates a second image using the aforementioned imaging signal by a second image processing process different from the first image processing, The processor is made to execute the above, The second generation process makes the color of the second image substantially the same as the color of the live view image. program.