Information processing apparatus and method, non-transitory computer readable storage medium and computer program product

Abstract

Information processing apparatus and method, non-transitory computer readable storage medium and computer program product are disclosed. The information processing apparatus includes: an acquisition component configured to acquire a plurality of pieces of input data having a first bit depth; a first conversion component configured to convert a bit depth of one piece of input data to a second bit depth lower than the first bit depth; a second conversion component configured to convert a bit depth of another piece of input data to the second bit depth, and extract a feature from data obtained by integrating the one piece of input data whose bit depth is converted to the second bit depth and the another piece of input data whose bit depth is converted to the second bit depth; and an expansion component configured to expand a bit depth of the feature to a third bit depth higher than the second bit depth.

Description

Technical Field

[0001] This disclosure relates to information processing apparatus, information processing method, non-transitory computer-readable storage medium, and computer program product for converting the bit depth of data. Background Technology

[0002] In recent years, the accuracy of image recognition techniques such as image classification, object detection, and object tracking has been significantly improved with the advent of deep neural networks (hereinafter abbreviated as DNN). Generally speaking, DNN operations are computationally very complex and memory-intensive. Therefore, DNN computations are often performed at a low bit depth, such as 8 bits. On the other hand, DNN inputs and outputs sometimes require high bit depths, such as 10 bits or 16 bits.

[0003] Japanese Patent Publication No. 2023-81714 discloses a technique related to neural networks when the bit depth of the data input to a neural network is greater than the bit depth processed by the neural network. This technique divides an input with a high bit depth into high and low bits to generate a low-bit input. A high-bit output is generated by inputting each division of the low-bit data into the neural network and cascading the output from the neural network in the bit direction.

[0004] However, in the above method, information from other bits (e.g., high bits) is lost during the computation of one set of bits (e.g., low bits), resulting in a significant decrease in output precision. That is, in the above method, precision decreases significantly due to the reduction in bit depth. Summary of the Invention

[0005] Therefore, this disclosure provides a technique that can suppress the decrease in output accuracy even when the bit depth of the input data is reduced.

[0006] A first aspect of this disclosure provides an information processing apparatus, comprising: at least one processor; and at least one memory storing instructions thereon, the instructions, when executed by the at least one processor, causing the information processing apparatus to at least: acquire a plurality of input data having a first bit depth; convert the bit depth of one input data to a second bit depth lower than the first bit depth; convert the bit depth of another input data to the second bit depth, and extract features from data obtained by integrating the one input data whose bit depth has been converted to the second bit depth and the other input data whose bit depth has been converted to the second bit depth; and extend the bit depth of the features to a third bit depth higher than the second bit depth.

[0007] A second aspect of this disclosure provides an information processing method comprising: acquiring multiple input data having a first bit depth; converting the bit depth of one input data to a second bit depth lower than the first bit depth; converting the bit depth of another input data to the second bit depth; extracting features from data obtained by integrating the one input data whose bit depth has been converted to the second bit depth and the other input data whose bit depth has been converted to the second bit depth; and expanding the bit depth of the features to a third bit depth higher than the second bit depth.

[0008] A third aspect of this disclosure provides a non-transitory computer-readable storage medium storing a computer program, which, when read and executed by a computer, causes the computer to perform the aforementioned information processing method.

[0009] The fourth aspect of this disclosure provides a computer program product including a computer program, which, when read and executed by a computer, causes the computer to perform the aforementioned information processing method.

[0010] The features of this disclosure will become clear from the following description of embodiments with reference to the accompanying drawings. The following description of the embodiments is by way of example. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the embodiments.

[0012] Figure 1 This is a block diagram illustrating the overall configuration of the main components of an image capturing device according to an embodiment.

[0013] Figure 2 This is a diagram used to illustrate the amount of defocus in an imaging optical system.

[0014] Figure 3 This is a block diagram illustrating the functional configuration of the image capture device according to the first embodiment.

[0015] Figure 4 This is a block diagram showing the functional configuration of the defocus range estimation unit.

[0016] Figure 5 This is a block diagram illustrating the functional configuration of the input data acquisition unit.

[0017] Figure 6 This is a flowchart of the defocus range estimation process according to the first embodiment.

[0018] Figure 7 This is a flowchart illustrating the input data acquisition and processing in S601.

[0019] Figure 8 This is a flowchart illustrating the bit depth conversion process in S602.

[0020] Figure 9A This is a flowchart illustrating the input consolidation process in S603, which is a flowchart of input consolidation processing performed through channel-wise concatenation.

[0021] Figure 9B This is a flowchart illustrating the input integration process in S603, which is a flowchart of input integration processing performed through spatial cascading.

[0022] Figure 9C This is a flowchart illustrating the input integration process in S603, which is a flowchart of input integration processing performed through element-wise summation.

[0023] Figure 10 This is a diagram illustrating the integration of inputs through channel-by-channel cascading.

[0024] Figure 11 This is a diagram illustrating the use of spatially cascaded input integration.

[0025] Figure 12 This is a diagram illustrating the integration of inputs through element-wise summation.

[0026] Figure 13 This is a flowchart illustrating the processing performed by the CNN used for feature extraction in S604.

[0027] Figure 14A This is a diagram illustrating the process of generating a BB image from input data, showing an image obtained by an image capture device imaging a person.

[0028] Figure 14B This is a diagram illustrating the process of generating a BB image from input data, showing an image overlaid with BB images of human body parts and the whole body detected from the image.

[0029] Figure 14C This is a diagram illustrating the process of generating a BB image from input data, showing a defocus map that indicates the amount of defocus in each region of the image.

[0030] Figure 14D This is a diagram illustrating the process of generating a BB plot from input data, showing an example of a BB plot.

[0031] Figure 15A It is a diagram showing the defocus range of each part of the person 1401.

[0032] Figure 15B It is a schematic diagram showing the estimated defocus range of a person's left eye, right eye, face, and whole body.

[0033] Figure 16A This is a diagram illustrating the details of the bit-depth transformation of the input data, showing the transformation performed by a nonlinear function corresponding to the first LUT applied in S6021.

[0034] Figure 16B This is a diagram illustrating the details of the bit-depth transformation of the input data, showing the transformation performed by a nonlinear function corresponding to the second LUT applied in S6022.

[0035] Figure 17 This is a flowchart illustrating the bit depth extension process in S605.

[0036] Figure 18 This is a flowchart illustrating another example of bit depth extension processing in S605.

[0037] Figure 19 This is a diagram illustrating the transition state of the bit depth expansion process.

[0038] Figure 20 This is a block diagram illustrating the functional configuration of a learning device for learning the defocus range.

[0039] Figure 21 This is a flowchart for the learning process used to learn the defocus range.

[0040] Figure 22 This is a block diagram illustrating the functional configuration of the image capture device according to the second embodiment.

[0041] Figure 23 This is a flowchart of the noise reduction process according to the second embodiment.

[0042] Figure 24 This is a flowchart illustrating the bit depth conversion in S2401.

[0043] Figure 25 It is a diagram that integrates the images of each input data point.

[0044] Figure 26 This is a flowchart illustrating the processing performed by the CNN used for feature extraction in S2403.

[0045] Figure 27 This is a diagram illustrating an example of integrating features and extending bit depth in S2404.

[0046] Figure 28 This is a block diagram illustrating the hardware configuration of the system control unit included in the image capture device 10. Detailed Implementation

[0047] In the following, embodiments will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments are not intended to limit the scope of the claims. Several features are described in the embodiments, but this is not to say that all such features are necessary, and multiple such features can be appropriately combined. Furthermore, in the drawings, the same or similar configurations are given the same reference numerals, and repeated descriptions thereof are omitted.

[0048] First Embodiment

[0049] This embodiment describes the following situation: an image capture device with interchangeable lenses considers the dispersion of the subject in the depth direction to estimate the defocus range and captures an image focused on the subject.

[0050] This embodiment will be described below with reference to the accompanying drawings. Figure 1 This is a block diagram illustrating the overall configuration of the main components including the image capture device 10. (Refer to...) Figure 1 The overall configuration of the image capture apparatus according to the first embodiment is described.

[0051] like Figure 1 As shown, the image capturing device 10 is, for example, a digital camera with interchangeable lenses.

[0052] The image capture device 10 includes a camera body 100, a lens unit 200, and a lens mounting mechanism 113. The camera body 100 is mechanically coupled to the lens unit 200 in a detachable manner and is also electrically connected to the lens unit 200 via the lens mounting mechanism 113 therebetween.

[0053] The camera body 100 includes an imaging element 101, a system control unit 102, a shutter 103, a memory 104, a power switch 105, a mode switching unit 106, a rear monitor 107, a touch panel 108, a viewfinder display unit 109, an eyepiece 110, and an eye proximity detection unit 111.

[0054] Imaging element 101 converts an optical signal (which is an optical image formed by light from a subject) into an electrical signal and outputs the electrical signal. Imaging element 101 can be an electronic device such as a complementary metal-oxide-semiconductor (CMOS) type image sensor or a charge-coupled device (CCD) type image sensor.

[0055] The system control unit 102 controls the camera body 100 and includes a processor such as a well-known central processing unit (CPU). The system control unit 102 also includes an image processing unit for the video signal obtained by the imaging element 101. The system control unit 102 further includes a phase-difference AF unit, which performs focus detection processing based on focus detection image data (signals for phase-difference AF) obtained from the imaging element 101 and the image processing unit using a phase-difference detection method. More specifically, the image processing unit generates a pair of image data using the luminous flux through a pair of pupil regions of the imaging optics system as focus detection image data. The phase-difference AF unit detects the amount of defocus based on the amount of shift in this pair of image data. In this way, the phase-difference AF unit according to this embodiment performs phase-difference AF (image-plane phase-difference AF) based on the output from the imaging element 101 without using a dedicated AF sensor.

[0056] The memory 104 stores programs, variables, constants, etc., used for the operation of the system control unit 102. The memory 104 may include, for example, electrically erasable and storable non-volatile memory. The memory 104 stores various parameters, such as ISO sensitivity settings, shooting modes, various correction data, etc.

[0057] The power switch 105 accepts power-on and power-off commands from the user for the camera body 100. The power switch 105 outputs the accepted commands to the system control unit 102.

[0058] The mode switching unit 106 receives operations from the user to switch and set various shooting modes, including live view shooting mode and moving image shooting mode. The mode switching unit 106 outputs the received operations to the system control unit 102.

[0059] The rear monitor 107 includes a display device, LEDs, etc. The rear monitor 107 displays captured information such as operating status and messages, indicated by characters, images, sounds, etc., based on the execution of the program in the system control unit 102. The display device can be a liquid crystal display, an organic electroluminescent (EL) display, etc.

[0060] A touch panel 108 is disposed on the display surface of the rear monitor 107. The touch panel 108 detects the contact of a finger or pen and notifies the system control unit 102 of the contact position on the rear monitor 107. As a result, the system control unit 102 executes an operation or function associated with the contact position.

[0061] The viewfinder display unit 109 displays shooting information according to the execution of the program in the system control unit 102, and together with the eyepiece 110, constitutes an electronic viewfinder (EVF).

[0062] The viewfinder display unit 109 is, for example, a small liquid crystal display device.

[0063] The eye proximity detection unit 111 detects the photographer's eye proximity status and outputs the eye proximity status to the system control unit 102. The system control unit 102 displays the shooting information mentioned above on the rear monitor 107 or the viewfinder display unit 109 based on the obtained eye proximity status.

[0064] Next, the configuration of the lens unit 200 will be described. The lens unit 200 guides incident light to the imaging element 101. The lens unit 200 includes a photographic lens 201, an aperture 202, a lens drive circuit 203, an aperture control circuit 204, and a lens control unit 205.

[0065] Although only one lens is illustrated for simplicity, the camera lens 201 can be a camera lens group that includes multiple lenses. When light from the subject is incident on the camera lens 201, the camera lens 201 forms an optical image of the light on the imaging element 101 via the aperture 202 and the shutter 103.

[0066] The lens control unit 205 controls the entire lens unit 200 based on instructions from the system control unit 102, etc. Specifically, to focus on a specific subject, the lens control unit 205 uses the lens drive circuit 203 to move the photographic lens of the lens unit 200 along the optical axis. To adjust depth of field and light intensity, the lens control unit 205 uses the aperture control circuit 204 to control the aperture 202. The lens control unit 205 includes a memory that stores various constants, variables, programs, etc., used for lens operation. The lens control unit 205 includes a non-volatile memory that holds information such as the maximum aperture value, minimum aperture value, focal length, etc., used for controlling the lens unit 200.

[0067] The system control unit 102 of the camera body 100 calculates the amount of defocus using output information from the imaging element 101. Subsequently, based on the calculated amount of defocus, the system control unit 102 controls the lens drive circuit 203 by communicating via the lens control unit 205 of the lens unit 200 in order to achieve focusing.

[0068] Figure 28This is a block diagram illustrating the hardware configuration of the system control unit 102 included in the image capture device 10. The system control unit 102 is an example of a computer (also known as an information processing device). The system control unit 102 includes a processor 2901, a memory 2902, a storage device 2903, a communication IF 2904, an input IF 2905, an output IF 2906, and a bus 2907. The processor 2901, memory 2902, storage device 2903, communication IF 2904, input IF 2905, and output IF 2906 are connected to enable them to send and receive information to each other via the bus 2907.

[0069] Processor 2901 is an arithmetic processing unit such as a central processing unit (CPU). Note that system control unit 102 may include other processors such as microprocessor units (MPUs), graphics processing units (GPUs), neural processing units (NPUs), quantum processing units (QPUs), etc., to replace or supplement the CPU. Processor 2901 implements the various functions of system control unit 102 by reading computer programs (hereinafter also referred to as programs) stored in storage device 2903 and loading them into memory 2902. Some or all of the functions of system control unit 102 may be implemented by one or more circuits such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs) including field-programmable gate arrays (FPGAs).

[0070] Memory 2902 corresponds to memory 104 and is, for example, a storage device capable of high-speed read and write operations, such as random access memory (RAM). When processor 2901 executes a program, memory 2902 serves as the working area. Memory 2902 temporarily stores the program and parameters required for program execution.

[0071] Storage device 2903 is, for example, a non-volatile storage device such as a hard disk drive (HDD) or a solid-state drive (SSD). Even without power, storage device 2903 retains the program, parameters required to execute the program, and the results of program execution. Storage device 2903 stores, for example, a trained model used to extract features (also called feature values), model parameters, etc.

[0072] The IF 2904 communication interface is used to enable communication with external devices via wired or wireless networks.

[0073] Input IF 2905 is an interface for accepting information input from an input device such as touch panel 108. The input device could be, for example, a mouse or keyboard.

[0074] Output IF 2906 is an interface used to output information such as images to a rear monitor 107, etc.

[0075] Description of the method for calculating defocus amount

[0076] Here, a description of the defocus amount used as depth information for the image in this embodiment will be given. Figure 2 It is a diagram used to illustrate the defocus of an imaging optical system.

[0077] Specifically, Figure 2 The relationship between the amount of defocus and the phase difference (image shift) between the first focus detection signal and the second focus detection signal obtained from the imaging element is shown.

[0078] Image plane 2000 is a plane on which imaging element 101 is deployed. The exit pupil of the imaging optics system is divided into two regions: a first pupil region 2011 and a second pupil region 2012. The defocus amount d can be the distance from the imaging position C to image plane 2000, or the magnitude of that distance (i.e., the absolute value of the distance), from the light flux from the subjects 2021 and 2022. In the foreground focal state where the imaging position C is on the subject side relative to image plane 2000, the defocus amount has a negative sign (d < 0). In the background focal state where the imaging position C is on the opposite side of the subject relative to image plane 2000, the defocus amount has a positive sign (d > 0). In the focused state where the imaging position C is on image plane 2000, d = 0. The imaging optics system is in focus relative to subject 2021 (d = 0) and in the foreground focal state relative to subject 2022 (d < 0). The foreground focus state (d < 0) and the background focus state (d > 0) are examples of the defocus state (|d| > 0).

[0079] In the foreground focus state (d < 0), the luminous flux from the subject 2022 that passes through the first pupil region 2011 (or the second pupil region 2012) converges once, and then spreads out to a width Γ1 or a width Γ2 centered on the centroid position G1 (or centroid position G2) of the luminous flux. This forms a blurred image on the image plane 2000. This blurred image is received by each first focus detection pixel (or each second focus detection pixel) on the imaging element, generating a first focus detection signal (or a second focus detection signal). That is, the first focus detection signal (or the second focus detection signal) is a signal representing the subject image, wherein the subject 2022 is blurred at the centroid position G1 (or centroid position G2) of the luminous flux on the image plane 2000 with a blur width Γ1 (or blur width Γ2).

[0080] The blur width Γ1 (or blur width Γ2) of the subject image increases approximately proportionally to the increase of the defocus amount d |d|. Similarly, the image shift p (= the difference in centroid positions of luminous flux G1 - G2) between the first and second focus detection signals |p| also increases approximately proportionally to the increase of the defocus amount d |d|. The relationship is similar in the back focus state (d > 0), but the direction of image shift between the first and second focus detection signals is opposite to that in the front focus state.

[0081] In this way, as the amount of defocus in the imaging signal increases, the amount of image shift between the first focus detection signal and the second focus detection signal also increases. Based on this relationship, the phase difference AF unit of the system control unit 102 performs focus detection using an image plane phase difference detection method to calculate the amount of defocus based on the amount of image shift between the first focus detection signal and the second focus detection signal obtained using the imaging element 101. Specifically, the phase difference AF unit of the system control unit 102 uses a conversion factor calculated based on the baseline length to convert the image shift amount into the detected amount of defocus. Note that in this embodiment, the product of the aperture F value and the permissible circle of confusion diameter δ in the optical system of the image capture device during image capture [Fδ] is used as the unit of defocus amount.

[0082] Functional configuration of the first embodiment

[0083] The functional configuration and operation of the first embodiment will be described below. Figure 3 This is a block diagram illustrating the functional configuration of an image capture apparatus 10 according to a first embodiment. The image capture apparatus 10 includes a defocus range estimation unit 301 and a control unit 302. The processor 2901 of the system control unit 102 can implement some or all of the functions of the defocus range estimation unit 301 and the control unit 302 by reading and executing programs. Some or all of the functions of the defocus range estimation unit 301 and the control unit 302 can be implemented by circuitry such as an ASIC.

[0084] The defocus range estimation unit 301 estimates the defocus range of the subject. The defocus range can be the range of the amount of defocus on the subject.

[0085] Figures 14A to 14D This is a diagram illustrating the process of generating a BB image from input data. The input data is a diagram illustrating the defocus of an image 1400 obtained by the image capture device 10 imaging a person 1401. Figure 14A An image 1400 is shown, obtained by the image capturing device 10 imaging a person 1401. Figure 14CA defocus map 1407 is shown, which is a map indicating the amount of defocus in each region of image 1400. The defocus map 1407 can indicate the magnitude of the defocus by shading.

[0086] Figure 15A and Figure 15B It is a diagram showing the defocus range of different parts of the subject. Figure 15A This is a diagram showing the defocus range of various parts of the person 1401. Defocus range 1500 indicates the defocus range of the person's left eye. Defocus range 1501 indicates the defocus range of the person's face. Defocus range 1502 indicates the defocus range of the person's entire body. Defocus ranges 1500, 1501, and 1502 each visualize the dispersion of an object in the depth direction when viewed from the image capture device 10. The focus position 1503, indicated by a thick line, indicates the focus position of the image capture device 10.

[0087] Figure 15B This is a schematic diagram showing the estimated defocus range of a person's left eye, right eye, face, and whole body. The horizontal axis represents the magnitude of the defocus. The proximal side on the horizontal axis indicates the side closer to the image capture device 10. On the other hand, the distal side on the horizontal axis indicates the side farther away from the image capture device 10. The length of the arrows extending horizontally represents the defocus range, which is the range of defocus values ​​for each part.

[0088] like Figure 15A As shown, for example, in the depth-direction dispersion of the entire body of person 1401 when viewed from the image capture device 10, the nearest side is, for example, the tip of person 1401's nose. On the other hand, the farthest side is, for example, the heel of person 1401. Therefore, the maximum value of the defocus amount of the entire body of person 1401 (the nearest value) is the defocus amount of the tip of person 1401's nose, and the minimum value of the defocus amount (the farthest value) is the defocus amount of the heel of person 1401. The range of values ​​defined by these values ​​is the defocus range of the entire body of person 1401.

[0089] In this way, the defocus range estimation unit 301 estimates the defocus range as a range of values ​​for the defocus amount by taking into account the perspective relationship of the estimated target, such as the subject's eyes, face, and whole body, in the depth direction.

[0090] The control unit 302 calculates the driving amount of the camera lens 201 and controls the focus position based on the defocus range estimated by the defocus range estimation unit 301. Additionally, when controlling the aperture 202, the control unit 302 adjusts the depth of field (DoF).

[0091] Figure 4This is a block diagram illustrating the functional configuration of the defocus range estimation unit 301. The defocus range estimation unit 301 includes an input data acquisition unit 401, a bit depth conversion unit 402, an input integration unit 403, a feature extraction unit 404, and a bit depth expansion unit 405.

[0092] The input data acquisition unit 401 acquires one or more pieces of input data necessary for estimating the defocus range. The input data may include, for example, data of an image (hereinafter also referred to as a picture) obtained by imaging the subject, a defocus map, etc. Multiple pieces of input data may have different bit depths.

[0093] Bit depth conversion unit 402 converts the bit depth of each of one or more input data obtained by input data acquisition unit 401. Bit depth conversion unit 402 can convert the bit depth of any one of the multiple input data. For example, bit depth conversion unit 402 converts the bit depth of a defocus image included in the input data into a lower bit depth.

[0094] When the bit depths of the input data are different, the bit depth conversion unit 402 can unify the bit depth by converting to a higher bit depth.

[0095] The input integration unit 403 integrates multiple input data. For example, the input integration unit 403 integrates input data obtained by the input data acquisition unit 401 and input data whose bit depth has been converted by the bit depth conversion unit 402.

[0096] Feature extraction unit 404 extracts one or more features (also called feature values) from input data whose bit depth has been transformed and integrated by input integration unit 403. Feature extraction unit 404 can use a trained model based on machine learning or the like to extract features. Feature extraction unit 404 can extract features related to the defocus range.

[0097] Bit depth expansion unit 405 expands and increases the bit depth of the feature data extracted by feature extraction unit 404. For example, bit depth expansion unit 405 expands the bit depth by integrating multiple features. For example, bit depth expansion unit 405 expands the bit depth of features related to defocus range. Bit depth expansion unit 405 can expand the bit depth of features that have been reduced by bit depth conversion unit 402 to the same bit depth as the original input data.

[0098] Figure 5 This is a block diagram illustrating the functional configuration of the input data acquisition unit 401. The input data acquisition unit 401 includes an image acquisition unit 501, a subject detection unit 502, a subject designation unit 503, a defocus image acquisition unit 504, an image cutout unit 505, and a defocus image cutout unit 506.

[0099] The image acquisition unit 501 acquires the image captured by the image capture device 10. The acquired image is, for example, an image 1400 in which a person appears.

[0100] The subject detection unit 502 detects a subject from the image acquired by the image acquisition unit 501. The subject may be, for example, a person. The subject detection unit 502 can detect objects such as people as subjects from the image using techniques such as Non-Patent Document 1 (Non-Patent Document 1: Liu et al., “SSD: Single Shot Multibox Detector”, ECCV 2016). The subject detection unit 502 obtains, for example, a bounding box (hereinafter abbreviated as BB) indicating the region of the subject through object detection. Alternatively, the subject detection unit 502 can detect a person's face, eyes, etc., using techniques such as Non-Patent Document 2 (Non-Patent Document 2: Jiankang Deng et al., “RetinaFace: Single-stage Dense Face Localisation in the Wild”, arXiv 2019). Figure 14B Image 1402 is shown, which overlays various body parts and full-body images of the person 1401 detected from image 1400. BB 1403 indicates the BB of the left eye. BB 1404 indicates the BB of the right eye. BB 1405 indicates the BB of the face. BB 1406 indicates the BB of the whole body.

[0101] The subject designation unit 503 designates the subject to be focused from the detected subjects. When the subject is displayed on the rear monitor 107, the subject designation unit 503 can designate the subject by receiving a touch from the user via the touch panel 108. Note that the subject designation unit 503 can designate the subject by automatically detecting the main subject in the image, rather than by touching the touch panel 108. The subject designation unit 503 can automatically detect the main subject in the image, for example, using the technology disclosed in Japanese Patent Publication No. 2017-98900.

[0102] The defocus map acquisition unit 504 acquires a defocus map corresponding to the image captured by the image capture device 10.

[0103] The image cropping unit 505 crops out the area where the subject appears from the image and adjusts its size to a predetermined size.

[0104] Similar to the image cropping unit 505, the defocus image cropping unit 506 crops out the area where the subject appears from the defocus image and adjusts the size of the defocus image to a predetermined size.

[0105] The BB image generation unit 507 generates a BB image based on the subject BB information obtained by the subject designation unit 503. Figure 14D An example of BB Figure 1408 is shown.

[0106] In a BB chart, predetermined values ​​are entered into the areas indicated by the BBs of various parts of the subject and the whole body (e.g., BB 1403 for the right eye, BB 1404 for the left eye, BB 1405 for the face, and BB 1406 for the whole body). Note that a BB chart does not need to be a single chart, and each part of the subject can have a chart.

[0107] Figure 6 This is a flowchart illustrating the defocus range estimation process according to the first embodiment. In the following description, the label of each process (step) is omitted by adding an "S" at the beginning of each process (step). However, the system control unit 102 of the image capture device 10 does not necessarily have to perform this operation. Figure 6 All steps described in the flowchart. Figure 6 The processes performed by the system control unit 102 are shown as steps.

[0108] In S601, the input data acquisition unit 401 acquires input data (also simply referred to as input). Input data may include, for example, images captured by the image capture device 10, defocus maps, etc. Figure 7 A flowchart illustrating the input data acquisition process in S601 is shown. (Refer to...) Figure 7 The process of obtaining and processing input data is explained.

[0109] In S6011, the image acquisition unit 501 acquires the captured image.

[0110] In S6012, the subject detection unit 502 detects the subject in the image and obtains the subject's BB.

[0111] In S6013, the subject designation unit 503 designates the subject to be captured.

[0112] In S6014, the defocus map acquisition unit 504 acquires the defocus map corresponding to the captured image.

[0113] In S6015, the image cropping unit 505 crops a portion of the image including the subject based on the subject BB obtained in S6013, and adjusts the image size to a predetermined size.

[0114] In S6016, the defocus image cutting unit 506 cuts out a portion of the defocus image of the subject based on the BB of the subject obtained in S6013, and adjusts the size of the defocus image to a predetermined size.

[0115] In S6017, the BB image generation unit 507 generates a BB image based on the subject's BB obtained in S6013, such as... Figure 14D The BB diagram shown.

[0116] Return to Figure 6 In S602, the bit depth conversion unit 402 converts the bit depth of the defocused image obtained in S6016. Here, it is assumed that the bit depth of the defocused image is L bits (e.g., 16 bits). It is assumed that the bit depth of the image obtained in S6011 is M bits (e.g., 8 bits). It is assumed that the feature extraction unit 404 can handle a bit depth of M bits. And it is assumed that L bits ≠ M bits. Specifically, L bits > M bits is true. Therefore, the bit depth of the image is M bits, which is the same as the bit depth that the feature extraction unit 404 can handle. On the other hand, the bit depth of the defocused image is different from the bit depth of the image and is higher than the bit depth that the feature extraction unit 404 can handle. Therefore, in S602, the bit depth conversion unit 402 converts the bit depth of the defocused image into a bit depth that the feature extraction unit 404 can handle. In other words, the bit depth conversion unit 402 converts the bit depth of the defocused image into a lower bit depth. Moreover, it can be said that the bit depth conversion unit 402 unifies the bit depth of multiple input data with different bit depths.

[0117] Figure 8 A flowchart illustrating the bit depth conversion process in S602 is shown. The bit depth conversion unit 402 simply or uniformly converts the L-bit defocused image to M-bit, resulting in information loss during quantization. This reduces the accuracy of defocus range estimation. Therefore, in S602, the bit depth conversion unit 402 applies at least one of several types of transformations to convert the bit depth of the defocused image. These transformations are, for example, nonlinear transformations. For instance, in S6021, the bit depth conversion unit 402 applies a first LUT to convert the bit depth of the defocused image. In S6022, the bit depth conversion unit 402 applies a second LUT, different from the first LUT, to convert the bit depth of the defocused image. LUT stands for Look-Up Table.

[0118] Figure 16A and Figure 16B This is a diagram illustrating the details of the bit-depth transformation of the input data. Figure 16A Transformation 1601 is shown using a nonlinear function corresponding to the first LUT applied in S6021. Figure 16B Transformation 1602 is shown using a nonlinear function corresponding to the second LUT applied in S6022.

[0119] Bit depth conversion unit 402 reduces information loss during quantization by applying multiple different types of bit depth conversions. Furthermore, for estimating the defocus range, accuracy near the focal plane (near the defocus value of 0) is particularly important. Therefore, bit depth conversion unit 402 converts the bit depth so that information about defocus values ​​with small absolute values ​​is less likely to be lost. Conversion 1601 increases the slope in regions with small absolute values ​​of defocus values ​​and decreases the slope in regions with large absolute values ​​of defocus values. Thus, conversion 1601 converts the bit depth so that information about defocus values ​​near the focal plane is less likely to be lost. On the other hand, the change in slope of conversion 1602 is smaller than the change in slope of conversion 1601.

[0120] Bit depth conversion unit 402 can use LUTs based on conversions 1601 and 1602 to convert bit depth. The number of bins in the LUT is not necessarily L bits. For example, if a LUT with fewer than L bins is prepared, bit depth conversion unit 402 can interpolate between bins using linear interpolation or the like. Furthermore, if the defocus value is negative, bit depth conversion unit 402 can calculate the absolute value of the defocus value, convert it to M bits using the LUT, and then assign a negative sign. When bit depth conversion unit 402 uses multiple conversions, the number of defocus maps increases with the number of conversions. For example, if bit depth conversion unit 402 applies two types of conversions, conversion 1601 and conversion 1602, the number of defocus maps increases from one to two.

[0121] In the following description, the M-bit defocus map obtained by transformation 1601 is referred to as the first defocus map. The M-bit defocus map obtained by transformation 1602 is referred to as the second defocus map. Note that transformation 1601, transformation 1602, transformation using the first LUT, and transformation using the second LUT are examples of the first nonlinear transformation.

[0122] Return to Figure 6 In S603, the input integration unit 403 integrates multiple input data. For example, the input integration unit 403 integrates the image and BB map obtained by the input data acquisition unit 401 with the first defocus map and the second defocus map whose bit depth has been converted by the bit depth conversion unit 402. In other words, the input integration unit 403 unifies multiple input data whose bit depth has been unified by the bit depth conversion unit 402.

[0123] Figures 9A to 9C This is a flowchart illustrating the input integration process in S603. References will be made below. Figures 9A to 9C Describes various types of integration processing.

[0124] Figure 9AThis is a flowchart of input integration processing through channel-by-channel cascading. Figure 10 This is a diagram illustrating the channel-by-channel integration of the input. In S6031, the input integration unit 403 generates the integrated input 1005 by cascading the first defocus map 1001, the second defocus map 1002, the image 1003, and the BB map 1004 in the channel direction. The integrated input 1005 can also be considered the output of the input integration unit 403. The number of channels of the input 1005 is the sum of the number of channels of the first defocus map 1001, the second defocus map 1002, the image 1003, and the BB map 1004.

[0125] Figure 9B This is a flowchart illustrating input integration processing via spatial cascading. Figure 11 This is a diagram illustrating the integration of input in the spatial direction. In S6032, the input integration unit 403 matches the number of channels of the data by broadcasting the first defocus map 1101, the second defocus map 1102, the image 1103, and the BB map 1104 in the channel direction. In S6033, the input integration unit 403 cascades the first defocus map 1101, the second defocus map 1102, the image 1103, and the BB map 1104 in the spatial direction to generate the integrated input 1105.

[0126] Figure 9C This is a flowchart of input integration processing through element-by-element summation. Figure 12 This diagram illustrates the integration of inputs through element-wise summation. In S6034, the input integration unit 403 matches the number of data channels by broadcasting the first defocus map 1101, the second defocus map 1102, the image 1103, and the BB map 1104 in the channel direction. In S6035, the input integration unit 403 combines the first defocus map 1204, the second defocus map 1203, the image 1201, and the BB map 1202 by calculating the element-wise sum to generate the integrated input 1205.

[0127] Return to Figure 6 In S604, the feature extraction unit 404 extracts features from the integrated input. The feature extraction unit 404 can use convolutional neural networks (hereinafter abbreviated as CNN) or similar methods to extract feature values.

[0128] Figure 13This is a flowchart illustrating the CNN processing for feature extraction in S604. The CNN includes processing via convolution operations and non-linear operations such as ReLU and max pooling. A CNN may include multiple of these elements. A CNN may include global average pooling, fully connected layers, etc. Feature extraction unit 404 performs convolution in S6041, ReLU in S6042, and max pooling in S6043. Subsequently, feature extraction unit 404 performs convolution in S6044 and ReLU in S6045. Feature extraction unit 404 again performs convolution in S6046 and ReLU in S6047. Note that feature extraction unit 404 may repeat convolution and ReLU three or more times. Feature extraction unit 404 performs global average pooling in S6048 and fully connected layer processing in S6049 to extract feature values.

[0129] The feature extraction unit 404 can be a multilayer perceptron or a multi-head self-attention system. The above example does not limit the feature extraction unit 404, and the configuration of the feature extraction unit 404 can be modified appropriately.

[0130] The feature extraction unit 404 can be computed using, for example, a bit depth of M bits.

[0131] Note that the number of elements in the feature output by the feature extraction unit 404 is the product of the number of parts for which the defocus range is estimated, the number of positions used to obtain the value of the defocus range, and the integration number N used for integrating the output, as described below. For example, when outputting for four parts including the left eye, right eye, face, and whole body, at two positions in the near and far ranges, and with an integration number of 2, the feature extraction unit 404 outputs with a number of 16 elements (= 4 × 2 × 2 = (number of parts) × (number of positions (= near, far)) × (integration number)).

[0132] Return to Figure 6 In S605, the bit depth extension unit 405 increases the bit depth of the features by extending the bit depth of the features obtained by the feature extraction unit 404, and outputs the estimated defocus range.

[0133] Figure 17 This is a flowchart illustrating the bit depth extension process in S605. Figure 19 This is a diagram showing the transition state of the bit depth extension process. Figure 19 Part (a) shows one or more M-bit features. Figure 19 Part (b) shows the state where the M-bit feature is decomposed into N groups of elements. Figure 19 Part (c) shows the following output, where element-wise product is used to integrate the decomposed elements. Figure 19 Part (d) shows the values ​​of the elements of the output feature values.

[0134] In the bit depth expansion process, in S6051, the bit depth expansion unit 405 decomposes the features extracted by the feature extraction unit 404 in the channel direction. Figure 19 The M-bit features shown in part (a) are used to decompose it into Figure 19 The multiple elements of group N shown in part (b). N is the integration number at the time of integration and is predetermined.

[0135] Next, in S6052, the bit depth extension unit 405 calculates the element-wise product of the decomposed features and integrates them to generate Figure 19 The output is shown in part (c). When the bit depth expansion unit 405 integrates features using element-wise multiplication, the bit depth after feature integration is M bits × N bits. For example, if M is 8 and N is 2, then the bit depth expansion unit 405 obtains a 16-bit output. The elements of the 16-bit output correspond one-to-one with the near and far sides of the defocused range of the desired output region. Figure 19 As shown in part (d), the 0th to 7th elements of the eigenvalue correspond one-to-one with the amount of defocusing on the near and far sides of each part. Note that the bit depth extension unit 405 can use the output obtained in M×N bits as is, or the bit depth of the output can be converted as needed by shifting bits, etc.

[0136] Figure 18 A flowchart illustrating another example of bit-depth extension processing in S605 is shown. The bit-depth extension unit 405 can perform, for example... Figure 18 The bit depth expansion process shown is illustrated.

[0137] Specifically, in S6051, the bit depth extension unit 405 decomposes the M-bit features extracted by the feature extraction unit 404 in the channel direction.

[0138] In S6053 and S6054, the bit depth expansion unit 405 applies the third LUT and the fourth LUT to the decomposed features. Here, the third LUT corresponds to the first LUT, and the fourth LUT corresponds to the second LUT. For example, the transformation using the third LUT is the inverse transformation of the transformation using the first LUT. For example, the transformation using the fourth LUT is the inverse transformation of the transformation using the second LUT. The transformation using the third and fourth LUTs is a nonlinear transformation and is an example of a second nonlinear transformation.

[0139] In S6055, the bit depth expansion unit 405 integrates the features to which the LUT has been applied by calculating the element-wise sum to generate the output. When M-bit features are integrated by element-wise summation with an integration number N, the number of bits in the integrated output is M+N bits. Note that the bit depth expansion unit 405 can integrate features by calculating the element-wise average of the features instead of the element-wise sum.

[0140] Therefore, in S605, the bit depth extension unit 405 can obtain the defocus range of each part with increased bit depth by extension. The bit depth extension unit 405 outputs the defocus range of each part with increased bit depth to, for example, the control unit 302.

[0141] The control unit 302 can calculate the driving amount of the photographic lens 201 and control the focus position based on the obtained defocus range of each part of any given subject. Furthermore, in the control (adjustment) of the aperture 202, depth of field (DoF) adjustment is performed. Since the defocus range is output with a bit depth greater than M bits that the feature extraction unit can calculate, finer adjustment of the lens driving amount is possible. Therefore, the image capture device 10 can easily focus on any given part of the subject.

[0142] Learning the defocus range

[0143] Next, the learning device and learning method for learning the defocus range of a subject will be described.

[0144] Figure 20 This is a block diagram illustrating the functional configuration of a learning device for learning the defocus range. The learning device 21 may be provided in the image capture device 10 or may be separate from the image capture device 10. The learning device 21 includes a defocus range estimation unit 2100, a ground truth acquisition unit 2101, a loss calculation unit 2102, a parameter update unit 2103, a parameter storage unit 2104, and a storage unit 2105.

[0145] The defocus range estimation unit 2100 has the same function as the defocus range estimation unit 301 during inference. The defocus range estimation unit 2100 takes the image and defocus map as input, estimates the defocus range of each part of the subject, and outputs the estimation result. The estimation result can be considered the output of the bit depth extension unit 405.

[0146] The true value acquisition unit 2101 acquires a true value label that includes a predetermined true value of the defocus range of each part of the subject.

[0147] Regarding the images prepared as training data and the defocus ranges used as ground truth labels, a value for the amount of defocus calculated for each focus detection region can be used, based on the focus detection signal obtained at the same timing as the image acquisition. For example, the image capture device 10 can calculate the amount of defocus. Alternatively, an external arithmetic device, such as a computer, can hold the focus detection signal and the image signal and calculate the amount of defocus.

[0148] When associating a ground truth label for the defocus range with an image, an appropriate range can be adopted as the ground truth value for the defocus amount of each part of the subject by referring to the amount of defocus in the region of the subject in the image and the amount of defocus in at least one of the background and foreground obstacles. Users can associate the ground truth label for the defocus amount with the image during visual inspection. Furthermore, the value calculated for each subject part region defined by image segmentation and excluding background and foreground obstacles, as well as for the region where the focus detection region overlaps, can be used as the defocus range for the ground truth label.

[0149] The loss calculation unit 2102 calculates the loss by comparing the estimation result output from the defocus range estimation unit 2100 with the true value label obtained by the true value acquisition unit 2101.

[0150] The parameter update unit 2103 updates the parameters of the defocus range estimation unit 2100 based on the loss calculated by the loss calculation unit 2102.

[0151] The parameter storage unit 2104 stores the parameters of the defocus range estimation unit 2100 in the storage unit 2105.

[0152] When the image capture device 10 and the learning device 21 are separate devices, the image capture device 10 can update the parameters of the defocus range estimation unit 301 using the parameters of the defocus range estimation unit 2100.

[0153] Figure 21 This is a flowchart of the learning process used to learn the defocus range. The processing flow in S601 to S605 is the same as that during inference. In the learning process, S601 is executed first.

[0154] In S2201, the truth value acquisition unit 2101 acquires truth value labels for the defocus range of each part of the subject. The truth value labels include the truth values ​​for the near and far defocus ranges of each part of the subject, such as... Figure 15B As shown in the diagram. Then, execute S602 through S605.

[0155] In S2202, loss calculation unit 2102 calculates the loss. The loss is calculated based on the estimated value of the defocus range obtained in S605 and the true value of the true value label of the defocus range obtained in S2201. The loss “Loss” can be obtained, for example, by the L1 norm shown in the following formula (1).

[0156] ...(1)

[0157] Here, the variables are defined as follows.

[0158] N: Number of parts of the subject

[0159] The true defocus value near the i-th location.

[0160] : Estimated defocus value of the proximal side of the i-th site

[0161] The true defocus value of the distal side of the i-th part.

[0162] : Estimated defocus value of the distal side of the i-th part

[0163] In S2203, parameter update unit 2103 updates the parameters of defocus range estimation unit 2100 based on the loss calculated in S2201. The parameters can be weights of elements such as convolutions in a neural network. Parameter update unit 2103 can update the parameters based on an error backpropagation method using momentum SGD or similar techniques.

[0164] In S2204, after determining that the loss obtained by the loss calculation unit 2102 has converged, the parameter update unit 2103 terminates the learning. If it is determined that the loss has not yet converged, the parameter update unit 2103 returns to S601. The parameter update unit 2103 can determine that the loss has converged when the loss drops below a certain value. The parameter update unit 2103 can also determine that the loss has converged when the number of repetitions from S601 to S2203 exceeds a predetermined number.

[0165] In S2205, the parameter storage unit 2104 stores the parameters of the defocus range estimation unit 2100. When the image capture device 10 and the learning device 21 are integrated, storing the parameters of the defocus range estimation unit 2100 is equivalent to storing the parameters of the defocus range estimation unit 301. When the image capture device 10 and the learning device 21 are separate devices, the parameters of the defocus range estimation unit 301 are subsequently updated using the parameters of the defocus range estimation unit 2100.

[0166] It should be noted that the subject does not necessarily have to be of only one type. Furthermore, the subject is not limited to humans, but can be of multiple types, including animals such as dogs and cats, and vehicles such as cars and trains. That is, this embodiment does not limit the number of categories and types of subjects.

[0167] Effects of the first embodiment

[0168] In this embodiment, features are extracted after the bit depth of the input data is converted to a lower bit depth, and the bit depth of the features is expanded. Therefore, this embodiment can suppress the loss of information in the input data during the bit depth conversion and suppress the reduction in the accuracy of the output features.

[0169] In this embodiment, element-wise multiplication and other methods are used to integrate multiple features extracted by the feature extraction unit 404. Therefore, this embodiment enables the conversion of bit depth without separating high and low bits. Thus, in this embodiment, even when the bit depth that the feature extraction unit 404 can handle is lower than the bit depth of the input defocus map, the defocus range can be estimated while suppressing a decrease in accuracy.

[0170] In this embodiment, the bit depth of the input data is converted to a lower bit depth through a nonlinear transformation. Furthermore, this embodiment expands the bit depth of the extracted features through a nonlinear transformation. Therefore, since this embodiment can suppress the loss of information in important ranges (e.g., near the focal plane), it can further suppress the degradation of output accuracy.

[0171] In this embodiment, the nonlinear transformation applied to the conversion from a lower bit depth to a higher bit depth can be, for example, the inverse of a nonlinear transformation applied to expand the bit depth to increase the bit depth. In this case, this embodiment can further suppress the loss of information caused by the conversion of bit depth.

[0172] In this embodiment, the bit depth of multiple input data is unified by reducing the bit depth of the bit depth conversion. Therefore, in this embodiment, multiple input data can be easily integrated.

[0173] Second Embodiment

[0174] The second embodiment performs each task of noise reduction processing to reduce noise in the image. Figure 22 This is a block diagram illustrating the functional configuration of an image capture apparatus according to a second embodiment. The image capture apparatus 23 according to the second embodiment includes an image acquisition unit 501, a bit depth conversion unit 402, an input integration unit 403, a feature extraction unit 404, and a bit depth expansion unit 405. Note that since the components according to the second embodiment have the same functions as those according to the first embodiment, the description is simplified.

[0175] The image acquisition unit 501 acquires the image captured by the image capture device 23 as input data.

[0176] Bit depth conversion unit 402 converts the bit depth of the image obtained by image acquisition unit 501 and generates multiple images.

[0177] The input integration unit 403 integrates the image whose bit depth has been converted by the bit depth conversion unit 402.

[0178] The feature extraction unit 404 extracts features from the image obtained by the input integration unit 403.

[0179] Bit depth extension unit 405 integrates the features extracted by feature extraction unit 404 and converts the bit depth.

[0180] Note that in this embodiment, the bit depth of the image is N bits. Assume that the feature extraction unit 404 can handle a bit depth of M bits. For example, N > M. Assume that the output bit depth obtained from the bit depth expansion unit 405 is K bits.

[0181] Figure 23 This is a flowchart of the noise reduction process according to the second embodiment.

[0182] In S601, the image acquisition unit 501 acquires input data including an image obtained by imaging the subject.

[0183] In S2401, the bit depth conversion unit 402 applies multiple different functions to the input data, which includes an N-bit image, and then quantizes it to, for example, convert the bit depth of the image to M bits, which is less than N bits.

[0184] Figure 24 This is a flowchart illustrating the bit depth conversion process in S2401. In S2501, the bit depth conversion unit 402 applies a first function, which is a non-linear function, to the image. In S2502, the bit depth conversion unit 402 applies a second function to the image, which is a different function from the first function and is a non-linear function. The non-linear function can be, for example, the function shown in formula (2).

[0185] ...(2)

[0186] Here, x is the input pixel value. α and β are each predetermined parameters. sign(·) is a function that returns the sign of the input data. The first function and the second function have different values ​​of α and β. Therefore, the bit depth conversion unit 402 can achieve different bit depth conversions for the image by applying the first function and the second function with different parameters α and β to the image.

[0187] In S2503, the bit depth conversion unit 402 quantizes the pixel values ​​of the image converted in S2501. In S2504, the bit depth conversion unit 402 quantizes the pixel values ​​of the image converted in S2502. Thus, the bit depth conversion unit 402 converts the bit depth of the image into a lower bit depth and generates two different types of data.

[0188] Note that in S2501 and S2502, although the bit-depth transformation unit 402 applies two functions as multiple different transformations, there is no limitation on the type and number of functions to be applied. Furthermore, the bit-depth transformation unit 402 can apply different LUTs instead of functions. The LUTs are generated, for example, based on the values ​​of the first and second functions. The parameters of the functions and the values ​​of the LUTs can be determined through learning for noise reduction.

[0189] Next, in S2402, the input integration unit 403 integrates the M-bit image acquired by the image acquisition unit 501 in S2501. Integration can be performed using the same method as in the first embodiment. For example, integration can be performed by cascading the M-bit image in the channel direction.

[0190] Figure 25 It is a diagram that integrates the images of each input data point. Figure 25 Images 2601 and 2602 are multiple M-bit images obtained in S2401. Integration input 2603 shows the integration state of images 2601 and 2602 in the channel direction. In other words, integration input 2603 can be considered the output of the input integration unit 403.

[0191] Next, in S2403, the feature extraction unit 404 extracts features from the integrated input image. The feature extraction unit 404 can use convolutional neural networks (hereinafter abbreviated as CNN) or the like to extract features.

[0192] Figure 26 This diagram illustrates the process performed by the CNN for feature extraction in S2403. The CNN can be a CNN used for noise reduction. Feature extraction unit 404 performs convolution in S2701, ReLU in S2702, and max pooling in S2703 to extract features while reducing spatial resolution. Feature extraction unit 404 performs convolution in S2704, ReLU in S2705, and max pooling in S2706 to extract features while further reducing spatial resolution. Subsequently, feature extraction unit 404 increases the resolution through upsampling in S2707. Subsequently, feature extraction unit 404 extracts features through convolution in S2708 and upsampling in S2709.

[0193] Subsequently, feature extraction unit 404 performs convolution in S2710 and ReLU in S2711 to extract features. Feature extraction unit 404 then performs convolution in S2712 (different from the convolution in S2710) and ReLU in S2713 (different from the ReLU in S2711) to extract features. Thus, feature extraction unit 404 outputs two types of M-bit features. These two types of features are referred to as the first feature and the second feature. Note that feature extraction unit 404 can output three or more types of features, and the output of feature extraction unit 404 can be appropriately modified.

[0194] Next, in S2404, the bit depth extension unit 405 integrates the multiple features obtained in S2403 and transforms the bit depth by extending it.

[0195] Figure 27 This diagram illustrates an example of integrating features and expanding the bit depth in S2404. The bit depth expansion unit 405 can obtain an M×N bit output by calculating the element-wise product of the first feature 2801 and the second feature 2802, each obtained in M-bit form. Here, N is the number of features for which the element-wise product is calculated. In this embodiment, N is 2. The bit depth expansion unit 405 can output the M×N bit output as is, or it can convert it to K bits and output it by shifting the bits, etc.

[0196] Non-Patent Literature 3 (Non-Patent Literature 3: Liangyu Chen et al., “Simple Baseline for Image Restoration”, in arXiv 2022) describes in detail a method for performing noise reduction learning on an image.

[0197] Effects of the second embodiment

[0198] In the second embodiment, element-wise multiplication and the like are used to integrate multiple features output from the feature extraction unit 404. Thus, the second embodiment enables bit depth conversion without separating high-order and low-order bits. Therefore, even when the bit depth that the feature extraction unit 404 can handle is lower than the bit depth of the input image, noise reduction can be performed while suppressing a decrease in accuracy.

[0199] The above embodiments can be combined appropriately. When combining embodiments, configurations such as user-selectable functions can be adopted.

[0200] According to this disclosure, even if the bit depth of the input data is reduced, the reduction in output precision can be suppressed.

[0201] Other embodiments

[0202] The embodiments of this disclosure can also be implemented by a computer that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more embodiments described above and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing the functions of one or more embodiments described above, and by a method executed by a computer of a system or device, for example, by reading and executing computer-executable instructions from a storage medium to perform the functions of one or more embodiments described above and / or controlling one or more circuits to perform the functions of one or more embodiments described above. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessor unit (MPU)) and may include a network of individual computers or individual processors to read and execute computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or storage medium. The storage medium may include, for example, a hard disk, random access memory (RAM), read-only memory (ROM), storage devices for distributed computing systems, optical discs (such as CDs, DVDs, or Blu-ray discs). TM One or more of the following: flash memory devices, memory cards, etc.

[0203] Other embodiments

[0204] The embodiments of the present invention can also be implemented by the following method: providing software (including computer program products of computer programs) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU) or microprocessor unit (MPU) of the system or device) reads and executes the computer program.

[0205] While this disclosure has been described with reference to embodiments, it should be understood that this disclosure is not limited to the disclosed embodiments. The scope of the following claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. An information processing device, characterized in that, include: The acquisition component is used to acquire multiple input data points with a first-order depth. The first conversion unit is used to convert the bit depth of an input data into a second bit depth that is lower than the first bit depth. The second conversion unit is used to convert the bit depth of another input data into a second bit depth, and to extract features from the data obtained by integrating the input data whose bit depth has been converted to the second bit depth and the other input data whose bit depth has been converted to the second bit depth. as well as An extension component is used to extend the bit depth of a feature to a third bit depth, which is higher than the second bit depth.

2. The information processing device according to claim 1, in, Bit depth conversion is performed using a first nonlinear transformation.

3. The information processing apparatus according to claim 1 is trained based on the output of a feature whose bit depth is expanded to a third bit depth and a preset true value.

4. The information processing device according to claim 1, in, Features are decomposed and integrated along the channel direction.

5. The information processing apparatus according to claim 4, in, Features are integrated by calculating the element-wise product of the decomposed features.

6. The information processing apparatus according to claim 4, in, A second nonlinear transformation is applied to the decomposed features.

7. The information processing apparatus according to claim 6, in, The bit depth of the feature is extended by calculating the element-wise sum of the feature to which the second nonlinear transformation has been applied.

8. The information processing apparatus according to claim 7, in, The bit depth of the feature is transformed into a third bit depth by a first nonlinear transformation, which is the inverse of the second nonlinear transformation.

9. The information processing apparatus according to claim 1, wherein the parameters are generated by learning.

10. The information processing apparatus according to claim 1, in, The multiple input data can be integrated by any of the following: concatenation of the multiple input data in the channel direction; concatenation of the multiple input data in the spatial direction; and concatenation of the multiple input data by element-wise summation.

11. The information processing apparatus according to claim 1, in, Multiple input data with different bit depths were obtained.

12. The information processing apparatus according to claim 11, The bit depth of the multiple input data is unified through bit depth conversion.

13. The information processing apparatus according to claim 1, in, Multiple input data points, including the captured image of the subject, are obtained, and Information about the defocus range of the subject is output.

14. The information processing apparatus according to claim 13, in, The bit depth of the characteristics of the defocus range of various parts of the subject is expanded and output.

15. The information processing apparatus according to claim 1, in, Multiple input data points, including a first input data point and a second input data point with a higher bit depth than the first input data point, are obtained, and The bit depth of the second input data is transformed.

16. The information processing apparatus according to claim 1, in, The bit depth of one or more of the input data is transformed using multiple different types of transformations.

17. An information processing method, characterized in that, include: Obtain multiple input data points with the first-order depth; Convert the bit depth of an input data point to a second bit depth that is lower than the first bit depth. The bit depth of another input data is converted to a second bit depth, and features are extracted from the data obtained by integrating the first input data whose bit depth has been converted to the second bit depth and the other input data whose bit depth has been converted to the second bit depth. as well as Expand the bit depth of the feature to a third bit depth, which is higher than the second bit depth.

18. A non-transitory computer-readable storage medium storing a computer program, characterized in that, When the computer program is read and executed by the computer, it causes the computer to perform the information processing method according to claim 17.

19. A computer program product comprising a computer program, characterized in that, When the computer program is read and executed by the computer, it causes the computer to perform the information processing method according to claim 17.

Images