Image processing device and method

Abstract

The present disclosure relates to an image processing device and method that make it possible to suppress a reduction in the quality of an image after defect correction while suppressing an increase in the amount of data outputted by an image sensor. The present invention converts the format of defect information about defective pixels in a pixel array that detects brightness and generates defect information for transmission, encodes image data that represents a brightness distribution detected at the pixel array and generates encoded image data at a bitrate set on the basis of a bitrate for the defect information for transmission, and multiplexes the defect information for transmission and the encoded image data and generates a bitstream. The present disclosure can be applied, for example, to an image processing device, an electronic apparatus, an image processing method, or a program.

Description

Image processing device and method

[0001] The present disclosure relates to an image processing device and method, and more particularly to an image processing device and method that can suppress a decrease in image quality after defect correction while suppressing an increase in the amount of image sensor output data.

[0002] In the past, the pixel array of an image sensor contained defective pixels whose pixel values ​​could not be detected correctly, and as a result, the image detected by the image sensor contained defective pixels whose pixel values ​​were abnormal. These pixel defects were classified into static pixel defects that occurred during the manufacturing of the image sensor and dynamic pixel defects that occurred for some reason after the manufacturing of the image sensor.

[0003] There is a method in which an image sensor grasps and stores the positions of such defective pixels, and corrects the pixel defects contained in the image based on that position information before outputting it to the outside (see, for example, Patent Document 1).

[0004] In response to this issue, recent advances in semiconductor technology have led to higher processor performance, and efforts are being made to correct such pixel defects in processors downstream of the image sensor. In such cases, defect information indicating the location of defective pixels must be transmitted from the image sensor to the downstream processor along with the image data. However, it can be difficult to increase the transmission bandwidth and transmission time. Therefore, a method has been devised to suppress the increase in data volume due to the transmission of defect information by combining, encoding, and transmitting the image data and defect information.

[0005] Japanese Patent Application Laid-Open No. 2021-61487

[0006] However, in this case, there is a risk that the defect information will be degraded, and the quality of the image after defect correction will be reduced.

[0007] The present disclosure has been made in consideration of such circumstances, and makes it possible to suppress a decrease in image quality after defect correction while suppressing an increase in the amount of image sensor output data.

[0008] An image processing device according to one aspect of the present technology is an image processing device that includes a defect information conversion unit that converts a format of defect information related to defective pixels in a pixel array for detecting luminance and generates defect information for transmission; an image encoding unit that encodes image data representing the distribution of luminance detected in the pixel array and generates image encoding data with a bit rate that is set based on the bit rate of the defect information for transmission; and a multiplexing unit that multiplexes the defect information for transmission and the image encoding data to generate a bit stream.

[0009] An image processing method according to one aspect of the present technology includes converting a format of defect information relating to defective pixels in a pixel array for detecting luminance to generate defect information for transmission, encoding image data representing the distribution of the luminance detected in the pixel array to generate image coding data having a bit rate set based on the bit rate of the defect information for transmission, and multiplexing the defect information for transmission and the image coding data to generate a bit stream.

[0010] According to another aspect of the present technology, an image processing device includes a demultiplexing unit that demultiplexes a bit stream and extracts transmission defect information and image coding data, a defect information inverse conversion unit that inversely converts the format of the transmission defect information and generates defect information related to defective pixels in a pixel array for detecting luminance, and an image decoding unit that decodes the image coding data using a bit rate of the image coding data estimated based on the bit rate of the transmission defect information, and generates image data representing the distribution of the luminance detected in the pixel array.

[0011] An image processing method according to another aspect of the present technology includes demultiplexing a bit stream to extract defect information for transmission and image coding data, inversely converting a format of the defect information for transmission to generate defect information related to defective pixels in a pixel array for detecting luminance, and decoding the image coding data using a bit rate of the image coding data estimated based on a bit rate of the defect information for transmission to generate image data representing the distribution of the luminance detected in the pixel array.

[0012] In an image processing device and method according to one aspect of the present technology, the format of defect information relating to defective pixels in a pixel array for detecting luminance is converted, defect information for transmission is generated, image data representing the distribution of luminance detected in the pixel array is encoded, image coding data having a bit rate set based on the bit rate of the defect information for transmission is generated, and the defect information for transmission and the image coding data are multiplexed to generate a bit stream.

[0013] In another aspect of the image processing device and method of the present technology, a bit stream is demultiplexed, transmission defect information and image coding data are extracted, the format of the transmission defect information is inversely converted, defect information regarding defective pixels in a pixel array for detecting luminance is generated, the image coding data is decoded using a bit rate of the image coding data estimated based on the bit rate of the transmission defect information, and image data representing the distribution of luminance detected in the pixel array is generated.

[0014] 1 is a block diagram showing an example of the main configuration of an image sensor. 2 is a block diagram showing an example of the main configuration of an image sensor and a signal processing unit. 3 is a diagram showing an example of pixel values. 4 is a diagram showing an example of defect information. 5 is a diagram showing an example of defect information. 6 is a diagram showing an example of a method for transmitting defect information for transmission and image data. 7 is a block diagram showing an example of the main configuration of an imaging device. 8 is a block diagram showing an example of the main configuration of an encoding unit. 9 is a block diagram showing an example of the main configuration of a decoding unit. 10 is a flowchart explaining an example of the flow of an imaging process. 11 is a flowchart explaining an example of the flow of an encoding process. 12 is a flowchart explaining an example of the flow of a decoding process. 13 is a diagram showing an example of an interface for transmitting image data in units of frames. 14 is a diagram showing an example of a block. 15 is a flowchart explaining an example of the flow of an encoding process. 16 is a flowchart explaining an example of the flow of a decoding process. 17 is a diagram showing an example of an interface for transmitting image data in units of frames. 18 is a diagram showing an example of a block. 19 is a flowchart explaining an example of the flow of an encoding process. 20 is a flowchart explaining an example of the flow of a decoding process. 21 is a diagram showing an example of an example of a bit stream. 22 is a diagram showing an example of a bit stream. 23 is a diagram showing an example of a syntax of defect information for transmission. 24 is a diagram showing an example of a syntax of defect information for transmission. 25 is a diagram showing an example of a syntax of defect information for transmission. 26 is a diagram showing an example of a bit stream. 27 is a diagram showing an example of a bit stream. 28 is a diagram showing an example of pre-processing. 29 is a block diagram showing an example of the main configuration of an encoding unit. 29 is a flowchart explaining an example of the flow of an encoding process. FIG. 1 is a block diagram showing an example of the main configuration of a decoding unit. FIG. 2 is a flowchart illustrating an example of the flow of decoding processing. FIG. 3 is a block diagram showing an example of the main configuration of a post-processing unit. FIG. 4 is a flowchart illustrating an example of the flow of post-processing. FIG. 4 is a block diagram showing an example of the main configuration of a post-processing unit. FIG. 5 is a flowchart illustrating an example of the flow of post-processing. FIG. 5 is a block diagram showing an example of the main configuration of a post-processing unit. FIG. 6 is a flowchart illustrating an example of the flow of post-processing. FIG. 6 is a diagram showing an example of a method for replacing defective pixel values. FIG. 7 is a block diagram showing an example of the main configuration of a coding unit. FIG. 7 is a flowchart illustrating an example of the flow of coding processing. FIG. 8 is a block diagram showing an example of the main configuration of a coding unit. FIG. 8 is a flowchart illustrating an example of the flow of coding processing. FIG. 9 is a block diagram showing an example of the main configuration of a decoding unit. FIG. 9 is a flowchart illustrating an example of the flow of decoding processing.1 is a block diagram showing an example of the main configuration of an image processing system;

[0015] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. Literature etc. supporting technical content and technical terminology 2. Pixel defects 3. Transmission of defect information for transmission and image data 4. Replacement of defective pixel values ​​5. Image processing system 6. Supplementary notes

[0016] <1. Literature, etc. supporting technical content and technical terminology> The scope of what is disclosed in the present technology includes not only the content described in the embodiments, but also the content described in the following patent documents, etc. that were publicly known at the time of filing, and the content of other documents referenced in the following patent documents.

[0017] Patent Document 1: (mentioned above)

[0018] In other words, the contents of the above-mentioned patent documents and the contents of other documents referenced in the above-mentioned patent documents are also used as the basis for determining the support requirements.

[0019] <2. Pixel Defects> <Correction of Pixel Defects in Image Sensors> Conventionally, the pixel array of an image sensor may contain defective pixels whose pixel values ​​cannot be correctly detected, and as a result, the image detected by the image sensor may contain pixel defects whose pixel values ​​are abnormal. These pixel defects are classified into static pixel defects that occur during the manufacture of the image sensor and dynamic pixel defects that occur for some reason after the manufacture of the image sensor.

[0020] Such pixel defects are classified into several types depending on the cause and manner of occurrence, such as point defects, stain defects, unevenness defects, and line defects. A point defect refers to a state in which the pixel value of one image is significantly different from the image values ​​of the eight surrounding pixels, resulting in a protruding (or depressed) value. A stain defect refers to a state in which each of multiple pixel values ​​in a certain area of ​​an image differs from the surrounding pixel values ​​by a smaller amount than the difference in the pixel value of a point defect. A unevenness defect refers to a state in which multiple pixels with even smaller pixel value differences than the pixels in a stain defect are gathered in a wider area than the stain defect. A line defect refers to a state in which pixel values ​​aligned in a column direction, row direction, or diagonal direction at an arbitrary angle in an image are significantly different from the surrounding pixel values, resulting in a protruding (or depressed) value.

[0021] For example, as described in Patent Document 1, there is a method in which an image sensor grasps and stores the positions of such defective pixels, and corrects pixel defects contained in an image based on that position information before outputting the image to the outside.

[0022] For example, in the image sensor 10 shown in FIG. 1 , the brightness detection unit 11 is configured with a pixel array, and each pixel photoelectrically converts incident light to generate a brightness signal, outputting image data of the captured image. A defect detector (static) 21 detects pixel defects in the image data during the manufacture of the image sensor 10 and stores the defect locations (positions of defective pixels) in the storage unit 13 of the image sensor 10. After the manufacture, a defect detection unit (dynamic) 12 of the image sensor 10 detects pixel defects at a predetermined timing and stores the defect locations (positions of defective pixels) in the storage unit 13. A defect correction unit 14 corrects the pixel values ​​of defective pixels in the image data (RAW image) output from the brightness detection unit 11, as indicated by the defect information supplied from the storage unit 13. The defect correction unit 14 then outputs the defect-corrected RAW image (i.e., image data in which the pixel values ​​of the defective pixels have been corrected) from the image sensor 10.

[0023] <Pixel Defect Correction in a Post-Stage Processor> In recent years, advances in semiconductor technology have led to increased processor performance, and efforts have been made to correct such pixel defects in a processor downstream of an image sensor. For example, as shown in FIG. 2 , a signal processing unit 30 is provided downstream of an image sensor 10, and pixel defects are corrected in a defect correction unit 33 of the signal processing unit 30. In this case, defect information, which is information about defective pixels, must be transmitted from the image sensor 10 to the signal processing unit 30 along with the image data. That is, the multiplexing unit 15 multiplexes the RAW image output from the brightness detection unit 11 with the defect information stored in the storage unit 13, and supplies the multiplexed signal to the signal processing unit 30 via a bus. The demultiplexing unit 31 of the signal processing unit 30 receives and demultiplexes the multiplexed signal to extract the defect information and the RAW image. The defect information is stored in the storage unit 32 and then supplied to the defect correction unit 33 at a predetermined timing. The defect correction unit 33 corrects the pixel defects in the RAW image based on the defect information, and outputs the defect-corrected RAW image from the signal processing unit 30.

[0024] <Defect Information> In this way, when pixel defects are corrected in the signal processing unit 30 , defect information is transmitted from the image sensor 10 to the signal processing unit 30 together with the image data.

[0025] The defect information includes map information of defective pixels corresponding to the pixel array of the brightness detection unit 11 to indicate the positions of the defective pixels. For example, assume that a RAW image like the one shown in FIG. 3 is output from the brightness detection unit 11. In FIG. 3, each square indicates information on a pixel-by-pixel basis, and the value inside the square indicates the pixel value. A pixel value of 1023 on a white background or a pixel value of 0 on a black background indicates that the pixel value of a defective pixel always outputs a saturated value (abnormal value).

[0026] In such a case, the defect detector 21 or the defect detection unit 12 distinguishes between defective and non-defective pixels and assigns an ID (binary notation) to each pixel, as shown in A of FIG. 4. For example, an ID of "0" (binary notation: "0") is assigned to normal pixels that are not defective, and an ID of "1" (binary notation: "1") is assigned to defective pixels. In other words, defect information such as that shown in B of FIG. 4 is generated for the brightness detection unit 11 that outputs the RAW image of FIG. 3. In this defect information, a value of "1" indicates a defective pixel, so the defect correction unit 33 corrects the pixel value at that position.

[0027] The defect information may also indicate the type of defect (e.g., point defect, spot defect, unevenness defect, line defect, etc.). For example, as shown in A of FIG. 5, different IDs may be assigned to no defect, point defect, spot defect or unevenness defect, and line defect, respectively. In this case, the ID is 2-bit information in binary notation. That is, defect information such as that shown in B of FIG. 5 is generated for the brightness detection unit 11 that outputs the RAW image of FIG. 3. In this defect information, values ​​"01," "10," and "11" indicate defective pixels, so the defect correction unit 33 corrects the pixel values ​​at those positions.

[0028] <Encoding of Multiplexed Signal> When pixel defects are corrected in the signal processing unit 30 as in the example of Figure 2, the amount of data transmitted from the image sensor 10 to the signal processing unit 30 increases by the amount of defect information transmitted. However, it has sometimes been difficult to increase the transmission bandwidth or transmission time. For example, when the configuration of Figure 2 is implemented within an imaging device, the bandwidth of the bus between the image sensor 10 and the signal processing unit 30 is generally fixed at the time of design. Furthermore, if there is a time lag limit, such as with moving images, the transmission time of RAW images via that bus is also limited. In this way, when the bandwidth is fixed and there is no room for transmission time, it has been difficult to increase them.

[0029] For example, while it is possible to mitigate bandwidth shortages by designing the bus bandwidth to be sufficiently wide, this could result in increased costs. In particular, the number of pixel defects varies from image sensor to image sensor. The I / F bandwidth must be designed to accommodate the worst-case number of defects, and when there are few or no pixel defects, only information indicating "no pixel defects" is transmitted. Expanding the bus width to accommodate this inefficient transmission of defect information could also be cost-inefficient.

[0030] In such cases, a method for suppressing the increase in data volume due to the transmission of defect information can be considered by encoding and transmitting a multiplexed signal that combines image data and defect information. However, with lossless encoding, it is difficult to sufficiently reduce the data volume of image data within an allowable short processing time. Therefore, relatively simple lossy encoding, such as reducing the bit length, is generally applied. When such lossy encoding is applied to a multiplexed signal, there is a risk of the defect information deteriorating (changing). When the defect information changes, there is a risk of the accuracy of information such as the location of defective pixels indicated by the defect information decreasing. This may result in a decrease in the accuracy of pixel defect correction performed based on the defect information. For example, there is a risk of problems occurring during pixel defect correction, such as not correcting the pixel values ​​of defective pixels or correcting the pixel values ​​of normal pixels. This may result in a decrease in the quality of the image after defect correction.

[0031] 6, the defect information is converted into defect information for transmission, and the image encoding rate is set based on the rate of the defect information for transmission (Method 1). That is, in order to reduce the data volume of the defect information, the defect information is converted into defect information for transmission using a reversible method. Then, based on the bit rate of the defect information for transmission, the bit rate after encoding of the RAW image is set so as not to increase the bit rate of the entire image (multiplexed signal), and the RAW image is encoded according to this setting.

[0032] For example, the first image processing device may include a defect information conversion unit that converts a format of defect information related to defective pixels in a pixel array for detecting luminance and generates defect information for transmission, an image encoding unit that encodes image data representing the distribution of luminance detected in the pixel array and generates image coded data with a bit rate set based on the bit rate of the defect information for transmission, and a multiplexing unit that multiplexes the defect information for transmission and the image coded data to generate a bit stream. Furthermore, the first image processing method may include the first image processing device converting a format of defect information related to defective pixels in a pixel array for detecting luminance and generating defect information for transmission, the first image processing device encoding image data representing the distribution of luminance detected in the pixel array and generating image coded data with a bit rate set based on the bit rate of the defect information for transmission, and multiplexing the defect information for transmission and the image coded data to generate a bit stream. The first program also causes the computer to perform processes including converting the format of defect information related to defective pixels in a pixel array for detecting luminance, generating defect information for transmission, encoding image data representing the distribution of luminance detected in the pixel array, generating image coding data with a bit rate set based on the bit rate of the defect information for transmission, and multiplexing the defect information for transmission and the image coding data to generate a bit stream.

[0033] In this way, the first image processing device can suppress an increase in the amount of data to be transmitted while suppressing a decrease in the accuracy of defect information, i.e., the first image processing device can suppress a decrease in the quality of the image after defect correction while suppressing an increase in the amount of image sensor output data.

[0034] For example, the second image processing device may include a demultiplexing unit that demultiplexes the bit stream and extracts transmission defect information and image coded data, a defect information inverse conversion unit that inversely converts the format of the transmission defect information and generates defect information related to defective pixels in a pixel array for detecting luminance, and an image decoding unit that decodes the image coded data using a bit rate estimated based on the bit rate of the transmission defect information to generate image data representing the distribution of luminance detected in the pixel array. Furthermore, the second image processing method may include the second image processing device demultiplexing the bit stream and extracting the transmission defect information and image coded data, the second image processing device inversely converting the format of the transmission defect information to generate defect information related to defective pixels in the pixel array for detecting luminance, and the second image processing device decoding the image coded data using the bit rate estimated based on the bit rate of the transmission defect information to generate image data representing the distribution of luminance detected in the pixel array. In addition, the second program causes the computer to perform processes including demultiplexing the bit stream to extract transmission defect information and image coding data, inversely converting the format of the transmission defect information to generate defect information related to defective pixels in a pixel array for detecting luminance, and decoding the image coding data using a bit rate of the image coding data estimated based on the bit rate of the transmission defect information to generate image data representing the distribution of luminance detected in the pixel array.

[0035] In this way, the second image processing device can suppress an increase in the amount of data to be transmitted while suppressing a decrease in the accuracy of defect information, i.e., the second image processing device can suppress a decrease in the quality of the image after defect correction while suppressing an increase in the amount of image sensor output data.

[0036] <Imaging Device> The present technology can be applied to any device. For example, the present technology can be applied to an imaging device that captures an image of a subject. Fig. 7 is a block diagram showing an example of the configuration of an imaging device that is one aspect of an image processing device to which the present technology is applied. The imaging device 100 shown in Fig. 7 has an image sensor 110 and a signal processing unit 120, and is a device that captures an image of a subject using the image sensor 110 and performs signal processing on the RAW image (captured image) using the signal processing unit 120.

[0037] Fig. 7 shows the main processing units, data flows, and the like related to the present technology, but is not limited to all that is shown in Fig. 7. In other words, the imaging device 100 may have processing units not shown in Fig. 7, may exchange data not shown in Fig. 7, or may execute any processing.

[0038] The image sensor 110 includes a brightness detection unit 111, a defect detection unit (Dynamic) 112, a storage unit 113, and an encoding unit 114. The signal processing unit 120 includes a decoding unit 121 and a defect correction unit 122. The image sensor 110 and the signal processing unit 120 are connected via a bus 130.

[0039] The luminance detection unit 111 has a pixel array, and converts incident light into a luminance value at each pixel by photoelectrically converting the luminance amount at that pixel. The luminance detection unit 111 supplies image data of the captured image (RAW image) including the luminance value obtained at each pixel to the encoding unit 114.

[0040] The defect detector (static) 141 detects pixel defects in the RAW image of the image sensor 110 during its manufacture, and stores the defect positions (positions of defective pixels) in the storage unit 113 of the image sensor 110. For example, the brightness detection unit 111 captures an image of a test pattern or the like, and the defect detector 141 determines whether or not a pixel defect has occurred in the captured image, and stores the positions of the defective pixels, the type of defect, and the like in the storage unit 113. In general, this pixel defect detection is often processed outside the image sensor 110 (imaging device 100).

[0041] The defect detection unit (Dynamic) 112 detects pixel defects in the RAW image at a predetermined timing after manufacture and stores the defect positions in the storage unit 113. The defect detection unit 112 detects pixel defects when pixel values ​​are corrupted for some reason after the shipment of the imaging device 100. Similar to the defect detector 141, the defect detection unit 112 detects whether pixel defects have occurred in the captured image and stores the positions of the defective pixels, the types of defects, and the like in the storage unit 113.

[0042] The storage unit 113 has a storage medium such as a semiconductor memory, and stores defect information such as the position of a defective pixel and the type of defect supplied from the defect detector 141 and the defect detection unit 112. The storage unit 113 supplies the stored defect information to the encoding unit 114 at a predetermined timing or based on a request from the encoding unit 114 or the like.

[0043] The encoding unit 114 executes processing related to encoding. For example, the encoding unit 114 may acquire captured image data (RAW image data) supplied from the brightness detection unit 111. The encoding unit 114 may read and acquire defect information stored in the storage unit 113. The encoding unit 114 may encode the acquired RAW image data to generate image encoded data. Furthermore, the encoding unit 114 may generate a bit stream including the image encoded data and the defect information, and supply the bit stream to the signal processing unit 120 (the decoding unit 121) via the bus 130.

[0044] The decoding unit 121 executes processing related to decoding. For example, the decoding unit 121 may acquire a bit stream supplied from (the encoding unit 114 of) the image sensor 110 via the bus 130. The decoding unit 121 may decode the bit stream to generate defect information and RAW image data, and supply them to the defect correction unit 122.

[0045] The defect correction unit 122 performs processing related to correction of pixel values ​​of defective pixels. For example, the defect correction unit 122 may acquire defect information and RAW image data supplied from the decoding unit 121. The defect correction unit 122 may correct pixel defects contained in the RAW image data based on the defect information. For example, the defect correction unit 122 may refer to pixel values ​​of pixels (also referred to as surrounding pixels) surrounding the defective pixel indicated by the defect information, predict true pixel values ​​that would be obtained if the pixel defect did not occur, and replace the predicted values ​​with the pixel values ​​of the defective pixel in the RAW image data. The defect correction unit 122 may output such defect-corrected RAW image data to the outside of the signal processing unit 120.

[0046] <Encoding Unit> Fig. 8 is a diagram showing an example of the main configuration of the encoding unit 114. As shown in Fig. 8, the encoding unit 114 (first image processing device) may have a defect information conversion unit 151, an image encoding unit 152, and a multiplexing unit 153.

[0047] The defect information conversion unit 151 performs processing related to the conversion of defect information. For example, the defect information conversion unit 151 may read and acquire defect information stored in the storage unit 113. This defect information includes information related to defective pixels detected by the brightness detection unit 111. For example, this defect information may include information indicating the position of the defective pixel and information indicating the type of defect. The defect information conversion unit 151 may convert the format of the acquired defect information to generate defect information for transmission. In other words, the defect information conversion unit 151 may convert the format of defect information related to defective pixels in a pixel array for detecting brightness, and generate defect information for transmission.

[0048] The defect information for transmission is information obtained by converting the format of the defect information in a reversible manner and includes at least the content of the defect information. The defect information conversion unit 151 may supply the generated defect information for transmission to the multiplexing unit 153. The defect information conversion unit 151 may also supply the bit rate of the defect information for transmission to the image encoding unit 152. The defect information conversion unit 151 may also supply the bit rate of the defect information for transmission to the multiplexing unit 153.

[0049] The image encoding unit 152 performs processing related to encoding of captured images (RAW images). For example, the image encoding unit 152 may acquire RAW image data supplied from the brightness detection unit 111. The image encoding unit 152 may encode the RAW image data to generate encoded image data. Essentially, any encoding method may be used. As long as the method satisfies the required conditions, such as processing speed and encoding efficiency, it may be a lossless method or a lossy method. For example, encoding methods such as COMP6, COMP7, COMP8, and MPC defined by MIPI (Mobile Industry Processor Interface) may be applied. In this case, the image encoding unit 152 may acquire the bit rate of the transmission defect information supplied from the defect information conversion unit 151 and set the bit rate of the encoded image data based on the acquired bit rate of the transmission defect information. In other words, the image encoding unit 152 may encode image data representing the distribution of brightness detected in the pixel array and generate encoded image data with a bit rate set based on the bit rate of the transmission defect information. The image encoding unit 152 may supply the generated encoded image data to the multiplexing unit 153.

[0050] The multiplexing unit 153 performs multiplexing processing. For example, the multiplexing unit 153 may acquire transmission defect information supplied from the defect information conversion unit 151. The multiplexing unit 153 may acquire image encoded data supplied from the image encoding unit 152. The multiplexing unit 153 may multiplex the transmission defect information and the image encoded data to generate a bit stream. Any method may be used to multiplex the transmission defect information and the image encoded data. For example, the transmission defect information may be added to the lower bits of the pixel-based image encoded data. The transmission defect information and the image encoded data may be linked so that the transmission defect information is processed first. Conversely, the transmission defect information and the image encoded data may be linked so that the transmission defect information is processed later. Of course, multiplexing methods other than these examples may also be used. The multiplexing unit 153 may also acquire the bit rate of the transmission defect information supplied from the defect information conversion unit 151 and store it in the bit stream. The multiplexing unit 153 may supply the generated bit stream to the signal processing unit 120 (the decoding unit 121 thereof) via the bus 130.

[0051] For example, the image encoding unit 152 may subtract the bit rate of the defect information for transmission from the target bit rate of the bit stream and use the difference as the bit rate of the encoded image data. By doing so, the image encoding unit 152 can set the bit rate of the encoded image data so that the bit rate of the bit stream does not increase (e.g., so that it is constant). Therefore, the encoding unit 114 can suppress an increase in the amount of image sensor output data. This can suppress an increase in transmission time and the occurrence of processing delays and failures. Furthermore, since there is no need to expand the bus bandwidth, an increase in costs can be suppressed. Furthermore, by transmitting the defect information for transmission, loss (deterioration) of defect information can be suppressed. In other words, the encoding unit 114 can suppress a decrease in the quality of the image after defect correction while suppressing an increase in the amount of image sensor output data.

[0052] <Decoding Unit> Fig. 9 is a diagram showing an example of the main configuration of the decoding unit 121. As shown in Fig. 9, the decoding unit 121 (second image processing device) may have a demultiplexing unit 171, a defect information inverse conversion unit 172, and an image decoding unit 173.

[0053] The demultiplexing unit 171 performs processing related to demultiplexing. For example, the demultiplexing unit 171 may acquire a bit stream supplied from (the multiplexing unit 153 of) the image sensor 110 via the bus 130. The demultiplexing unit 171 may demultiplex the bit stream to extract transmission defect information and image coding data. In this case, the demultiplexing unit 171 may acquire a bit rate of the transmission defect information supplied from the defect information inverse conversion unit 172 and demultiplex the bit stream based on the bit rate. The demultiplexing unit 171 may supply the extracted transmission defect information to the defect information inverse conversion unit 172. The demultiplexing unit 171 may supply the extracted image coding data to the image decoding unit 173.

[0054] The defect information inverse conversion unit 172 performs processing related to the inverse conversion of the transmission defect information. For example, the defect information inverse conversion unit 172 may acquire the transmission defect information supplied from the demultiplexing unit 171. The defect information inverse conversion unit 172 may inversely convert the format of the acquired transmission defect information to generate defect information. That is, the defect information inverse conversion unit 172 may inversely convert the format of the transmission defect information to generate defect information related to defective pixels in a pixel array for detecting luminance. Note that this "inverse conversion" is the inverse process of the defect information format conversion performed by the defect information conversion unit 151. Furthermore, the defect information format conversion performed by the defect information conversion unit 151 is performed in a reversible manner. Therefore, by inversely converting the format of the transmission defect information, the defect information inverse conversion unit 172 can generate (restore) defect information having the same content as the defect information before the format conversion performed by the defect information conversion unit 151. The defect information inverse conversion unit 172 may supply the generated defect information to the defect correction unit 122. The defect information inverse conversion unit 172 may supply the bit rate of the transmission defect information to be inversely converted to the demultiplexing unit 171. The defect information inverse conversion unit 172 may supply the bit rate of the transmission defect information to be inversely converted to the image decoding unit 173.

[0055] The image decoding unit 173 performs processing related to decoding of image coded data. For example, the image decoding unit 173 may acquire image coded data supplied from the demultiplexing unit 171. The image decoding unit 173 may decode the acquired image coded data to generate RAW image data. This decoding method may be any method that corresponds to the coding method applied by the image coding unit 152. In this case, the image decoding unit 173 may acquire the bit rate of the defect information for transmission supplied from the defect information inverse conversion unit 172, estimate the bit rate of the image coded data based on the bit rate, and decode the image coded data using the estimated bit rate. In other words, the image decoding unit 173 may decode the image coded data using the bit rate of the image coded data estimated based on the bit rate of the defect information for transmission, and generate image data representing the distribution of luminance detected in the pixel array. The image decoding unit 173 may supply the generated RAW image data to the defect correction unit 122.

[0056] For example, the image decoding unit 173 may subtract the bit rate of the transmission defect information from the bit rate of the bit stream and estimate the difference as the bit rate of the image encoding data. By doing so, the image decoding unit 173 can correctly estimate the bit rate of the image encoding data, which is set so that the bit rate of the bit stream does not increase (e.g., so that it is constant). Therefore, the decoding unit 121 can suppress an increase in the amount of image sensor output data. Therefore, an increase in transmission time can be suppressed, and the occurrence of processing delays and failures can be suppressed. Furthermore, since there is no need to expand the bus bandwidth, an increase in costs can be suppressed. Furthermore, by transmitting the transmission defect information, loss (deterioration) of defect information can be suppressed. In other words, the decoding unit 121 can suppress a decrease in the quality of the image after defect correction while suppressing an increase in the amount of image sensor output data.

[0057] <Flow of Image Capture Processing> An example of the flow of image capture processing executed by the image capture device 100 will be described with reference to the flowchart of FIG.

[0058] When the image capturing process is started, in step S101, the brightness detection unit 111 of the image sensor 110 captures an image of a subject and generates a captured image (RAW image).

[0059] In step S102, the encoding unit 114 performs encoding processing to encode the RAW image using the defect information from the brightness detection unit 111 and generate a bit stream. The encoding unit 114 supplies the bit stream to the signal processing unit 120 via the bus 130.

[0060] In step S103, the decoding unit 121 of the signal processing unit 120 acquires the bit stream transmitted via the bus 130, performs a decoding process, and decodes the bit stream to generate (restore) defect information and a RAW image.

[0061] In step S104, the defect correction unit 122 corrects pixel defects in the RAW image using the defect information.

[0062] When the process of step S104 is completed, the imaging process ends.

[0063] <Flow of Encoding Process> Next, an example of the flow of the encoding process executed in step S102 of FIG. 10 will be described with reference to the flowchart of FIG.

[0064] When the encoding process starts, in step S121, the defect information conversion unit 151 of the encoding unit 114 converts the format of the defect information from the luminance detection unit 111 into the format of defect information for transmission. That is, the defect information conversion unit 151 converts the format of defect information related to defective pixels in the pixel array for detecting luminance, and generates defect information for transmission.

[0065] In step S122, the image encoding unit 152 generates encoded image data by encoding the RAW image at a bit rate based on the bit rate of the generated defect information for transmission. That is, the image encoding unit 152 encodes image data representing the luminance distribution detected in the pixel array, and generates encoded image data at a bit rate set based on the bit rate of the defect information for transmission.

[0066] In step S123, the multiplexing unit 153 multiplexes the transmission defect information and the coded image data to generate a bit stream.

[0067] In step S124, the multiplexing unit 153 transmits the generated bit stream to the signal processing unit 120 via the bus 130.

[0068] When the process of step S124 is completed, the encoding process ends and the process returns to FIG.

[0069] By performing each process in this manner, the encoding unit 114 can suppress a decrease in the quality of the image after defect correction while suppressing an increase in the amount of image sensor output data, as described above.

[0070] <Flow of Decoding Process> Next, an example of the flow of the decoding process executed in step S103 of FIG. 10 will be described with reference to the flowchart of FIG.

[0071] When the decoding process starts, the demultiplexing unit 171 of the decoding unit 121 receives the bit stream transmitted from the image sensor 110 via the bus 130 in step S141.

[0072] In step S142, the demultiplexer 171 demultiplexes the bit stream and discriminates the transmission defect information from the coded image data.

[0073] In step S143, the defect information inverse converter 172 inversely converts the transmission defect information into defect information, i.e., the defect information inverse converter 172 inversely converts the format of the transmission defect information to generate defect information related to defective pixels in the pixel array for detecting luminance.

[0074] In step S144, the image decoding unit 173 decodes the image coded data at a bit rate based on the bit rate of the defect information for transmission to generate a RAW image. That is, the image decoding unit 173 decodes the image coded data using the bit rate of the image coded data estimated based on the bit rate of the defect information for transmission to generate image data representing the distribution of luminance detected in the pixel array.

[0075] When the process of step S144 ends, the decoding process ends and the process returns to FIG.

[0076] By performing each process in this manner, the decoding unit 121 can suppress a decrease in the quality of the image after defect correction while suppressing an increase in the amount of image sensor output data, as described above.

[0077] <Method 1-1> The defect information may be transmitted from the image sensor 110 to the signal processing unit 120 in any data unit. For example, the defect information may be transmitted as embedded data of CIS-2 defined by MIPI. In this CIS-2, as shown in FIG. 13 , embedded data (shaded areas in the figure) can be transmitted frame by frame before and after pixel data.

[0078] However, when defect information is transmitted on a frame-by-frame basis, a time lag occurs between the transmission of the defect information and the transmission of the pixel values. Therefore, the signal processing unit 120 must retain the acquired defect information until it is time to use it. Furthermore, with this method, defect information for one frame is transmitted all at once, so it is necessary to retain the defect information for that one frame. This increases the required memory capacity, potentially resulting in increased costs.

[0079] Therefore, when the above-mentioned method 1 is applied, defect information may be processed in predetermined block units as shown in the second row from the top of the table in FIG. 6 (method 1-1).

[0080] For example, in the first image processing device, the defect information conversion unit may convert the format of the defect information for each of a plurality of predetermined blocks into which a picture of image data is divided, thereby generating defect information for transmission. Also, in the second image processing device, the defect information inverse conversion unit may inversely convert the format of the defect information for each of a plurality of predetermined blocks into which a picture of image data is divided, thereby generating defect information.

[0081] The "block," which is the unit of processing this defect information, may be of any size and shape as long as it is a partial region within one frame (a region smaller than one frame). For example, a block may be a pixel region of eight pixels horizontally and two pixels vertically (8x2 pixels) as shown in FIG. 14A. Alternatively, a block may be a pixel region of two pixels horizontally and two pixels vertically (2x2 pixels) as shown in FIG. 14B. Alternatively, a block may be a pixel region of four pixels horizontally and two pixels vertically (4x2 pixels) as shown in FIG. 14C. Alternatively, a block may be a pixel region of eight pixels horizontally and one pixel vertically (8x1 pixel) as shown in FIG. 14D. Of course, blocks may be of other sizes and shapes.

[0082] In this way, the defect information is divided into blocks and supplied to the signal processing unit 120, thereby suppressing an increase in the amount of data held at one time in the signal processing unit 120. Also, an increase in the time lag between the timing of transmission and the timing of use of the defect information can be suppressed. Therefore, the first image processing device and the second image processing device can suppress an increase in the memory capacity required to hold the defect information. Therefore, an increase in costs can be suppressed.

[0083] Furthermore, the "block" that is the processing unit for this defect information may be the same as the processing unit for image coding (hereinafter also referred to as the coding block). That is, in the first image processing device, the block may be the same as the coding block that is the processing unit for coding image data by the image coding unit. Furthermore, in the second image processing device, the block may be the same as the coding block that is the processing unit for decoding image coding data by the image decoding unit (i.e., the processing unit for generating image data).

[0084] In this way, by making the processing unit of defect information and the processing unit of image encoding / decoding the same, it is possible to further suppress an increase in the time lag between the timing of transmission and the timing of use of defect information. Therefore, the first image processing device and the second image processing device can suppress an increase in the memory capacity required to store defect information, and therefore, an increase in costs can be suppressed.

[0085] When the processing unit (block) for the defect information processing and the image encoding / decoding are the same, the processing order (scanning order) of each pixel may be the same, which further reduces the time lag between the timing of transmitting the defect information and the timing of using it.

[0086] <Configuration> In this case, the configuration of the encoding unit 114 may be the same as the example shown in Fig. 8. Furthermore, the configuration of the decoding unit 121 may be the same as the example shown in Fig. 9.

[0087] <Flow of Encoding Process> An example of the flow of the encoding process executed in step S102 of FIG. 10 when defect information is processed for each block identical to the encoding block will be described with reference to the flowchart of FIG.

[0088] When the encoding process starts, the defect information conversion unit 151 of the encoding unit 114 converts the format of the defect information of the processing target block in the pixel array of the luminance detection unit 111 into the format of defect information for transmission in step S161.

[0089] In step S162, the image encoding unit 152 generates encoded image data by encoding the RAW image of the processing target block at a bit rate based on the bit rate of the generated defect information for transmission.

[0090] In step S163, the multiplexing unit 153 multiplexes the transmission defect information and the coded image data of the block to be processed to generate a bit stream.

[0091] In step S164, the multiplexing unit 153 transmits the generated bit stream of the block to be processed to the signal processing unit 120 via the bus 130.

[0092] When the process of step S164 ends, the encoding process ends and the process returns to Fig. 10. The encoding unit 114 performs such encoding process for each block.

[0093] By performing each process in this manner, the encoding unit 114 can suppress an increase in memory capacity for storing defect information, as described above.

[0094] <Flow of Decoding Process> An example of the flow of the decoding process executed in step S103 of FIG. 10 when defect information is processed for each block identical to the coded block will be described with reference to the flowchart of FIG.

[0095] When the decoding process starts, in step S181, the demultiplexing unit 171 of the decoding unit 121 receives the bit stream of the block to be processed transmitted from the image sensor 110 via the bus 130.

[0096] In step S182, the demultiplexer 171 demultiplexes the bit stream of the target block to discriminate between the transmission defect information and the coded image data for the target block.

[0097] In step S183, the defect information inverse converter 172 inversely converts the transmission defect information of the target block into defect information, i.e., the defect information inverse converter 172 inversely converts the format of the transmission defect information to generate defect information corresponding to the target block.

[0098] In step S184, the image decoding unit 173 decodes the image coded data of the target block at a bit rate based on the bit rate of the transmission defect information to generate a RAW image of the target block. That is, the image decoding unit 173 decodes the image coded data of the target block using the bit rate of the image coded data estimated based on the bit rate of the transmission defect information to generate image data of the target block.

[0099] When the process of step S184 ends, the decoding process ends and the process returns to Fig. 10. The decoding unit 121 executes such a decoding process for each block.

[0100] By performing each process in this manner, the decoding unit 121 can suppress an increase in memory capacity for storing defect information, as described above.

[0101] <Method 1-2> The format of the transmission defect information may be any format. For example, when the above-mentioned method 1 is applied, the defect information may be converted into transmission defect information in a format corresponding to the number of defective pixels, as shown in the third row from the top of the table in FIG. 6 (Method 1-2). In other words, as shown in the example in the table in FIG. 17, the format of the transmission defect information may be different depending on whether the number of defective pixels contained in the RAW image (the processing target block) is 0, 1, 2, or 3 or more. In other words, the format of the transmission defect information may depend on the number of defective pixels present in the pixel array.

[0102] For example, the smaller the number of defective pixels in the pixel array (or the target block), the smaller the data volume of the transmission defect information. In the example of FIG. 17, the smaller the number of defective pixels, the shorter the bit string of the transmission defect information. Generally, an image sensor 110 in a normal state has a small number of defective pixels. In other words, it is highly likely that a format with a short bit string will be applied. Therefore, by using such a format, it is possible to prevent an increase in the data volume of the transmission defect information. Therefore, it is possible to prevent a reduction in the bit rate allocated to the image encoding data, and to prevent a reduction in the quality of the RAW image due to encoding and decoding.

[0103] <Defect Information for Transmission> FIG. 18A shows an example of the syntax of the defect information for transmission when the defect information is processed for each 8x2 pixel block (FIG. 14A). In this syntax example, "defectNum" is a parameter indicating the number of defective pixels in the block. "defectPoxX" is a parameter indicating the X coordinate of the defective pixel (the horizontal position within the block). "defectPosY" is a parameter indicating the Y coordinate of the defective pixel (the vertical position within the block). "defectPosBitMap" indicates map information in which the "identification values ​​indicating whether a pixel is defective" assigned to each pixel in the block are arranged according to the pixel arrangement (or scan order). In this way, parameters such as "defectPoxX," "defectPosY," and "defectPosBitMap" are set as necessary in the defect information for transmission. These parameters indicate the position of the defective pixel. In addition, a bit pattern (VLC code value) indicating the number of defective pixels is set in the defect information for transmission, as shown in FIG. 18B. In this way, the defect information to be transmitted may include first information indicating the number of defective pixels and second information indicating the positions of the defective pixels.

[0104] As shown in the example of B in FIG. 18 , the first information indicating the number of defective pixels may include a predetermined bit pattern corresponding to the number of defective pixels. The bit pattern may be shorter as the number of defective pixels decreases. Generally, an image sensor 110 in a normal state has a small number of defective pixels. In other words, a short bit pattern is likely to be applied. This makes it possible to prevent an increase in the amount of data for the defect information to be transmitted. This makes it possible to prevent a reduction in the bit rate allocated to the encoded image data, thereby preventing a reduction in the quality of the RAW image due to encoding and decoding.

[0105] The method of expressing the positions of defective pixels may vary depending on the number of defective pixels. For example, as shown in the syntax of A in FIG. 18 , when the number of defective pixels is two or less, the positions of the defective pixels may be indicated by coordinates. When the number of defective pixels is three or more, the positions of the defective pixels may be indicated by map information. In other words, when the second information indicating the positions of defective pixels indicates that the number of defective pixels is less than a predetermined standard, the positions of the defective pixels may be indicated by coordinates. When the second information indicates that the number of defective pixels is equal to or greater than a predetermined standard, the positions of the defective pixels may be indicated by a map showing the distribution of defective pixels.

[0106] 17, if there is no defective pixel, the second information indicating the position of the defective pixel may be omitted. In other words, if there is no defective pixel in the pixel array (or the target block) in the transmission defect information, the second information may be omitted.

[0107] By using this syntax for the second information, the data volume of the defect information for transmission can be reduced as the number of defective pixels decreases. Generally, an image sensor 110 in a normal state has a small number of defective pixels. In other words, it is highly likely that a short bit pattern will be applied. Therefore, an increase in the data volume of the defect information for transmission can be suppressed. Therefore, it is possible to suppress a reduction in the bit rate allocated to the image encoding data, and to suppress a reduction in the quality of the RAW image due to encoding and decoding.

[0108] <Bitstream> Figures 19 and 20 show examples of bitstreams when the example of Figure 18 is applied (when the block is 8x2 pixels). Figure 19A shows an example of a bitstream in which transmission defect information and image coding data are multiplexed when the number of defective pixels is 0. In this case, the transmission defect information is a single "0". Therefore, in a RAW image, MIPI COMP7 compression is applied to the first pixel, and MIPI COMP8 compression is applied to the remaining pixels.

[0109] 19B shows an example of a bit stream in which transmission defect information and image encoded data are multiplexed when the number of defective pixels is one. As shown in the upper part of FIG. 19B, in this case, the defective pixel is the third pixel from the left in the first column from the top. In this case, the transmission defect information is a bit pattern "100100" consisting of a bit pattern "10" indicating the number of defective pixels, "010" indicating the X coordinate of the defective pixel, and "0" indicating the Y coordinate of the defective pixel. Therefore, for the RAW image, MIPI COMP7 compression is applied to the first six pixels, and MIPI COMP8 compression is applied to the remaining pixels.

[0110] 20A shows an example of a bit stream in which transmission defect information and image coding data are multiplexed when the number of defective pixels is two. As shown in the upper part of FIG. 20A, in this case, the third pixel from the left in the first column from the top and the seventh pixel from the left in the second column from the top are defective pixels. In this case, the transmission defect information is a bit pattern "11001001101" consisting of a bit pattern "110" indicating the number of defective pixels, "010" indicating the X coordinate of the first defective pixel and "0" indicating its Y coordinate, and "110" indicating the X coordinate of the second defective pixel and "1" indicating its Y coordinate. Therefore, MIPI COMP7 compression is applied to the first 11 pixels of the RAW image, and MIPI COMP8 compression is applied to the remaining pixels.

[0111] 20B shows an example of a bit stream in which transmission defect information and image coding data are multiplexed when the number of defective pixels is three. As shown in the upper part of FIG. 20B, in this case, the third pixel from the left in the first column from the top, the first pixel from the left in the second column from the top, and the seventh pixel from the left in the second column from the top are defective pixels. In this case, the transmission defect information is a bit pattern "1110010000010000010" composed of a bit pattern "111" indicating the number of defective pixels and defective pixel map information "0010000010000010." Therefore, MIPI COMP6 compression is applied to the first three pixels of the RAW image, and MIPI COMP7 compression is applied to the remaining pixels.

[0112] In this way, the increase in the amount of data of the defect information for transmission in the bit stream can be suppressed, and the deterioration of the defect information for transmission can be suppressed, so that the increase in the amount of image sensor output data can be suppressed while the reduction in the quality of the image after defect correction can be suppressed.

[0113] Of course, these are just examples, and as mentioned above, any multiplexing method can be used and is not limited to these examples.

[0114] <For Other Blocks> A of FIG. 21 is a diagram showing an example of the syntax of the transmission defect information when the defect information is processed for each 2x2 pixel block (B of FIG. 14). B of FIG. 21 is a diagram showing an example of a bit pattern (first information) indicating the number of defective pixels in that case. A of FIG. 22 is a diagram showing an example of the syntax of the transmission defect information when the defect information is processed for each 4x2 pixel block (C of FIG. 14). B of FIG. 22 is a diagram showing an example of a bit pattern (first information) indicating the number of defective pixels in that case. A of FIG. 23 is a diagram showing an example of the syntax of the transmission defect information when the defect information is processed for each 8x1 pixel block (D of FIG. 14). B of FIG. 23 is a diagram showing an example of the bit pattern (first information) indicating the number of defective pixels in that case.

[0115] As in these examples, the syntax and bit pattern (first information) may be changed in accordance with the shape and size of the block so as to suppress an increase in the amount of data of the defect information for transmission.

[0116] Furthermore, information indicating the type of defect may be stored in the transmission defect information, as in the syntax shown in A of FIG. 24. The syntax in A of FIG. 24 shows an example of the syntax of the transmission defect information when the defect information is processed for each 8x2 pixel block (A of FIG. 14). In this example, "defetType" is a parameter indicating the type of defect. The value of this parameter identifies the type of defect, such as a point defect, a stain defect, an unevenness defect, a line defect, etc. In other words, the transmission defect information may further include third information indicating the type of defect.

[0117] By using such information to perform defect correction, defect correction can be performed more easily and more accurately.

[0118] <Encoding Method> As described above, any encoding / decoding method may be applied to a RAW image. For example, as shown in FIGS. 25 and 26 , an encoding / decoding method using 8×2 pixel blocks as the processing unit may be applied. In other words, in this case, the processing unit of the defect information and the processing unit of the encoding / decoding are the same (8×2 pixels). A in FIG. 25 shows an example of a bitstream when the number of defective pixels is 0, corresponding to example A in FIG. 19 . B in FIG. 25 shows an example of a bitstream when the number of defective pixels is 1, corresponding to example B in FIG. 19 . A in FIG. 26 shows an example of a bitstream when the number of defective pixels is 2, corresponding to example A in FIG. 20 . B in FIG. 26 shows an example of a bitstream when the number of defective pixels is 3, corresponding to example B in FIG. 20 . In this way, a method in which image data and defect information are always equal at a fixed processing unit may be adopted.

[0119] <Method 1-3> Generally, when a defective pixel is present in the pixel array of the brightness detection unit 111, the pixel value of the defective pixel may differ significantly from the pixel values ​​of its surrounding pixels, as shown in the example of A in FIG. 27 . In the example of A in FIG. 27 , white circles indicate pixel values ​​of normal pixels (pixels that are not defective), and black circles indicate pixel values ​​of the defective pixel. Generally, in image encoding, data volume compression is often achieved by utilizing the correlation (similarity) between pixel values ​​of surrounding pixels. Therefore, as shown in the example of A in FIG. 27 , there is a risk that the encoding efficiency of the RAW image may decrease due to an increase in the difference between the pixel value of the defective pixel and the pixel values ​​of the surrounding pixels.

[0120] Therefore, when the above-mentioned method 1 is applied, pre-encoding processing may be performed on the image as shown in the fourth row from the top of the table in FIG. 6 (method 1-3).

[0121] For example, the first image processing device may further include a preprocessing unit that performs predetermined preprocessing on the image data. The image encoding unit may then encode the image data that has been subjected to the preprocessing to generate encoded image data. In this manner, the first image processing device can suppress a decrease in encoding efficiency of the image data due to pixel values ​​of defective pixels. Furthermore, it can suppress a decrease in the quality of the image after decoding.

[0122] <Encoding Unit> Fig. 28 is a block diagram showing an example of the main configuration of the encoding unit 114. As shown in Fig. 28, in this case, the encoding unit 114 has a pre-processing unit 211 in addition to the configuration in the example of Fig. 8.

[0123] The preprocessing unit 211 performs processing related to preprocessing of a RAW image. For example, the preprocessing unit 211 may acquire a RAW image supplied from the brightness detection unit 111. The preprocessing unit 211 may perform predetermined image processing on the acquired RAW image as preprocessing. The preprocessing unit 211 may supply the RAW image that has been subjected to the preprocessing (hereinafter also referred to as a preprocessed RAW image) to the image encoding unit 152.

[0124] In this case, the image encoding unit 152 generates encoded image data by encoding the preprocessed RAW image supplied from the preprocessing unit 211. This makes it possible to suppress a decrease in encoding efficiency.

[0125] The multiplexing unit 153 multiplexes the image encoded data with the defect information for transmission. That is, the preprocessed RAW image (encoded data) is transmitted to the signal processing unit 120. Therefore, an increase in the bit rate of the bit stream can be suppressed. Therefore, a decrease in the quality of the image after decoding can be suppressed.

[0126] <Flow of Encoding Process> An example of the flow of the encoding process executed in step S102 of FIG. 10 in this case will be described with reference to the flowchart of FIG.

[0127] When the encoding process starts, in step S201, the preprocessing unit 211 of the encoding unit 114 preprocesses the RAW image and generates a preprocessed RAW image.

[0128] In step S202, the defect information conversion unit 151 converts the format of the defect information into the format of defect information for transmission.

[0129] In step S203, the image encoding unit 152 generates encoded image data by encoding the preprocessed RAW image at a bit rate based on the bit rate of the generated defect information for transmission.

[0130] In step S204, the multiplexing unit 153 multiplexes the transmission defect information and the coded image data to generate a bit stream.

[0131] In step S205, the multiplexing unit 153 transmits the generated bit stream to the signal processing unit 120 via the bus 130.

[0132] When the process of step S205 is completed, the encoding process ends and the process returns to FIG.

[0133] By performing each process in this manner, the encoding unit 114 can suppress a reduction in the encoding efficiency of the RAW image, as described above, and as a result, can suppress a reduction in the quality of the image after decoding.

[0134] <Preprocessing> The preprocessing thus executed by the preprocessing unit 211 may be any processing that corrects the pixel values ​​of defective pixels in a RAW image and suppresses a decrease in the encoding efficiency of the RAW image.

[0135] For example, the preprocessing unit 211 may replace the pixel value of a defective pixel with a copy of the pixel value of a pixel adjacent to the defective pixel. That is, the preprocessing unit 211 may copy the pixel value of a pixel adjacent to the defective pixel and replace the pixel value of the defective pixel with the copied pixel value. The pixel adjacent to the defective pixel may be adjacent in any direction relative to the defective pixel. For example, the preprocessing unit 211 may copy the pixel value of the pixel (also referred to as the previous value) that is processed immediately before the defective pixel in the processing order. For example, as shown in B of FIG. 27 , the preprocessing unit 211 may copy the pixel value of the pixel immediately to the left of the defective pixel and replace the pixel value of the defective pixel with the copied pixel value.

[0136] The preprocessing unit 211 may also replace the pixel value of a defective pixel with a calculated value based on the pixel values ​​of neighboring pixels located around the defective pixel. This calculated value may be derived by any calculation, including, for example, a statistical value using the pixel values ​​of the neighboring pixels. For example, this calculated value may be the average of the pixel values ​​of the neighboring pixels derived using an FIR (Finite Impulse Response) filter or an adaptive filter. For example, as shown in FIG. 27C, the average of the pixel values ​​of the pixels adjacent to the left and right of the defective pixel may be derived, and the pixel value of the defective pixel may be replaced with the derived average. This calculated value may also be the median of the pixel values ​​of the neighboring pixels. This calculated value may also be a value derived using a learning model that uses the pixel values ​​of the neighboring pixels as input (i.e., the output of the learning model).

[0137] Alternatively, the preprocessing unit 211 may replace the pixel value of the defective pixel with a predetermined value. This predetermined value may be any value, such as the median value of the range of pixel values.

[0138] By performing preprocessing on the pixel values ​​of defective pixels in this manner, it is possible to suppress an increase in the difference in pixel values ​​between the defective pixel and its surrounding pixels. Typically, this difference can be reduced. Therefore, as described above, the encoding unit 114 can suppress a decrease in the encoding efficiency of the RAW image, and as a result, it can suppress a decrease in the quality of the decoded image.

[0139] The peripheral pixels may be located physically near the defective pixel or may be located near the defective pixel in terms of processing order. The peripheral pixels may be located near the defective pixel (not relatively far from the defective pixel), and their positional relationship (e.g., relative direction) may be arbitrary. For example, the size and shape of the peripheral range from the reference pixel (the range that can be peripheral pixels relative to the reference pixel) may be predetermined. For example, the range may be one-dimensional or two-dimensional. The range may be centered around the reference pixel, or may be a range biased in a direction relative to the reference pixel. For example, if the range is one-dimensional in the horizontal direction, the range may be the left side of the reference pixel, the right side of the reference pixel, or the reference pixel may be the center of the range. Only pixels processed before the reference pixel may be considered peripheral pixels.

[0140] <Method 1-4> Post-processing may be performed on the decoded RAW image. When the above-mentioned method 1 is applied, post-decoding processing may be performed on defective pixel values ​​of the image (Method 1-4), as shown in the fifth row from the top of the table in Fig. 6. For example, the second image processing device may further include a post-processing unit that performs predetermined post-processing on image data generated by decoding the image encoding data.

[0141] <Decoding Unit> Fig. 30 is a block diagram showing an example of the main configuration of the decoding unit 121. As shown in Fig. 30, the decoding unit 121 in this case has a post-processing unit 221 in addition to the configuration in the example of Fig. 9.

[0142] The post-processing unit 221 performs post-processing on the RAW image. For example, the image decoding unit 173 may decode the image encoding data to generate a RAW image and supply it to the post-processing unit 221. The post-processing unit 221 may acquire the RAW image. The defect information inverse conversion unit 172 may supply the generated defect information to the post-processing unit 221. The post-processing unit 221 may acquire the defect information. The post-processing unit 221 may perform predetermined image processing on the acquired RAW image as post-coding processing. For example, the post-processing unit 221 may perform predetermined processing on pixel values ​​of defective pixels included in the RAW image based on the acquired defect information. The post-processing unit 221 may supply the RAW image that has been subjected to the post-processing (hereinafter also referred to as a post-processed RAW image) to the defect correction unit 122.

[0143] This post-processing may be any type of processing as long as it is performed on a RAW image. For example, the post-processing unit 221 may perform processing on the RAW image that is useful for defect correction. The post-processing unit 221 may identify defective pixels based on the defect information and perform processing such as modifying the pixel values.

[0144] In this way, the decoding unit 121 can perform post-processing on the RAW image and supply the RAW image to the defect correction unit 122. For example, the decoding unit 121 may include the contents of the defect information in the RAW image through this post-processing. In this way, the defect correction unit 122 can perform defect correction without needing the defect information (using only the post-processed RAW image).

[0145] <Flow of Decoding Process> An example of the flow of the decoding process executed in step S103 of FIG. 10 in this case will be described with reference to the flowchart of FIG.

[0146] When the decoding process is started, the processes of steps S221 to S224 are executed in the same manner as the processes of steps S141 to S144 in FIG.

[0147] In step S225, the post-processing unit 221 performs post-processing on the RAW image obtained by the processing in step S224, and generates a post-processed RAW image.

[0148] When the process of step S225 ends, the decoding process ends and the process returns to FIG.

[0149] By performing each process in this manner, the decoding unit 121 can perform post-processing on the restored RAW image as described above.

[0150] <Method 1-4-1> For example, this post-processing may be a process of replacing pixel values ​​of defective pixels contained in a RAW image with identification values ​​for identifying the defective pixels. That is, when the above-described Method 1-4 is applied, the defective pixel values ​​may be replaced with identification values ​​(Method 1-4-1) as shown in the sixth row from the top of Fig. 6. For example, in the second image processing device, the post-processing may be a process of replacing pixel values ​​of defective pixels in image data (RAW image) generated by decoding image encoding data with identification values ​​for identifying the defective pixels.

[0151] By performing such post-processing on the RAW image by the post-processing unit 221, the defect correction unit 122 can identify defective pixels based on the identification value. In other words, the post-processing unit 221 can include the contents of the defect information in the RAW image. Therefore, the defect correction unit 122 can identify defective pixels included in the post-processed RAW image without needing the defect information. Therefore, the defect correction unit 122 can easily perform defect correction without needing the defect information.

[0152] The identification value may be any value. For example, the identification value may be a predetermined fixed value. Alternatively, the identification value may be set externally (for example, by a user or an application).

[0153] Note that the pixel values ​​of normal pixels that are not defective pixels may have the same value as this identification value. In this case, identification values ​​may exist not only for defective pixels but also for normal pixels in the post-processed RAW image. However, since the defect correction unit 122 cannot distinguish between them, there is a risk that the pixel values ​​of normal pixels may also be corrected. As a result, such correction may degrade the quality of the RAW image. Therefore, before the pixel values ​​of such defective pixels are replaced with identification values, pixel values ​​of normal pixels that are not defective pixels and have the same value as the identification value may be changed to other values. For example, in the second image processing device, as post-processing, the post-processing unit may change pixel values ​​that are the same as the identification values ​​of normal pixels that are not defective pixels in image data (RAW image) generated by decoding image encoding data to other values, and then replace the pixel values ​​of the defective pixels with the identification values.

[0154] This allows for more accurate defect correction, thereby preventing degradation of the quality of the RAW image.

[0155] This changed value (other value) may be any value that is different from the identification value. However, if this change results in a large change in pixel value, there is a risk that the quality of the RAW image will be reduced. Therefore, in order to prevent a reduction in the quality of the RAW image, it is desirable to change the value to a value close to the value before the change (identification value), such as "identification value +1" or "identification value -1."

[0156] 32 is a block diagram showing an example of the main configuration of the post-processing unit 221. As shown in Fig. 32, the post-processing unit 221 in this case has a normal pixel value changing unit 231 and an identification value replacing unit 232.

[0157] The normal pixel value modification unit 231 executes processing related to modification of normal pixel values. For example, the normal pixel value modification unit 231 may acquire a RAW image supplied from the image decoding unit 173. This RAW image is a RAW image generated by the image decoding unit 173 decoding encoded image data. If the acquired RAW image contains a pixel value of a normal pixel having the same value as the identification value, the normal pixel value modification unit 231 may modify the pixel value to another value.

[0158] The identification value may be changeable. For example, it may be set by a user, an application, or the like. In this case, the normal pixel value modification unit 231 is supplied with an identification value set externally (for example, by a user or an application). The normal pixel value modification unit 231 may use the supplied identification value to modify the pixel values ​​of normal pixels as described above.

[0159] The normal pixel value changing unit 231 may supply the RAW image in which the pixel values ​​of the normal pixels have been changed as necessary as described above to the identification value replacing unit 232 .

[0160] The discrimination value replacement unit 232 performs processing related to the replacement of discrimination values. For example, the discrimination value replacement unit 232 may acquire a RAW image supplied from the normal pixel value modification unit 231. Alternatively, the discrimination value replacement unit 232 may acquire defect information supplied from the defect information inverse conversion unit 172. The discrimination value replacement unit 232 may identify defective pixels in the acquired RAW image based on the acquired defect information and replace the pixel values ​​with discrimination values. The discrimination value replacement unit 232 may supply the RAW image subjected to such processing to the defect correction unit 122 as a post-processed RAW image.

[0161] In this way, the post-processing unit 221 can replace the pixel value of the defective pixel with the identification value, and the defect correction unit 122 can easily perform defect correction without needing defect information.

[0162] <Post-Processing Flow> An example of the flow of post-processing executed in step S225 of FIG. 31 in this case will be described with reference to the flowchart of FIG.

[0163] When post-processing is started, the normal pixel value changing unit 231 of the post-processing unit 221 changes the pixel values ​​of normal pixels that have the same value as the identification value in step S241.

[0164] In step S242, the identification value replacement unit 232 identifies defective pixels based on the identification information, and replaces the pixel values ​​with identification values.

[0165] When the process of step S242 ends, the post-processing ends and the process returns to FIG.

[0166] By performing each process in this manner, the post-processing unit 221 can replace the pixel value of the defective pixel with the identification value, and the defect correction unit 122 can easily perform defect correction without needing defect information.

[0167] <Method 1-4-2> This post-processing may also be a process of replacing the pixel value of a defective pixel included in a RAW image with a neighboring reference value derived by referring to neighboring pixel values ​​of the defective pixel. That is, when the above-described Method 1-4 is applied, the defective pixel value may be replaced with a neighboring reference value (Method 1-4-2), as shown in the seventh row from the top of Figure 6. For example, in the second image processing device, the post-processing may replace the pixel value of a defective pixel in image data (RAW image) generated by decoding image encoding data with a neighboring reference value derived by referring to neighboring pixel values ​​of the defective pixel.

[0168] In other words, the pixel value of the defective pixel is corrected based on the values ​​of the surrounding pixels. Therefore, in this case, the defect correction unit 122 may be omitted.

[0169] The neighboring reference value may be any value derived by referring to neighboring pixel values. For example, the neighboring reference value may be a copy of the value of a pixel adjacent to the defective pixel. The pixel adjacent to the defective pixel may be adjacent to the defective pixel in any direction, including above, below, left, right, or diagonally. Furthermore, the pixel adjacent to the defective pixel may be the pixel that is processed immediately before the defective pixel in the processing order.

[0170] The neighboring reference value may be a calculated value based on the neighboring pixel values ​​of the defective pixel. The calculated value may be a value derived by any calculation. For example, the calculated value may be a statistical value using the pixel values ​​of the neighboring pixels of the defective pixel. For example, the calculated value may be the average of the pixel values ​​of the neighboring pixels derived using an FIR filter or an adaptive filter. The calculated value may also be the median of the pixel values ​​of the neighboring pixels. The calculated value may also be a value derived using a learning model that uses the pixel values ​​of the neighboring pixels as input (i.e., the output of the learning model).

[0171] The peripheral pixels may be located physically near the defective pixel or may be located near the defective pixel in terms of processing order. The peripheral pixels may be located near the defective pixel (not relatively far from the defective pixel), and their positional relationship (e.g., relative direction) may be arbitrary. For example, the size and shape of the peripheral range from the reference pixel (the range that can be peripheral pixels relative to the reference pixel) may be predetermined. For example, the range may be one-dimensional or two-dimensional. The range may be centered around the reference pixel, or may be a range biased in a direction relative to the reference pixel. For example, if the range is one-dimensional in the horizontal direction, the range may be the left side of the reference pixel, the right side of the reference pixel, or the reference pixel may be the center of the range. Only pixels processed before the reference pixel may be considered peripheral pixels.

[0172] In this way, the pixel value of the defective pixel is corrected based on the values ​​of the surrounding pixels, so that the defect correction unit 122 (or its defect correction) can be omitted.

[0173] 34 is a block diagram showing an example of the main configuration of the post-processing unit 221 in this case. As shown in FIG. 34, the post-processing unit 221 in this case has a neighboring reference value replacing unit 241.

[0174] The neighboring reference value replacement unit 241 performs processing related to replacement of neighboring reference values. For example, the neighboring reference value replacement unit 241 may acquire a RAW image supplied from the image decoding unit 173. This RAW image is a RAW image generated by the image decoding unit 173 by decoding image encoded data. The neighboring reference value replacement unit 241 may also acquire defect information supplied from the defect information inverse conversion unit 172. The neighboring reference value replacement unit 241 may identify defective pixels in the acquired RAW image based on the acquired defect information and replace the pixel values ​​with neighboring reference values. The neighboring reference value replacement unit 241 may supply the RAW image subjected to such processing to the defect correction unit 122 as a post-processed RAW image.

[0175] In this way, the post-processing unit 221 can replace the pixel value of the defective pixel with the surrounding reference value, and therefore the defect correction unit 122 (defect correction by the defect correction unit 122) can be omitted.

[0176] <Post-Processing Flow> An example of the flow of post-processing executed in step S225 of FIG. 31 in this case will be described with reference to the flowchart of FIG.

[0177] When post-processing is started, the surrounding reference value replacement unit 241 of the post-processing unit 221 replaces the defective pixel value with the surrounding reference value in step S261.

[0178] When the process of step S261 ends, the post-processing ends and the process returns to FIG.

[0179] By performing each process in this manner, the post-processing unit 221 can replace the pixel value of the defective pixel with the surrounding reference value, thereby eliminating the need for the defect correction unit 122 (to correct the defect).

[0180] <Method 1-4-3> Alternatively, multiple modes may be prepared as candidates for this post-processing, and a mode selected from the candidates may be applied. That is, when the above-described Method 1-4 is applied, a post-processing mode may be selected (Method 1-4-3) as shown in the bottom row of Fig. 6. For example, in the second image processing device, a post-processing unit may select a post-processing mode to apply, and perform post-processing in the selected mode.

[0181] This allows for a wider variety of post-processing to be performed on RAW images, making it possible to perform more appropriate post-processing in a wider variety of situations and suppressing degradation of the RAW image quality.

[0182] Note that the post-processing mode (i.e., processing content that can be executed as post-processing) may be any mode. Furthermore, any number of post-processing options may be provided. For example, in the second image processing device, the post-processing unit may select one from an identification value replacement mode in which the pixel value of a defective pixel in image data is replaced with an identification value for identifying the defective pixel, a surrounding reference mode in which the pixel value of a defective pixel in image data is replaced with a surrounding reference value derived by referencing pixel values ​​of the defective pixel, and a through mode in which replacement of the pixel value of the defective pixel is omitted, and perform post-processing in the selected mode.

[0183] The discrimination value replacement mode is a mode in which the post-processing described as method 1-4-1 is performed. The peripheral reference mode is a mode in which the post-processing described as method 1-4-2 is performed. The through mode is a mode in which the RAW image is supplied to the defect correction unit 122 without performing post-processing.

[0184] 36 is a block diagram showing an example of the main configuration of the post-processing unit 221. As shown in Fig. 36, the post-processing unit 221 in this case includes a normal pixel value changing unit 231, an identification value replacing unit 232, a neighboring reference value replacing unit 241, and a mode selecting unit 251.

[0185] The normal pixel value changing unit 231 and the discrimination value replacing unit 232 are as described with reference to Fig. 32 . The surrounding reference value replacing unit 241 is as described with reference to Fig. 34 . However, the discrimination value replacing unit 232 may supply a RAW image in which the pixel values ​​of defective pixels have been replaced with discrimination values ​​to the mode selecting unit 251. Furthermore, the surrounding reference value replacing unit 241 may supply a RAW image in which the pixel values ​​of defective pixels have been replaced with surrounding reference values ​​to the mode selecting unit 251.

[0186] The mode selection unit 251 executes processing related to the selection of a post-processing mode. For example, the mode selection unit 251 may acquire a RAW image supplied from the identification value replacement unit 232. In this RAW image, the pixel values ​​of defective pixels have been replaced with identification values. The mode selection unit 251 may also acquire a RAW image supplied from the surrounding reference value replacement unit 241. In this RAW image, the pixel values ​​of defective pixels have been replaced with surrounding reference values. Furthermore, the mode selection unit 251 may acquire a RAW image supplied to the post-processing unit 221 from the image decoding unit 173.

[0187] The mode selection unit 251 may then acquire a mode selection instruction from outside (for example, a user, an application, or the like). The mode selection unit 251 selects information to be output according to the mode specified by the mode selection instruction. In other words, the post-processing unit 221 selects a post-processing mode to be applied by the mode selection unit 251 and performs post-processing in the selected mode.

[0188] For example, when the discrimination value replacement mode is selected by this mode selection instruction, the mode selection unit 251 may supply the RAW image supplied from the discrimination value replacement unit 232 to the defect correction unit 122 as a post-processed RAW image. Furthermore, when the peripheral reference mode is selected by this mode selection instruction, the mode selection unit 251 may supply the RAW image supplied from the peripheral reference value replacement unit 241 to the defect correction unit 122 as a post-processed RAW image. Furthermore, when the through mode is selected by this mode selection instruction, the mode selection unit 251 may supply the RAW image supplied from the image decoding unit 173 to the defect correction unit 122 as a post-processed RAW image.

[0189] This allows the post-processing unit 221 to perform a wider variety of post-processing on the RAW image, thereby enabling the post-processing unit 221 to perform more appropriate post-processing in a wider variety of situations and suppress degradation of the quality of the RAW image.

[0190] <Post-Processing Flow> An example of the flow of post-processing executed in step S225 of FIG. 31 in this case will be described with reference to the flowchart of FIG.

[0191] When post-processing is started, the mode selection unit 251 of the post-processing unit 221 selects a post-processing mode to be applied in accordance with the mode selection instruction in step S281.

[0192] In step S282, the mode selection unit 251 determines whether or not the identification value replacement mode has been selected. If it is determined that the identification value replacement mode has been selected, the process proceeds to step S283.

[0193] In step S283, the normal pixel value changing unit 231 changes the pixel value of the normal pixel having the same value as the identification value to another value.

[0194] In step S284, the identification value replacement unit 232 replaces the pixel value of the defective pixel with the identification value.

[0195] When the process of step S284 ends, the post-processing ends and the process returns to FIG.

[0196] If it is determined in step S282 that the identification value replacement mode has not been selected, the process proceeds to step S285.

[0197] In step S285, the mode selection unit 251 determines whether or not the surrounding reference mode has been selected. If it is determined that the surrounding reference mode has been selected, the process proceeds to step S286.

[0198] In step S286, the surrounding reference value replacement unit 241 replaces the pixel value of the defective pixel with the surrounding reference value.

[0199] When the process of step S286 ends, the post-processing ends and the process returns to FIG.

[0200] Furthermore, if it is determined in step S285 that the peripheral reference mode has not been selected (that is, the through mode has been selected), post-processing is not performed, and the RAW image supplied from the image decoding unit 173 is supplied to the defect correction unit 122. Then, the post-processing ends, and the process returns to Fig. 31 .

[0201] By performing each process in this manner, the post-processing unit 221 can perform a wider variety of post-processing on the RAW image, thereby enabling the post-processing unit 221 to perform more appropriate post-processing in a wider variety of situations and suppressing degradation of the quality of the RAW image.

[0202] <Scope of application of explanation> The explanation given for the higher-level method in <3. Transmission of defect information and image data for transmission> also applies to the lower-level methods belonging to that method, unless a contradiction arises. For example, when it is explained that "Method 1 may be applied," it means that one or more of Methods 1-1 to 1-4 may be applied. Of course, it also means that even lower-level methods (e.g., Methods 1-4-1 to 1-4-3) may be applied.

[0203] <Combinations> Note that each of the methods described above in <3. Transmission of defect information and image data for transmission> may be applied in combination with any other method, as long as no contradiction occurs. Three or more methods may be applied in combination. For example, any two or more of Methods 1-1 to 1-4 may be applied in combination. Furthermore, the methods that can be combined are not limited to those shown in the table of FIG. 6 as "methods," but may include all elements described in this specification. Furthermore, each of the methods described above in <3. Transmission of defect information and image data for transmission> may be applied in combination with methods other than those described above.

[0204] <4. Replacement of Defective Pixel Values> <Method 2> In order to prevent a decrease in the coding efficiency of the bit stream, the defect information may be included in the RAW image, and a bit stream containing image coding data obtained by coding the RAW image including the defect information may be transmitted from image sensor 110 to signal processing unit 120 via bus 130. For example, as shown in the top row of the table in Fig. 38 , defective pixel values ​​may be replaced with identification values ​​before coding (Method 2). In other words, in this bit stream, the defect information to be transmitted is not multiplexed into the image coding data as in Method 1.

[0205] For example, a first image processing device may include an identification value replacement unit that replaces pixel values ​​of defective pixels in image data representing a luminance distribution detected in a pixel array for detecting luminance with an identification value for identifying the defective pixel, and an encoding unit that encodes the image data in which the pixel values ​​of the defective pixels have been replaced with the identification value to generate a bit stream. A first image processing method may include the first image processing device replacing pixel values ​​of defective pixels in image data representing a luminance distribution detected in a pixel array for detecting luminance with an identification value for identifying the defective pixel, and the first image processing device encoding the image data in which the pixel values ​​of the defective pixels have been replaced with the identification value to generate a bit stream. A first program may cause a computer to execute a process including replacing pixel values ​​of defective pixels in image data representing a luminance distribution detected in the pixel array for detecting luminance with an identification value for identifying the defective pixel, and encoding the image data in which the pixel values ​​of the defective pixels have been replaced with the identification value to generate a bit stream.

[0206] In this way, the first image processing device can suppress an increase in the amount of data to be transmitted while suppressing a decrease in the accuracy of defect information, i.e., the first image processing device can suppress a decrease in the quality of the image after defect correction while suppressing an increase in the amount of image sensor output data.

[0207] The identification value may be any value. For example, the identification value may be a predetermined fixed value. Alternatively, the identification value may be an externally set value set by an external device (for example, a user or an application).

[0208] Note that the pixel value of a normal pixel that is not a defective pixel may have the same value as this identification value. In such a case, it may be difficult to distinguish whether the identification value is the replacement value for the defective pixel or the pixel value of a normal pixel. For example, when defect correction is performed on such a RAW image, the pixel values ​​of the normal pixels may also be corrected, resulting in a reduction in the quality of the RAW image. Therefore, before the pixel value of such a defective pixel is replaced with the identification value, the pixel value of a normal pixel that is not a defective pixel and has the same value as the identification value may be changed to another value. For example, the first image processing device may further include a normal pixel value changer that changes a pixel value in the image data that is the same as the identification value of a normal pixel that is not a defective pixel to another value. Then, the identification value replacement unit may replace the pixel value of the defective pixel in the image data, in which the pixel value that is the same as the identification value of a normal pixel that is not a defective pixel has been changed to another value, with the identification value.

[0209] This allows for more accurate identification of defective pixels during defect correction, thereby preventing degradation of the quality of the RAW image.

[0210] <Encoding unit> The present technology can be applied to any device. For example, the present technology can be applied to an imaging device that captures an image of a subject. Even when Method 2 is applied, the imaging device 100 has a configuration similar to that of the example in FIG. 7 .

[0211] Fig. 39 is a block diagram showing an example of the main configuration of the encoding unit 114 when Method 2 is applied. As shown in Fig. 39 , in this case, the encoding unit 114 has a normal pixel value modification unit 311, a discrimination value replacement unit 312, and an encoding unit 313.

[0212] The normal pixel value modification unit 311 is a processing unit basically similar to the normal pixel value modification unit 231 ( FIG. 32 ) of the post-processing unit 221, and executes the same processing as the normal pixel value modification unit 231. For example, the normal pixel value modification unit 311 may acquire a RAW image supplied from the brightness detection unit 111. If the acquired RAW image contains a pixel value of a normal pixel having the same value as the identification value, the normal pixel value modification unit 311 may change the pixel value to another value. In other words, the normal pixel value modification unit 311 may change the pixel value of a normal pixel in the image data that is not a defective pixel and is the same as the identification value to another value.

[0213] This identification value and other values ​​are the same as in the example of FIG. 32. That is, the identification value may be changeable. For example, it may be set by the user, an application, or the like. Furthermore, the other values ​​may be any value that is different from the identification value, but in order to prevent a decrease in the quality of the RAW image, it is desirable to change them to a value close to the value before the change (identification value), such as "identification value +1" or "identification value -1."

[0214] The normal pixel value changing unit 311 may supply the RAW image in which the pixel values ​​of the normal pixels have been changed as necessary as described above to the identification value replacing unit 312 .

[0215] The identification value replacement unit 312 is a processing unit basically similar to the identification value replacement unit 232 ( FIG. 32 ) of the post-processing unit 221 and performs the same processing as the identification value replacement unit 232. For example, the identification value replacement unit 312 may acquire a RAW image supplied from the normal pixel value modification unit 311. Alternatively, the identification value replacement unit 312 may read and acquire defect information stored in the storage unit 113. The identification value replacement unit 312 may identify defective pixels in the acquired RAW image based on the acquired defect information and replace the pixel values ​​with identification values. That is, the identification value replacement unit 312 may replace the pixel values ​​of defective pixels in a pixel array for detecting luminance in image data representing the distribution of the luminance with identification values ​​for identifying the defective pixels. The identification value replacement unit 312 may supply the RAW image subjected to such processing to the encoding unit 313 as a defective pixel-processed RAW image.

[0216] The encoding unit 313 performs processing related to encoding of a RAW image. For example, the encoding unit 313 may acquire a RAW image after defective pixel processing supplied from the identification value replacement unit 312. As described above, in this RAW image after defective pixel processing, the pixel values ​​of the defective pixels are replaced with identification values. In other words, this RAW image after defective pixel processing is a RAW image that includes the contents of defect information. The encoding unit 313 encodes the acquired RAW image after defective pixel processing to generate a bitstream. In other words, the encoding unit 313 can apply a general image encoding method. The encoding unit 313 may supply the generated bitstream to the signal processing unit 120 via the bus 130.

[0217] By doing so, the encoding unit 114 can omit transmission of defect information. That is, the encoding unit 114 can suppress an increase in the amount of data to be transmitted while suppressing a decrease in the accuracy of the defect information. That is, the encoding unit 114 can suppress a decrease in the quality of the image after defect correction while suppressing an increase in the amount of image sensor output data.

[0218] <Flow of Encoding Process> An example of the flow of the encoding process executed in step S102 of FIG. 10 in this case will be described with reference to the flowchart of FIG.

[0219] When the encoding process starts, in step S301, the normal pixel value changing unit 311 of the encoding unit 114 changes the pixel values ​​of normal pixels that have the same value as the identification value of the RAW image.

[0220] In step S302, the discrimination value replacement unit 312 replaces the defective pixel values ​​of the RAW image with discrimination values, and generates a RAW image that has undergone the defective pixel processing.

[0221] In step S303, the encoding unit 313 encodes the RAW image that has undergone the defective pixel processing to generate a bit stream.

[0222] In step S304, the encoding unit 313 transmits the generated bit stream to the signal processing unit 120 via the bus 130.

[0223] When the process of step S304 is completed, the encoding process ends and the process returns to FIG.

[0224] By performing each process in this manner, the encoding unit 114 can suppress a decrease in the quality of the image after defect correction while suppressing an increase in the amount of image sensor output data, as described above.

[0225] <Regarding Decoding> In this case, the decoding unit 121 of the signal processing unit 120 simply decodes the bit stream using a decoding method corresponding to the encoding method of the encoding unit 114. As a result, a RAW image after defective pixel processing is generated (restored). Therefore, the decoding unit 121 simply supplies the RAW image after defective pixel processing to the defect correction unit 122. The defect correction unit 122 can detect the identification value of the RAW image after defective pixel processing and correct the pixel values ​​of the defective pixels. Therefore, the decoding unit 121 can apply a decoding method for general images.

[0226] <Method 2-1> When the above-described Method 2-1 is applied, the defect information may be encoded and the bit length of the image encoding may be set based on the bit length of the defect information (Method 2-1), as shown in the bottom row of the table in Fig. 38. For example, in the first image processing device, the encoding unit may include a discrimination unit that discriminates, based on an identification value, image data in which the pixel values ​​of defective pixels have been replaced with the identification values, into defect information about the defective pixels and the image data, a defect information encoding unit that encodes the defect information and generates defect information encoded data, an image encoding unit that encodes the image data and generates image encoded data, and a multiplexing unit that multiplexes the defect information encoded data and the image encoded data to generate a bit stream.

[0227] By doing so, the encoding unit 114 can suppress an increase in the amount of image sensor output data while suppressing a decrease in the quality of the image after defect correction.

[0228] <Encoding Unit> Fig. 41 is a block diagram showing an example of the main configuration of the encoding unit 313. As shown in Fig. 41, the encoding unit 313 has a discrimination unit 321, a defect information encoding unit 322, an image encoding unit 323, and a multiplexing unit 324.

[0229] The discriminator 321 performs processing related to data discrimination. For example, the discriminator 321 may acquire a defective pixel-processed RAW image supplied from the discrimination value replacement unit 312. The discriminator 321 may detect defective pixels included in the defective pixel-processed RAW image based on the discrimination value, and discriminate the defective pixel-processed RAW image into defect information related to the defective pixels and the RAW image. The discriminator 321 may supply the defect information to the defect information encoder 322. The discriminator 321 may supply the RAW image to the image encoder 323.

[0230] The defect information encoding unit 322 executes processing related to the encoding of defect information. For example, the defect information encoding unit 322 may acquire defect information supplied from the discrimination unit 321. The defect information encoding unit 322 may encode the defect information to generate defect information encoded data. Any encoding method may be used. For example, a lossless encoding method may be applied to suppress degradation of the defect information. The defect information encoding unit 322 may supply the generated defect information encoded data to the multiplexing unit 324. The defect information encoding unit 322 may also supply the bit length of the defect information encoded data to the image encoding unit 323.

[0231] The image encoding unit 323 performs processing related to encoding of a RAW image. For example, the image encoding unit 323 may acquire a RAW image supplied from the discrimination unit 321. The image encoding unit 323 may encode the acquired RAW image to generate encoded image data. Any encoding method may be used. For example, a lossy encoding method may be applied to suppress an increase in the amount of data. In addition, at this time, the image encoding unit 323 may acquire the bit length of the defect information encoded data supplied from the defect information encoding unit 322 and set the bit length of the encoded image data based on the acquired bit length. In other words, the image encoding unit 323 may encode image data representing the distribution of luminance detected in the pixel array and generate encoded image data with a bit length set based on the bit length of the encoded defect information data. The image encoding unit 323 may supply the generated encoded image data to the multiplexing unit 324.

[0232] The multiplexing unit 324 performs processing related to data multiplexing. For example, the multiplexing unit 324 may acquire defect information encoded data supplied from the defect information encoding unit 322. The multiplexing unit 324 may acquire image encoded data supplied from the image encoding unit 323. The multiplexing unit 324 may multiplex the acquired defect information encoded data and image encoded data to generate a bit stream. The multiplexing unit 324 may supply the generated bit stream to the signal processing unit 120 (the decoding unit 121 thereof) via the bus 130.

[0233] By doing so, the encoding unit 114 can suppress an increase in the amount of image sensor output data while suppressing a decrease in the quality of the image after defect correction.

[0234] <Flow of Encoding Process> An example of the flow of the encoding process executed in step S102 of FIG. 10 in this case will be described with reference to the flowchart of FIG.

[0235] When the encoding process starts, in step S321, the normal pixel value changing unit 311 of the encoding unit 114 changes the pixel values ​​of normal pixels that have the same value as the identification value of the RAW image.

[0236] In step S322, the identification value replacement unit 312 replaces the defective pixel values ​​of the RAW image with the identification values, and generates a RAW image that has undergone the defective pixel processing.

[0237] In step S323, the discrimination unit 321 of the encoding unit 313 discriminates between the defect information and the image data based on the discrimination value.

[0238] In step S324, the defect information encoding unit 322 encodes the defect information.

[0239] In step S325, the image encoding unit 323 encodes the image data based on the bit length of the defect information.

[0240] In step S326, the multiplexing unit 324 multiplexes the defect information coded data and the image coded data to generate a bit stream.

[0241] In step S327, the multiplexing unit 324 transmits the generated bit stream to the signal processing unit 120 via the bus 130.

[0242] When the process of step S327 ends, the encoding process ends and the process returns to FIG.

[0243] By performing each process in this manner, the encoding unit 114 can suppress a decrease in the quality of the image after defect correction while suppressing an increase in the amount of image sensor output data, as described above.

[0244] 43 is a block diagram showing an example of the main configuration of the decoding unit 121. As shown in Fig. 43, the decoding unit 121 has a demultiplexing unit 331, a defect information decoding unit 332, and an image decoding unit 333.

[0245] The demultiplexing unit 331 performs processing related to demultiplexing. For example, the demultiplexing unit 331 may acquire a bit stream supplied from (the multiplexing unit 324 of) the image sensor 110 via the bus 130. The demultiplexing unit 331 may demultiplex the bit stream to extract defect information coded data and image coded data. The demultiplexing unit 331 may supply the extracted defect information coded data to the defect information decoding unit 332. The demultiplexing unit 331 may supply the extracted image coded data to the image decoding unit 333.

[0246] The defect information decoding unit 332 executes processing related to the decoding of the defect information encoded data. For example, the defect information decoding unit 332 may acquire the defect information encoded data supplied from the demultiplexing unit 331. The defect information decoding unit 332 may decode the acquired defect information encoded data to generate defect information. The defect information decoding unit 332 may supply the generated defect information to the defect correction unit 122. The defect information decoding unit 332 may supply the bit length of the defect information encoded data to be decoded to the image decoding unit 333.

[0247] The image decoding unit 333 performs processing related to decoding of image coded data. For example, the image decoding unit 333 may acquire image coded data supplied from the demultiplexing unit 331. The image decoding unit 333 may decode the acquired image coded data to generate RAW image data. This decoding method may be any method that corresponds to the coding method applied by the image coding unit 323. In this case, the image decoding unit 333 may acquire the bit length of the defect information coded data supplied from the defect information decoding unit 332, estimate the bit length of the image coded data based on the bit length, and decode the image coded data using the estimated bit length. The image decoding unit 333 may supply the generated RAW image data to the defect correction unit 122.

[0248] By doing so, the decoding unit 121 can suppress a decrease in the quality of the image after defect correction while suppressing an increase in the amount of image sensor output data.

[0249] <Flow of Decoding Process> An example of the flow of the decoding process executed in step S103 of FIG. 10 in this case will be described with reference to the flowchart of FIG.

[0250] When the decoding process starts, the demultiplexing unit 331 of the decoding unit 121 receives the bit stream transmitted from the image sensor 110 via the bus 130 in step S341.

[0251] In step S342, the demultiplexer 331 demultiplexes the bit stream and distinguishes between the defect information coded data and the image coded data.

[0252] In step S343, the defect information decoding section 332 decodes the defect information encoded data to generate defect information.

[0253] In step S344, the image decoding unit 333 decodes the image coded data based on the bit length of the defect information coded data to generate a RAW image.

[0254] When the process of step S344 ends, the decoding process ends and the process returns to FIG.

[0255] By performing each process in this manner, the decoding unit 121 can suppress a decrease in the quality of the image after defect correction while suppressing an increase in the amount of image sensor output data, as described above.

[0256] <Scope of application of explanation> The explanation given for the higher-level method in <4. Replacement of defective pixel values> also applies to the lower-level methods belonging to that method, unless a contradiction arises. For example, when it is explained that "Method 2 may be applied," it means that Method 2-1 may also be applied.

[0257] <Combinations> Note that each of the methods described above in <4. Replacement of Defective Pixel Values> may be applied in combination with any other method as long as no contradictions arise. Furthermore, the techniques that can be combined are not limited to those shown in the table of FIG. 38 as "methods," but may include all elements described in this specification. Furthermore, each of the methods described above in <4. Replacement of Defective Pixel Values> may be applied in combination with other methods. For example, any of the methods described in <3. Transmission of Defect Information and Image Data for Transmission> may be applied in appropriate combination with any of the methods described in <4. Replacement of Defective Pixel Values>.

[0258] 5. Image Processing System Although the imaging device 100 has been described above as an example, the present technology may be applied to any configuration. For example, the present technology may be applied to a system configured with multiple devices. For example, the brightness detection unit 111, the encoding unit 114, the decoding unit 121, and the defect correction unit 122 of the imaging device 100 shown in FIG. 7 may be configured as independent devices, and the present technology may be applied to an image processing system configured with these devices.

[0259] Fig. 45 is a block diagram showing an example of the configuration of an image processing system, which is one aspect of a system to which the present technology is applied. The image processing system 500 shown in Fig. 45 is a system that executes processing similar to that of the imaging device 100 of Fig. 7. As shown in Fig. 45, the image processing system 500 includes an imaging device 511, an encoding device 512, a decoding device 513, and a signal processing device 514.

[0260] The image capturing device 511 has the same function as the brightness detection unit 111 of the image capturing device 100 , captures an image of a subject, generates a RAW image, and supplies the RAW image to the encoding device 512 .

[0261] The encoding device 512 has the same function as the encoding unit 114 of the image capturing device 100, and generates a bit stream including encoded data of the RAW image and defect information of the image capturing device 511. The encoding device 512 supplies the bit stream to the decoding device 513 via a network 520.

[0262] The decoding device 513 has the same function as the decoding unit 121 of the imaging device 100, and acquires and decodes the bit stream to generate defect information and a RAW image. The decoding device 513 supplies the defect information and the RAW image to the signal processing device 514.

[0263] The signal processor 514 acquires the defect information and the RAW image, and uses them to perform defect correction.

[0264] The present technology described in <3. Transmission of defect information for transmission and image data> or <4. Replacement of defective pixel values> may be applied to the encoding device 512 or the decoding device 513 of the image processing system 500 configured as described above. By doing so, the image processing system 500 can suppress a decrease in the quality of the image after defect correction while suppressing an increase in the amount of image sensor output data, similar to the case of the imaging device 100.

[0265] 6. Supplementary Notes Image Although the above description has been given using a RAW image as an example of an image corresponding to defect information, the image corresponding to defect information may be any image and is not limited to a RAW image.

[0266] <Computer> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware, and general-purpose personal computers, etc., that can execute various functions by installing various programs.

[0267] FIG. 46 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0268] In a computer 900 shown in FIG. 46, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0269] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0270] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 912 includes, for example, a display, a speaker, and an output terminal. The storage unit 913 includes, for example, a hard disk, a RAM disk, and a non-volatile memory. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0271] In a computer configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data necessary for the CPU 901 to execute various processes as appropriate.

[0272] The program executed by the computer can be applied by recording it on, for example, a removable medium 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable medium 921 into the drive 915.

[0273] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0274] Alternatively, this program can be installed in advance in the ROM 902 or the storage unit 913 .

[0275] <Application of the Present Technology> The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0276] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set in which other functions are added to a unit (e.g., a video set).

[0277] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.

[0278] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0279] <Fields and uses to which this technology can be applied> Systems, devices, processing units, etc. to which this technology is applied can be used in any field, for example, transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, the uses thereof are also arbitrary.

[0280] <Others> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the value that this "flag" can take may be, for example, two values, 1 / 0, or three or more values. That is, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, identification information (including flags) can be included not only in a bitstream, but also in a bitstream that includes differential information of the identification information relative to certain reference information. Therefore, in this specification, "flag" and "identification information" encompass not only the information itself, but also differential information relative to the reference information.

[0281] Furthermore, various information (e.g., metadata) related to the coded data (bitstream) may be transmitted or recorded in any form as long as it is associated with the coded data. Here, the term "associate" means, for example, making one piece of data available (linked) when processing the other piece of data. That is, data associated with each other may be combined into one piece of data or may be individual pieces of data. For example, information associated with coded data (image) may be transmitted over a transmission path separate from that of the coded data (image). Furthermore, for example, information associated with coded data (image) may be recorded on a recording medium separate from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may refer not to the entire data, but to only a portion of the data. For example, an image and information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0282] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0283] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0284] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0285] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.

[0286] Also, for example, each step of a single flowchart may be executed by a single device, or may be shared and executed by multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes may be executed by a single device, or may be shared and executed by multiple devices. In other words, multiple processes included in a single step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0287] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0288] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0289] The present technology may also be configured as follows. (1) An image processing device including: a defect information conversion unit that converts a format of defect information related to defective pixels in a pixel array for detecting luminance, and generates defect information for transmission; an image encoding unit that encodes image data representing the distribution of luminance detected in the pixel array, and generates image encoded data having a bit rate set based on the bit rate of the defect information for transmission; and a multiplexing unit that multiplexes the defect information for transmission and the image encoded data to generate a bit stream. (2) The image processing device according to (1), in which the defect information conversion unit generates the defect information for transmission for each predetermined block obtained by dividing a picture of the image data into a plurality of blocks. (3) The image processing device according to (2), in which the block is the same as an encoding block, which is a processing unit for encoding the image data by the image encoding unit. (4) The image processing device according to any one of (1) to (3), in which the format of the defect information for transmission depends on the number of defective pixels present in the pixel array. (5) The image processing device according to (4), in which the amount of data of the defect information for transmission decreases as the number of defective pixels present in the pixel array decreases. (6) The image processing device according to (4) or (5), wherein the defect information for transmission includes first information indicating the number of defective pixels and second information indicating the positions of the defective pixels. (7) The image processing device according to (6), wherein the first information includes a predetermined bit pattern according to the number of defective pixels. (8) The image processing device according to (7), wherein the bit pattern is shorter as the number of defective pixels is smaller. (9) The image processing device according to any of (6) to (8), wherein the second information indicates the positions of the defective pixels using coordinates when the number of defective pixels is smaller than a predetermined standard. (10) The image processing device according to (9), wherein the second information for transmission omits the second information when the defective pixel does not exist in the pixel array. (11) The image processing device according to any of (6) to (10), wherein the second information indicates the positions of the defective pixels using a map showing the distribution of the defective pixels when the number of the defective pixels is equal to or greater than a predetermined standard.(12) The image processing device according to any one of (6) to (11), wherein the defect information to be transmitted further includes third information indicating a type of defect. (13) The image processing device according to any one of (1) to (12), further comprising a preprocessing unit that performs predetermined preprocessing on the image data, wherein the image encoding unit is configured to encode the image data that has been subjected to the preprocessing to generate the image encoded data. (14) The image processing device according to (13), wherein the preprocessing unit replaces the pixel value of the defective pixel with a copy of the pixel value of a pixel adjacent to the defective pixel. (15) The image processing device according to (13) or (14), wherein the preprocessing unit replaces the pixel value of the defective pixel with a calculated value based on pixel values ​​of pixels surrounding the defective pixel. (16) The image processing device according to any one of (13) to (15), wherein the preprocessing unit replaces the pixel value of the defective pixel with a predetermined value. (17) An image processing method comprising: converting a format of defect information regarding defective pixels in a pixel array for detecting luminance to generate defect information for transmission, encoding image data representing the distribution of luminance detected in the pixel array to generate image coded data with a bit rate set based on the bit rate of the defect information for transmission, and multiplexing the defect information for transmission and the image coded data to generate a bit stream. (18) A program for causing a computer to execute processes comprising: converting a format of defect information regarding defective pixels in a pixel array for detecting luminance to generate defect information for transmission, encoding image data representing the distribution of luminance detected in the pixel array to generate image coded data with a bit rate set based on the bit rate of the defect information for transmission, and multiplexing the defect information for transmission and the image coded data to generate a bit stream.

[0290] (21) An image processing device comprising: a demultiplexing unit that demultiplexes a bit stream and extracts transmission defect information and image coding data; a defect information inverse conversion unit that inversely converts a format of the transmission defect information and generates defect information related to defective pixels in a pixel array that detects luminance; and an image decoding unit that decodes the image coding data using a bit rate of the image coding data estimated based on the bit rate of the transmission defect information to generate image data representing the distribution of the luminance detected in the pixel array. (22) The image processing device according to (21), wherein the defect information inverse conversion unit generates the defect information for each predetermined block obtained by dividing a picture of the image data into a plurality of blocks. (23) The image processing device according to (22), wherein the blocks are the same as coding blocks that are processing units by which the image decoding unit generates the image data. (24) The image processing device according to any of (21) to (23), wherein the format of the transmission defect information depends on the number of defective pixels present in the pixel array. (25) The image processing device according to (24), wherein the amount of data in the defect information for transmission is smaller as the number of defective pixels present in the pixel array is smaller. (26) The image processing device according to (24) or (25), wherein the defect information for transmission includes first information indicating the number of defective pixels and second information indicating the positions of the defective pixels. (27) The image processing device according to (26), wherein the first information includes a predetermined bit pattern according to the number of defective pixels. (28) The image processing device according to (27), wherein the bit pattern is shorter as the number of defective pixels is smaller. (29) The image processing device according to any of (26) to (28), wherein the second information indicates the positions of the defective pixels using coordinates when the number of defective pixels is smaller than a predetermined standard. (30) The image processing device according to (29), wherein the second information is omitted from the defect information for transmission if the defective pixel is not present in the pixel array. (31) The image processing device according to (26), wherein the second information indicates the positions of the defective pixels using a map showing a distribution of the defective pixels when the number of the defective pixels is equal to or greater than a predetermined standard.(32) The image processing device according to any one of (26) to (31), wherein the defect information for transmission further includes third information indicating a type of defect. (33) The image processing device according to any one of (21) to (32), further comprising a post-processing unit that performs predetermined post-processing on the generated image data. (34) The image processing device according to (33), wherein the post-processing unit replaces a pixel value of the defective pixel in the image data with an identification value for identifying the defective pixel. (35) The image processing device according to (34), wherein the post-processing unit changes a pixel value in the image data that is the same as the identification value of a normal pixel that is not the defective pixel to another value, and then replaces the pixel value of the defective pixel with the identification value. (36) The image processing device according to any one of (33) to (35), wherein the post-processing unit replaces the pixel value of the defective pixel in the image data with a surrounding reference value derived by referring to pixel values ​​of pixels surrounding the defective pixel. (37) The image processing device according to (36), wherein the neighboring reference value is a copy of a pixel value adjacent to the defective pixel. (38) The image processing device according to (36) or (37), wherein the neighboring reference value is a calculated value based on neighboring pixel values ​​of the defective pixel. (39) The image processing device according to any of (33) to (38), wherein the post-processing unit selects a mode of the post-processing to be applied and performs the post-processing in the selected mode. (40) The image processing device according to (39), wherein the post-processing unit selects one from an identification value replacement mode that replaces a pixel value of the defective pixel in the image data with an identification value for identifying the defective pixel, a neighboring reference mode that replaces a pixel value of the defective pixel in the image data with a neighboring reference value derived by referencing neighboring pixel values ​​of the defective pixel, and a through mode that omits replacement of the pixel value of the defective pixel, and performs the post-processing in the selected mode.(41) An image processing method comprising: demultiplexing a bit stream to extract transmission defect information and image coded data, inversely converting a format of the transmission defect information to generate defect information related to defective pixels in a pixel array for detecting luminance, and decoding the image coded data using a bit rate of the image coded data estimated based on the bit rate of the transmission defect information to generate image data representing the distribution of the luminance detected in the pixel array. (42) A program for causing a computer to execute processes comprising: demultiplexing a bit stream to extract transmission defect information and image coded data, inversely converting a format of the transmission defect information to generate defect information related to defective pixels in a pixel array for detecting luminance, and decoding the image coded data using the bit rate of the image coded data estimated based on the bit rate of the transmission defect information to generate image data representing the distribution of the luminance detected in the pixel array.

[0291] (51) An image processing device comprising: an identification value replacement unit that replaces pixel values ​​of defective pixels in a pixel array in image data representing a distribution of luminance detected in the pixel array for detecting luminance, with an identification value for identifying the defective pixel; and an encoding unit that encodes the image data in which the pixel values ​​of the defective pixels have been replaced with the identification values ​​to generate a bit stream. (52) The image processing device according to (51), in which the identification value is a predetermined fixed value. (53) The image processing device according to (51) or (52), in which the identification value is an externally set value that is set externally. (54) The image processing device according to any of (51) to (53), further comprising a normal pixel value change unit that changes pixel values ​​in the image data that are the same as the identification values ​​of normal pixels that are not defective pixels to other values, and the identification value replacement unit replaces pixel values ​​of the defective pixels in the image data in which the pixel values ​​that are the same as the identification values ​​of normal pixels that are not defective pixels have been changed to other values, with the identification value. (55) The image processing device according to any one of (51) to (54), wherein the encoding unit comprises: a discriminator that discriminates, based on the identification value, the image data in which the pixel values ​​of the defective pixels have been replaced with the identification value, into defect information about the defective pixels and the image data; a defect information encoding unit that encodes the defect information and generates defect information encoded data; an image encoding unit that encodes the image data and generates image encoded data; and a multiplexer that multiplexes the defect information encoded data and the image encoded data and generates the bit stream. (56) An image processing method comprising: replacing pixel values ​​of defective pixels in a pixel array in image data representing a distribution of luminance detected in the pixel array for detecting luminance, with an identification value for identifying the defective pixel; and encoding the image data in which the pixel values ​​of the defective pixels have been replaced with the identification value to generate a bit stream.(57) A program for causing a computer to execute a process including: replacing pixel values ​​of defective pixels in a pixel array for detecting luminance in image data representing the distribution of the luminance detected in the pixel array with an identification value for identifying the defective pixels; and encoding the image data in which the pixel values ​​of the defective pixels have been replaced with the identification value to generate a bit stream.

[0292] REFERENCE SIGNS LIST 100 Imaging device, 110 Image sensor, 111 Brightness detection unit, 112 Defect detection unit, 113 Memory unit, 114 Encoding unit, 120 Signal processing unit, 121 Decoding unit, 122 Defect correction unit, 130 Bus, 140 Defect detector, 151 Defect information conversion unit, 152 Image encoding unit, 153 Multiplexing unit, 171 Demultiplexing unit, 172 Defect information inverse conversion unit, 173 Image decoding unit, 211 Pre-processing unit, 221 Post-processing unit, 231 Normal pixel value polarization unit, 232 Discrimination value replacement unit, 241 Surrounding reference value replacement unit, 251 Mode selection unit, 311 Normal pixel value change unit, 312 Discrimination value replacement unit, 313 Encoding unit, 321 Discrimination unit, 322 Defect information encoding unit, 323 image encoding unit, 324 multiplexing unit, 331 demultiplexing unit, 332 defect information decoding unit, 333 image decoding unit, 500 image processing system, 511 imaging device, 512 encoding device, 513 decoding device, 514 signal processing device, 520 network, 900 computer

Claims

a defect information conversion unit that converts a format of defect information relating to defective pixels in a pixel array for detecting luminance, and generates defect information for transmission; an image encoding unit that encodes image data representing the luminance distribution detected in the pixel array and generates encoded image data with a bit rate set based on a bit rate of the defect information for transmission; a multiplexing unit that multiplexes the transmission defect information and the image coding data to generate a bit stream; An image processing device comprising: 　 The defect information conversion unit generates the transmission defect information for each of a plurality of predetermined blocks obtained by dividing a picture of the image data. The image processing device according to claim 1 . 　 The block is the same as an encoding block, which is a processing unit for encoding the image data by the image encoding unit. The image processing device according to claim 2 . 　 The format of the defect information to be transmitted depends on the number of the defective pixels present in the pixel array. 　 The image processing device according to claim 1 . 　 The smaller the number of defective pixels present in the pixel array, the smaller the amount of data of the defect information to be transmitted. The image processing device according to claim 4 . 　 The defect information to be transmitted includes first information indicating the number of the defective pixels and second information indicating the positions of the defective pixels. The image processing device according to claim 4 . 　 The first information includes a predetermined bit pattern that is shorter as the number of defective pixels is smaller. The image processing device according to claim 6 . 　 The second information indicates the positions of the defective pixels using coordinates when the number of the defective pixels is less than a predetermined standard. The image processing device according to claim 6 . 　 When the number of the defective pixels is equal to or greater than a predetermined standard, the second information indicates the positions of the defective pixels using a map showing the distribution of the defective pixels. The image processing device according to claim 6 . 　 The defect information for transmission further includes third information indicating a type of defect. The image processing device according to claim 6 . 　 a pre-processing unit that performs predetermined pre-processing on the image data; The image encoding unit is configured to encode the image data that has been subjected to the preprocessing, and generate the encoded image data. The image processing device according to claim 1 . 　 The preprocessing unit replaces the pixel value of the defective pixel with a copy of the pixel value of a pixel adjacent to the defective pixel. 　 The image processing device according to claim 11 . 　 The preprocessing unit replaces the pixel value of the defective pixel with a calculated value based on pixel values ​​of pixels surrounding the defective pixel. 　 The image processing device according to claim 11 . 　 converting a format of defect information relating to defective pixels in a pixel array for detecting luminance, and generating defect information for transmission; encoding image data representing the luminance distribution detected in the pixel array to generate image encoded data with a bit rate set based on a bit rate of the defect information for transmission; multiplexing the transmission defect information and the image coding data to generate a bit stream; An image processing method comprising: a demultiplexing unit for demultiplexing the bit stream and extracting the transmission defect information and the image coding data; a defect information inverse conversion unit that inversely converts the format of the defect information for transmission and generates defect information regarding defective pixels in a pixel array for detecting luminance; an image decoding unit that decodes the coded image data using a bit rate of the coded image data estimated based on a bit rate of the defect information for transmission, and generates image data that represents the distribution of the luminance detected in the pixel array; An image processing device comprising: 　 The image data is then processed by a post-processing unit. The image processing device according to claim 15. 　 The post-processing unit replaces pixel values ​​of the defective pixels in the image data with identification values ​​for identifying the defective pixels. 　 The image processing device according to claim 16. 　 The post-processing unit replaces the pixel value of the defective pixel in the image data with a surrounding reference value derived by referring to surrounding pixel values ​​of the defective pixel. 　 The image processing device according to claim 16. 　 The post-processing unit selects a mode of the post-processing to be applied and performs the post-processing in the selected mode. The image processing device according to claim 16. demultiplexing the bitstream to extract the transmission impairment information and the image coding data; converting the format of the defect information for transmission inversely to generate defect information regarding defective pixels in the pixel array for detecting luminance; decoding the image coded data using a bit rate of the image coded data estimated based on a bit rate of the defect information for transmission, and generating image data representing the distribution of the luminance detected in the pixel array; An image processing method comprising:

Images