Image shooting method and device and electronic equipment

Abstract

The invention discloses an image shooting method and device and electronic equipment, and belongs to the technical field of image processing. The method comprises the steps of obtaining a dead pixel map based on a collected test image; the dead pixel map is used for indicating defect data of the image sensor; the defect data comprises at least one of bad points, bad rows or bad lines; based on the position of each piece of defect data, generating defect coordinates of each piece of defect data; one defect coordinate represents a dead pixel, a bad row coordinate or a bad row coordinate; storing each defect coordinate in an external nonvolatile memory; and under the condition that the image sensor is powered on, reading each defect coordinate of the image sensor from the external nonvolatile memory, correcting the shot to-be-corrected image based on each defect coordinate, and obtaining and outputting the corrected image. Compared with storing the whole dead pixel map, the defect data are stored in the form of defect coordinates, so that the required storage space can be effectively reduced, and the hardware cost is effectively reduced.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to an image shooting method, device and electronic equipment. Background Art

[0002] The presence of bad pixels in image sensors may lead to bad pixels (bad pixels for short) in the captured images. Bad pixels refer to pixels in the image sensor that cannot be normally sensitive to light. A bad pixel map refers to an image with a resolution the same as the camera output resolution of the image sensor, which is used to indicate bad pixels in the image sensor. Different marks are used to distinguish bad pixels from normal pixels in the bad pixel map. For example, 0 represents a normal pixel and 1 represents a bad pixel. The resolution of the bad pixel map is related to the camera output resolution. For image sensors with higher camera output resolutions, a non-volatile memory with a large storage space (for example, Flash) is required to store the bad pixel map, which increases hardware costs. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes an image shooting method, device and electronic device, which can effectively reduce the required storage space and hardware cost by storing each defect data in the form of defect coordinates compared to storing the entire bad pixel map.

[0004] In a first aspect, the present application provides an image capturing method, the method comprising:

[0005] Based on the collected test image, a bad pixel map is obtained; the bad pixel map is used to indicate defect data of the image sensor; the defect data includes at least one of a bad pixel and a bad line;

[0006] Based on the position of each defect data, a defect coordinate of each defect data is generated; a defect coordinate represents the coordinate of a bad point or a bad line;

[0007] Storing each defect coordinate in an external non-volatile memory;

[0008] When the image sensor is powered on, the coordinates of each defect of the image sensor are read from the external non-volatile memory, so as to correct the captured image to be corrected based on the coordinates of each defect, and obtain and output the corrected image.

[0009] According to the image capture method of the present application, a bad pixel map is obtained based on the collected test image; the bad pixel map is used to indicate the defect data of the image sensor; the defect data includes at least one of bad pixels and bad lines, and based on the position of each defect data, the defect coordinates of each defect data are generated; a defect coordinate represents the coordinates of the bad pixel or the bad line; each defect coordinate is stored in an external non-volatile memory. After all, the defect data in the bad pixel map is small in number. No matter what the camera output resolution is, compared with storing the entire bad pixel map, storing each defect data in the form of defect coordinates can effectively reduce the required storage space and effectively reduce the hardware cost.

[0010] According to an embodiment of the present application, each defect coordinate of the image sensor is read from an external non-volatile memory to correct the captured image to be corrected based on each defect coordinate, and obtain and output the corrected image, including:

[0011] The coordinates of each defect in the external non-volatile memory are cached in the block random access memory BRAM; the image to be corrected is corrected based on the following method:

[0012] Loading at least one set of defect coordinates from the BRAM into the register in sequence;

[0013] In each clock cycle, the pixel value of the pixel point corresponding to the clock cycle is read from the image to be corrected;

[0014] For each pixel point in each clock cycle, when it is determined that the first defect coordinate in the register is the coordinate of the bad pixel and the coordinate of the pixel point is the same as the first defect coordinate, or when it is determined that the first defect coordinate is the coordinate of the bad line and the pixel point is located on the bad line, the pixel value of the pixel point is corrected to obtain and output the corrected pixel value of the pixel point;

[0015] When the first defect coordinate in the register is determined to be the coordinate of the bad point but the coordinate of the pixel point is different from the first defect coordinate, or when the first defect coordinate is determined to be the coordinate of the bad line but the pixel point is not located on the bad line, the pixel value of the pixel point is output.

[0016] According to an embodiment of the present application, when it is determined that the first defect coordinate in the register is the coordinate of the bad pixel and the coordinate of the pixel point is the same as the first defect coordinate, the pixel value of the pixel point is corrected, and the corrected pixel value of the pixel point is obtained and output, the method further includes:

[0017] Delete the first defect coordinates from the register;

[0018] When the register is not empty, the next defect coordinate stored in the register is updated to be the first defect coordinate of the register;

[0019] When the register is empty, at least one set of defect coordinates after the deleted defect coordinates are loaded from the BRAM to the register in sequence.

[0020] According to an embodiment of the present application, when it is determined that the first defect coordinates are the coordinates of a bad line and the pixel point is located on the bad line, the pixel value of the pixel point is corrected, and after obtaining and outputting the corrected pixel value of the pixel point, the method further includes:

[0021] When the coordinate of the pixel point is determined to be the last coordinate of the bad line, the first defective coordinate is deleted from the register;

[0022] When the register is not empty, the next defect coordinate stored in the register is updated to be the first defect coordinate of the register;

[0023] When the register is empty, at least one set of defect coordinates after the deleted defect coordinates are loaded from the BRAM to the register in sequence.

[0024] According to an embodiment of the present application, each defect coordinate in an external non-volatile memory is cached in a block random access memory BRAM, including:

[0025] Based on the number of channels and the stitching method, determine the target number of BRAMs required and the depth and bit width of each BRAM;

[0026] Instantiate each BRAM based on its bit width and depth;

[0027] Read each bit of the horizontal coordinate and the vertical coordinate of each defect coordinate from the BRAM;

[0028] Based on the splicing method, the respective proportions of the horizontal coordinates and the vertical coordinates of the defect coordinates are spliced ​​to obtain the splicing results;

[0029] Each splicing result is stored in the corresponding BRAM.

[0030] According to an embodiment of the present application, determining the target number of required BRAMs and the depth and bit width of each BRAM includes:

[0031] When the number of channels is one and the stitching mode is the first stitching mode, the number of targets is determined to be one, the depth of the BRAM is the number of defect coordinates, and the width is a first value; the first value is the sum of the number of bits of the horizontal coordinate and the vertical coordinate;

[0032] When there are multiple channels and the splicing method is the first splicing method, determine the target number to be the number of channels, the depth of the BRAM to be the second value, and the width to be the first value; or,

[0033] When the number of channels is multiple and the splicing mode is the second splicing mode, the target number is determined to be two, the depth of the BRAM is the second value, and the width is the product of the number of channels and the number of bits of the horizontal coordinate or the vertical coordinate;

[0034] The second value is obtained by rounding up the ratio of the number of defect coordinates to the number of channels; the first splicing method is to splice the horizontal coordinate and vertical coordinate of a defect coordinate; the second splicing method is to splice the horizontal coordinates or vertical coordinates of the number of channels.

[0035] According to an embodiment of the present application, the test image includes at least one of a dark field image, a middle gray image, and a saturated image under a uniform field.

[0036] In a second aspect, the present application provides an image capturing device, the device comprising:

[0037] A first processing module is used to obtain a bad pixel map based on the collected test image; the bad pixel map is used to indicate defect data of the image sensor; the defect data includes at least one of a bad pixel and a bad line;

[0038] The second processing module is used to generate defect coordinates of each defect data based on the position of each defect data; a defect coordinate represents the coordinates of a bad point or a bad line;

[0039] A storage module, used for storing the coordinates of each defect in an external non-volatile memory;

[0040] The third processing module is used to read the defect coordinates of the image sensor from the external non-volatile memory when the image sensor is powered on, so as to correct the captured image to be corrected based on the defect coordinates, and obtain and output the corrected image.

[0041] According to the image capture device of the present application, a bad pixel map is obtained based on a collected test image; the bad pixel map is used to indicate defect data of the image sensor; the defect data includes at least one of bad pixels and bad lines, and defect coordinates of each defect data are generated based on the position of each defect data; a defect coordinate represents the coordinates of a bad pixel or a bad line; each defect coordinate is stored in an external non-volatile memory. After all, the defect data in the bad pixel map is small in number. No matter what the camera output resolution is, compared with storing the entire bad pixel map, storing each defect data in the form of defect coordinates can effectively reduce the required storage space and effectively reduce the hardware cost.

[0042] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image capturing method provided in the first aspect is implemented.

[0043] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image capturing method provided in the first aspect above.

[0044] In a fifth aspect, the present application provides a chip, the chip comprising a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run a program or instruction to implement the image capture method provided in the first aspect.

[0045] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the image capture method provided in the first aspect above.

[0046] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0047] A bad pixel map is obtained based on the collected test image; the bad pixel map is used to indicate defect data of the image sensor; the defect data includes at least one of bad pixels and bad lines, and defect coordinates of each defect data are generated based on the position of each defect data; a defect coordinate represents the coordinate of a bad pixel or a bad line; each defect coordinate is stored in an external non-volatile memory. After all, the defect data in the bad pixel map is small in number. No matter what the camera output resolution is, compared with storing the entire bad pixel map, storing each defect data in the form of defect coordinates can effectively reduce the required storage space and effectively reduce the hardware cost.

[0048] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0050] Figure 1 This is one of the flowcharts of the image capturing method provided in the embodiment of the present application;

[0051] Figure 2 is a schematic diagram of a bad pixel map provided in an embodiment of the present application;

[0052] Figure 3 is a structural schematic diagram of an image capturing device provided in an embodiment of the present application;

[0053] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0055] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0056] Image sensors, as a key component for image acquisition, are widely used in various products and applications, such as industrial inspection, automotive assisted driving, security monitoring and other industries. The production process of image sensors has a great impact on the yield of image sensors. Among them, bad pixels, bad lines (bad rows and bad columns), and bad clusters are common defects in image sensors. These defects cause some pixels (isolated points, clusters, and lines in different forms) to be unable to sense light normally, resulting in inconsistent responses with other normal pixels. With the continuous advancement of image sensor technology, the number of bad pixels, bad lines, and bad clusters on image sensors has been greatly reduced, but they still exist. For applications with high image quality requirements, these defects still need to be corrected.

[0057] Cameras for industrial applications have high requirements for real-time image processing, so they often use Field-Programmable Gate Arrays (FPGAs) as core processing devices, which can complete image processing with very low latency, but require the use of logic and storage resources on the FPGA. The current conventional solution is to mark bad pixels, bad clusters, and bad lines in advance to form a bad pixel map (the resolution of the bad pixel map is the same as the resolution of the camera output image, and different marks are used to distinguish bad pixels from normal pixels, for example, 0 represents a normal pixel and 1 represents a bad pixel).

[0058] like Figure 2As shown, an embodiment of the present application provides a schematic diagram of a bad pixel map, where W (width) = 640, H height = 512, 0 represents a normal pixel, and 1 represents a bad pixel. The bad pixel map is usually stored in a non-volatile memory (e.g., Flash) of the camera as a basis for correcting the pixel value of the pixel. The bad pixel map in the non-volatile memory is loaded as soon as the camera is powered on. It needs to be read into the DDR (Double Data Rate) memory first, and then part of the data is cached in the BRAM (Block Random Access Memory) of the FPGA. On the one hand, the system design is complex and the data access delay is high. On the other hand, the bad pixel map changes with the resolution of the camera output image. For a high-resolution camera, a non-volatile memory with a large storage space is required, which increases the hardware cost.

[0059] To solve the above problems, embodiments of the present application provide an image capturing method, an image capturing device, an electronic device, and a readable storage medium.

[0060] The image capturing method, image capturing device, electronic device and readable storage medium provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0061] The image capturing method may be applied to a terminal having an image capturing function, and may be specifically executed by hardware or software in the terminal.

[0062] The image capturing method provided in the embodiment of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the image capturing method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The image capturing method provided in the embodiment of the present application is described below using the electronic device as an example of the execution subject.

[0063] like Figure 1 As shown, the image capturing method includes: step 110, step 120, step 130 and step 140.

[0064] Step 110: Obtain a bad pixel map based on the collected test image; the bad pixel map is used to indicate defect data of the image sensor; the defect data includes at least one of a bad pixel and a bad line.

[0065] The test image in the embodiment of the present application includes at least one of a dark field image, a middle gray image, and a saturated image under a uniform field (the illumination, color, and brightness of each part are very consistent).

[0066] Among them, dark field images refer to images taken under almost no light.

[0067] A neutral gray image is an image taken when the camera receives a uniform light source of medium intensity.

[0068] A saturated image is an image captured by the camera where the exposure intensity has exceeded the dynamic range of the sensor, causing some areas of the image to reach the maximum brightness value. In this case, the brightness value in the image cannot be increased any further and the sensor will appear to be "saturated".

[0069] The bad pixels (bad clusters are multiple bad pixels) and bad lines of the above test images can be calibrated according to the bad pixel definition in the image sensor manual or the detection standards customized by the camera company. After combining the test images, the bad pixel map of the image sensor can be obtained.

[0070] A bad pixel refers to a pixel that cannot sense light normally.

[0071] A bad line can be a bad row or a bad column.

[0072] A bad row refers to a pixel row in which all pixels in the entire pixel row are bad pixels.

[0073] A bad column refers to a pixel column in which all pixels in the entire pixel column are bad pixels.

[0074] The resolution of the bad pixel map is the same as the camera output resolution of the image sensor. The bad pixel map is used to indicate defect data of the image sensor, and the defect data includes at least one of bad pixels, bad rows or bad lines. For example, 0 represents a normal pixel, normal row or normal column, and 1 represents a bad pixel, bad row or bad line.

[0075] Step 120 : Generate defect coordinates of each defect data based on the position of each defect data; a defect coordinate represents the coordinates of a bad pixel or a bad line.

[0076] The embodiment of the present application generates defect coordinates corresponding to each defect data, that is, one defect coordinate is used to represent a bad pixel, one defect coordinate is used to represent a bad row, and one defect coordinate is used to represent a bad column.

[0077] Specifically, for example, for a bad pixel, the coordinates (X, Y) of each point need to be recorded, where X represents the horizontal coordinate and Y represents the vertical coordinate.

[0078] The bad row records the row coordinates. The X coordinate of the bad row can be specially marked to distinguish it from the X coordinate of the bad pixel. For example, the minimum X coordinate of the bad pixel starts from 1, while the X coordinate of the bad row can start from 0, for example (0, Y), indicating that the Yth row is a bad row.

[0079] Similarly, the bad column records the column coordinates, and the Y coordinate of the bad column can be specially marked to distinguish it from the Y coordinate of the bad point. For example, the minimum value of the Y coordinate of the bad point starts from 1, while the Y coordinate of the bad column can start from 0, for example (X, 0), indicating that the Xth column is a bad column.

[0080] In the embodiment of the present application, bad pixels, bad rows and bad columns only need to be recorded once, and there is no need to repeatedly record the coordinates of each bad pixel contained in the bad row or bad column.

[0081] As shown in Table 1, the present embodiment provides coordinates of various defects. Table 1 is for the above Figure 2 The defect map in the generated

[0082] X coordinate Y coordinate 4 0 9 2 2 4 8 4 7 5 8 5 ... ... 639 511

[0083] Table 1

[0084] Figure 2 The 4th pixel column is a bad column, so the defect coordinates of the defective column can be (4,0).

[0085] Figure 2 The pixel in the 9th column and the 2nd row is a bad pixel, and the defect coordinates of the bad pixel are (9, 2); the pixel in the 2nd column and the 4th row is a bad pixel, and the defect coordinates of the bad pixel are (2, 4); the pixel in the 8th column and the 4th row is a bad pixel, and the defect coordinates of the bad pixel are (8, 4); the pixel in the 7th column and the 5th row is a bad pixel, and the defect coordinates of the bad pixel are (7, 5); the pixel in the 8th column and the 5th row is a bad pixel, and the defect coordinates of the bad pixel are (8, 5); ...; the pixel in the 639th column and the 511th row is a bad pixel, and the defect coordinates of the bad pixel are (639, 511).

[0086] Step 130: Store the coordinates of each defect in an external non-volatile memory.

[0087] After obtaining the coordinates of each defect, the embodiment of the present application can store the defect data in a non-volatile memory (NVM) in the order from top to bottom and from left to right of the image.

[0088] Non-volatile memory refers to memory that can maintain its data without losing it when the power is off or the power fails. This means that the data in the memory will not be cleared whether the power is turned off, the power is cut off or restarted.

[0089] The non-volatile memory in the embodiment of the present application may be any one or more of a flash memory, a read-only memory (ROM), a magnetic memory, an optical disc, etc., without limitation thereto.

[0090] Step 140 : When the image sensor is powered on, the coordinates of each defect of the image sensor are read from the external non-volatile memory, so as to correct the captured image to be corrected based on the coordinates of each defect, and obtain and output the corrected image.

[0091] In the embodiment of the present application, when the image sensor is powered on, the processing chip FPGA can read the defect coordinates of the image sensor from the external non-volatile memory. Since the image sensor is a sensor with defect data, the image captured by the image sensor is the image to be corrected. The captured image to be corrected needs to be corrected according to the defect coordinates, that is, the pixel values ​​of the pixels corresponding to the defect coordinates are corrected to obtain a corrected image. The image sensor can output the corrected image.

[0092] The embodiment of the present application obtains a bad pixel map based on the collected test image; the bad pixel map is used to indicate the defect data of the image sensor; the defect data includes at least one of bad pixels, bad rows or bad lines, and based on the position of each defect data, the defect coordinates of each defect data are generated; a defect coordinate represents the coordinates of a bad pixel, a bad row or a bad row; each defect coordinate is stored in an external non-volatile memory. After all, the defect data in the bad pixel map is small. No matter what the camera output resolution is, compared with storing the entire bad pixel map, storing each defect data in the form of defect coordinates can effectively reduce the required storage space and effectively reduce the hardware cost.

[0093] In some embodiments, the coordinates of each defect of the image sensor are read from an external non-volatile memory to correct the captured image to be corrected based on the coordinates of each defect, and the corrected image is obtained and output, including:

[0094] The coordinates of each defect in the external non-volatile memory are cached in the block random access memory BRAM; the image to be corrected is corrected based on the following method:

[0095] Loading at least one set of defect coordinates from the BRAM into the register in sequence;

[0096] In each clock cycle, the pixel value of the pixel point corresponding to the clock cycle is read from the image to be corrected; the number of pixel values ​​in each clock cycle is the number of channels of the image sensor;

[0097] For each pixel point in each clock cycle, when it is determined that the first defect coordinate in the register is the coordinate of the bad pixel and the coordinate of the pixel point is the same as the first defect coordinate, or when it is determined that the first defect coordinate is the coordinate of the bad line and the pixel point is located on the bad line, the pixel value of the pixel point is corrected to obtain and output the corrected pixel value of the pixel point;

[0098] When the first defect coordinate in the register is determined to be the coordinate of the bad point but the coordinate of the pixel point is different from the first defect coordinate, or when the first defect coordinate is determined to be the coordinate of the bad line but the pixel point is not located on the bad line, the pixel value of the pixel point is output.

[0099] As soon as the machine is powered on, the FPGA reads the coordinates, X coordinates and Y coordinates of each defect from the external non-volatile memory, and caches the coordinates of each defect in the external non-volatile memory in the block random access memory BRAM.

[0100] In some embodiments, each defect coordinate in an external non-volatile memory is cached in a block random access memory BRAM, including:

[0101] Based on the number of channels and the stitching method, determine the target number of BRAMs required and the depth and bit width of each BRAM;

[0102] Instantiate each BRAM based on its bit width and depth;

[0103] Read each bit of the horizontal coordinate and the vertical coordinate of each defect coordinate from the BRAM;

[0104] Based on the splicing method, the respective proportions of the horizontal coordinates and the vertical coordinates of the defect coordinates are spliced ​​to obtain the splicing results;

[0105] Each splicing result is stored in the corresponding BRAM.

[0106] The number of channels refers to the amount of image data read in one cycle, where the image data refers to the pixel value of the pixel.

[0107] The stitching method includes a first stitching method and a second stitching method. The first stitching method is to stitch the horizontal coordinate and the vertical coordinate of a defect coordinate; the second stitching method is to stitch the horizontal coordinate or the vertical coordinate of the number of channels. For example, if there are 4 channels, then 4 X coordinates are stitched and 4 Y coordinates are stitched.

[0108] For image sensors with a single channel (one tap channel), in order to ensure the synchronization of the X coordinate and the Y coordinate, the X coordinate and the Y coordinate can be spliced ​​and stored in a BRAM. When the BRAM is instantiated, the bit width can be set to the sum of the proportion of the X coordinate and the proportion of the Y coordinate. The depth is designed according to the number S of defect coordinates.

[0109] For image sensors with multiple channels (multiple tap channels), the X-coordinate and Y-coordinate data can be spliced ​​and stored in a BRAM. When the BRAM is instantiated, the bit width can be set to the sum of the proportion of the X-coordinate and the proportion of the Y-coordinate. The depth is □ the number of defect coordinates / the number of Taps N□, "□□" means rounding up, so that N BRAMs are instantiated according to the number of Taps.

[0110] For image sensors with multiple channels, the horizontal coordinates or vertical coordinates of the channels can also be spliced, and a bit width is instantiated as the splicing bit width. The splicing bit width is the product of the number of channels and the number of bits that the horizontal coordinate or vertical coordinate accounts for. The depth is the number of defect coordinates / the number of Taps N. Similarly, the Y coordinates of all Taps are spliced ​​and BRAM is instantiated. A total of 2 BRAMs need to be instantiated.

[0111] BRAM Instantiation refers to the explicit instantiation of block RAM (BRAM) resources as available hardware modules in FPGA design. In FPGA design, BRAM is usually used to store data. It is an efficient storage unit built into the FPGA that can provide high-speed read and write operations. Through BRAM instantiation, designers can directly configure and use the FPGA's BRAM resources in the hardware description language, thereby achieving high-speed access to data.

[0112] After the BRAM is instantiated, each splicing result may be stored in the corresponding instantiated BRAM.

[0113] In some embodiments, determining a target number of BRAMs required and a depth and a bit width of each BRAM includes:

[0114] When the number of channels is one and the stitching mode is the first stitching mode, the number of targets is determined to be one, the depth of the BRAM is the number of defect coordinates, and the width is a first value; the first value is the sum of the number of bits of the horizontal coordinate and the vertical coordinate;

[0115] When there are multiple channels and the splicing method is the first splicing method, determine the target number to be the number of channels, the depth of the BRAM to be the second value, and the width to be the first value; or,

[0116] When the number of channels is multiple and the splicing mode is the second splicing mode, the target number is determined to be two, the depth of the BRAM is the second value, and the width is the product of the number of channels and the number of bits of the horizontal coordinate or the vertical coordinate;

[0117] The second value is obtained by rounding up the ratio of the number of defect coordinates to the number of channels.

[0118] Continuing with the above embodiment, when the number of channels is one and the splicing mode is the first splicing mode, the target number of BRAMs is determined to be one, the depth of the BRAM is the number of defect coordinates S, and the width is the first value; the first value is the sum of the number of bits of the horizontal coordinate and the vertical coordinate. Specifically, assuming that the X coordinate is in the upper 16 bits, the Y coordinate is in the lower 16 bits, and the number of defect coordinates is S, then the bit width can be set to 32 bits when the BRAM is instantiated, and the depth is the number of defect coordinates S.

[0119] When there are multiple channels, there are two splicing methods as described above and will not be described here.

[0120] When there are multiple channels and the stitching method is the first stitching method, the target number is determined to be the number of channels, the depth of the BRAM is the second value, which is obtained by rounding up the ratio of the number of defect coordinates to the number of channels, and the width is the first value, which is the sum of the number of bits of the horizontal coordinate and the vertical coordinate.

[0121] When the number of channels is multiple and the splicing method is the second splicing method, the target number is determined to be two, the depth of the BRAM is the second value, and the width is the product of the number of channels and the number of bits of the horizontal coordinate or the vertical coordinate. Specifically, for example, if the number of channels is 4, the X coordinate is in the upper 16 bits, and the Y coordinate is in the lower 16 bits, then the bit width of the BRAM is 4×16=64 bits.

[0122] In practical applications, in the case of multiple channels, the first splicing mode and the second splicing mode can be selected to perform data caching.

[0123] After the image sensor is powered on and the image to be corrected is captured, in each clock cycle, the pixel value of the pixel corresponding to the clock cycle is read from the image to be corrected; the reading order can be from top to bottom and from left to right, and the number of pixel values ​​in each clock cycle is the number of channels of the image sensor, that is, a single Tap processes the pixel value of one pixel per clock, and a multi-Tap processes the pixel value of N pixels per clock, where N is the number of Taps. Row count row_cot and column count col_cnt are performed on the pixels processed in real time.

[0124] At the same time, read at least one set of defect coordinates (no less than 1 set, it is recommended to read 2 sets) from BRMA to the register in advance. A set of defect coordinates includes multiple defect coordinates, usually N, where N is the number of taps. The register is a small and fast storage area in the central processing unit (CPU), usually used to store temporary data required for the current calculation.

[0125] For each pixel point in each clock cycle, when it is determined that the first defect coordinate in the register is the coordinate of the bad pixel and the coordinate of the pixel point is the same as the first defect coordinate, that is, the X coordinate of the pixel point is the same as the X coordinate of the first defect coordinate, and the Y coordinate of the pixel point is also the same as the Y coordinate of the first defect coordinate, it means that the pixel point is a bad pixel, and the pixel value of the pixel point needs to be corrected to obtain and output the corrected pixel value of the pixel point.

[0126] When it is determined that the first defect coordinate is the coordinate of a bad line (bad row or bad column) and the pixel is located on the bad line, the pixel value of the pixel is corrected to obtain and output the corrected pixel value of the pixel. The X coordinate of the bad row is 0, and the Y coordinate is not 0. Assuming that the defect coordinate of a bad row is (0,7), and the coordinate of a pixel is (2,7), it can be found that the pixel is located in the bad row, so the pixel value of the pixel needs to be corrected.

[0127] Similarly, the Y coordinate of the bad column is 0, and the X coordinate is not 0. Assuming that the defect coordinates of a bad column are (3,0), and the coordinates of a pixel are (3,5), it can be found that the pixel is located in the bad column, so the pixel value of the pixel needs to be corrected.

[0128] Normal pixels in the conventional minimum gradient direction may be used to replace bad pixel data through reasonable calculations, or other methods may be used for correction, without limitation.

[0129] In some embodiments, when it is determined that the first defect coordinate in the register is the coordinate of the bad pixel and the coordinate of the pixel point is the same as the first defect coordinate, the pixel value of the pixel point is corrected, and the corrected pixel value of the pixel point is obtained and output, the method further includes:

[0130] Delete the first defect coordinates from the register;

[0131] When the register is not empty, the next defect coordinate stored in the register is updated to be the first defect coordinate of the register;

[0132] When the register is empty, at least one set of defect coordinates after the deleted defect coordinates are loaded from the BRAM to the register in sequence.

[0133] Continuing with the above embodiment, after the pixel value of the bad pixel is corrected, the defect coordinate of the bad pixel needs to be deleted from the register, that is, the first defect coordinate in the register is deleted. If the register is not empty, the next defect coordinate stored in the register is used as the new first defect coordinate.

[0134] When the register is empty, at least one set of defect coordinates after the deleted defect coordinates are sequentially loaded from the BRAM to the register, that is, at least one set of defect coordinates is re-read from the register to perform judgment in the next clock cycle.

[0135] When the row count and the column count are the maximum counts of the image to be corrected, the correction of the image to be corrected is completed.

[0136] In some embodiments, when it is determined that the first defect coordinate is the coordinate of a bad line and the pixel point is located on the bad line, the pixel value of the pixel point is corrected, and after obtaining and outputting the corrected pixel value of the pixel point, the method further includes:

[0137] When the coordinate of the pixel point is determined to be the last coordinate of the bad line, the first defective coordinate is deleted from the register;

[0138] When the register is not empty, the next defect coordinate stored in the register is updated to be the first defect coordinate of the register;

[0139] When the register is empty, at least one defect coordinate after the deleted defect coordinate is loaded from the BRAM to the register in sequence.

[0140] Continuing with the above embodiment, it should be noted that the defect coordinates also need to determine whether it is a bad line (there is 0 or other identifier in the X coordinate or Y coordinate). If it is a bad line, the defect coordinates need to be retained until the last pixel point on the bad line is processed before the coordinates can be removed from the register.

[0141] Through the above process, it can be found that in the process of capturing and correcting images, the present application does not need to load defect data into the DDR memory, which greatly improves the processing speed and reduces data access delay.

[0142] The image capturing method provided in the embodiment of the present application may be executed by an image capturing device. In the embodiment of the present application, an image capturing device executing the image capturing method is taken as an example to illustrate the image capturing device provided in the embodiment of the present application.

[0143] The embodiment of the present application also provides an image capturing device.

[0144] like Figure 3 As shown, the image capturing device includes: a first processing module 310 , a second processing module 320 , a storage module 330 and a third processing module 340 .

[0145] The first processing module 310 is used to obtain a bad pixel map based on the collected test image; the bad pixel map is used to indicate defect data of the image sensor; the defect data includes at least one of a bad pixel and a bad line;

[0146] The second processing module 320 is used to generate defect coordinates of each defect data based on the position of each defect data; a defect coordinate represents the coordinates of a bad point or a bad line;

[0147] The storage module 330 is used to store the coordinates of each defect in an external non-volatile memory;

[0148] The third processing module 340 is used to read the defect coordinates of the image sensor from the external non-volatile memory when the image sensor is powered on, so as to correct the captured image to be corrected based on the defect coordinates, and obtain and output the corrected image.

[0149] According to the image capture device provided by the embodiment of the present application, a bad pixel map is obtained based on the collected test image; the bad pixel map is used to indicate the defect data of the image sensor; the defect data includes at least one of bad pixels and bad lines, and based on the position of each defect data, defect coordinates of each defect data are generated; a defect coordinate represents the coordinates of a bad pixel or a bad line; each defect coordinate is stored in an external non-volatile memory. After all, the defect data in the bad pixel map is small in number. No matter what the camera output resolution is, compared with storing the entire bad pixel map, storing each defect data in the form of defect coordinates can effectively reduce the required storage space and effectively reduce the hardware cost.

[0150] In some embodiments, the third processing module 340 includes:

[0151] A first processing submodule is used to cache each defect coordinate in the external non-volatile memory in a block random access memory BRAM;

[0152] The first processing submodule is used to correct the image to be corrected based on the following method:

[0153] Loading at least one set of defect coordinates from the BRAM into the register in sequence;

[0154] In each clock cycle, the pixel value of the pixel point corresponding to the clock cycle is read from the image to be corrected;

[0155] For each pixel point in each clock cycle, when it is determined that the first defect coordinate in the register is the coordinate of the bad pixel and the coordinate of the pixel point is the same as the first defect coordinate, or when it is determined that the first defect coordinate is the coordinate of the bad line and the pixel point is located on the bad line, the pixel value of the pixel point is corrected to obtain and output the corrected pixel value of the pixel point;

[0156] When the first defect coordinate in the register is determined to be the coordinate of the bad point but the coordinate of the pixel point is different from the first defect coordinate, or when the first defect coordinate is determined to be the coordinate of the bad line but the pixel point is not located on the bad line, the pixel value of the pixel point is output.

[0157] In some embodiments, the second processing submodule is specifically used for: when it is determined that the first defect coordinate in the register is the coordinate of a bad pixel and the coordinate of the pixel point is the same as the first defect coordinate, correcting the pixel value of the pixel point, obtaining and outputting the corrected pixel value of the pixel point, and then deleting the first defect coordinate from the register; when the register is not empty, updating the next defect coordinate stored in the register to be the first defect coordinate of the register; when the register is empty, loading at least one set of defect coordinates after the deleted defect coordinate from the BRAM to the register in sequence.

[0158] In some embodiments, the second processing submodule is further configured to:

[0159] When it is determined that the first defect coordinate is the coordinate of the bad line and the pixel point is located on the bad line, the pixel value of the pixel point is corrected, and after the corrected pixel value of the pixel point is obtained and output, when it is determined that the coordinate of the pixel point is the last coordinate of the bad line, the first defect coordinate is deleted from the register;

[0160] When the register is not empty, the next defect coordinate stored in the register is updated to be the first defect coordinate of the register;

[0161] When the register is empty, at least one set of defect coordinates after the deleted defect coordinates are loaded from the BRAM to the register in sequence.

[0162] In some embodiments, the first processing submodule is specifically used for:

[0163] Based on the number of channels and the stitching method, determine the target number of BRAMs required and the depth and bit width of each BRAM;

[0164] Instantiate each BRAM based on its bit width and depth;

[0165] Read each bit of the horizontal coordinate and the vertical coordinate of each defect coordinate from the BRAM;

[0166] Based on the splicing method, the respective proportions of the horizontal coordinates and the vertical coordinates of the defect coordinates are spliced ​​to obtain the splicing results;

[0167] Each splicing result is stored in the corresponding BRAM.

[0168] In some embodiments, the first processing submodule is further configured to:

[0169] When the number of channels is one and the stitching mode is the first stitching mode, the number of targets is determined to be one, the depth of the BRAM is the number of defect coordinates, and the width is a first value; the first value is the sum of the number of bits of the horizontal coordinate and the vertical coordinate;

[0170] When there are multiple channels and the splicing method is the first splicing method, determine the target number to be the number of channels, the depth of the BRAM to be the second value, and the width to be the first value; or,

[0171] When the number of channels is multiple and the splicing mode is the second splicing mode, the target number is determined to be two, the depth of the BRAM is the second value, and the width is the product of the number of channels and the number of bits of the horizontal coordinate or the vertical coordinate;

[0172] The second value is obtained by rounding up the ratio of the number of defect coordinates to the number of channels; the first splicing method is to splice the horizontal coordinate and vertical coordinate of a defect coordinate; the second splicing method is to splice the horizontal coordinates or vertical coordinates of the number of channels.

[0173] In some embodiments, the test image includes at least one of a dark field image, a middle gray image, and a saturated image under a uniform field.

[0174] The image capture device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0175] The image capture device in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0176] The image capture device provided in the embodiment of the present application can achieve Figure 1 to Figure 2 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0177] In some embodiments, Figure 4 As shown, an embodiment of the present application further provides an electronic device 400, including a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the program is executed by the processor 401, each process of the above-mentioned image capturing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0178] Processor 401 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 401 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0179] The memory 402 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0180] The memory 402 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 401. The processor 401 is used to execute the computer program stored in the memory 402 to implement the steps shown in the above method embodiment.

[0181] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0182] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned image capture method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0183] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0184] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned image capturing method when executed by a processor.

[0185] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0186] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image capture method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0187] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0188] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0189] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0190] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0191] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0192] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. An image capturing method, characterized in that: The method comprises: Based on the collected test image, a bad pixel map is obtained; the bad pixel map is used to indicate defect data of the image sensor; the defect data includes at least one of a bad pixel and a bad line; Based on the position of each defect data, a defect coordinate of each defect data is generated; a defect coordinate represents the coordinate of a bad point or a bad line; Storing each defect coordinate in an external non-volatile memory; When the image sensor is powered on, the defect coordinates of the image sensor are read from the external non-volatile memory, so as to correct the captured image to be corrected based on the defect coordinates, and obtain and output the corrected image.

2. The image capturing method according to claim 1, characterized in that: The step of reading the defect coordinates of the image sensor from the external non-volatile memory, correcting the captured image to be corrected based on the defect coordinates, and obtaining and outputting the corrected image comprises: The coordinates of each defect in the external non-volatile memory are cached in a block random access memory BRAM; and the image to be corrected is corrected based on the following method: sequentially loading at least one set of defect coordinates from the BRAM into a register; In each clock cycle, a pixel value of a pixel point corresponding to the clock cycle is read from the image to be corrected; For each pixel point in each clock cycle, when it is determined that the first defect coordinate in the register is the coordinate of the bad pixel and the coordinate of the pixel point is the same as the first defect coordinate, or when it is determined that the first defect coordinate is the coordinate of the bad line and the pixel point is located on the bad line, the pixel value of the pixel point is corrected to obtain and output the corrected pixel value of the pixel point; When it is determined that the first defect coordinate in the register is the coordinate of a bad point but the coordinate of the pixel point is different from the first defect coordinate, or when it is determined that the first defect coordinate is the coordinate of a bad line but the pixel point is not located on the bad line, the pixel value of the pixel point is output.

3. The image capturing method according to claim 2, characterized in that: In the case where it is determined that the first defect coordinate in the register is the coordinate of a bad pixel and the coordinate of the pixel point is the same as the first defect coordinate, after correcting the pixel value of the pixel point and obtaining and outputting the corrected pixel value of the pixel point, the method further includes: Deleting the first defect coordinate from the register; If the register is not empty, updating the next defect coordinate stored in the register to be the first defect coordinate of the register; When the register is empty, at least one set of defect coordinates after the deleted defect coordinates are loaded from the BRAM to the register in sequence.

4. The image capturing method according to claim 2, characterized in that: In the case where it is determined that the first defect coordinate is the coordinate of the bad line and the pixel point is located on the bad line, the pixel value of the pixel point is corrected, and after the corrected pixel value of the pixel point is obtained and output, the method further includes: In the case where it is determined that the coordinate of the pixel point is the last coordinate of the bad line, deleting the first defective coordinate from the register; If the register is not empty, updating the next defect coordinate stored in the register to be the first defect coordinate of the register; When the register is empty, at least one set of defect coordinates after the deleted defect coordinates are loaded from the BRAM to the register in sequence.

5. The image capturing method according to claim 2, characterized in that: The step of caching each defect coordinate in the external non-volatile memory in a block random access memory BRAM comprises: Based on the number of channels and the stitching method, determine the target number of BRAMs required and the depth and bit width of each BRAM; Instantiate each BRAM based on its bit width and depth; Reading each bit of the horizontal coordinate and the vertical coordinate of each defect coordinate from the BRAM; Based on the splicing method, the respective proportions of the horizontal coordinates and the vertical coordinates of the respective defect coordinates are spliced ​​to obtain the respective splicing results; Each splicing result is stored in the corresponding BRAM.

6. The image capturing method according to claim 5, characterized in that: The determining of the target number of required BRAMs and the depth and bit width of each BRAM includes: When the number of channels is one and the stitching mode is the first stitching mode, the target number is determined to be one, the depth of the BRAM is the number of defect coordinates, and the width is a first value; the first value is the sum of the number of bits of the horizontal coordinate and the vertical coordinate; When the number of channels is multiple and the splicing mode is the first splicing mode, determine the target number to be the number of channels, the depth of the BRAM to be the second value, and the width to be the first value; or, In the case where the number of channels is multiple and the splicing mode is the second splicing mode, the target number is determined to be two, the depth of the BRAM is the second value, and the width is the product of the number of channels and the number of bits of the horizontal coordinate or the vertical coordinate; The second value is obtained by rounding up the ratio of the number of defect coordinates to the number of channels; the first splicing method is to splice the horizontal coordinate and vertical coordinate of a defect coordinate; the second splicing method is to splice the horizontal coordinates or vertical coordinates of the number of channels.

7. The image capturing method according to any one of claims 1 to 6, characterized in that: The test image includes at least one of a dark field image, a middle gray image and a saturated image under a uniform field.

8. An image capturing device, characterized in that: include A first processing module, used to obtain a bad pixel map based on the collected test image; the bad pixel map is used to indicate defect data of the image sensor; the defect data includes at least one of a bad pixel and a bad line; A second processing module, used for generating defect coordinates of each defect data based on the position of each defect data; A defect coordinate represents the coordinates of a bad point or bad line; A storage module, used for storing the coordinates of each defect in an external non-volatile memory; The third processing module is used to read the defect coordinates of the image sensor from the external non-volatile memory when the image sensor is powered on, so as to correct the captured image to be corrected based on the defect coordinates, and obtain and output the corrected image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the image capturing method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image capturing method according to any one of claims 1 to 7 is implemented.

Images