Spatial filtering method, system, and medium based on smart cmos image sensor

Abstract

The application provides a spatial filtering method, system and medium based on an intelligent CMOS image sensor, the method comprising: collecting an image based on the CMOS image sensor, and dividing the collected image into units to obtain a plurality of pixel units; configuring a plurality of spatial filtering kernels, constructing a plurality of filtering modes based on the plurality of spatial filtering kernels, and configuring a corresponding filtering mode based on filtering condition information; performing pixel current integration processing on the spatial filtering kernel based on the filtering mode configuration integration time ratio to obtain processing data; controlling the capacitance polarity switching based on the processing data to obtain a filtering result; evaluating the filtering result based on pixel evaluation standard information, and dynamically adjusting the integration time ratio based on the evaluation result; through precise control of the integration time ratio, pixel-level parallel spatial filtering calculation of the CMOS image sensor is realized, a plurality of configurable filtering algorithms are supported, and high frame rate and good image quality indicators are maintained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a spatial filtering method, system and medium based on an intelligent CMOS image sensor. BACKGROUND

[0002] With the rapid development of Internet of Things technology, multimedia Internet of Things (M-IoT) devices equipped with image sensors are increasingly popular. Traditional CMOS image sensors (CIS) need to transmit the collected image data to an external digital processor for processing, and this architecture has problems such as high power consumption, limited frame rate, and low algorithm parallelism. In particular, in applications such as robots and autonomous driving that require high real-time performance, this separate processing architecture has become a performance bottleneck.

[0003] Existing solutions mainly include two categories: one is to implement image processing algorithms on a digital processor (such as DSP, FPGA), and the other is to integrate specific functional analog processing circuits inside the sensor. The former is flexible but has high power consumption and delay; the latter is efficient but has single function, making it difficult to adapt to diverse image processing needs.

[0004] Therefore, there is an urgent need to develop an intelligent image sensor solution that has high energy efficiency, high frame rate, and algorithm flexibility. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a spatial filtering method, system and medium based on an intelligent CMOS image sensor, which realizes pixel-level parallel spatial filtering calculation of the CMOS image sensor by precisely controlling the integration time ratio, supports a variety of configurable filtering algorithms, and at the same time maintains high frame rate and good image quality indicators.

[0006] The embodiments of the present application also provide a spatial filtering method based on an intelligent CMOS image sensor, comprising:

[0007] Collecting an image based on a CMOS image sensor and dividing the collected image into units to obtain a plurality of pixel units;

[0008] Configuring a plurality of spatial filtering cores, constructing a plurality of filtering modes based on the plurality of spatial filtering cores, and configuring a corresponding filtering mode based on filtering condition information;

[0009] Performing pixel current integration processing on the spatial filtering cores based on the integration time ratio configured based on the filtering mode to obtain processing data;

[0010] Controlling the capacitor polarity switching based on the processing data, performing absolute value calculation of the voltage across the capacitor, and obtaining a filtering result;

[0011] The filtering result is evaluated based on the pixel evaluation standard information to obtain an evaluation result, and the integral time ratio is dynamically adjusted based on the evaluation result.

[0012] Optionally, in the spatial filtering method based on the intelligent CMOS image sensor, an image is captured based on the CMOS image sensor, and the captured image is divided into units to obtain a plurality of pixel units, which specifically includes:

[0013] An image edge region is obtained based on an edge detection algorithm based on the CMOS image sensor capturing an image.

[0014] A plurality of first pixel units are obtained by dividing the image edge region into pixel blocks based on a first set size.

[0015] Texture features of all image pixels are extracted based on a gradient detection algorithm, and a gradient analysis is performed based on the texture features to screen an image smooth region.

[0016] A plurality of second pixel units are obtained by dividing the image smooth region into pixel blocks based on a second set size.

[0017] The plurality of first pixel units and the plurality of second pixel units are added to generate a plurality of final pixel units.

[0018] Optionally, in the spatial filtering method based on the intelligent CMOS image sensor, a plurality of spatial filtering kernels are configured, a plurality of filtering modes are constructed based on the plurality of spatial filtering kernels, and a corresponding filtering mode is configured based on filtering condition information, which specifically includes:

[0019] A plurality of spatial filtering kernels are configured, including a denoising kernel, an edge detection kernel, and a composite kernel.

[0020] The denoising kernel includes a mean kernel and a Gaussian kernel, the edge detection kernel includes a Roberts operator, a Prewitt operator, and a Sobel operator, and the composite kernel includes a Gaussian-Laplacian operator and a Gaussian-Sobel operator.

[0021] A single-kernel filtering mode is constructed based on a single filtering kernel, a double-kernel independent filtering mode is constructed by randomly combining two filtering kernels, and a double-kernel cascade mode is constructed by randomly combining two filtering kernels.

[0022] The single-kernel filtering mode, the double-kernel independent filtering mode, and the double-kernel cascade mode form a plurality of filtering modes.

[0023] Filtering parameters are set based on filtering requirements to obtain filtering condition information.

[0024] The single-core filtering mode, the dual-core independent filtering mode and the dual-core cascade mode are selected according to the filtering condition information.

[0025] Optionally, in the spatial filtering method based on the intelligent CMOS image sensor, the pixel current integration processing is performed on the spatial filtering core based on the integration time proportion configured according to the filtering mode, and processing data is obtained, and specifically, the method comprises the following steps:

[0026] The integration time is set, and the integration time is divided into a first stage and a second stage;

[0027] The filtering core is selected based on the filtering mode, and positive core coefficients and negative core coefficients of the filtering core are obtained;

[0028] The pixel current corresponding to the positive core coefficient is integrated based on the first stage, and the integration time of the first stage is proportional to the core coefficient value;

[0029] The pixel current corresponding to the negative core coefficient is integrated based on the second stage, and the integration time is proportional to the absolute value of the negative core coefficient;

[0030] The processing data of the pixel current is obtained based on the integration processing result of the first stage and the integration processing result of the second stage.

[0031] Optionally, in the spatial filtering method based on the intelligent CMOS image sensor, the processing data is used to control the capacitor polarity switching, the absolute value of the voltage across the capacitor is calculated, and the filtering result is obtained, and specifically, the method comprises the following steps:

[0032] The switching time node is generated based on the clock and the digital timer control switch timing;

[0033] The capacitor polarity is switched based on the switching time node, and the voltage value across the capacitor is detected;

[0034] The voltage value across the capacitor is calculated to obtain the voltage absolute value;

[0035] The filtering interval is set, the filtering interval in which the voltage absolute value is located is analyzed, and the filtering result is generated.

[0036] Optionally, in the spatial filtering method based on the intelligent CMOS image sensor, the filtering result is evaluated based on the pixel evaluation standard information to obtain an evaluation result, and the integration time proportion is dynamically adjusted based on the evaluation result, and specifically, the method comprises the following steps:

[0037] The filtering result is evaluated based on the pixel evaluation standard information, and the filtering difference value is analyzed;

[0038] The filtering difference value is compared with the set filtering threshold value to obtain the filtering deviation rate;

[0039] Determine whether the filter deviation rate is greater than or equal to the set filter deviation rate threshold;

[0040] If the error rate is greater than or equal to the set filter deviation rate threshold, correction information is generated, and the integration time ratio is adjusted based on the correction information.

[0041] If the result is less than the set filter deviation rate threshold, the filtering result will be transmitted to the terminal in real time.

[0042] Secondly, embodiments of this application provide a spatial filtering system based on a smart CMOS image sensor. The system includes a memory and a processor. The memory includes a program for a spatial filtering method based on a smart CMOS image sensor. When the program for the spatial filtering method based on a smart CMOS image sensor is executed by the processor, it performs the following steps:

[0043] Images are acquired using a CMOS image sensor, and the acquired images are divided into units to obtain several pixel units;

[0044] Configure multiple spatial filter cores, construct various filtering modes based on multiple spatial filter cores, and configure the corresponding filtering modes based on filtering condition information;

[0045] Based on the filter mode configuration and the integral time ratio, pixel current integration is performed on the spatial filter kernel to obtain processed data;

[0046] Based on the data processing, the polarity switching of the capacitor is controlled, and the absolute value of the voltage across the capacitor is calculated to obtain the filtering result.

[0047] The filtering results are evaluated based on pixel evaluation criteria information to obtain evaluation results, and the integration time ratio is dynamically adjusted based on the evaluation results.

[0048] Optionally, in the spatial filtering system based on a smart CMOS image sensor described in this application embodiment, an image is acquired based on the CMOS image sensor, and the acquired image is divided into units to obtain a number of pixel units, specifically including:

[0049] Images are acquired using a CMOS image sensor, and edge pixels are detected using an edge detection algorithm to obtain the image edge region.

[0050] The image edge region is divided into pixel blocks based on a first set size to obtain multiple first pixel units;

[0051] The gradient detection algorithm is used to extract texture features from all image pixels, and gradient analysis is performed based on the texture features to filter out smooth regions in the image.

[0052] The smooth region of the image is divided into pixel blocks based on the second set size to obtain multiple second pixel units;

[0053] Multiple first pixel units are added to multiple second pixel units to generate several final pixel units.

[0054] Optionally, in the spatial filtering system based on a smart CMOS image sensor described in this application embodiment, multiple spatial filtering cores are configured, multiple filtering modes are constructed based on the multiple spatial filtering cores, and the corresponding filtering modes are configured based on filtering condition information, specifically including:

[0055] Multiple spatial filtering kernels are configured, including a denoising kernel, an edge detection kernel, and a composite kernel;

[0056] The denoising kernels include mean kernels and Gaussian kernels, the edge detection kernels include Roberts operator, Prewitt operator and Sobel operator, and the composite kernels include Gaussian-Laplacian operator and Gaussian-Sobel operator;

[0057] A single-core filtering mode is constructed based on a single filter core; a dual-core independent filtering mode is constructed by randomly combining two filter cores; and a dual-core cascaded mode is constructed by randomly combining two filter cores.

[0058] Multiple filtering modes are formed based on single-core filtering mode, dual-core independent filtering mode and dual-core cascaded mode;

[0059] Based on the filtering requirements, filter parameters are set to obtain filter condition information;

[0060] Select one of the following modes based on the filtering conditions: single-core filtering mode, dual-core independent filtering mode, or dual-core cascaded mode.

[0061] Thirdly, embodiments of this application also provide a computer-readable storage medium, which includes a spatial filtering method program based on a smart CMOS image sensor. When the spatial filtering method program based on a smart CMOS image sensor is executed by a processor, it implements the steps of the spatial filtering method based on a smart CMOS image sensor as described in any of the preceding claims.

[0062] As can be seen from the above, the spatial filtering method, system, and medium based on a smart CMOS image sensor provided in this application involves acquiring images using a CMOS image sensor, dividing the acquired images into units to obtain several pixel units; configuring multiple spatial filtering kernels, constructing multiple filtering modes based on the multiple spatial filtering kernels, and configuring the corresponding filtering modes based on filtering condition information; performing pixel current integration processing on the spatial filtering kernels based on the integration time ratio configured according to the filtering mode to obtain processed data; controlling capacitor polarity switching based on the processed data to obtain filtering results; evaluating the filtering results based on pixel evaluation standard information to obtain evaluation results; and dynamically adjusting the integration time ratio based on the evaluation results. By precisely controlling the integration time ratio, pixel-level parallel spatial filtering calculation of the CMOS image sensor is realized, supporting multiple configurable filtering algorithms while maintaining high frame rate and good image quality indicators. Attached Figure Description

[0063] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 A flowchart of a spatial filtering method based on a smart CMOS image sensor provided in this application embodiment;

[0065] Figure 2 A flowchart of an image unit partitioning method based on a spatial filtering method of a smart CMOS image sensor provided in this application embodiment;

[0066] Figure 3 A flowchart illustrating the filtering mode configuration method of the spatial filtering method based on a smart CMOS image sensor provided in this application embodiment. Detailed Implementation

[0067] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0068] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0069] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a spatial filtering method based on a smart CMOS image sensor, as described in some embodiments of this application. This spatial filtering method based on a smart CMOS image sensor is used in a terminal device and includes the following steps:

[0070] S101 acquires images based on a CMOS image sensor and divides the acquired images into units to obtain several pixel units;

[0071] S102, configure multiple spatial filter cores, construct multiple filtering modes based on multiple spatial filter cores, and configure the corresponding filtering modes based on filtering condition information;

[0072] S103, Based on the filter mode configuration integration time ratio, perform pixel current integration processing on the spatial filter kernel to obtain processed data;

[0073] S104, based on the processed data, controls the capacitor polarity switching, calculates the absolute value of the voltage across the capacitor, and obtains the filtering result;

[0074] S105 evaluates the filtering results based on pixel evaluation standard information, obtains the evaluation results, and dynamically adjusts the integration time ratio based on the evaluation results.

[0075] It should be noted that each pixel unit integrates a complete 3×3 core spatial filter circuit, which performs convolution operations while converting photoelectric data, thus eliminating data transmission bottlenecks.

[0076] By precisely controlling the integration time ratio using time-proportional integration technology, the kernel coefficients can be simulated and weighted, avoiding the use of complex analog multipliers.

[0077] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating an image unit partitioning method based on a spatial filtering method using a smart CMOS image sensor, as described in some embodiments of this application. According to embodiments of the present invention, an image is acquired using a CMOS image sensor, and the acquired image is partitioned into units to obtain several pixel units, specifically including:

[0078] S201: Acquires images based on a CMOS image sensor, detects edge pixels based on an edge detection algorithm, and obtains the image edge region;

[0079] S202, the image edge region is divided into pixel blocks based on the first set size to obtain multiple first pixel units;

[0080] S203: Extract texture features from all image pixels based on gradient detection algorithm, perform gradient analysis based on texture features, and filter smooth regions of image.

[0081] S204, the smooth region of the image is divided into pixel blocks based on the second set size to obtain multiple second pixel units;

[0082] S205, add multiple first pixel units and multiple second pixel units together to generate several final pixel units.

[0083] It should be noted that by performing edge detection and smoothing region analysis on the image, different size divisions are established, thereby accurately dividing the image into pixel blocks and obtaining several pixel units. This ensures that different pixel units can be filtered in parallel, improving the filtering accuracy.

[0084] Please refer to Figure 3 , Figure 3 This is a flowchart illustrating a filtering mode configuration method for a spatial filtering method based on a smart CMOS image sensor, as described in some embodiments of this application. According to embodiments of the present invention, multiple spatial filtering kernels are configured, multiple filtering modes are constructed based on these kernels, and the corresponding filtering modes are configured based on filtering condition information. Specifically, this includes:

[0085] S301 is configured with multiple spatial filtering cores, including a denoising core, an edge detection core, and a composite core.

[0086] Among them, the denoising kernels include the mean kernel and the Gaussian kernel, the edge detection kernels include the Roberts operator, the Prewitt operator and the Sobel operator, and the composite kernels include the Gaussian-Laplacian operator and the Gaussian-Sobel operator;

[0087] S302, based on a single filter core, a single-core filtering mode is constructed; two filter cores are randomly combined to construct a dual-core independent filtering mode; two filter cores are randomly combined to construct a dual-core cascaded mode.

[0088] S303 forms multiple filtering modes based on single-core filtering mode, dual-core independent filtering mode and dual-core cascaded mode;

[0089] S304, Based on the filtering requirements, set the filtering parameters to obtain filtering condition information;

[0090] S305 selects one of the following modes based on the filtering conditions: single-core filtering mode, dual-core independent filtering mode, and dual-core cascaded mode.

[0091] It should be noted that each pixel unit includes a photodiode and a pixel-level spatial filtering circuit; the pixel-level spatial filtering circuit includes a configurable capacitor transimpedance amplifier (CTIA), a sample-and-hold circuit, a switch control circuit, a maximum value selection circuit, and a voltage-to-current converter; the CTIA can perform a 3×3 core spatial filtering convolution operation during photocurrent integration. It supports three operating modes: single-core filtering, dual-core independent filtering, and dual-core cascaded filtering.

[0092] Furthermore, the principle of single-core filtering mode is as follows:

[0093] Taking 3×3 mean filtering as an example, the implementation steps include:

[0094] 1. Configure the integral time series: T1=9μs, divided into 9 equal parts (1μs each);

[0095] 2. The switches of each adjacent pixel are turned on in sequence, and the integrating capacitor is charged linearly;

[0096] 3. The output voltage is read out after sampling and holding;

[0097] The mean kernel calculation process can be expressed as:

[0098] ,

[0099] This represents the voltage value output by the integrating capacitor, and indicates the result after mean filtering.

[0100] Function: Generated by integrating the pixel current through the capacitor (charge accumulation), reflecting the weighted average brightness of neighboring pixels;

[0101] This represents the photocurrent input of the kth neighboring pixel (k=1,2,…,9, corresponding to 9 pixels in a 3×3 neighborhood).

[0102] Function: The photocurrent of each pixel is multiplied by the weight of the mean kernel (1 / 9 in this case) and then integrated to jointly determine the output voltage Vout.

[0103] Physical meaning: A photodiode converts light signals into current signals, and the magnitude of the current is proportional to the pixel brightness.

[0104] The principle of dual-core independent mode is as follows:

[0105] Taking Sobel omnidirectional edge detection as an example:

[0106] First, calculate the gradient in the x-direction, as shown in the following formula:

[0107] ,

[0108] , , This represents the photocurrent input of the three adjacent pixels above the current pixel (corresponding to the top row of the Sobel kernel).

[0109] Specifically, : Top left pixel (corresponding kernel coefficient -1);

[0110] : The pixel directly above (corresponding kernel coefficient -2);

[0111] : Top right pixel (corresponding kernel coefficient -1);

[0112] , , This represents the photocurrent input of the three adjacent pixels below the current pixel (corresponding to the bottom row of the Sobel kernel).

[0113] Specifically, : The bottom left pixel (corresponding kernel coefficient +1);

[0114] : The pixel directly below (corresponding kernel coefficient +2);

[0115] : The bottom right pixel (corresponding kernel coefficient +1);

[0116] Points allocation time: (2T / 8), / (T / 8), / (-T / 8), (-2T / 8),

[0117] 2. Sampling preserves Gx results.

[0118] 3. Calculate the gradient in the y-direction, using the following formula:

[0119] ,

[0120] 4. Select the maximum value for output, as shown in the formula below:

[0121] ,

[0122] Represents the horizontal gradient, defined as the gradient value obtained by convolving the image with a Sobel horizontal kernel, used to detect vertical edges in the image (such as the left and right boundaries of an object).

[0123] Represents the vertical gradient, defined as the gradient value obtained by convolving the image with a Sobel vertical kernel, used to detect horizontal edges (such as the upper and lower boundaries of objects) in the image.

[0124] The principle of dual-core cascade mode is as follows:

[0125] Taking LoG (Gaussian-Laplace) filtering as an example:

[0126] The first-stage Gaussian filter formula is as follows:

[0127] ,

[0128] This indicates the input current, which typically refers to the current signal generated by the photoelectric conversion of image pixels;

[0129] It is the result current after Gaussian filtering and weighting.

[0130] Integration time is allocated proportionally to kernel coefficients.

[0131] The second-stage Laplace filter is calculated as follows:

[0132] ,

[0133] This represents the image intensity or current output value after the first stage of Gaussian filtering, which is the absolute value of the input signal for the second stage of Laplacian filtering and the final output result.

[0134] According to an embodiment of the present invention, pixel current integration processing is performed on the spatial filter kernel based on the integration time ratio configured by the filtering mode to obtain processed data, specifically including:

[0135] Set a scoring time, and divide the scoring time into a first stage and a second stage;

[0136] Select a matching filter kernel based on the filtering mode, and obtain the positive and negative kernel coefficients of the filter kernel;

[0137] The first stage integrates the pixel current corresponding to the kernel coefficient, and the integration time of the first stage is proportional to the kernel coefficient value.

[0138] The second stage involves integrating the pixel current corresponding to the negative kernel coefficient, and the integration time is proportional to the absolute value of the negative kernel coefficient.

[0139] Based on the integration results of the first stage and the integration results of the second stage, the pixel current processing data is obtained.

[0140] It should be noted that during the positive kernel coefficient stage, the pixel unit performs normal integration of the photocurrent, and the charge accumulates on the capacitor (with positive polarity).

[0141] During the negative core coefficient stage, the pixel unit performs reverse integration of the photocurrent through circuit switching (such as reverse bias or switch control), and the charge accumulates on the same capacitor with negative polarity (or the positive charge is canceled by differential circuit).

[0142] The charge accumulation results of the two stages correspond to the weighted sum of the input signal and the convolution kernel, which realizes the multiplication-accumulation (MAC) operation in spatial filtering.

[0143] According to an embodiment of the present invention, based on the processed data, the polarity switching of the control capacitor is adjusted, and the absolute value of the voltage across the capacitor is calculated to obtain the filtering result, specifically including:

[0144] The switching timing is controlled based on a clock and a digital timer to generate switching time nodes;

[0145] The polarity of the capacitor is switched based on the switching time point, and the voltage value across the capacitor is detected.

[0146] The absolute value of the voltage across the capacitor is calculated to obtain the absolute voltage value.

[0147] Set the filtering interval, analyze the filtering interval in which the absolute value of the voltage falls, and generate the filtering result.

[0148] It should be noted that by dividing the total integration time into two stages (positive coefficient integration stage and negative coefficient integration stage), each pixel unit accumulates a charge proportional to the kernel coefficient in different stages, and finally achieves the superposition of positive and negative weights through capacitor polarity switching (such as reverse integration or charge accumulation).

[0149] The absolute value calculation process can be described by the following formula:

[0150] ,

[0151] This represents the original grayscale value or the voltage value obtained by integration of a pixel, and represents the pixel intensity at position i in the input image.

[0152] Represents the maximum pixel coordinates in the X direction of the image, which may be used for normalization or defining the boundary for gradient calculation.

[0153] T represents the total integration time, which is divided into a positive kernel stage and a negative kernel stage, used to simulate the filter weighting process.

[0154] The maximum possible value of the filtered result (Maximum luminance / intensity or filtered level) is often used for normalization or to identify strong edges.

[0155] This refers to the reference voltage, used for comparing voltages across a capacitor, performing absolute value calculations, or as a reference point during analog-to-digital conversion.

[0156] C is the integrating capacitor, used in the pixel current integration process to store charge and form voltage; This is the integral output voltage.

[0157] According to an embodiment of the present invention, the filtering result is evaluated based on pixel evaluation standard information to obtain an evaluation result, and the integration time ratio is dynamically adjusted based on the evaluation result, specifically including:

[0158] The filtering results are evaluated based on pixel evaluation criteria information, and the filtering difference values ​​are analyzed.

[0159] The filter difference value is compared with the set filter threshold to obtain the filter deviation rate;

[0160] Determine whether the filter deviation rate is greater than or equal to the set filter deviation rate threshold;

[0161] If the error rate is greater than or equal to the set filter deviation rate threshold, correction information is generated, and the integration time ratio is adjusted based on the correction information.

[0162] If the result is less than the set filter deviation rate threshold, the filtering result will be transmitted to the terminal in real time.

[0163] It should be noted that the pixel-level spatial filtering circuit achieves weighted calculation of different kernel coefficients by adjusting the integration time ratio. The specific formula is as follows:

[0164] ,

[0165] This is the output voltage across the capacitor after integration, representing the spatial filtering output voltage of a specific pixel i. This value is derived from charge integration and reflects the weighted sum between the kernel weight and the pixel current.

[0166] This serves as a reference voltage, used as a benchmark for judging or comparing voltage differences in filter circuits. In absolute value calculations, it may be used as a reference for differential amplifiers, or for voltage adjustment.

[0167] C refers to the integrating capacitor in the circuit, which is the dielectric material through which charge accumulates during the integration of photocurrent.

[0168] The weight coefficient (or kernel coefficient) represents the position of the nth pixel in the filter kernel; its value can be positive or negative.

[0169] in, and These represent the positive and negative regions of the kernel coefficients, respectively. and For the corresponding integration time, This represents the photodiode current.

[0170] Secondly, embodiments of this application provide a spatial filtering system based on a smart CMOS image sensor. The system includes a memory and a processor. The memory includes a program for a spatial filtering method based on a smart CMOS image sensor. When the program for the spatial filtering method based on a smart CMOS image sensor is executed by the processor, it performs the following steps:

[0171] Images are acquired using a CMOS image sensor, and the acquired images are divided into units to obtain several pixel units;

[0172] Configure multiple spatial filter cores, construct various filtering modes based on multiple spatial filter cores, and configure the corresponding filtering modes based on filtering condition information;

[0173] Based on the filter mode configuration and the integral time ratio, pixel current integration is performed on the spatial filter kernel to obtain processed data;

[0174] Based on the data processing, the polarity switching of the capacitor is controlled, and the absolute value of the voltage across the capacitor is calculated to obtain the filtering result.

[0175] The filtering results are evaluated based on pixel evaluation criteria information to obtain evaluation results, and the integration time ratio is dynamically adjusted based on the evaluation results.

[0176] It should be noted that each pixel unit integrates a complete 3×3 core spatial filter circuit, which performs convolution operations while converting photoelectric data, thus eliminating data transmission bottlenecks.

[0177] By precisely controlling the integration time ratio using time-proportional integration technology, the kernel coefficients can be simulated and weighted, avoiding the use of complex analog multipliers.

[0178] According to an embodiment of the present invention, an image is acquired based on a CMOS image sensor, and the acquired image is divided into units to obtain a plurality of pixel units, specifically including:

[0179] Images are acquired using a CMOS image sensor, and edge pixels are detected using an edge detection algorithm to obtain the image edge region.

[0180] The image edge region is divided into pixel blocks based on a first set size to obtain multiple first pixel units;

[0181] The gradient detection algorithm is used to extract texture features from all image pixels, and gradient analysis is performed based on the texture features to filter out smooth regions in the image.

[0182] The smooth region of the image is divided into pixel blocks based on the second set size to obtain multiple second pixel units;

[0183] Multiple first pixel units are added to multiple second pixel units to generate several final pixel units.

[0184] It should be noted that by performing edge detection and smoothing region analysis on the image, different size divisions are established, thereby accurately dividing the image into pixel blocks and obtaining several pixel units. This ensures that different pixel units can be filtered in parallel, improving the filtering accuracy.

[0185] According to an embodiment of the present invention, multiple spatial filter kernels are configured, multiple filtering modes are constructed based on the multiple spatial filter kernels, and the corresponding filtering modes are configured based on filtering condition information, specifically including:

[0186] Multiple spatial filtering kernels are configured, including a denoising kernel, an edge detection kernel, and a composite kernel;

[0187] Among them, the denoising kernels include the mean kernel and the Gaussian kernel, the edge detection kernels include the Roberts operator, the Prewitt operator and the Sobel operator, and the composite kernels include the Gaussian-Laplacian operator and the Gaussian-Sobel operator;

[0188] A single-core filtering mode is constructed based on a single filter core; a dual-core independent filtering mode is constructed by randomly combining two filter cores; and a dual-core cascaded mode is constructed by randomly combining two filter cores.

[0189] Multiple filtering modes are formed based on single-core filtering mode, dual-core independent filtering mode and dual-core cascaded mode;

[0190] Based on the filtering requirements, filter parameters are set to obtain filter condition information;

[0191] Select one of the following modes based on the filtering conditions: single-core filtering mode, dual-core independent filtering mode, or dual-core cascaded mode.

[0192] It should be noted that each pixel unit includes a photodiode and a pixel-level spatial filtering circuit; the pixel-level spatial filtering circuit includes a configurable capacitor transimpedance amplifier (CTIA), a sample-and-hold circuit, a switch control circuit, a maximum value selection circuit, and a voltage-to-current converter; the CTIA can perform a 3×3 core spatial filtering convolution operation during photocurrent integration. It supports three operating modes: single-core filtering, dual-core independent filtering, and dual-core cascaded filtering.

[0193] Furthermore, the principle of single-core filtering mode is as follows:

[0194] Taking 3×3 mean filtering as an example, the implementation steps include:

[0195] 1. Configure the integral time series: T1=9μs, divided into 9 equal parts (1μs each);

[0196] 2. The switches of each adjacent pixel are turned on in sequence, and the integrating capacitor is charged linearly;

[0197] 3. The output voltage is read out after sampling and holding;

[0198] The mean kernel calculation process can be expressed as:

[0199] ,

[0200] The principle of dual-core independent mode is as follows:

[0201] Taking Sobel omnidirectional edge detection as an example:

[0202] First, calculate the gradient in the x-direction, as shown in the following formula:

[0203] ,

[0204] Points allocation time: (2T / 8), / (T / 8), / (-T / 8), (-2T / 8),

[0205] 2. Sampling preserves Gx results.

[0206] 3. Calculate the gradient in the y-direction, using the following formula:

[0207] ,

[0208] 4. Select the maximum value for output, as shown in the formula below:

[0209] ,

[0210] The principle of dual-core cascade mode is as follows:

[0211] Taking LoG (Gaussian-Laplace) filtering as an example:

[0212] The first-stage Gaussian filter formula is as follows:

[0213] ,

[0214] Integration time is allocated proportionally to kernel coefficients.

[0215] The second-stage Laplace filter is calculated as follows:

[0216] ,

[0217] Output the absolute value of the final result.

[0218] According to an embodiment of the present invention, pixel current integration processing is performed on the spatial filter kernel based on the integration time ratio configured by the filtering mode to obtain processed data, specifically including:

[0219] Set a scoring time, and divide the scoring time into a first stage and a second stage;

[0220] Select a matching filter kernel based on the filtering mode, and obtain the positive and negative kernel coefficients of the filter kernel;

[0221] The first stage integrates the pixel current corresponding to the kernel coefficient, and the integration time of the first stage is proportional to the kernel coefficient value.

[0222] The second stage involves integrating the pixel current corresponding to the negative kernel coefficient, and the integration time is proportional to the absolute value of the negative kernel coefficient.

[0223] Based on the integration results of the first stage and the integration results of the second stage, the pixel current processing data is obtained.

[0224] It should be noted that during the positive kernel coefficient stage, the pixel unit performs normal integration of the photocurrent, and the charge accumulates on the capacitor (with positive polarity).

[0225] During the negative core coefficient stage, the pixel unit performs reverse integration of the photocurrent through circuit switching (such as reverse bias or switch control), and the charge accumulates on the same capacitor with negative polarity (or the positive charge is canceled by differential circuit).

[0226] The charge accumulation results of the two stages correspond to the weighted sum of the input signal and the convolution kernel, which realizes the multiplication-accumulation (MAC) operation in spatial filtering.

[0227] According to an embodiment of the present invention, based on the processed data, the polarity switching of the control capacitor is adjusted, and the absolute value of the voltage across the capacitor is calculated to obtain the filtering result, specifically including:

[0228] The switching timing is controlled based on a clock and a digital timer to generate switching time nodes;

[0229] The polarity of the capacitor is switched based on the switching time point, and the voltage value across the capacitor is detected.

[0230] The absolute value of the voltage across the capacitor is calculated to obtain the absolute voltage value.

[0231] Set the filtering interval, analyze the filtering interval in which the absolute value of the voltage falls, and generate the filtering result.

[0232] It should be noted that by dividing the total integration time into two stages (positive coefficient integration stage and negative coefficient integration stage), each pixel unit accumulates a charge proportional to the kernel coefficient in different stages, and finally achieves the superposition of positive and negative weights through capacitor polarity switching (such as reverse integration or charge accumulation).

[0233] The absolute value calculation process can be described by the following formula:

[0234] ,

[0235] Where C is the integrating capacitor and T is the total integration time. This is the integral output voltage.

[0236] According to an embodiment of the present invention, the filtering result is evaluated based on pixel evaluation standard information to obtain an evaluation result, and the integration time ratio is dynamically adjusted based on the evaluation result, specifically including:

[0237] The filtering results are evaluated based on pixel evaluation criteria information, and the filtering difference values ​​are analyzed.

[0238] The filter difference value is compared with the set filter threshold to obtain the filter deviation rate;

[0239] Determine whether the filter deviation rate is greater than or equal to the set filter deviation rate threshold;

[0240] If the error rate is greater than or equal to the set filter deviation rate threshold, correction information is generated, and the integration time ratio is adjusted based on the correction information.

[0241] If the result is less than the set filter deviation rate threshold, the filtering result will be transmitted to the terminal in real time.

[0242] It should be noted that the pixel-level spatial filtering circuit achieves weighted calculation of different kernel coefficients by adjusting the integration time ratio. The specific formula is as follows:

[0243] ,

[0244] in, and These represent the positive and negative regions of the kernel coefficients, respectively. and For the corresponding integration time, This represents the photodiode current.

[0245] A third aspect of the present invention provides a computer-readable storage medium including a spatial filtering method program based on a smart CMOS image sensor. When the spatial filtering method program based on a smart CMOS image sensor is executed by a processor, it implements the steps of the spatial filtering method based on a smart CMOS image sensor as described in any of the above claims.

[0246] This invention discloses a spatial filtering method, system, and medium based on an intelligent CMOS image sensor. The method involves acquiring images using a CMOS image sensor and dividing the acquired images into pixel units. Multiple spatial filtering kernels are configured, and various filtering modes are constructed based on these kernels. The corresponding filtering mode is configured based on filtering condition information. The spatial filtering kernels are then subjected to pixel current integration processing using an integration time ratio configured according to the filtering mode, resulting in processed data. The capacitor polarity is switched based on the processed data to obtain the filtering result. The filtering result is evaluated based on pixel evaluation criteria information, resulting in an evaluation result. The integration time ratio is then dynamically adjusted based on the evaluation result. By precisely controlling the integration time ratio, pixel-level parallel spatial filtering calculations of the CMOS image sensor are achieved, supporting multiple configurable filtering algorithms while maintaining high frame rates and good image quality.

[0247] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0248] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0249] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0250] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0251] Alternatively, if the integrated units of the present invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A spatial filtering method based on a smart CMOS image sensor, characterized in that, include: Images are acquired using a CMOS image sensor, and the acquired images are divided into units to obtain several pixel units; Configure multiple spatial filter cores, construct various filtering modes based on multiple spatial filter cores, and configure the corresponding filtering modes based on filtering condition information; Based on the filter mode configuration and the integral time ratio, pixel current integration is performed on the spatial filter kernel to obtain processed data; Based on the data processing, the polarity switching of the capacitor is controlled, and the absolute value of the voltage across the capacitor is calculated to obtain the filtering result. The filtering results are evaluated based on pixel evaluation criteria information to obtain evaluation results, and the integration time ratio is dynamically adjusted based on the evaluation results.

2. The spatial filtering method based on a smart CMOS image sensor according to claim 1, characterized in that, Images are acquired using a CMOS image sensor, and the acquired images are divided into units to obtain several pixel units, specifically including: Images are acquired using a CMOS image sensor, and edge pixels are detected using an edge detection algorithm to obtain the image edge region. The image edge region is divided into pixel blocks based on a first set size to obtain multiple first pixel units; The gradient detection algorithm is used to extract texture features from all image pixels, and gradient analysis is performed based on the texture features to filter out smooth regions in the image. The smooth region of the image is divided into pixel blocks based on the second set size to obtain multiple second pixel units; Multiple first pixel units are added to multiple second pixel units to generate several final pixel units.

3. The spatial filtering method based on a smart CMOS image sensor according to claim 2, characterized in that, Configure multiple spatial filter cores, construct various filtering modes based on these cores, and configure the corresponding filtering modes based on filtering condition information. Specifically, this includes: Multiple spatial filtering kernels are configured, including a denoising kernel, an edge detection kernel, and a composite kernel; The denoising kernels include mean kernels and Gaussian kernels, the edge detection kernels include Roberts operator, Prewitt operator and Sobel operator, and the composite kernels include Gaussian-Laplacian operator and Gaussian-Sobel operator; A single-core filtering mode is constructed based on a single filter core; a dual-core independent filtering mode is constructed by randomly combining two filter cores; and a dual-core cascaded mode is constructed by randomly combining two filter cores. Multiple filtering modes are formed based on single-core filtering mode, dual-core independent filtering mode and dual-core cascaded mode; Based on the filtering requirements, filter parameters are set to obtain filter condition information; Select one of the following modes based on the filtering conditions: single-core filtering mode, dual-core independent filtering mode, or dual-core cascaded mode.

4. The spatial filtering method based on a smart CMOS image sensor according to claim 3, characterized in that, Based on the filter mode configuration and the integration time ratio, pixel current integration is performed on the spatial filter kernel to obtain processed data, specifically including: Set a scoring time, and divide the scoring time into a first stage and a second stage; Select a matching filter kernel based on the filtering mode, and obtain the positive and negative kernel coefficients of the filter kernel; The first stage integrates the pixel current corresponding to the kernel coefficient, and the integration time of the first stage is proportional to the kernel coefficient value. The second stage integrates the pixel current corresponding to the negative kernel coefficient, and the integration time of the second stage is proportional to the absolute value of the negative kernel coefficient. Based on the integration results of the first stage and the integration results of the second stage, the pixel current processing data is obtained.

5. The spatial filtering method based on a smart CMOS image sensor according to claim 4, characterized in that, Based on the data processing, the polarity switching of the capacitor is controlled, and the absolute value of the voltage across the capacitor is calculated to obtain the filtering result, which specifically includes: The switching timing is controlled based on a clock and a digital timer to generate switching time nodes; The polarity of the capacitor is switched based on the switching time point, and the voltage value across the capacitor is detected. The absolute value of the voltage across the capacitor is calculated to obtain the absolute voltage value. Set the filtering interval, analyze the filtering interval in which the absolute value of the voltage falls, and generate the filtering result.

6. The spatial filtering method based on a smart CMOS image sensor according to claim 5, characterized in that, The filtering results are evaluated based on pixel evaluation criteria information to obtain the evaluation result. The integration time ratio is then dynamically adjusted based on the evaluation result, specifically including: The filtering results are evaluated based on pixel evaluation criteria information, and the filtering difference values ​​are analyzed. The filter difference value is compared with the set filter threshold to obtain the filter deviation rate; Determine whether the filter deviation rate is greater than or equal to the set filter deviation rate threshold; If the error rate is greater than or equal to the set filter deviation rate threshold, correction information is generated, and the integration time ratio is adjusted based on the correction information. If the result is less than the set filter deviation rate threshold, the filtering result will be transmitted to the terminal in real time.

7. A spatial filtering system based on a smart CMOS image sensor, characterized in that, The system includes a memory and a processor. The memory contains a program for a spatial filtering method based on a smart CMOS image sensor. When the processor executes the program for the spatial filtering method based on the smart CMOS image sensor, it performs the following steps: Images are acquired using a CMOS image sensor, and the acquired images are divided into units to obtain several pixel units; Configure multiple spatial filter cores, construct various filtering modes based on multiple spatial filter cores, and configure the corresponding filtering modes based on filtering condition information; Based on the filter mode configuration and the integral time ratio, pixel current integration is performed on the spatial filter kernel to obtain processed data; Based on the data processing, the polarity switching of the capacitor is controlled, and the absolute value of the voltage across the capacitor is calculated to obtain the filtering result. The filtering results are evaluated based on pixel evaluation criteria information to obtain evaluation results, and the integration time ratio is dynamically adjusted based on the evaluation results.

8. The spatial filtering system based on a smart CMOS image sensor according to claim 7, characterized in that, Images are acquired using a CMOS image sensor, and the acquired images are divided into units to obtain several pixel units, specifically including: Images are acquired using a CMOS image sensor, and edge pixels are detected using an edge detection algorithm to obtain the image edge region. The image edge region is divided into pixel blocks based on a first set size to obtain multiple first pixel units; The gradient detection algorithm is used to extract texture features from all image pixels, and gradient analysis is performed based on the texture features to filter out smooth regions in the image. The smooth region of the image is divided into pixel blocks based on the second set size to obtain multiple second pixel units; Multiple first pixel units are added to multiple second pixel units to generate several final pixel units.

9. The spatial filtering system based on a smart CMOS image sensor according to claim 8, characterized in that, Configure multiple spatial filter cores, construct various filtering modes based on these cores, and configure the corresponding filtering modes based on filtering condition information. Specifically, this includes: Multiple spatial filtering kernels are configured, including a denoising kernel, an edge detection kernel, and a composite kernel; The denoising kernels include mean kernels and Gaussian kernels, the edge detection kernels include Roberts operator, Prewitt operator and Sobel operator, and the composite kernels include Gaussian-Laplacian operator and Gaussian-Sobel operator; A single-core filtering mode is constructed based on a single filter core; a dual-core independent filtering mode is constructed by randomly combining two filter cores; and a dual-core cascaded mode is constructed by randomly combining two filter cores. Multiple filtering modes are formed based on single-core filtering mode, dual-core independent filtering mode and dual-core cascaded mode; Based on the filtering requirements, filter parameters are set to obtain filter condition information; Select one of the following modes based on the filtering conditions: single-core filtering mode, dual-core independent filtering mode, or dual-core cascaded mode.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a spatial filtering method program based on a smart CMOS image sensor, which, when executed by a processor, implements the steps of the spatial filtering method based on a smart CMOS image sensor as described in any one of claims 1 to 6.

Images