Imaging method, sensor, 3D shape reconstruction method and system

Abstract

The present invention proposes a novel complementary metal oxide semiconductor (CMOS) sensor that can detect "bright" pixels and output only the light intensity and position of selected pixels. The detection function is achieved by applying a threshold criterion method. The present invention proposes a novel CMOS architecture where two sets of column processing circuits share a CMOS focal plane array (FPA) for respectively selecting, processing, and exporting data from the upper and lower halves of the FPA. The CMOS architecture includes: a re-wiring architecture, multiple I / O deployments, a parallel shift FIFO storage buffer, and an interleaved timing scheme.

Description

Technical Field

[0001] The present invention generally relates to the fields of intelligent complementary metal oxide semiconductor (CMOS) image sensors and 3D measurement and / or reconstruction technologies. More specifically, the present invention relates to an imaging method, an image sensor, a 3D shape reconstruction method, and an imaging system. Background Art

[0002] Conventional image sensors output an entire image, which may contain a lot of useless information. For example, in a 3D laser scanner, when a laser line scans an object to be photographed, the positions and intensities of bright pixels are the required information, while dark pixels do not need to be further processed and calculated. In this case, outputting the intensities of dark pixels will result in high bandwidth requirements and low readout speeds of the sensor.

[0003] To solve this problem, the present invention proposes an imaging method and a novel intelligent CMOS image sensor that reduce the output bandwidth requirements and improve the speed of the analog-to-digital converter (ADC), as well as a method and system for reconstructing 3D information of an object using the high-speed intelligent CMOS image sensor and structured light according to the present invention.

[0004] Summary of the Application

[0005] One aspect of the present invention proposes an imaging method with pixel selection, including: selecting pixels from one or more pixels according to a rule; only outputting the positions of the selected pixels or the positions and intensities of the selected pixels; exporting data through multiple parallel I / Os; and facilitating data export through a fast export architecture.

[0006] In some embodiments, before outputting the selected pixels, the method further includes at least one or more of the following: converting the intensity of the selected pixels into a digital signal through an analog-to-digital converter; in the case of facilitating data export, rewiring the selected pixels on a row by distributing the data of the selected pixels into multiple units; and storing the data of the selected pixels in a storage buffer.

[0007] In some embodiments, outputting the selected pixels includes at least one or more of the following: exporting data in one or more columns through one parallel I / O, wherein, in the case where the selected pixels have been rewired, outputting the intensity of the selected pixels in a rewired unit through one I / O channel among the multiple parallel I / Os, and the position of the selected pixels is encoded in a column in one parallel I / O; and outputting a global flag, which is used to indicate one or more of the following: the number of selected pixels to be exported through the parallel I / O, whether there are selected pixels to be exported, and the working mode of data export.

[0008] In some embodiments, pixels are selected according to at least one of the following rules: the intensity of a pixel is greater than a first threshold; or the intensity difference between a pixel and its adjacent column pixels is greater than a second threshold. The first threshold and the second threshold are set to user-defined values, or the intensity when the light source associated with one or more pixels is turned off, or the average intensity of all pixels in the region when the light source is turned off, where the region is a row, a column, or an image.

[0009] In some embodiments, rewiring the selected pixels on a row by allocating the data of the selected pixels to multiple units includes: dividing the data of several connected selected pixels in a row into one or more units; and evenly distributing the divided data of the several selected pixels to one or more parallel I / Os for data export.

[0010] In some embodiments, converting the intensity of the selected pixels into digital signals by an analog-to-digital converter includes at least one or more of the following: for each of the one or more pixels: generating a flag associated with the pixel;

[0011] if the pixel is selected, setting the flag to active, and if the pixel is not selected, setting the flag to inactive; and converting the intensity of the pixel into a digital signal when the flag associated with the pixel is active; performing AD conversion on the data corresponding to a parallel I / O simultaneously by one or more parallel analog-to-digital converters; and outputting one bit of digital data per cycle by a parallel analog-to-digital converter until the data is completely converted into digital data, outputting n bits of digital data per cycle by n parallel AD converters until the data is completely converted into digital data.

[0012] In some embodiments, the analog-to-digital converter is a successive approximation analog-to-digital converter, and the method further includes: selecting a pixel from the one or more pixels while converting the pixel into a digital signal by the successive approximation analog-to-digital converter.

[0013] In some embodiments, AD conversion and data communication adopt interleaved timing: when the analog-to-digital converter is converting data, the data of the next row is started to be read out.

[0014] In some embodiments, storing data of the selected pixels into a storage buffer includes at least one or more of the following: moving data of pixels corresponding to an I / O into one or more storage buffers, where the number of the storage buffers is less than the number of pixels corresponding to the same I / O; and / or moving data of the selected pixels into the storage buffer through a controller based on carry look-ahead adder logic; when the storage buffer is a FIFO memory, moving data in and out one by one; when the storage buffer is a FIFO memory, moving in and out a batch of multi-bit data in parallel; and emptying the data in the storage buffer when the next intensity data is converted into digital data.

[0015] In some embodiments, the method further includes: controlling the operation timing through a clock signal; and eliminating signal delay by adding a buffer, where the buffer has a hierarchical structure.

[0016] Another aspect of the present invention further provides an image sensor, including: one or more pixels in a pixel array; a pixel selection circuit associated with the pixel array for selecting pixels according to rules; one or more parallel I / Os associated with the pixel selection circuit for outputting the positions of the selected pixels or the positions and intensities of the selected pixels; and a fast export architecture associated with the plurality of parallel I / Os for facilitating data export.

[0017] In some embodiments, the image sensor further includes at least one or more of the following: one or more analog-to-digital converters associated with the pixel selection circuit for converting the intensity of the selected pixels into digital signals; one or more rewiring circuits in the fast export architecture for rewiring the selected pixels; one or more storage buffers associated with the one or more parallel I / Os for storing the selected pixels before output through the one or more parallel I / Os; and one or more column processing circuits including the pixel selection circuit, the one or more parallel I / Os, and the fast export architecture, where pixels in one row or multiple rows or all rows in a column share one column processing circuit.

[0018] In some embodiments, the plurality of parallel I / Os further includes at least one of the following: a parallel I / O for exporting data in one or more columns, where when the selected pixels have been rewired, the intensity of the selected pixels in a rewiring unit is output through one I / O channel in the plurality of parallel I / Os, and the position of the selected pixels is encoded in a column in one parallel I / O; and an output global flag for indicating one or more of the following: the number of selected pixels to be exported through the parallel I / O, whether there are selected pixels to be exported, and the working mode of data export.

[0019] In some embodiments, the pixel selection circuit is configured to select a pixel according to at least one of the following rules: the intensity of the pixel is greater than a first threshold; or the intensity difference between the pixel and the pixels in its adjacent columns is greater than a second threshold.

[0020] In some embodiments, one or more rewiring circuits further include: dividing the data of several selected adjacent pixels in a row into one or more rewiring units; and evenly distributing the divided data of the several selected pixels to one or more parallel I / Os for data export.

[0021] In some embodiments, one or more analog-to-digital converters are further configured to: for each of the one or more pixels: generate a flag associated with the pixel; set the flag to active if the pixel is selected, and set the flag to inactive if the pixel is not selected; and convert the intensity of the pixel into a digital signal when the flag associated with the pixel is active.

[0022] In some embodiments, when the one or more ADCs convert the intensity of the selected pixel into a digital signal, the image sensor further includes at least one or more of the following: one or more parallel ADCs for simultaneously performing AD conversion on the data corresponding to one parallel I / O; a parallel ADC for outputting one-bit digital data in each cycle until the data is completely converted into digital data, and multiple parallel ADCs for simultaneously outputting multiple-bit digital data in each cycle until the data is completely converted into digital data; and one or more successive approximation analog-to-digital converters, wherein the comparator of the successive approximation analog-to-digital converter is used to simultaneously perform comparison for bright pixel selection and AD conversion.

[0023] In some embodiments, AD conversion and data communication adopt interleaved timing: when the analog-to-digital converter is converting data, the data of the next row starts to be read out.

[0024] In some embodiments, when the one or more storage buffers store the data, the number of the storage buffers is less than the number of pixels corresponding to the same I / O; the data is moved into and out of the storage buffer through a controller based on carry look-ahead adder logic; when the storage buffer is a FIFO memory, the data is moved into and out of the buffer one by one; when the storage buffer is a FIFO memory, a batch of multi-bit data is moved into and out of the buffer in parallel; and when the next intensity data is converted into digital data, the data in the storage buffer is cleared.

[0025] In some embodiments, the operation timing is controlled by a clock signal; and a buffer is added to eliminate signal delay, wherein the buffer has a hierarchical structure.

[0026] Another aspect of the present invention also provides a 3D shape reconstruction method, including: calculating the geometry of an object scanned by feature light according to the position of selected pixels in an image sensor and / or the position and intensity of the selected pixels; wherein, the position of the selected pixels in the image sensor and / or the position and intensity of the selected pixels are obtained according to the method described in any of the above embodiments.

[0027] In some embodiments, calculating the geometry of an object scanned by feature light according to the position of selected pixels in an image sensor and / or the position and intensity of the selected pixels includes: forming pixel rays by the selected pixels and the camera center; intersecting the pixel rays in different image sensors at a point, or intersecting the pixel rays with the plane of the incident light; and calculating the geometric position of the point according to the calibration information of the image sensor.

[0028] Another aspect of the present invention also provides an imaging system, including: one or more image sensors including one or more pixels; one or more light sources; and one or more computing units adapted to the one or more image sensors; wherein, the one or more image sensors are the image sensors described in any of the above embodiments, and the one or more computing units are configured to execute the method described in any of the above embodiments.

[0029] The present invention provides a novel CMOS sensor, which can detect "bright" pixels and only output the light intensity and position of the selected pixels. The detection function is realized by applying a threshold criterion method. The present invention provides a novel CMOS junction architecture. Two groups of column processing circuits share a CMOS focal plane array (FPA) for respectively selecting, processing, and deriving data of the upper half and the lower half of the FPA. The present invention provides some solutions to achieve high-speed, low-power consumption, and high-efficiency CMOS; wherein, the deployment of multiple I / Os can reduce the pressure of data in the transmission row; the rewiring scheme is to evenly distribute the "selected" pixels to multiple I / Os; the shift-in and shift-out registers can maximize the storage efficiency; and the interleaved timing scheme can reduce the requirement for the ADC speed. Description of the Drawings

[0030] The present invention is further described in conjunction with non-limiting embodiments shown in the drawings, wherein:

[0031] Figure 1 Shows the way of selecting intensity using a dual-rule strategy according to an embodiment of the present invention;

[0032] Figure 2 Shows a laser line reflected on the sensor according to an embodiment of the present invention;

[0033] Figure 3.1 Shows the process of implementing the dual-rule strategy according to an embodiment of the present invention;

[0034] Figure 3.2 shows a CMOS architecture for implementing a dual - rule strategy according to an embodiment of the present invention;

[0035] Figure 4 shows a general CMOS architecture according to an embodiment of the present invention;

[0036] Figure 5 shows an example of a re - wiring scheme according to an embodiment of the present invention;

[0037] Figure 6 shows an example of multiple re - wiring units in a row according to an embodiment of the present invention;

[0038] Figure 7 shows an interleaved timing scheme for ADC and data readout according to an embodiment of the present invention;

[0039] Figure 8 shows a storage buffer in a chain structure and a CLA control circuit for accessing and shifting data in the storage buffer according to an embodiment of the present invention;

[0040] Figure 9 shows a shift register for parallel data in - shift and out - shift according to an embodiment of the present invention;

[0041] Figure 10 shows a 3D scanning system with a monocular intelligent image sensor according to an embodiment of the present invention; and

[0042] Figure 11 shows a 3D scanning system with a binocular intelligent image sensor according to an embodiment of the present invention. Detailed implementation manners

[0043] To facilitate those skilled in the art to better understand the present invention, the following further illustrates the subject matter of the present invention in conjunction with the accompanying drawings and embodiments.

[0044] The present invention relates to a novel CMOS sensor for selecting / detecting "bright" pixels on an image plane according to a threshold criterion and only outputting the intensity and position of the selected pixels; it also relates to methods for detecting pixels that meet the threshold criterion, methods for encoding positions, methods for reducing output bandwidth requirements, and methods for improving ADC speed, thereby achieving a high frame rate (> 10k fps); it also relates to methods and systems for using a CMOS image sensor and structured light to reconstruct 3D information of an object.

[0045] I. Overview

[0046] Traditional image sensors output the entire image, which may contain a lot of useless information. For example, in a 3D laser scanner, when the laser line scans the object being photographed, the positions and intensities of the bright pixels are the required information, while the dark pixels do not need to be further processed and calculated. In this case, outputting the intensities of the dark pixels will result in high bandwidth requirements and low readout speeds for the sensor.

[0047] To solve this problem, the present invention proposes a novel intelligent CMOS image sensor that can select bright pixels inside the CMOS chip and output only the intensities and positions of the selected bright pixels.

[0048] In each frame period, the light irradiating the pixels is converted into a voltage corresponding to the light intensity. The "bright" pixels are selected by a selection circuit (which is also called a column-based comparator in Section 3 and is responsible for selecting pixels whose intensities meet the threshold requirements), and only the selected pixels are sent to the ADC for data conversion. The fast export architecture helps with data export to evenly distribute the load of data output to multiple parallel I / Os. The pixels on a row are re-wired to multiple windows or cells, and one I / O channel is responsible for outputting the intensities and positions of the selected pixels in one window or cell. To reduce the bandwidth requirements, a memory with a fixed length is used to store the data of the selected pixels. The control circuit is used to control the access and store the data of the selected pixels in the storage buffer.

[0049] The method and concept proposed by the present invention have a wide range of application fields. For example, they can be applied to 3D scanning, such as high-speed 3D reconstruction of objects in a scene and tracking of moving objects, etc.

[0050] II. Pixel Selection Method

[0051] In each frame period, the intelligent CMOS sensor does not output the intensities of all pixels, but only the selected intensities and the corresponding pixel positions. Referring to Figures 1 to 3.1 / 3.2, this section will introduce the method of selecting pixels on the CMOS sensor in one frame period.

[0052] In some cases, the selection strategy can follow one or both of the following two rules: (1) the intensity of a certain pixel is greater than the threshold Δ 1 ; or (2) the intensity difference between a certain pixel and the pixel in the next column (or row) is greater than the threshold Δ 2 . If either rule is satisfied, then the pixel is selected and output. Rule (l) is to detect the peak intensity, the peak intensity when the pixel is saturated, and these peak intensities may correspond to several pixels in the row. Rule (2) aims to detect the pixels corresponding to the rising and falling of the intensity curve in the row. As Figure 1 shown, Rule (1) detects the intensities of I j+2 、Ij+3 , I j+4 , I j+5 , I j+6 (Pixels marked as solid circles), while rule (2) detects pixels marked as hollow circles with an intensity of I i , I i+1 , I i+2 , I i+3 , I i+4 , I j+6 , I j , I j+1 , I j+7 , I j+8 pixels.

[0053] In some cases, the "dual - rule" selection process is as follows: (1) When processing the n - th row, check the intensity of each pixel to see if its intensity is greater than the threshold Δ 1 (Check if I(n,m)>Δ 1 ). If so, go to step (3); if not, proceed to the next step. (2) Check the intensity difference between each pixel and its left - hand neighbor pixel to see if the intensity difference is greater than the threshold Δ 2 (Check if |I(n,m)-I(n,m - 1)|>Δ 2 ). If so, go to step (3); if not, perform step (4). (3) Generate a flag for each selected pixel, guide the analog intensity value into the ADC for digital conversion, and then export the intensity and location of the pixel. (4) If the n - th row is not the last row, continue processing the (n + 1)-th row, repeating steps (1) to (3). Figure 2 , Figure 3.1 and Figure 3.2 the architectures implementing the dual - rule strategy shown in. Figure 3.2 is an example diagram of a CMOS architecture for implementing the dual - rule strategy according to an embodiment of the present invention.

[0054] The threshold used in the frame period can be adjusted manually or adaptively. In some cases, the threshold can be determined according to the background. For example, the threshold can be set to the intensity when the light source is off; or, the average intensity of all pixels in a row (column or image) when the light source is off.

[0055] In each frame period, only the intensity of the selected pixels is converted into digital signals through the ADC, thus reducing power consumption. Thereafter, the selected intensities and the corresponding pixel locations are exported.

[0056] III. CMOS Sensor

[0057] 1) CMOS Sensor Architecture

[0058] The overall architecture of the CMOS sensor is as Figure 4As shown. The focal plane array (FPA) is located in the middle of the structure. The resolution of the FPA is H×W. In this article, 256*256 is taken as an example for easy explanation of details. At the top and bottom of the FPA, there are two groups of column processing circuits, and each group of column processing circuits is responsible for selecting, processing, and exporting the data of half of the FPA. This configuration reduces the wiring length of the row pixels and reduces the requirement for the ADC speed.

[0059] The column-based comparator is adjacent to the FPA and is responsible for selecting the pixels whose intensity meets the threshold requirement (see Figure 4 ). A flag is generated for each pixel selected by the column-based comparator, and the analog value of the intensity is guided into the column-parallel successive approximation analog-to-digital converter (SAR ADC) for digital conversion. The comparator of the SAR ADC can be used to perform the comparison and selection of bright pixels and the AD conversion simultaneously. In this case, the result of the comparison of the SAR ADC can be used in the AD conversion, so this method can improve the speed of the ADC. In other words, the comparison in this way does not take extra time.

[0060] Then, the digital value is exported to the I / O and transmitted outside the chip. However, considering high-speed image output (e.g., 20,000Hz), the transmission of the flag and the ADC data requires high power consumption and high bandwidth. To solve this problem, the present invention proposes a novel CMOS architecture, which is described in detail in the following subsections.

[0061] 2) I / O Encoding

[0062] Assume that there is only 1 I / O in each group of column processing circuits for transmitting the data of an image with a resolution of H×W (e.g., 2048×2048) and a frame rate of f (e.g., 20,000Hz). Then, the length of the output data (d) of one pixel is 19 bits, where 8 bits are used to represent the intensity and 11 bits are used to encode its position in the row. Let n be the maximum number of selected pixels whose intensity satisfies the double rule in the row. Assume n = 48 pixels, and the bandwidth requirement of the image sensor is H / 2×n×f×d = 2048 / 2×48×20000×19 = 18.68Gbps, which is too large for a single I / O to handle. In the above formula, the number of rows H is divided by 2 because each group of column processing circuits is only responsible for processing half of the total number of rows.

[0063] In the present invention, multiple I / Os are used to transfer data to accelerate data transfer. By introducing m I / Os, the bandwidth requirement can be reduced by more than m times because the bit length (d) of the output data decreases as the number of I / O channels (m) increases. For example, when 128 I / Os are used, each I / O is responsible for reading the pixels on 16 columns in an imager with a size of 2048×2048. The length (d) of the output data becomes 12 bits, i.e., 8 bits are used to represent the intensity and 4 bits are used to address the pixels within the I / O window (2048 / 128 = 16 pixels). Therefore, if 48 pixels are evenly distributed in the window of 128 I / Os, the average bandwidth requirement for each I / O becomes: H / 2×n×f×d / m = 2048 / 2×48×20000×12 / 128 = 92.16 Mbps. However, in applications, these 48 pixels are obviously not evenly distributed. For example, when the image sensor tracks a laser beam with a width of 16 pixels, the selected 16 pixels may be within the width of a single I / O channel. In this case, the maximum required bandwidth for the I / O is H / 2×n×f×d = 2048 / 2×16×20000×12 = 3.93 Gbps, which is still too large for a single I / O to handle. This situation is common. For example, when a laser beam is reflected on the image sensor, the bright pixels are usually connected together, which causes some I / Os to be overloaded. To solve this problem, the present invention proposes a rewiring scheme to balance the workloads of multiple parallel I / Os.

[0064] 3) Rewiring architecture

[0065] In practical applications, the selected bright pixels are usually connected to each other, resulting in a high bandwidth requirement for the I / O channels even when multiple parallel I / O channels are provided. The purpose of the rewiring architecture is to scatter the data of the connected pixels into different new windows and evenly distribute the data to the parallel I / Os for facilitating data export. Figure 5 The rewiring architecture is shown in. In this architecture, the rows are divided into several windows, where each window consists of multiple connected pixels. Then, the data of the connected pixels is scattered and distributed to different new cells corresponding to different I / Os. For example, Figure 5 it is shown in that 48 pixels are divided into 3 windows. Using the rewiring scheme, the data of the 1st pixel is guided to the 1st position of the 1st new cell, the data of the 2nd pixel is guided to the 1st position of the 2nd new cell, the data of the 3rd pixel is guided to the 1st position of the 3rd new cell, the data of the 4th pixel is guided to the 2nd position of the 1st new cell, and so on.

[0066] By applying a rewiring scheme, the data connecting pixels (e.g., pixels 6, 7, 8 and 46, 47, 48) is segmented and evenly distributed into different windows.

[0067] In a row, the pixels after rewiring are formed into a rewiring block by using a unified rewiring scheme. A row can be composed of one or more rewiring blocks. In an embodiment of the present invention, Figure 6 A row in an image with 768 columns is shown. This row includes 3 rewiring blocks, where each rewiring block rewires the pixels on 256 columns (for better illustration, Figure 6 the pixels in different rewiring blocks are marked with different colors). The structure of each rewiring block is the same, so it is easy to handle when designing the circuit.

[0068] 4) Storage buffer

[0069] Since the CMOS sensor performs ADC and only outputs the intensities of selected pixels, it is not necessary to send all the pixels in the unit of the parallel I / O channel. A small storage buffer is added after the window to store the intensities and positions of the selected pixels. Because the rewiring circuit evenly distributes the selected "bright" pixels, the size (length) of the storage buffer (denoted as l m ) is smaller than the size of the unit (l w ). In an embodiment of the present invention, the memory length corresponding to each I / O pin is 3, that is, the maximum number of selected bright pixels in a unit is 3. Then, for 128 I / Os, the maximum number of bright pixels in a row is 3×128 = 384. In addition, there is also an l g -bit memory for storing the global flag. In an embodiment of the present invention, the global flag can indicate whether there is data to be output, or / and the number of data to be output in the storage buffer. For example, when l g = 1, if the flag is 1, there is data to be output; otherwise, the memory is empty. When l g = 2, the global flags 00, 01, 10 or 11 respectively indicate that there are 0, 1, 2 or 3 data to be output through the I / O. Therefore, the bit length of the memory in each window is l m *(b I + log 2 l w ) + l g bits, where b I is the depth of the intensity, log 2 l wis the depth of the address in the window. In the embodiment of the present invention, the length of the memory corresponding to each I / O pin is 3, so the bit length of the storage buffer in each window is 37 bits (3 pixels * (8-bit intensity + 4-bit address) + 1-bit global flag). The maximum bandwidth of each I / O is H / 2×f×d=2048 / 2×20000×37=757.76Mbps (taking 2048 columns and 20kfps as an example), which can be implemented by a conventional FPGA circuit.

[0070] In an embodiment of the present invention, a controller based on carry lookahead adder (CLA) logic is used to save the data in the rewired window to the storage buffer. The enabling logic of the controller is as follows: (1) If the flag of the first pixel in the window (for convenience, the flag is flag l ) is 1, the data of the first pixel in the window (ADC 1 ) is saved to the first memory (MEM 1 );(2) If flag 1 =0 and flag 2 =1, the data of the second pixel in the window (ADC 2 ) boot to MEM 1 ; (3) If flag 1 =0 and flag 2 =0 and flag 3 =1, then ADC 3 Boot to MEM 1 , and so on. Assume that MEM 1 Filled with ADC i , then: (1) if flag i+1 =1, then ADC i+1 Boot to MEM 2 ; (2) If flag i+1 =0 and flag i+2 =1, then ADC i+2 Boot to MEM 2 , and so on. Repeat the above process until MEM 3 Flag indicating that the pixel is filled with data or the last pixel has been checked.

[0071] In an embodiment of the present invention, the storage buffer is a FIFO memory and is a chain structure. Figure 8 An example of this structure is shown in Figure 1, where the bit length of the storage buffer is 37 bits. The controller is used to organize the data I / O, shifting in and out, because the three data from the parallel ADC should be stored serially. The controller's enable logic is as follows: (1) If the flag of the first pixel in the window (for convenience, marked as flag1 ) is 1, then the data of the first pixel in the window (ADC 1 ) is directed to the storage buffer, and the controller generates a signal (Sclk_mem) for shifting in the data ADC 1 ; otherwise, (2) if flag 1 = 0 and flag 2 = 1, then ADC 2 is shifted into the storage buffer; otherwise, (3) if flag 1 = 0 and flag 2 = 0 and flag 3 = 1, then ADC 3 is shifted into the storage buffer, and so on. Assuming the storage buffer is filled with ADC i , then: (l) if flag i+1 = 1, the controller generates the 'Sclk_mem' signal for shifting out the data ADC i and shifting in the data ADC i+1 , otherwise, (2) if flag i+1 = 0 and flag i+2 = 1, the controller generates the 'Sclk_mem' signal for shifting out the data ADC i and shifting in the data ADC i+2 , and so on. The above process is repeated until the storage buffer is filled with 3 data or the flags of the last pixel have been checked, and then the controller generates an I / O enable signal (I / O en) for data transfer through I / O. After processing one line, all the memory is refreshed so that when processing the next line, the memory will be filled with new data.

[0072] To achieve a high frame rate and prevent data loss, before shifting in the data of the next line into the buffer, the data of the selected pixels in the current line should be shifted out of the buffer. Therefore, the data should be transferred immediately after AD conversion. The data includes a global flag (l g bit), an address (l m × log 2 l w bit) and an intensity (l m × b IHowever, the data shifting speed may be affected by the generation of digital data. In an embodiment of the present invention, multiple parallel SAR ADCs are employed to select bright pixels and perform AD conversion. During this period, a global flag and an address can be generated immediately. Then, the SAR ADC starts the AD conversion. In each conversion cycle, only 1-bit digital data is generated. When n parallel SAR ADCs are used in each I / O window, n-bit digital data can be obtained in each ADC cycle. In an embodiment of the present invention, a shift register is proposed to solve the above problems. The operation of the shift register is as follows. In the row processing cycle, first, a global flag (l g bit) and an address are generated and loaded into the shift register in parallel in the first ADC cycle. Then, in the next cycle, as Figure 9 shown, m bits of data are shifted out in parallel while n bits of newly converted ADC data are shifted in. The above process is repeated until the last n bits of ADC data are shifted out. The number of bits shifted out (m) needs to be carefully set to avoid data overwrite. Assume that the sensor has W columns, the required speed is f fps, and the size of the digital data for I / O is l m *(b I +log 2 l w )+l g bits. Then, at least data bits should be shifted out per unit time, that is, at least data bits should be shifted out of the buffer in each cycle, where t cycle is the conversion time of the ADC. The lower limit frequency of the shift is

[0073] 5) Interleaved Timing

[0074] To achieve a high frame rate, the total time used for row reset, comparator, data readout, and ADC should be less than the average row processing time. For example, for an image array with a size of 2048×2048 and a frame rate of 20,000 Hz, the average row processing time is 1 / (20,000 Hz×2048 rows / 2) = 48.83 ns. Assume that the times used for row reset, comparator, and data readout are 7.5 ns, 1 ns, and 40 ns respectively. Then, the available time for the ADC is only 0.3 ns. This means that the speed of the ADC needs to exceed 3 GSPS, which is obviously impossible. To reduce the requirement for the ADC speed, the present invention proposes as Figure 7The interleaved timing shown. In this interleaved timing, the data readout and ADC for each row are not serial: after the data of the n - l-th row is read out, the sensor starts to reset the n-th row; at the same time, the ADC processes the (n - 1)-th row. By applying this interleaved timing, the requirement for the ADC speed can be reduced. In this way, there is 48.83 ns of time for the ADC, so the requirement for the ADC speed is reduced to 1 / 48.83 ns = 20.48 MPSP.

[0075] The timing signal needs to drive and control operations in a large number of rows and columns, such as row selection and column-based ADC, etc., which leads to signal delay and affects the frame rate. To eliminate the delay, in the present invention, a buffer is added in the signal conversion. The buffer has a hierarchical structure, and the lowest-level buffer is configured to enable signals for only a few rows or columns. In other words, the signal is sent through the hierarchical buffer, and the buffer is enabled only when the corresponding few rows or columns are selected. This greatly reduces the load of the control signal, thereby reducing the delay. For example, in an image sensor with 256 rows, the input signal needs to control the row selection of 128 rows. By using a four-level hierarchical buffer structure, the input only needs to drive the row selection signal for 16 rows within a given time. At the first moment, the first buffer is enabled, and rows 0 to 15 wait to be selected. Then, this buffer is disabled and the next buffer is enabled, so that the input can control the row selection of rows 16 to 31. This process is iterated until the row selection of all rows is completed.

[0076] IV. 3D Reconstruction Using an Intelligent CMOS Sensor

[0077] In an embodiment of the present invention, referring to Figure 10 and Figure 11 , an imaging system includes one or more light sources for irradiating a region to be measured with characteristic light; one or more image sensors including a plurality of pixels; and one or more computing units for calculating the 3D position of the characteristic light.

[0078] In an embodiment of the present invention, the light source generated by a laser or LED light can be visible light or invisible light, and the shape of the light source can be selected within the following range: point, line or curve. The light generated by the light source can be continuous wave or discrete light pulses. The generated light can scan the region to be measured or be in a fixed direction. In the embodiment, a moving light beam or light pulse can be generated in different ways. For example, the following types of light sources can generate a moving light beam or light pulse: (a) a galvanometer that rotates automatically; (b) a projector; (c) a motor that rotates automatically. When the light is a moving light beam or light pulse, the angle or position of the light can be measured by a sensor, such as an encoder.

[0079] In an embodiment of the present invention, light generated by a light source is used to irradiate an intelligent image sensor, and then the intensity and position of the irradiated bright pixels are exported to a calculation unit using the methods proposed in the second and third sections.

[0080] In an embodiment of the present invention, the calculation unit is used to calculate the 3D shape or 3D contour of an object irradiated by characteristic light. In each frame, the intensity and position of the irradiated bright pixels are obtained, and the angle or position of the light can also be obtained according to pre-calibration or a sensor (e.g., an encoder); then, based on triangulation, the 3D position of the reflection point of the light wave on the object can be easily calculated.

[0081] In the 3D reconstruction method of the present invention, a monocular system using an intelligent image sensor is as Figure 10 shown. The light beam scans the area to be measured, and the angular position of the light beam can be measured by an encoder. The frame update signal of the intelligent sensor is synchronized with the angle update signal of the galvanometer. Therefore, in each frame period, the position of the bright pixels and the angle of the light beam can be obtained. Then, the direction of the light beam, the pixel, and the optical center of the camera form a triangulation system for calculating the 3D position of the light beam reflection point. As an example, in Figure 10 , the pixel at position u is exported as a bright pixel. The plane 2 of the incident light St can be determined according to the calibration data and the encoder. In the camera model, the center O c of the camera 3 is known. Then, the pixel ray O c u intersects the plane S t at point p. According to the line-plane intersection equation, the 3D position of point p is determined. In this way, all the exported bright pixels can be calculated, and thus the 3D contour of the irradiated area can be obtained.

[0082] In some embodiments, Figure 11 shows a schematic diagram of a 3D scanning system with a dual intelligent image sensor. In one frame period, the intensity and position of the bright pixels of the dual sensors can be obtained. For the bright pixel u exported by the left sensor, its corresponding matching pixel v can be found in the right sensor using epipolar geometry. Then, the pixel ray O L u and the pixel ray O R v intersect at point p. According to the line-line intersection equation, the 3D position of point p is determined. In this way, all the exported bright pixels can be calculated, and thus the 3D contour of the irradiated area can be obtained.

[0083] Additional exemplary embodiments

[0084] The following examples are provided for further description of the present invention:

[0085] Example 1. An image sensor includes one or more pixels that integrate light intensity during exposure, wherein:

[0086] (a) The image sensor selects pixels whose intensity meets certain conditions;

[0087] (b) The image sensor only exports the positions or positions and intensities of the selected pixels;

[0088] (c) The image sensor exports data through a parallel export port; and

[0089] (d) The image sensor adopts a fast data transmission structure to achieve a high frame rate.

[0090] Example 2. The image sensor according to Example 1, wherein the image sensor selects pixels according to at least one of a plurality of rules: the intensity of the pixel is greater than a threshold; or, the intensity difference between the pixel and its adjacent pixels is greater than a certain threshold.

[0091] Example 3. The image sensor according to Example 1, wherein the image sensor selects pixels row by row.

[0092] Example 4. The image sensor according to Example 1, wherein the image sensor uses a column processing circuit to select pixels, that is, one or more rows or all rows in a column share a processing circuit, namely the column processing circuit; the processing circuit can be a comparator.

[0093] Example 5. The image sensor according to Example 1, wherein the image sensor simultaneously exports data through one or more parallel export ports (for example, multiple parallel I / Os), and one export port is responsible for sending data of one or more columns.

[0094] Example 6. The image sensor according to Example 5, wherein the address only encodes the pixels corresponding to one parallel port.

[0095] Example 7. The image sensor according to Example 1, wherein only the intensity of the selected pixels is converted into a digital signal.

[0096] Example 8. The image sensor according to Example 7, wherein (1) if a pixel is selected by the sensor, a flag is generated and the flag is set to active (i.e., high or 1); if a pixel is not selected by the sensor, a flag is generated and the flag is set to inactive (i.e., low or 0); and (2) AD conversion is only performed when the flag is detected as active.

[0097] Example 9. The image sensor according to Example 7, wherein the comparator of the SAR ADC is used to simultaneously perform the comparison for selecting bright pixels and AD conversion.

[0098] Example 10. The image sensor according to Example 7, wherein interleaved timing is used for AD conversion and data communication. For example, when AD conversion is performed on data, reading of data of the next row is started.

[0099] Example 11. The image sensor according to Example 7, wherein one or more parallel devices (such as multiple parallel ADCs) are responsible for simultaneously performing AD conversion on data corresponding to one parallel data export port.

[0100] Example 12. The image sensor according to Example 11, wherein one AD converter outputs 1-bit digital data in each cycle until the data is completely converted into digital data; n parallel AD converters output n-bit digital data in each cycle until the data is completely converted into digital data.

[0101] Example 13. The image sensor according to Example 5, wherein the fast data export architecture includes a rewiring circuit to evenly distribute the read load to the parallel export ports, thereby achieving a high frame rate.

[0102] Example 14. The image sensor according to Example 13, wherein one row of pixels is rewired such that the data of the connected pixels is split and distributed to different parallel export ports.

[0103] Example 15. The image sensor according to Example 1, wherein the data is moved into a storage buffer before being exported.

[0104] Example 16. The image sensor according to Example 15, wherein the number of storage buffers is less than the number of pixels corresponding to the same export port; the data of the selected pixels is moved into the buffer by a controller that can be based on CLA logic.

[0105] Example 17. The image sensor according to Example 16, wherein the storage buffer is a FIFO memory, and the data is controlled to be moved into and out of the buffer.

[0106] Example 18. The image sensor according to Example 17, wherein the buffer (such as a register) moves out multiple bits of the original data while one or more bits of new data are moved in, such that when starting to process a new batch of pixels, all the original data has been moved out.

[0107] Example 19. The image sensor according to Example 1, wherein the sensor also outputs a global flag for indicating one or more of the following meanings: the number of selected pixels to be exported, or whether there are selected pixels to be exported, or the working mode of data export.

[0108] Example 20. The image sensor according to Example 1, wherein a clock is generated to synchronize row selection, AD conversion, data export, etc.

[0109] Example 21. The image sensor according to Example 20, wherein a buffer is added to the clock to eliminate the delay at high frame rates.

[0110] Example 22. A method for high-speed 3D shape reconstruction, wherein the geometry of an object scanned with characteristic light is calculated based on information related to pixel positions and / or light intensities, comprising:

[0111] a) obtaining the position and / or intensity of selected pixels in an image sensor described in any one of Examples 1-18;

[0112] b) forming pixel rays from the selected pixels and the camera center;

[0113] c) intersecting the matching pixel rays in different image sensors at a point, or intersecting the matching pixel rays with the plane of the incident light; and

[0114] d) calculating the geometric position of the point based on the calibration information of the image sensor.

[0115] Example 23. An imaging system, comprising: one or more image sensors described in any one of Examples 1-21; one or more light sources; and one or more computing units; wherein the one or more computing units are configured to perform the method of Example 22.

[0116] The description of the specific embodiments is intended to help understand the core idea of the present invention. It should be noted that those skilled in the art can make improvements and modifications without departing from the technical principle of the present invention, and such improvements and modifications shall be considered within the protection scope of the present invention.

Claims

1. An imaging method, characterized in that, comprising: selecting pixels from one or more pixels according to rules; only outputting the positions of the selected pixels or the positions and intensities of the selected pixels; exporting data through multiple parallel I / Os; and facilitating data export through a fast export architecture; wherein, outputting the selected pixels includes at least one or more of the following: exporting data in one or more columns through one parallel I / O, wherein, when the selected pixels have been re-wired, outputting the intensities of the selected pixels in one re-wiring unit through one I / O channel among the multiple parallel I / Os; wherein, re-wiring includes dividing the connected selected pixels into multiple units, scattering and evenly distributing them to different parallel I / Os for data export; representing the positions of the selected pixels with the encoding of the columns in the parallel I / O; and outputting a global flag, the global flag being used to indicate one or more of the following: the number of selected pixels to be exported through the parallel I / O, whether there are selected pixels to be exported, and the working mode of data export.

2. The imaging method according to claim 1, characterized in that, before outputting the selected pixels, the method further includes at least one or more of the following: converting the intensities of the selected pixels into digital signals through an analog-to-digital converter; when facilitating data export, re-wiring the selected pixels on a row by distributing the data of the selected pixels to multiple units; the re-wiring being: dividing the connected selected pixels into multiple units, scattering and evenly distributing them to different parallel I / Os for data export; and storing the data of the selected pixels into a storage buffer.

3. The imaging method according to claim 1, characterized in that, selecting pixels according to at least one of the rules: the intensity of the pixel is greater than a first threshold; or the intensity difference between the pixel and its adjacent column pixels is greater than a second threshold; wherein, the first threshold and the second threshold are set as user-defined values, or are the intensities when the light source related to one or more pixels is turned off, or are the average intensities of all pixels in the area when the light source is turned off, wherein the area is a row, a column or an image.

4. The imaging method according to claim 2, characterized in that, the re-wiring of the selected pixels on a row by distributing the data of the selected pixels to multiple units includes: dividing the data of several connected selected pixels in a row into one or more units; and evenly distributing the divided data of the several selected pixels to one or more parallel I / Os for data export.

5. The imaging method according to claim 2, characterized in that, the converting of the intensities of the selected pixels into digital signals through an analog-to-digital converter includes: for each of the one or more pixels: generating a flag related to the pixel; if the pixel is selected, setting the flag to active, and if the pixel is not selected, setting the flag to non-active; when the flag related to the pixel is active, converting the intensity of the pixel into a digital signal.

6. The imaging method according to claim 2, wherein, the conversion of the intensity of the selected pixels into digital signals by the analog-to-digital converter includes: performing AD conversion on the data corresponding to one parallel I / O simultaneously through one or more parallel analog-to-digital converters; or outputting one bit of digital data in each cycle through one parallel analog-to-digital converter until the data is completely converted into digital data, and outputting n bits of digital data simultaneously in each cycle through n parallel AD converters until the data is completely converted into digital data.

7. The imaging method according to claim 2, wherein, the analog-to-digital converter is a successive approximation analog-to-digital converter, and the method further includes: selecting pixels from the one or more pixels, and simultaneously converting the pixels into digital signals through the successive approximation analog-to-digital converter.

8. The imaging method according to claim 2, wherein, the AD conversion and data communication adopt an interleaved timing sequence: when the analog-to-digital converter is converting data, the data of the next line starts to be read out.

9. The imaging method according to claim 2, wherein, the storing of the data of the selected pixels into the storage buffer includes: shifting the data of the pixels corresponding to one I / O into one or more storage buffers, wherein the number of the storage buffers is less than the number of the pixels corresponding to the same I / O.

10. The imaging method according to claim 2, wherein, the storing of the data of the selected pixels into the storage buffer includes: shifting the data of the selected pixels into the storage buffer through a controller based on carry look-ahead adder logic; in the case where the storage buffer is a FIFO memory, shifting data in and out one by one, and emptying the data in the storage buffer when the next intensity data is converted into digital data; or in the case where the storage buffer is a FIFO memory, shifting in and out a batch of multiple bits of data in parallel, and emptying the data in the storage buffer when the next intensity data is converted into digital data.

11. The imaging method according to claim 1, wherein, further includes: controlling the operation timing through a clock signal; and eliminating signal delay by adding a buffer, wherein the buffer has a hierarchical structure.

12. An image sensor, wherein, includes: one or more pixels in a pixel array; a pixel selection circuit associated with the pixel array for selecting pixels according to rules; one or more parallel I / Os associated with the pixel selection circuit for outputting the positions of the selected pixels or the positions and intensities of the selected pixels; and a fast export architecture associated with the multiple parallel I / Os for facilitating data export; wherein, the multiple parallel I / Os further include at least one of the following: one parallel I / O for exporting data in one or more columns, wherein When the selected pixel has been rewired, output the intensity of the selected pixel in a rewiring unit through one I / O channel among the multiple parallel I / Os; wherein, rewiring includes dividing the connected selected pixels into multiple units, scattering and evenly distributing them to different parallel I / Os for data export; and represent the position of the selected pixel with the encoding of the column in the parallel I / O; and output a global flag, which is used to indicate one or more of the following: the number of selected pixels to be exported through the parallel I / O, whether there are selected pixels to be exported, and the working mode of data export.

13. The image sensor according to claim 12, wherein, further includes at least one or more of the following: one or more analog-to-digital converters associated with the pixel selection circuit for converting the intensity of the selected pixel into a digital signal; one or more rewiring circuits in the fast export architecture for rewiring the selected pixel; the rewiring is: dividing the connected selected pixels into multiple units, scattering and evenly distributing them to different parallel I / Os for data export; one or more storage buffers associated with the one or more parallel I / Os for storing the selected pixel before output through the one or more parallel I / Os; and one or more column processing circuits, including the pixel selection circuit, the one or more parallel I / Os and the fast export architecture, wherein, the pixels in one or more or all rows in a column share one column processing circuit.

14. The image sensor according to claim 12, wherein, the pixel selection circuit is used to select pixels according to at least one of the following rules: the intensity of the pixel is greater than the first threshold; or the intensity difference between the pixel and the pixels in its adjacent column is greater than the second threshold.

15. The image sensor according to claim 13, wherein, the one or more rewiring circuits are further used for: dividing the data of several adjacent selected pixels in a row into one or more rewiring units; and evenly distributing the divided data of the several selected pixels to one or more parallel I / Os for data export.

16. The image sensor according to claim 13, wherein, the one or more analog-to-digital converters are further used for: for each pixel in the one or more pixels: generating a flag associated with the pixel; if the pixel is selected, setting the flag to active, and if the pixel is not selected, setting the flag to inactive; and when the flag associated with the pixel is active, converting the intensity of the pixel into a digital signal.

17. The image sensor according to claim 13, wherein, when the one or more analog-to-digital converters convert the intensity of the selected pixel into a digital signal, the image sensor further includes: one or more parallel analog-to-digital converters for simultaneously performing AD conversion on the data corresponding to one parallel I / O.

18. The image sensor according to claim 13, wherein, In the case where the one or more analog-to-digital converters convert the intensity of the selected pixel into a digital signal, the image sensor further comprises: A parallel analog-to-digital converter for outputting one-bit digital data in each cycle until the data is completely converted into digital data, and a plurality of parallel analog-to-digital converters for simultaneously outputting multiple bits of digital data in each cycle until the data is completely converted into digital data.

19. The image sensor according to claim 13, characterized in that In the case where the one or more analog-to-digital converters convert the intensity of the selected pixel into a digital signal, the image sensor further comprises: One or more successive approximation analog-to-digital converters, wherein the comparator of the successive approximation analog-to-digital converter is used for simultaneous pixel comparison selection and AD conversion.

20. The image sensor according to claim 13, characterized in that AD conversion and data communication adopt interleaved timing: when the analog-to-digital converter is converting data, the data of the next line starts to be read out.

21. The image sensor according to claim 13, characterized in that In the case where the one or more storage buffers store the data, the number of the storage buffers is less than the number of pixels corresponding to the same I / O; the data is shifted into and out of the storage buffer by a controller based on carry look-ahead adder logic; In the case where the storage buffer is a FIFO memory, the data is shifted into and out of the buffer one by one; when the next intensity data is converted into digital data, the data in the storage buffer is cleared; or In the case where the storage buffer is a FIFO memory, a batch of multiple-bit data is shifted into and out of the buffer in parallel, and when the next intensity data is converted into digital data, the data in the storage buffer is cleared.

22. The image sensor according to claim 13, characterized in that the operation timing is controlled by a clock signal; and a buffer is added to eliminate signal delay, wherein the buffer has a hierarchical structure.

23. A 3D shape reconstruction method, characterized in that comprises: Calculating the geometry of an object scanned by characteristic light according to the position of the selected pixel and / or the position and intensity of the selected pixel in the image sensor; wherein the position of the selected pixel and / or the position and intensity of the selected pixel in the image sensor are obtained according to the imaging method according to any one of claims 1 to 11.

24. The 3D shape reconstruction method according to claim 23, characterized in that the calculating the geometry of the object scanned by characteristic light according to the position of the selected pixel and / or the position and intensity of the selected pixel in the image sensor comprises: Forming a pixel ray by the selected pixel and the camera center; Intersecting the pixel rays in different image sensors at a point, or intersecting the pixel ray with the plane of the incident light; and Calculating the geometric position of the point according to the calibration information of the image sensor.

25. An imaging system, characterized in that comprises: One or more image sensors each including one or more pixels; One or more light sources; and One or more computing units adapted to the one or more image sensors; wherein, The one or more image sensors and the one or more computing units are configured to perform the imaging method according to any one of claims 1 to 11.

Images