A TDI data caching method, system, electronic device and storage medium

Abstract

The present application discloses a TDI data caching method, system, electronic device and storage medium, and relates to the field of data processing. The method includes: obtaining the first data and the second data output by the TDI sensor, and storing the first data row by row in the cache space of different addresses; reading the stored data to be superimposed, superimposing it with the second data of the target in the same row area, and forming the data to be superimposed for the next superimposition process, and re-storing it in the original cache space of the data to be superimposed; repeating the superposition process until the first data of the target in the same row area and all the second data are superimposed, and outputting the image data after the superposition is completed. Based on the above processing, the present application realizes the reuse of the same cache space in the TDI data caching process, which can effectively reduce the demand for cache resources of the back-end processor compared to the existing method of separately caching and superimposing multiple rows of pixel data in parallel.

Description

Technical Field

[0001] The present invention belongs to the field of data processing, and in particular relates to a TDI data caching method, system, electronic equipment and storage medium. Background Art

[0002] TDI (Time Delayed Integration) technology is based on multiple exposures and multi-frame accumulation of the same detected target to improve signal brightness. It can output higher signal-to-noise ratio signals when the light is dark and the exposure time is insufficient, improve the shooting environment conditions, and limit the signal-to-noise ratio of the exported image to be too low. Taking a multi-line TDI camera with four rows of pixels as an example, its basic principle is roughly as follows: for any row of pixels pixel0, pixel1, pixel2 and pixel3 in its internal TDI sensor, the detected target is divided into four parts ABCD, and the running speed of the detected target is adapted to the scanning line frequency of the multi-line TDI camera. At four consecutive scanning moments, a part A of the detected target is captured by pixel rows pixel0, pixel1, pixel2 and pixel3 respectively, and then pixel0, pixelell, pixel12 and pixel3 capture a part A of the detected target for integration and superposition, and finally obtain the image information of a part A of the detected target.

[0003] As the requirements for detection accuracy and efficiency in the field of machine vision continue to increase, multi-line TDI cameras are increasingly being used. Compared to ordinary line array cameras that generally have only one row of photosensitive pixels, a typical feature of a multi-line TDI camera is that its image sensor (i.e., TDI sensor) has multiple rows of photosensitive pixels. For example, multi-line TDI cameras with 2 or 4 rows of photosensitive pixels are already quite common, and there are even more multi-line TDI cameras with 12 or even 16 rows of photosensitive pixels.

[0004] Compared with multi-line TDI cameras with a higher number of rows (for example, 256 rows), multi-line TDI cameras with low-order pixel rows are characterized in that their graphic sensors themselves usually do not have the TDI overlay function. It is necessary to read the graphic data corresponding to all rows of pixels from the graphic sensor and then complete the TDI overlay operation in the back-end processor, which places high demands on the cache space of the back-end processor.

[0005] In order to solve the above problems, the present application proposes a TDI data caching method to reduce the cache space occupied by TDI data in the back-end processor. Summary of the invention

[0006] The present application proposes a TDI data caching method, system, electronic device and storage medium, which are used to reduce the cache space occupied by TDI data in a back-end processor.

[0007] To achieve the above objectives, this application proposes the following technical solutions:

[0008] In a first aspect of the present application, a TDI data caching method is provided, comprising:

[0009] Acquire first data and second data output by the TDI sensor, and store the first data line by line in cache spaces at different addresses;

[0010] Read the stored data to be superimposed, superimpose it with the second data of the target in the same row area, and form the data to be superimposed in the next superimposition process, and re-store it in the original cache space of the data to be superimposed; repeat the superimposition process until the first data of the target in the same row area is superimposed with all the second data, and output the image data after the superposition is completed;

[0011] Among them, the first data represents the image data generated by the first row of pixels in the TDI sensor capturing targets in different row areas; the second data represents the image data generated by the non-first row of pixels in the TDI sensor capturing targets in different row areas; the data to be superimposed includes the first data, and the image data after the first data and the second data are superimposed.

[0012] Optionally, the method further includes:

[0013] The third data output by the TDI sensor is discarded; wherein the third data represents the image data generated when the row pixels in the TDI sensor fail to capture the row area target.

[0014] Optionally, the method is applied to FPGA.

[0015] Optionally, the time for reading all rows of pixels in the TDI sensor once is used as a reading cycle; the method further includes:

[0016] In each reading cycle, the first data is stored in a blank buffer space, the second data is superimposed with the data to be superimposed of the target in the same row area in the previous reading cycle, and the superimposed image data is stored again in the buffer space of the data to be superimposed, and the third data is discarded;

[0017] The blank cache space indicates a cache space where no image data is stored; and the third data indicates image data generated when row pixels in the TDI sensor do not capture row area targets.

[0018] Optionally, the method further includes:

[0019] Based on the sequence of the read cycles, determining operations to be performed on different rows of pixels in the TDI sensor in different read cycles;

[0020] The execution operation includes storing the first data in a blank buffer space, superimposing the first data with the data to be superimposed of the same row area target in the previous reading cycle, and re-storing the superimposed image data in the buffer space of the data to be superimposed, and discarding the third data.

[0021] Optionally, the method further includes:

[0022] The execution operation is performed on pixels in different rows based on a dual-port RAM.

[0023] Optionally, the method further includes:

[0024] The execution operation is performed on different rows of pixels based on a single-port RAM in a double frequency mode; wherein the double frequency mode means that the data reading and writing frequency of the single-port RAM is twice the data output frequency of the TDI sensor.

[0025] In a second aspect of the present application, a TDI data cache system is provided, comprising:

[0026] A data acquisition module, used for acquiring first data and second data output by the TDI sensor, and storing the first data line by line in cache spaces at different addresses;

[0027] A data superposition module is used to read the stored data to be superimposed, superimpose it with the second data of the target in the same row area, and form the data to be superimposed in the next superposition process, and store it again in the original cache space of the data to be superimposed; repeat the superposition process until the first data of the target in the same row area is superimposed with all the second data, and output the image data after the superposition is completed;

[0028] Among them, the first data represents the image data generated by the first row of pixels in the TDI sensor capturing targets in different row areas; the second data represents the image data generated by the non-first row of pixels in the TDI sensor capturing targets in different row areas; the data to be superimposed includes the first data, and the image data after the first data and the second data are superimposed.

[0029] In a third aspect of the present application, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0030] Memory, used to store computer programs;

[0031] The processor is used to implement any one of the TDI data caching methods when executing a program stored in the memory.

[0032] In a fourth aspect of the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, any one of the TDI data caching methods described above is implemented.

[0033] The beneficial effects of this application are as follows:

[0034] The present application provides a TDI data caching method, including: obtaining the first data and the second data output by the TDI sensor, and storing the first data row by row in a cache space with different addresses; reading the stored data to be superimposed, superimposing it with the second data of the target in the same row area, and forming the data to be superimposed for the next superimposition process, and re-storing it in the original cache space of the data to be superimposed; repeating the superposition process until the first data of the target in the same row area and all the second data are superimposed, and outputting the image data after the superposition is completed. Wherein, the first data represents the image data generated by the first row of pixels in the TDI sensor collecting targets in different rows of regions; the second data represents the image data generated by the non-first row of pixels in the TDI sensor collecting targets in different rows of regions; the data to be superimposed includes the first data, and the image data after the first data and the second data are superimposed.

[0035] Based on the above processing, the present application superimposes the data to be superimposed containing the same surface information of the object to be measured (i.e., the same row of area targets) in sequence and caches them in the cache space of the corresponding first data. In the process of caching TDI data, the same cache space is reused. Compared with the existing method of separately caching and superimposing multiple rows of pixel data in parallel, the present application can effectively reduce the demand for cache resources and can greatly reduce the pressure on the cache space demand of the back-end processor during the TDI image generation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0037] Figure 1 It is a schematic diagram of an existing TDI data caching process provided by the present application;

[0038] Figure 2 is a schematic diagram of another existing TDI data caching process provided by the present application;

[0039] Figure 3 is a flow chart of a TDI data caching method provided by the present application;

[0040] Figure 4 It is a schematic diagram of a TDI data caching process provided by this application;

[0041] Figure 5 is a schematic diagram of a color TDI sensor provided by the present application;

[0042] Figure 6 is a schematic diagram of another color TDI sensor provided by the present application;

[0043] Figure 7 is a schematic diagram of a grayscale and color mixed TDI sensor provided by the present application;

[0044] Figure 8 It is a structural diagram of a TDI data caching system provided by the present application;

[0045] Fig. 9 It is a structural diagram of an electronic device provided by this application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0047] In the field of machine vision, compared with high-end multi-line TDI cameras (such as 256 lines of pixels), the image sensors of multi-line TDI cameras with 2-16 lines of pixels usually do not have TDI functions. Generally, the image data of all lines of pixels need to be read out from the image sensor, and then the TDI operation is completed in the back-end processor, which places high demands on the cache resources in the back-end processor. Especially in the fields of photovoltaic, lithium battery, and PCB inspection, multi-light working mode (i.e., multi-light source or multi-exposure working mode) is usually adopted. The number of pixel rows superimposed by TDI is doubled (2 lights) or tripled (3 lights) compared to the single-light mode, which requires a large amount of cache resources of the back-end processor. A back-end processor with larger cache resources needs to be selected to complete the TDI superposition operation, which places higher requirements on machine vision products in the photovoltaic and lithium battery industries, and greatly increases the cost of machine vision products.

[0048] The image superposition generation process of existing low-level TDI cameras usually adopts the method of multi-line cache parallel superposition. That is, the image data generated by each row of pixels in the image sensor are stored in the cache space of different addresses in the back-end processor. Then, when all the row pixel data of a certain row area target (usually the detection area corresponding to a row of pixels) of the detected target are stored in the back-end processor, the corresponding row pixel data is superimposed by TDI to form a superimposed row image.

[0049] However, the above-mentioned image overlay generation method requires a large amount of cache space in the back-end processor. Especially in the multi-lamp working mode, the order of TDI overlay increases with the number of lights, and the demand for cache space also increases sharply, which puts higher requirements on the limited cache resources in the back-end processor. In some work scenarios, it is necessary to give up the use of some FPGAs (Field Programmable Gate Arrays) with low storage resources and select FPGAs with larger cache resources as back-end processors. Therefore, the current TDI cache method not only increases the cost and power consumption in the machine vision inspection process, but also makes the back-end processors with low cache resources unusable, resulting in a waste of resources.

[0050] like Figure 1 As shown in the figure, the internal arrangement of a sensor with 4 rows of photosensitive pixels is typical at present. L1-L4 represent 4 photosensitive pixel rows in the image sensor respectively, and the row interval and pixel size are the same. In the normal single-lamp mode, when the ratio of the row frequency to the object movement rate is 1:1. Assuming that the object moves from top to bottom, taking the pixel row L4 as the reference, the number of delayed rows of L1, L2, and L3 are 6, 4, and 2 respectively. That is, the image data output by L1 needs to be cached for 7 rows, the image data output by L2 needs to be cached for 5 rows, and the image data output by L3 needs to be cached for 3 rows. When the image data of the 7th row of L4 is input to the back-end processor, the image data of the 1st row of L1, the image data of the 3rd row of L2, the image data of the 5th row of L3, and the image data of the 7th row of L4 are superimposed, thereby completing the TDI operation. Among them, the image data superposition here means the addition of the pixel values ​​corresponding to the image data.

[0051] When the line frequency is high and the back-end processing clock frequency is reduced, ping-pong cache is required. The image data input to L4 requires 2 lines of cache space, and the corresponding L3, L2, and L1 require 4, 6, and 8 lines of cache space respectively. The total number of cache lines in the back-end processor is 2+4+6+8=20 lines. Even if the L4 line does not use ping-pong cache and only uses a single line cache, the corresponding L3, L2, and L1 require 3, 5, and 7 lines of cache space respectively, and the total number of lines corresponding to the total cache space is 1+3+5+7=16 lines.

[0052] For the photovoltaic and lithium battery industries, a multi-light mode with multi-angle lighting is usually adopted. At this time, the ratio of line frequency to object movement rate is M:1, where M is the number of light sources. Figure 2The figure shows the TDI superposition relationship in the 3-lamp mode. The order of TDI superposition increases sharply. Taking the L4 line as the benchmark, in the 3-lamp working mode, the delay lines of L1, L2, and L3 are 18, 12, and 6 respectively. The single-line cache calculation is used for the L4 line. The corresponding L3, L2, and L1 in the back-end processor require 7, 13, and 19 lines of cache space respectively. The number of lines corresponding to the total cache space is 1+7+13+19=40 lines.

[0053] As can be seen from the above embodiments, as the number of TDI stacking orders increases, the existing multi-line cache method places high demands on the cache resources of the back-end processor. For multi-line TDI cameras without internal TDI functions, the image data of all row pixels needs to be read from the sensor and the TDI operation needs to be completed in the back-end processor (usually FPGA). In order to support the working requirements of the multi-lamp mode, a larger FPGA has to be selected, which adds great pressure on cost and power consumption.

[0054] In order to solve the above problems, the present application provides a TDI data caching method, such as Figure 3 As shown, the method comprises the following steps:

[0055] S1. Acquire first data and second data output by a TDI sensor, and store the first data line by line in cache spaces at different addresses.

[0056] The first data represents image data generated by the first row of pixels in the TDI sensor capturing targets in different rows; the second data represents image data generated by non-first row of pixels in the TDI sensor capturing targets in different rows.

[0057] S2, reading the stored data to be superimposed, superimposing it with the second data of the target in the same row area, and forming the data to be superimposed in the next superimposition process, and re-storing it in the original cache space of the data to be superimposed; repeating the superimposition process until the first data of the target in the same row area is superimposed with all the second data, and outputting the image data after the superposition is completed.

[0058] The data to be superimposed include first data and image data obtained by superimposing the first data and the second data.

[0059] Based on the above processing, the present application superimposes the data to be superimposed containing the same surface information of the detected target (i.e., the row area targets are the same) in sequence and caches them in the cache space of the corresponding first data. In the process of caching TDI data, the same cache space is reused. Compared with the existing method of separately caching and superimposing multiple rows of pixel data, the present application can effectively reduce the demand for cache resources and can greatly reduce the pressure on the cache space demand of the back-end processor during the TDI image generation process.

[0060] For step S1, for the grayscale TDI sensor, the first row of pixels represents the row of pixels that first obtains the detected target image information. Figure 1 For example, L1 is the first row of pixels (note that in this application, pixel rows are used as the unit of image data), and L2, L3, and L4 are all non-first row pixels. Similarly, the image data of row L1 belongs to the first data, and the image data of row L2 after the third exposure, the image data of row L3 after the fifth exposure, and the image data of row L4 after the seventh exposure all belong to the second data. In addition, for a color TDI sensor, the first row of pixels represents a row of pixels that first obtains the image information of the detected target for multiple rows of pixels that can be superimposed by TDI. Row-by-row storage means that under the same exposure light source, one row of pixels is used as a fixed unit to generate image data, and based on the differences in exposure light sources and row area targets, the image data corresponding to different row area targets are stored separately.

[0061] A row area target means that during the detection process, a row of pixels in the TDI sensor corresponds to the area of ​​the detected target. A same row area target means that under the exposure conditions of the same exposure light source, different rows of pixels in the same TDI sensor capture the same area of ​​the detected target. Figure 1 As shown, the image data of L1 at the first exposure time and the image data of L2 at the third exposure time belong to the same row region.

[0062] It should be noted that if the exposure light sources are different during the detection process, the image data generated by the same or different rows of pixels do not belong to the same row area target. Even for the same area of ​​the detected target, the image data generated by the same row of pixels under different exposure light sources do not belong to the image data of the same row area target.

[0063] For step S2, the superposition process stops until all rows of pixels corresponding to the same row area target (corresponding to the detection area of ​​a row of pixels under the same light source) complete TDI superposition. Figure 1 For example, the 1st row of image data of L1, the 3rd row of image data of L2, the 5th row of image data of L3 and the 7th row of image data of L4 are all cached after superposition, and the superimposed image data are output to the next processing module for the next step of data processing or data transmission.

[0064] The data to be superimposed include three types: first data, image data obtained by superimposing the first data and a single second data, and image data obtained by superimposing the first data and multiple second data. Specifically, when the first data of the first row of pixels that have been stored are superimposed with the second data of the second row of pixels, the data to be superimposed corresponds to the first data; when the first data and the second data of the second row of pixels are superimposed and stored, and then superimposed with the second data corresponding to the third row of pixels again, the data to be superimposed corresponds to the image data obtained by superimposing the first data and the single second data; when the first data and multiple second data of multiple rows of pixels are superimposed and stored, and then superimposed with another second data corresponding to the next row of pixels again, the data to be superimposed corresponds to the image data obtained by superimposing the first data and multiple second data.

[0065] In addition, for the same row area target, the TDI superposition process is achieved by accumulating the image data of the first row of pixels (i.e., the first data) and the image data of the remaining rows of pixels (the second data). In the first superposition process, the data to be superimposed are all the first data, so the original cache space of the data to be superimposed is represented as the cache space corresponding to the original address of the first data of the same row area target.

[0066] For the TDI sensor, the processing of step S2 can be understood as follows: in the order in which the pixel rows capture the detected target, the back-end processor reads the image data generated by the stored previous row of pixels (including the first row of pixels, and the superposition of the first row of pixels and the subsequent row of pixels), and superimposes it with the image data generated by the next row of pixels of the same row area target to form a new row of pixel generated data, which is stored again in the cache space corresponding to the first row of pixels. Repeat the above superposition process until all row pixels of the same row area target are superimposed, and output the superimposed image data to the next processor.

[0067] In some embodiments, the TDI cache method provided by the present application may further include the following steps:

[0068] Step a: collect the corresponding data of the row area targets passed in sequence under the front row of pixels (ie, the first row of pixels), and store them row by row into the cache space of different addresses.

[0069] Step b, superimpose the corresponding data of the next row of pixels of the time-sharing repeated row area target on the corresponding data of the previous row of pixels of the time-sharing repeated row area target, and form a new row of corresponding data, and re-store it in the cache space of the original address of the corresponding data of the front row of pixels.

[0070] The front row of pixels is the row of pixels where the TDI sensor first obtains the target image. Before superposition, the corresponding data of the previous row is read first; the superposition is repeated until the last row of pixels, and the corresponding data is output row by row.

[0071] For step a, the time-sharing repeated row area target means that under the same exposure light source, different rows of pixels in the same TDI sensor capture the same area of ​​the detected target.

[0072] In actual work, step b includes the following:

[0073] Superimpose the corresponding data of the second row of pixels of the time-sharing repeated row area target on the corresponding data of the first row of pixels of the time-sharing repeated row area target to form a new row of corresponding data, and store it in the cache space of the original address of the corresponding data of the first row of pixels;

[0074] Superimpose the corresponding data of the time-sharing repeated row area target of the third row of pixels onto the superimposed corresponding data of the first two rows of pixels (i.e., the first row and the second row of pixels) of the time-sharing repeated row area target to form a new row of corresponding data, and store it in the cache space of the original address of the corresponding data of the first row of pixels;

[0075] Similarly, the corresponding data of the time-sharing repeated row area target of the Fth row of pixels are superimposed on the corresponding data of the first F-1 rows of pixels of the time-sharing repeated row area target, forming a new row of corresponding data, and stored in the cache space of the original address of the corresponding data of the first row of pixels;

[0076] If the TDI sensor has a total of N rows of pixels and the total number of superposition orders is N, the final operation content of step b is: superimpose the corresponding data of the time-sharing repeated row area target of the Nth row of pixels (the last row) onto the corresponding data of the first N-1 rows of pixels of the time-sharing repeated row area target after superposition, to form the corresponding data of the row area target after superposition is completed, and output it to the next processing module.

[0077] In some embodiments, the TDI cache method provided by the present application further includes the following steps:

[0078] S3, discarding the third data output by the TDI sensor; wherein the third data represents the image data generated by the row pixels in the TDI sensor without capturing the row area target. Specifically, the third data represents the data generated by the row pixels in the TDI sensor and does not contain the valid image information of the detected target. Figure 1 For example, during the first exposure time, the image data of L2, L3 and L4 are all third data.

[0079] In actual work, the TDI data caching method provided in this application can be applied to the hardware side, for example, an FPGA connected to a TDI sensor, wherein the cache resource of the FPGA is represented as RAM (Random Access Memory).

[0080] After one exposure is completed, the time to read all row pixels in the TDI sensor (i.e., the image data generated by the row pixels) is taken as the reading cycle. Figure 2 For example, in each reading cycle, the backend processor reads the image data generated by four rows of pixels, namely, L1, L2, L3 and L4. That is, after the same exposure, it takes the image sensor to output the image data generated by all rows of pixels to the backend processor.

[0081] Normally, the output of row pixel image data in an image sensor is in the order of the pixel rows. Figure 2 For example, in each reading cycle, L1, L2, L3 and L4 of the image sensor are divided into 4 line cycles and output sequentially without overlapping each other. Among them, the line cycle represents the time for the image sensor to send and transmit a line of pixel image data. In addition, for some image sensors, in a single line cycle, it can simultaneously output 2 lines, 3 lines or more lines of line pixel image data.

[0082] In the TDI cache solution provided in the present application, only the image data of the first row of pixels in the image sensor is cached in the new cache space, and the image data of the remaining rows of pixels are not cached in the new cache space. Figure 2 For example, in the present application, the image data of L2, L3 and L4 are not stored in the new cache space, but only the image data of L1 is stored in the new cache space.

[0083] In some embodiments, the TDI cache solution provided by the present application is completed by the back-end processor in multiple read cycles. The TDI cache method provided by the present application also includes the following contents:

[0084] Step c: in each reading cycle, the first data is stored in a blank buffer space, the second data is superimposed with the data to be superimposed of the target in the same row area in the previous reading cycle, and the superimposed image data is stored again in the buffer space of the data to be superimposed, and the third data is discarded.

[0085] The blank cache space refers to a cache space that does not store image data; the blank cache space includes a cache space that has never stored image data, and a cache space that has stored image data but has sent the image data to the next processor after TDI overlay is completed.

[0086] Specifically, the back-end processor performs the process of step c above. For each row of pixel image data sent by the image sensor in each reading cycle, the back-end processor stores the first data in a blank buffer space, superimposes the second data with the data to be superimposed of the target in the same row area in the previous reading cycle, and stores the superimposed image data again in the buffer space of the data to be superimposed, and discards the third image data, etc.

[0087] In some embodiments, the TDI cache solution provided by the present application also includes the following:

[0088] Step d, based on the order of the reading cycles, determining the execution operations for different rows of pixels in the TDI sensor in different reading cycles, wherein the execution operations include storing the first data in a blank buffer space, superimposing the data to be superimposed on the target in the same row area in the previous reading cycle, and re-storing the superimposed image data in the buffer space of the data to be superimposed, and discarding the third data.

[0089] In actual work, the back-end processor determines the operations to be performed on different rows of pixels in the TDI sensor in different reading cycles according to the time sequence of different reading cycles, combined with factors such as the total number of pixel rows in the image sensor, the ratio of pixel row interval to pixel, and the multiple exposure ratio. Among them, the multiple exposure ratio represents the number of exposures when the image sensor moves a single row of pixels, and this ratio is usually the same as the number of light sources. For example, in single-lamp mode, the ratio of the line frequency of the TDI sensor to the object movement rate is 1:1; in multi-lamp mode with multi-angle lighting, the number of light sources is M, and the ratio of the line frequency to the object movement rate is M:1.

[0090] Specifically, if it is assumed that the total number of pixel rows is N, the ratio of the pixel row interval to the pixel is (Q-1) (Q is a positive integer greater than or equal to 1), and the number of light sources is M, the ratio of the line frequency of the TDI sensor to the object movement rate is M:1, and the specific processing process of step b is as follows (corresponding to the contents of the aforementioned steps S1-S2).

[0091] Step 1. In the initial (M×Q) read cycles, the back-end processor only needs to cache the first row of pixel image data (i.e., the first data) of each read cycle, and write them to the cache space at different addresses in the RAM respectively. Since the image data of non-first row pixels does not contain the surface information of the object under test, it belongs to the third data, and the image data of non-first row pixels is discarded. Among them, the cache space of the first row of pixel image data in the 1st read cycle is used as the 1st row cache space; the cache space of the first row of pixel image data in the 2nd read cycle is used as the 2nd row cache space..., and the cache space of the first row of pixel image data in the Cth read cycle is used as the Cth row cache space.

[0092] Step 2: For the (M×Q+1)th read cycle, the back-end processor caches the first row of pixel image data and writes it into a new RAM cache space (i.e., a blank cache space), and this cache space is the (M×Q+1)th row cache space. When acquiring the second row of image data, the first row of pixel image data is read from the first row of RAM cache space, and the two rows of image data are superimposed in the lookup table or adder in the FPGA, and the superimposed data is rewritten into the first row of cache space, thereby realizing the TDI superposition of the first row of pixels and the second row of pixels, and the image data of the remaining rows of pixels are directly discarded.

[0093] In one implementation, the back-end processor implements the execution operation on different rows of pixels based on the dual-port RAM. Specifically, one port in the dual-port RAM is used to read the written image data from the RAM, and when the row pixels sent by the image sensor are superimposed with the image data read from the RAM, the superimposed image data is rewritten into the RAM from the other port of the dual-port RAM.

[0094] In some embodiments, the back-end processor can also implement execution operations on different rows of pixels based on the single-port RAM in double frequency mode. The double frequency mode means that the data read and write frequency of the single-port RAM is twice the data output frequency of the image sensor, that is, the time for the single-port RAM to read and write data for a row of pixel image data is the same as the time for the image sensor to send a row of pixel image data. In addition, the data reading and writing process of the single-port RAM is to read and write the clock alternately, that is, read one clock cycle and write one clock cycle alternately, instead of writing and reading in units of pixel row image data.

[0095] Step 3. For the (M×Q+2)th to (2×M×Q)th reading cycles, the first data corresponding to the first row of pixels is input into the backend processor by the image sensor and written into the new cache space in the RAM, for example, written into the (M×Q+2)th to (2×M×Q)th row of cache space respectively. After the image data of the second row of pixels is input into the backend processor, it is superimposed with the first row of pixel image data containing the same surface information of the object under test, and rewritten into the original cache space of the first row of pixel image data. For example, the first row of pixel image data is written into the Dth (where M×Q+2≤D≤2×M×Q)th row of cache space, and the second row of pixel image data corresponding to the first row of pixels is superimposed with the first row of pixel image data of the (DM×Q)th row of cache space, and the superimposed image data is rewritten into the (DM×Q)th row of cache space. In addition, the backend processor discards the remaining image data except for the first and second rows of pixels.

[0096] Step 4: For the (2×M×Q+1)th reading cycle, the first data corresponding to the first row of pixels is input into the backend processor by the image sensor and written into the new cache space in the RAM. After the image data of the second row of pixels is input, it is superimposed with the first row of pixels in the (2×M×Q+1)th row of cache space and written into the (2×M×Q+1)th row of cache space in the RAM. After the image data of the third row of pixels is input, it is superimposed with the image data of the first row of cache space (note that the image data of the first row of cache space at this time is the image data after the first row of pixels and the second row of pixels are superimposed in step 2), and the superimposed image data is rewritten into the first row of cache space. In addition, the remaining pixel row images except the first row, second row and third row of pixels are discarded.

[0097] Similarly, for subsequent read cycles, the back-end processor continues to execute the processing of steps S1-S2 according to the above processing. It is worth noting that the steps S1-S2 provided in the present application do not require the limiting conditions of the operation time sequence, that is, in the same cache cycle, steps S1-S2 can be executed simultaneously, or step S2 can be executed before step S1. The present application realizes the reuse of the same cache space in RAM, and adopts the method of reading, superimposing and rewriting the image data that needs to be superimposed. Compared with the traditional multi-line cache parallel superposition method, it greatly reduces the cache resource demand of RAM.

[0098] like Figure 4 As shown, in the working scenario of an image sensor with 4 pixel rows, a ratio of pixel row interval to pixel size of 1, and a number of light sources of 3, the TDI cache process provided by the present application is specifically as follows. In each reading cycle, the back-end processor receives image data of 4 rows of pixels, L1, L2, L3 and L4, and the data output order of the image sensor is L1 (output first), L2, L3, L4 (output last).

[0099] 1. In the first 6 read cycles, the back-end processor only needs to write the image data corresponding to the 6 rows of pixels of L1 into different cache spaces of RAM (such as the cache spaces of rows 1 to 6), and the image data of rows L2, L3 and L4 are directly discarded.

[0100] 2. For the 7th read cycle: L1 image data is input by the sensor and directly written into the 7th row cache space in RAM. At the same time, L2 image data is input by the sensor, and the image data in the 1st row cache space in RAM is also read. The two rows of image data are added and written back into the 1st row cache space, thus realizing the TDI superposition of L1 and L2. L3 and L4 image data are directly discarded.

[0101] 3. The processing process of the 8th to 12th read cycles is the same as that of the 7th read cycle. The L1 image data is input by the image sensor and directly written into the E-th row (E is the number of the current read cycle) cache space. While the L2 row image data is read from the sensor, the cached E-6 row cache space image data is read. The two rows of image data are added and rewritten into the E-6 row cache space to achieve TDI superposition of L1 and L2. The L3 and L4 rows are directly discarded.

[0102] 4. In the 13th read cycle, the L1 image data is input from the image sensor and directly written into the 13th row cache space. While the L2 image data is input from the sensor, the cached 7th row (E-6th row) is read, and the two rows are added and written back to the 7th row cache space to realize the TDI superposition of L1 and L2. While the L3 row is read from the sensor, the cached 1st row (E-12th row, note that the image data in the 1st row cache space is the image data after L1 and L2 complete the TDI superposition operation) is read, and the two rows are added and written back to the 1st row cache space to realize the TDI superposition of L1, L2 and L3. The image data of L4 is directly discarded.

[0103] 5. The 14th to 18th read cycles are the same as the 13th read cycle. The L1 image data is input from the image sensor and directly written into the E-th row (E is the sequence number of the current read cycle, and the range is 14-18) cache space. While the L2 image data is input from the image sensor, the image data of the E-6th row cache space is read, and the two rows of image data are added and written back into the E-6th row cache space to realize the TDI superposition of L1 and L2; while the L3 image data is input from the sensor, the image data of the E-12th row cache space (note that the image data of the E-12th row cache space is the image data after the TDI superposition operation of L1 and L2 is completed) is read, and the two rows are added and written back into the E-12th row cache space to realize the TDI superposition of L1, L2 and L3 image data; the L4 image data is directly discarded.

[0104] 6. Reading cycle of row 19: L1 image data is input by the image sensor and directly written into the cache space of row 19. When L2 image data is input from the sensor, the image data of row 13 cache space is read, and the two rows are added and written back into the cache space of row 13 to realize TDI superposition of L1 and L2. When L3 image data is input by the image sensor, the image data of row 7 cache space (note that the image data of row 7 cache space is already the image data after L1 and L2 complete TDI superposition operation) is read, and the two rows are added and written back into the cache space of row 7 to realize TDI superposition of L1, L2 and L3. When L4 image data is input by the sensor, the image data of row 1 cache space (row E-18, note that the image data of row 1 cache space is already the image data after L1, L2 and L3 complete TDI superposition operation) is read, and the two rows are added to realize TDI superposition of L1, L2, L3 and L4 and output to the next level processing module at the same time. The next level processing module refers to a module for processing the image data after TDI superposition, for example, it may be an image algorithm module or a data sending module in the FPGA.

[0105] 7. The 20th read cycle and subsequent read cycles are processed in the same way as the 19th read cycle: the L1 line image data is input by the sensor and directly written into the E-th line (E is the sequence number of the current read cycle: 20-∞) cache space. While the L2 image data is input by the sensor, the image data of the cached E-6th line cache space is read, and the two lines are added and written back into the E-6th line cache space to achieve TDI superposition of L1 and L2. While the L3 image data is input by the sensor, the cached E-12th line cache space is read. The image data between the two rows (the image data after L1 and L2 complete the TDI overlay operation) is read, and the two rows are added and written back to the E-12 row cache space to realize the TDI overlay of L1, L2 and L3; while the L4 image data is input by the sensor, the image data of the cached E-18 row cache space (note that at this time it is the image data after L1, L2 and L3 complete the TDI overlay operation) is read, and the two rows are added to realize the TDI overlay of L1, L2, L3 and L4 and output to the next level processing module at the same time.

[0106] In the solution provided by the present application, the TDI delay order of adjacent pixel rows in the image sensor (i.e., the number of pixel rows that need to be delayed when TDI is superimposed) is M×Q. Among them, M is the number of light sources or the ratio of line frequency to object motion rate, and Q-1 is the ratio of pixel row interval to pixel. For example, in single-lamp mode, the ratio of line frequency to object motion rate is 1, the ratio of pixel row interval to pixel is 1, and the Q value is 2. At this time, the delay order between L1 and L2 is 2nd order, the delay between L1 and L3 is 4th order, and the delay order between L1 and L4 is 6th order, that is, the delay relationship between L1, L2, L3, and L4 is 6, 4, 2, and 0. In 3-lamp mode, the ratio of line frequency to object motion rate is 3. At this time, the delay relationship between L1, L2, L3, and L4 is 18, 12, 6, and 0. It is worth noting that the delay relationship corresponds to the starting address of reading the cached space. For example, when the delay order of adjacent pixel rows is 2, the order of the current read cycle is E. At this time, the row cache spaces corresponding to L2, L3, and L4 are E-2, E-4, and E-6. When the delay order is 6, the row cache space reading addresses corresponding to L2, L3, and L4 are E-6, E-12, and E-18.

[0107] In some embodiments, when the image sensor simultaneously outputs image data of two or more rows of pixels in one row cycle, such as image data of L1 and L2 are output simultaneously, the RAM must complete the writing of L1 image data on the one hand, and simultaneously process the reading and writing of L2 image data on the other hand.

[0108] In view of this situation, in the TDI cache solution provided by this application, the cache space of the FPGA is set up in a single-row cascade to support simultaneous operations on 2 or more rows of pixel data at the same time. Specifically, a single row means that the total cache space of the FPGA is composed of multiple independent RAMs, each of which can cache the image data of a single row of pixels. When the application needs to store a certain number of pixel rows, at least that many RAMs are set in the FPGA, that is, one RAM stores one row. Cascading means the cascading of data between multiple RAMs. When a new row of pixel image data comes from the front end, the previous row of data read out by the previous RAM is simultaneously written into the next RAM, that is, the image data of multiple rows of pixels are translated in multiple RAMs.

[0109] It is worth noting that when the FPGA cache space adopts a single-row cascade setting, it only changes the connection relationship of the memory cells in the RAM and the processing process of writing image data into the RAM, while the processing process of the TDI cache method is consistent with the above and no adjustment is required.

[0110] In the TDI cache solution provided by the present application, for the writing process of the image data (i.e., the first data) of the first row of pixel rows, there is no need to re-read the cache space of the cached data, but directly write it into the corresponding row cache space according to the sequence number of the read cycle, which is equivalent to clearing the previous row cache space (i.e., the blank cache space of the present application). Because the image data of the first row of pixel rows belongs to the latest surface information of the object under test in each read cycle, it is the first row that needs to be TDI superimposed, and there is no need to superimpose it with the data of other rows of pixels.

[0111] In some embodiments, the address of the write cache space is E%LB, where LB is the size of the single-row cache of the first row of pixel row image data, and the address of the cache space of the remaining rows of pixels is (EM×Q)%LB. Figure 1 Taking the corresponding image sensor as an example, the addresses of the L2, L3, and L4 read and write cache spaces are (E-2)%LB, (E-4)%LB, and (E-6)%LB, respectively.

[0112] The size of the entire single-line cache space LB is the overall cache space of the first line of pixel image data in the original multi-line cache mode, specifically (N-1)×M×Q+1 line cache space. Figure 2 For example, in the 3-light mode, the total cache space of the present application solution only needs 19 cache lines, compared with the 40 cache lines under the traditional multi-line cache method ( Figure 1 Taking the single-light mode as an example, the method of the present application only requires 7 lines of cache space, while the existing multi-line cache method requires 16 lines of cache space), thus reducing the cache space by half.

[0113] In some embodiments, when the cache method in the FPGA does not use a single-line cache, but uses other cache methods such as a ping-pong cache. The TDI cache method provided in this application only needs to adaptively increase or decrease the amount of cache space in the RAM without changing the TDI cache processing process of this solution. Taking the ping-pong cache as an example, it is only necessary to increase the number of cache space rows corresponding to each pixel row by one.

[0114] In the TDI cache method provided in this application, starting from the moment of starting image acquisition, the image data of L2, L3 and L4 rows need to be delayed by M×Q, 2×M×Q and 3×M×Q orders respectively before being output effectively, and the image data of the previous pixel rows need to be discarded and do not participate in TDI superposition. Figure 1 Taking the image sensor as an example, starting from the 7th reading cycle, each reading cycle will output a line of data after L1, L2, L3 and L4 are superimposed and output to the next level processing module.

[0115] In some embodiments, the TDI caching method provided in the present application is not only applicable to ordinary grayscale TDI image sensors, but also to different types of color TDI sensors.

[0116] Type 1 color TDI sensor uses three separate sensors. The pixel array of each sensor is covered with a light shielding film of one color among R, G, and B. The RGB three-color information collected by the three separate CCD sensors is subsequently fused to obtain a TDI color image.

[0117] For type one color TDI sensor, it is only necessary to execute the TDI buffering method of the present application for each CCD sensor respectively.

[0118] Type II color TDI sensor has three rows of R, G, and B pixels that respond to red, green, and blue light, respectively, arranged alternately from top to bottom. Figure 5 As shown, when the object is moving, the three rows R, G, and B respectively obtain the pixel data of the three colors at the same position of the object, and then the data of the three colors are spliced ​​together to obtain the color pixel data of this position of the object.

[0119] For type 2 color TDI sensors, the pixel rows corresponding to the three colors of R, G, and B are used as three separate image sensors, and then the cache superposition is performed separately using the TDI cache method provided in this application; after the TDI superposition of the grayscale values ​​corresponding to each color is completed, the data is spliced ​​and processed to obtain a TDI color image.

[0120] Type III color TDI sensor performs interlaced accumulation of image data output by Bayer type color area array CMOS sensor, such as Figure 6 As shown, the odd and even rows of the CMOS image are processed respectively through two parallel channels to complete the digital domain time delay integration (D-TDI) of the RG channel and the GB channel. The D-TDI integration results are channel multiplexed to realize any even-level time delay integration of the color CMOS chip to obtain a Bayer-type color image.

[0121] For type 3 color TDI sensors, the pixel rows corresponding to the two color combinations of RG and GB are used as two separate image sensors, and then the scheme is used to cache and superimpose each separate image sensor. After the grayscale value corresponding to each color is TDI superimposed, it is fused to obtain a TDI color image. It should be noted that in the process of TDI caching for each separate image sensor, the pixel rows and the numerical values ​​of the reading cycle need to be adaptively adjusted.

[0122] Type 4 grayscale and color mixed TDI sensor. Figure 7 As shown in the figure, the pixel units of the first three levels of the TDI-CMOS sensor are covered with R, G, and B monochrome filters respectively, and the remaining levels still maintain the original pixel structure. A memory array is used outside the pixel array to store the R, G, and B primary color components and grayscale accumulation signals. In the grayscale TDI image mode, the grayscale signal after TDI accumulation is output to form a complete grayscale TDI image. In the color TDI image mode, the RGB three-color composition ratio of the scene element is calculated based on the RGB three-color color components of the scene element collected in the first three levels, and is proportionally fused with the grayscale signals accumulated in the subsequent levels to output a complete color TDI image.

[0123] For the type 4 grayscale and color mixed TDI sensor, when executing the TDI cache method provided by this application, the first three rows of RGB pixels in each reading cycle are first stored in separate cache spaces, without TDI superposition. The fourth and subsequent pixel rows are used as separate image sensors to execute the TDI cache solution of this application, in which the fourth row of pixels is used as the first row of pixels in the TDI cache method. After completing the TDI superposition, continue to generate a color TDI image according to the original color TDI image mode.

[0124] It is worth noting that the first row of pixels in the TDI cache method of the present application represents the first row of pixels that can be TDI superimposed, which is usually the first row of pixels on a grayscale image sensor, but is not limited to the first row of pixels in a color camera or other application scenarios.

[0125] To sum up, the TDI cache method provided by the present application adds the concept of progressive successive overlay compared to the traditional multi-line cache overlay method. The same buffer is reused, and the new image data that needs to be overlaid is read, overlaid and rewritten at the same time, thereby achieving the purpose of completing TDI overlay with a single-line buffer and greatly reducing the cache space capacity. Compared with the traditional multi-line cache parallel overlay method, the cache space consumption can be reduced by more than half, which greatly reduces the use of RAM cache resources in the FPGA, especially in multi-light mode, greatly reducing the cost and power consumption pressure of the machine vision inspection system.

[0126] In some embodiments, the present application also provides a TDI cache system, such as Figure 8 As shown, the calibration system includes:

[0127] The data acquisition module 801 is used to acquire first data and second data output by the TDI sensor, and store the first data line by line in cache spaces at different addresses.

[0128] The data superposition module 802 is used to read the stored data to be superimposed, superimpose it with the second data of the target in the same row area, and form the data to be superimposed in the next superposition process, and store it again in the original cache space of the data to be superimposed; repeat the superposition process until the first data of the target in the same row area is superimposed with all the second data, and output the image data after the superposition is completed.

[0129] Among them, the first data represents the image data generated by the first row of pixels in the TDI sensor capturing targets in different row areas; the second data represents the image data generated by the non-first row of pixels in the TDI sensor capturing targets in different row areas; the data to be superimposed includes the first data, and the image data after the first data and the second data are superimposed.

[0130] The present application also provides an electronic device, such as Fig. 9 As shown, it includes a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0131] Memory 903, used for storing computer programs;

[0132] The processor 901 is used to implement any of the above-mentioned TDI data caching methods when executing the program stored in the memory 903.

[0133] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0134] The communication interface is used for communication between the above electronic device and other devices.

[0135] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0136] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0137] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned TDI data caching method steps is implemented.

[0138] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the TDI data caching method steps in the above embodiments.

[0139] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A TDI data caching method, characterized in that: include: Acquire first data and second data output by the TDI sensor, and store the first data row by row in cache spaces at different addresses, and do not cache the second data in the cache space; Read the stored data to be superimposed, superimpose it with the second data of the target in the same row area, and form the data to be superimposed in the next superimposition process, and store it again in the original buffer space of the data to be superimposed; Repeat the superposition process until the first data of the target in the same row area is superimposed with all the second data, and output the image data after the superposition is completed; The first data represents image data generated by the first row of pixels in the TDI sensor capturing targets in different rows; the second data represents image data generated by the non-first row of pixels in the TDI sensor capturing targets in different rows; the data to be superimposed includes the first data, and image data after the first data and the second data are superimposed; The time for reading all rows of pixels in the TDI sensor once is used as a reading cycle; the method further includes: Determine the operations to be performed on different rows of pixels in the TDI sensor in different reading cycles according to the time sequence of different reading cycles, combined with the total number of pixel rows in the TDI sensor, the ratio of the pixel row interval to the pixel, and the multiple exposure ratio; wherein the multiple exposure ratio represents the number of exposures when the TDI sensor moves a single row of pixels; Among them, the execution operation includes storing the first data in a blank cache space, superimposing it with the data to be superimposed of the same row area target in the previous reading cycle, and re-storing the superimposed image data in the cache space of the data to be superimposed, and discarding the third data; the blank cache space represents the cache space where no image data is stored; the third data represents the image data generated by the row pixels in the TDI sensor that do not capture the row area target.

2. The method according to claim 1, characterized in that The method is applied to FPGA.

3. The method according to claim 1, characterized in that The method further comprises: The execution operation is performed on pixels in different rows based on a dual-port RAM.

4. The method according to claim 3, characterized in that The method further comprises: The execution operation is performed on different rows of pixels based on a single-port RAM in a double frequency mode; wherein the double frequency mode means that the data reading and writing frequency of the single-port RAM is twice the data output frequency of the TDI sensor.

5. A TDI data cache system, characterized in that: include: A data acquisition module, used for acquiring first data and second data output by the TDI sensor, and storing the first data line by line in cache spaces at different addresses, and not caching the second data in the cache space; A data superposition module is used to read the stored data to be superimposed, superimpose it with the second data of the target in the same row area, and form the data to be superimposed in the next superposition process, and store it again in the original cache space of the data to be superimposed; Repeat the superposition process until the first data of the target in the same row area is superimposed with all the second data, and output the image data after the superposition is completed; The first data represents image data generated by the first row of pixels in the TDI sensor capturing targets in different rows; the second data represents image data generated by the non-first row of pixels in the TDI sensor capturing targets in different rows; the data to be superimposed includes the first data, and image data after the first data and the second data are superimposed; The time for reading all rows of pixels in the TDI sensor once is used as a reading cycle; the system further includes: A first determination module is used to determine the execution operation of different rows of pixels in the TDI sensor in different reading cycles according to the time sequence of different reading cycles, combined with the total number of pixel rows in the TDI sensor, the ratio of the pixel row interval to the pixel, and the multiple exposure ratio; wherein the multiple exposure ratio represents the number of exposures when the TDI sensor moves a single row of pixels; Among them, the execution operation includes storing the first data in a blank cache space, superimposing it with the data to be superimposed of the same row area target in the previous reading cycle, and re-storing the superimposed image data in the cache space of the data to be superimposed, and discarding the third data; the blank cache space represents the cache space where no image data is stored; the third data represents the image data generated by the row pixels in the TDI sensor that do not capture the row area target.

6. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the TDI data caching method described in any one of claims 1-4 when executing the program stored in the memory.

7. A computer-readable storage medium, characterized in that: A computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the TDI data caching method according to any one of claims 1 to 4 is implemented.

Images