A multi-mode image acquisition and processing method for visible light sensors

Abstract

This invention discloses a multi-mode image acquisition and processing method for a visible light sensor, comprising: an FPGA receiving the functional mode and image window center coordinates set by a processor, determining the image window size, and calculating the corresponding row and column information; the FPGA generating periodic exposure start and stop signals to expose all rows containing the window; after exposure, the image sensor internally splits the image data of each row into N channels and outputs them in parallel according to a fixed format; the FPGA simultaneously receives image data from the N channels of the sensor and stores it in N on-chip RAMs; the FPGA determines the number of RAMs to be read and the RAM address reading range based on the column range of the window area, and reads the pixel data of a single row of the window; the FPGA and the processor process the window pixel data according to the image processing method set by the internal functional mode flag. This invention can meet the requirements of both target recognition and pose measurement functional modes for image size and update rate, and is beneficial to reducing the weight, power consumption, and size of the entire device.

Description

Technical Field

[0001] This invention relates to a multi-mode image acquisition and processing method for a visible light sensor, belonging to the field of image acquisition and processing. Background Technology

[0002] With the rapid development of aerospace technology, the space domain has become a new battleground for strategic competition among major powers. Deploying visible light sensors on orbital spacecraft enables target acquisition and tracking, providing warnings, deterring, and even destroying high-orbit operational spacecraft that maliciously approach or harass our high-orbit satellites, thus protecting the normal operation of our high-value satellites. A star sensor is a high-precision attitude measurement device for spacecraft. It acquires images of stars through image sensors and obtains three-axis inertial attitude through star image processing. Visible light sensors with multiple functional modes, such as target recognition and attitude measurement, can reduce the number of payloads, effectively reducing the mass, power consumption, and size of the spacecraft.

[0003] In attitude measurement mode, to ensure measurement accuracy, full-image or large-window images need to be processed to ensure a sufficient number of stars can be extracted for attitude calculation. In target recognition mode, a small-window image can be used to locate the target, but a high image update rate is required to achieve fast and accurate target tracking. Different functional modes place different demands on the image update rate and image processing methods.

[0004] In recent years, research progress has been made in multi-mode image acquisition and processing methods for star sensors, but existing technologies all have certain application limitations and shortcomings. CN103968845A discloses a processor and FPGA parallel multi-mode star map processing method for star sensors. By dividing and optimizing the star map processing algorithm, it realizes pose measurement functions in both capture and tracking modes. However, this scheme cannot meet the requirements of high frame rate window images in the target recognition function mode. CN107478218A discloses a variable update rate image acquisition method for star sensors by changing the underlying hardware driver timing of the image sensor. However, this method is only applicable to specific types of image sensors and lacks versatility. CN104202513A discloses an FPGA-based image acquisition and processing method for star sensors in full-map mode and window image mode. When capturing the window image, it is necessary to read out the complete single-row image from multiple on-chip RAM caches and store it in on-chip RAM again. Then, the window image is captured according to the start column position and the end column position, which increases processing time and resource consumption. Summary of the Invention

[0005] The technical problem solved by the present invention is to overcome the shortcomings of the prior art and provide a multi-mode image acquisition and processing method for visible light sensors. The image window size is determined according to different functional modes, and by changing the image update rate and image processing method, the visible light sensor can simultaneously have target recognition and star-sensor attitude calculation functions.

[0006] The technical solution of this invention is: a multi-mode image acquisition and processing method for a visible light sensor, wherein the visible light sensor includes: an image sensor, a processor, and an FPGA; and the multi-mode image acquisition and processing method includes:

[0007] The FPGA receives the function mode and image window center coordinates set by the receiver processor, determines the image window size according to the requirements of different function modes for image update rate and image size, and calculates the row and column information corresponding to the window area.

[0008] The FPGA generates periodic exposure start and stop signals to control the image sensor to expose all rows of the window. After the exposure is completed, the image sensor internally splits the image data of each row into N channels and outputs them in parallel according to a fixed format.

[0009] The FPGA simultaneously receives image data from N channels of the sensor and stores it in N on-chip RAMs. After waiting for the data in the N RAMs to form a complete single-row image, the FPGA determines the number of RAMs to be read and the RAM address reading range based on the column range of the window area, and reads out the pixel data of the single-row window.

[0010] The FPGA and processor process the window pixel data according to the image processing method set by the internal function mode flag bit. The processing method includes: target recognition, star point extraction for pose measurement in capture mode, and star point extraction for pose measurement in tracking mode.

[0011] Preferably, the calculation of the row and column information corresponding to the image window region is specifically as follows:

[0012] Calculate the starting row number, starting column number, ending row number, and ending column number of the window based on the window size and center coordinates;

[0013] The time for the FPGA to read out a single row of data from the window is calculated based on the start column number and the end column number. At the same time, the number and index address of the output channel where the start column and the end column of the window are located are calculated based on the number of sensor image output channels in a single row and the number of image data in each channel.

[0014] Preferably, during the exposure idle time, the interval time for updating each row of data of the image sensor, the start and end row numbers of the window are written to the image sensor register through the SPI interface, and the image sensor will only expose and output data for the pixels from the start row to the end row of the window in the whole image.

[0015] Preferably, when performing target recognition on window pixel data:

[0016] During the process of receiving pixel data, the FPGA clusters the rows and columns of pixels that are higher than the background threshold. Pixels belonging to the same target are clustered into a pixel block. False targets are eliminated based on the size and energy of the pixel block. The effective target feature information is calculated and stored. The processor reads the target feature information stored in the FPGA to perform target recognition.

[0017] Target characteristic information includes: the size, energy, and centroid coordinates of each target.

[0018] Preferably, when using the capture mode to extract star points from the window pixel data for pose measurement:

[0019] During the process of receiving pixel data, the FPGA uses a first-order low-pass filter to filter the image, compares the filtered pixel grayscale value with the background threshold, clusters the rows and columns of filtered pixels that are higher than the background threshold, and clusters pixels belonging to the same star point into a pixel block. False star points are eliminated based on the size and energy of the pixel block, and the information of the effective star points is calculated and stored. The processor reads the star point information stored in the FPGA and performs attitude calculation.

[0020] Among them, the star point information includes the energy and centroid coordinates of each star point.

[0021] Preferably, when extracting star points from window pixel data using a tracking mode for pose measurement:

[0022] During the process of receiving pixel data, the FPGA performs star point window segmentation according to the coordinates and size of multiple star point windows set by the processor. The FPGA calculates the background threshold based on the boundary pixels of each window image, and uses a connected component algorithm to traverse the pixel blocks that are higher than the background threshold and belong to the same star point. False star points are eliminated based on the size and energy of the pixel blocks, and the information of the valid star points is calculated and stored. The processor reads the star point information stored by the FPGA and performs attitude calculation.

[0023] Preferably, the star point information includes the energy and centroid coordinates of each star point.

[0024] Preferably, the visible light sensor image exposure, acquisition and processing process adopts a three-stage pipelined parallel architecture. When the image sensor outputs the N-channel data of the (n+1)th image and performs the exposure integration of the (n+2)th image, the FPGA performs the reception, windowing and image preprocessing of the (n+1)th image in parallel. At the same time, the processor reads the FPGA preprocessing result and processes the nth image in parallel, where n≥0.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] (1) Configure the row interval and number of exposure rows of sensor image data output before image exposure. During image acquisition, only the column where the window is located is read out. The image window size determines the image output and readout time. A high image update rate is achieved under a small window, which can meet the needs of multiple functional modes.

[0027] (2) Based on the calculated output channel numbers and index addresses of the start and end columns of the window, the FPGA can directly read the window image from the complete single-row image cached in N on-chip RAMs. This process does not require reading and storing the complete single-row image from N on-chip RAMs and then cropping the window image based on the start and end column numbers, thus saving image acquisition time and hardware resources.

[0028] (3) According to different functional modes, the processor and FPGA switch the window image size and update rate, and select different image algorithms for parallel processing. This not only ensures the image processing speed, but also enables the visible light sensor to have both target recognition and star attitude calculation functions, which is beneficial to reduce the overall weight, power consumption and volume of the aircraft. Attached Figure Description

[0029] Figure 1 This is a basic hardware circuit block diagram of the visible light sensor of the present invention;

[0030] Figure 2 This is an overall flowchart of the present invention;

[0031] Figure 3 This is a schematic diagram of the sensor image output and FPGA image reception of the present invention;

[0032] Figure 4 This is a diagram of the parallel architecture for image exposure, acquisition, and processing of the visible light sensor of the present invention. Detailed Implementation

[0033] Therefore, a multi-mode image acquisition and processing method for visible light sensors is proposed. According to the functional mode set by the processor, different sized window images are cropped at a variable update rate to meet the requirements of both target recognition and attitude calculation for image size and update rate. According to different functional modes, the FPGA and the processor adopt corresponding image processing methods, so that the visible light sensor can simultaneously have target recognition and star-sensor attitude calculation functions.

[0034] The technical solution of this invention is: a multi-mode image acquisition and processing method for visible light sensors. The steps are as follows:

[0035] (1) The FPGA receiver processor sets the function mode and the center coordinates of the image window. In the target recognition mode, a small window is used, and in the pose measurement mode, a large window is used. The starting row number, starting column number, ending row number, and ending column number of the window are calculated according to the window size and center coordinates. Then proceed to step (2).

[0036] (2) Calculate the time for the FPGA to read out a single row of data in the window image based on the starting column number and the ending column number, and use it as the row interval time for updating the sensor image; at the same time, calculate the output channel number and index address of the starting column and the ending column of the window based on the number of output channels of the single row of sensor image and the number of image data in each channel, and then proceed to step (3);

[0037] (3) During the exposure idle time, the row interval time of image update and the start and end row numbers of the window are written to the image sensor register through the SPI interface. The image sensor will only expose and output data to the pixels from the start row to the end row, and then proceed to step (4).

[0038] (4) The FPGA generates periodic exposure start and stop signals to control the image sensor to expose all rows of the window. After the exposure is completed, the image sensor splits the image data of each row into N channels and outputs them in parallel according to a fixed format, and then proceeds to step (5).

[0039] (5) The FPGA simultaneously receives image data from N channels of the sensor, stores it in N on-chip RAMs, waits for the data in the N on-chip RAMs to form a complete row of images, and then proceeds to step (6).

[0040] (6) The FPGA determines the number of RAMs to be read and the RAM address reading range based on the output channel number and index address of the start column and end column of the window in step (2), directly reads the pixel data of the single row window, and then executes step (7). This process does not require reading the complete single row image from N on-chip RAMs and storing it in on-chip RAM, and then cropping the window image according to the start column number and end column number.

[0041] (7) The FPGA selects an image processing method for the window image obtained in step (6) according to the function mode flag. If it is the target recognition function, then step (8) is executed; if it is to use the capture mode to extract star points for attitude measurement, then step (9) is executed; if it is to use the tracking mode to extract star points for attitude measurement, then step (10) is executed.

[0042] (8) During the process of receiving pixel data, the FPGA compares the pixel gray value with the background threshold. Pixels with a gray value higher than the background threshold are extracted as targets, and the obtained target feature information is stored. Then, step (12) is executed. The target feature information includes: the size, energy and centroid coordinates of each target.

[0043] (9) During the process of receiving pixel data, the FPGA uses a first-order low-pass filter to filter the image, compares the gray value of the filtered pixel with the background threshold, extracts star points for pixels that are higher than the background threshold, and stores the obtained star point information. Then, step (13) is executed. The star point information is the energy and center coordinates of each star point.

[0044] (10) During the process of receiving pixel data, the FPGA performs star point window capture. After a single window capture is completed, step (11) is executed. The star point window capture refers to the FPGA capturing star point pixels according to the coordinates and size of multiple star point windows set by the processor and storing them in the on-chip RAM.

[0045] (11) The FPGA extracts star points one by one from the captured star point window image, stores the obtained star point information, and then executes step (13).

[0046] (12) The processor reads multiple target feature information stored in the FPGA and identifies the target to be tracked and its centroid.

[0047] (13) The processor reads the star point information stored in the FPGA and performs attitude calculation.

[0048] The present invention will now be described in detail with reference to the accompanying drawings.

[0049] Figure 1 This is a basic hardware circuit block diagram of the visible light sensor of the present invention, including an image sensor, a processor, and an FPGA. The FPGA outputs an exposure signal to control the image sensor to image targets and stars in space. The row address of the image sensor exposure and the data update interval for each row can be configured through the SPI interface. The FPGA acquires pixel data from multiple channels output by the image sensor, and captures window images of different sizes according to different update rates based on the function mode set by the processor, and selects different image preprocessing methods. The FPGA has an internal storage unit to cache the acquired images and processing results. The processor reads the processing result data from the FPGA's on-chip storage unit to perform target recognition and attitude calculation.

[0050] like Figure 2 As shown, this invention proposes a multi-mode image acquisition and processing method for visible light sensors, the steps of which are as follows:

[0051] (1) The FPGA receives the function mode and the center coordinates of the image window set by the processor. The target recognition mode uses a small window and the attitude measurement mode uses a large window. The starting row number row_saddr, the ending row number row_eaddr, the starting column number col_saddr, and the ending column number col_eaddr are calculated according to the window size and center coordinates. Then, proceed to step (2). The function mode flag is located inside the FPGA. The processor sets the mode flag while switching modes.

[0052] (2) Calculate the time Trd for the FPGA to read a single row of data from the window diagram based on the starting column number col_saddr and the ending column number col_eaddr. Trd is equal to the product of (col_eaddr-col_saddr) and the FPGA main clock cycle. Then proceed to step (3).

[0053] (3) According to Figure 3 The sensor image data output format shown is used to calculate the output channel number ram_schan and index address ram_schan_addr of the starting column of the calculation window, and the output channel number ram_echan and index address ram_echan_addr of the ending column of the calculation window. The calculation is described below, and then proceed to step (4):

[0054] ram_schan=col_saddr / chan_Len

[0055] ram_echan=col_eaddr / chan_Len

[0056] ram_schan_addr=col_saddr%chan_Len

[0057] ram_echan_addr=col_eaddr%chan_Len

[0058] in:

[0059] chan_Len = Line_Len / N

[0060] In the formula, Line_Len and chan_Len are the number of effective pixels in each row and each channel, respectively, N is the number of sensor image channels, and % represents the modulo operation.

[0061] (4) During the exposure idle time, the row interval time of image update and the start and end row numbers of the window are written to the image sensor register through the SPI interface. The image sensor will only expose and output data to the pixels of row_saddr to row_eaddr, and then proceed to step (5). The row interval time is the time Trd of the FPGA readout window map single row data calculated in step (2).

[0062] (5) The FPGA generates periodic exposure start and stop signals to control the exposure of the image sensor. After the exposure is completed, the image data of the sensor is processed according to... Figure 3 The output format is shown. The sensor internally splits each line of image data into N channels for parallel output, and then proceeds to step (6).

[0063] (6) Figure 3 As shown, the FPGA simultaneously receives image data from N channels of the sensor, stores it in N on-chip RAMs, waits for the data from the N RAMs to form a complete row of images, and then proceeds to step (7);

[0064] (7) Figure 3 As shown, the FPGA determines the number of RAMs to be read and the reading range of each RAM address based on ram_schan, ram_schan_addr, ram_schan and ram_schan_addr, directly reads the pixel data of a single row window, and then executes step (8). The ram_schan, ram_schan_addr, ram_schan and ram_schan_addr are the channel number and index address calculated in step (3). This process does not require reading out the complete single row image and storing it in the on-chip RAM, and then cropping the window image based on the starting column number col_saddr and the ending column number col_eaddr.

[0065] (8) Figure 2 As shown, the FPGA selects an image preprocessing method for the window image obtained in step (7) according to the function mode flag. If it is the target recognition function, then step (9) is executed; if it is to use the capture mode to extract star points for pose measurement, then step (10) is executed; if it is to use the tracking mode to extract star points for pose measurement, then step (11) is executed.

[0066] (9) During the process of receiving pixel data, the FPGA compares the pixel grayscale value with the background threshold. Pixels with grayscale values ​​higher than the background threshold are extracted as targets, and the obtained target feature information is stored. Then, step (14) is executed. The specific processing of target extraction is as follows:

[0067] During the process of receiving pixel data, the FPGA clusters the rows and columns of pixels that are higher than the background threshold. Pixels belonging to the same target are clustered into a pixel block. False targets are eliminated based on the size and energy of the pixel block, and the feature information of the effective targets is calculated.

[0068] (10) Use capture mode to extract star points for pose measurement: During the process of receiving pixel data, the FPGA uses a first-order low-pass filter to filter the image, compares the gray value of the filtered pixel with the background threshold, and then executes step (12).

[0069] (11) Star points are extracted using tracking mode for pose measurement: During the process of receiving pixel data, the FPGA performs star point window cropping. After a single window is cropped, step (13) is executed. The window coordinates and size are written to the FPGA after the processor receives the window interrupt. The window interrupt is generated by the FPGA before the exposure ends. The star point window cropping refers to the FPGA cropping star point pixels according to the multiple star point window coordinates and sizes set by the processor and storing them in the on-chip RAM.

[0070] (12) The FPGA extracts star points from pixels above the background threshold and stores the obtained star point information. Then, step (15) is executed. The specific processing of star point extraction is as follows:

[0071] During the process of receiving pixel data, the FPGA uses a first-order low-pass filter to filter the image, compares the gray value of the filtered pixels with the background threshold, clusters the rows and columns of filtered pixels that are higher than the background threshold, and clusters pixels belonging to the same star point into a pixel block. False star points are eliminated according to the size and energy of the pixel block, and the information of the effective star points is calculated and stored.

[0072] (13) The FPGA extracts star points one by one from the captured star point window image and stores the obtained star point information, and then executes step (16); the star point window image is the image data obtained in step (11), and the specific processing is as follows:

[0073] The FPGA calculates the background threshold based on the boundary pixels of each window image, traverses the pixel blocks that are higher than the background threshold and belong to the same star point through the connected component algorithm, eliminates false star points based on the size and energy of the pixel blocks, and calculates the information of the valid star points.

[0074] (14) The processor reads multiple target feature information stored in the FPGA and identifies the target to be tracked and its centroid. The target feature information is the size, energy and centroid coordinates of each target extracted in step (9).

[0075] (15) The processor reads the star point information stored in the capture mode of the FPGA and performs attitude calculation. The star point information is the energy and centroid coordinates of each star point obtained in step (12).

[0076] (16) The processor reads the star point information stored in the tracking mode of the FPGA and performs attitude calculation. The star point information is the energy and centroid coordinates of each star point obtained in step (13).

[0077] The target feature information data and star point data obtained in steps (14), (15) and (16) are directly stored in the FPGA on-chip RAM and do not require external memory.

[0078] The visible light sensor image exposure, acquisition, and processing process employs a three-stage pipelined parallel architecture, such as... Figure 4 As shown, when the image sensor outputs the N channels of the (n+1)th image and performs exposure integration of the (n+2)th image, the FPGA performs the reception, windowing, and image preprocessing of the (n+1)th image in parallel. At the same time, the processor reads the FPGA preprocessing results and processes the nth image in parallel, where n≥0.

[0079] Before image exposure, configure the row spacing and number of exposure rows for sensor image data output. When acquiring the image, the FPGA determines the number of RAM channels to be read and the RAM address reading range based on the output channel number and index address of the window start column and end column in step (2). It directly reads the single-row window image data. This process does not require reading the complete single-row image from N on-chip RAMs and storing it in on-chip RAM, and then cropping the window image according to the start column number and end column number.

[0080] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A multi-mode image acquisition and processing method for a visible light sensor, the visible light sensor comprising: Image sensor, processor, FPGA, characterized in that: the multi-mode image acquisition and processing method includes: The FPGA receives the function mode and image window center coordinates set by the receiver processor, determines the image window size according to the requirements of different function modes for image update rate and image size, and calculates the row and column information corresponding to the window area. The FPGA generates periodic exposure start and stop signals to control the image sensor to expose all rows of the window. After the exposure is completed, the image sensor internally splits the image data of each row into N channels and outputs them in parallel according to a fixed format. The FPGA simultaneously receives image data from N channels of the sensor and stores it in N on-chip RAMs. After waiting for the data in the N RAMs to form a complete single-row image, the FPGA determines the number of RAMs to be read and the RAM address reading range based on the column range of the window area, and reads out the pixel data of the single-row window. The FPGA and processor process the window pixel data according to the image processing method set by the internal function mode flag bit. The processing method includes: target recognition, star point extraction for pose measurement in capture mode, and star point extraction for pose measurement in tracking mode.

2. The multi-mode image acquisition and processing method for a visible light sensor according to claim 1, characterized in that: The specific steps for calculating the row and column information corresponding to the image window region are as follows: Calculate the starting row number, starting column number, ending row number, and ending column number of the window based on the window size and center coordinates; The time for the FPGA to read out a single row of data from the window is calculated based on the start column number and the end column number. At the same time, the number and index address of the output channel where the start column and the end column of the window are located are calculated based on the number of sensor image output channels in a single row and the number of image data in each channel.

3. The multi-mode image acquisition and processing method for a visible light sensor according to claim 1, characterized in that: During the exposure idle time, the interval time for updating each row of data of the image sensor, the start and end row numbers of the window are written to the image sensor register through the SPI interface. The image sensor will only expose and output data for the pixels from the start row to the end row of the window in the whole image.

4. The multi-mode image acquisition and processing method for a visible light sensor according to claim 1, characterized in that: When performing target recognition on window pixel data: During the process of receiving pixel data, the FPGA clusters the rows and columns of pixels that are higher than the background threshold. Pixels belonging to the same target are clustered into a pixel block. False targets are eliminated based on the size and energy of the pixel block, and the effective target feature information is calculated and stored. The processor reads the target feature information stored in the FPGA to perform target recognition; Target characteristic information includes: the size, energy, and centroid coordinates of each target.

5. The multi-mode image acquisition and processing method for a visible light sensor according to claim 1, characterized in that: When using capture mode to extract star points from window pixel data for pose measurement: During the process of receiving pixel data, the FPGA uses a first-order low-pass filter to filter the image, compares the filtered pixel grayscale value with the background threshold, clusters the rows and columns of filtered pixels that are higher than the background threshold, and clusters pixels belonging to the same star point into a pixel block. False star points are eliminated based on the size and energy of the pixel block, and the information of the effective star points is calculated and stored. The processor reads the star point information stored in the FPGA and performs attitude calculation. Among them, the star point information includes the energy and centroid coordinates of each star point.

6. The multi-mode image acquisition and processing method for a visible light sensor according to claim 1, characterized in that: When using tracking mode to extract star points for pose measurement from window pixel data: During the process of receiving pixel data, the FPGA performs star point window segmentation according to the coordinates and size of multiple star point windows set by the processor. The FPGA calculates the background threshold based on the boundary pixels of each window image, and uses a connected component algorithm to traverse the pixel blocks that are higher than the background threshold and belong to the same star point. False star points are eliminated based on the size and energy of the pixel blocks, and the information of the valid star points is calculated and stored. The processor reads the star point information stored by the FPGA and performs attitude calculation.

7. A multi-mode image acquisition and processing method for a visible light sensor according to any one of claims 5 and 6, characterized in that: The star point information includes the energy and centroid coordinates of each star point.

8. The multi-mode image acquisition and processing method for a visible light sensor according to claim 1, characterized in that: The visible light sensor image exposure, acquisition and processing process adopts a three-stage pipelined parallel architecture. While the image sensor outputs the N-channel data of the (n+1)th image and performs the exposure integration of the (n+2)th image, the FPGA performs the reception, windowing and image preprocessing of the (n+1)th image in parallel. At the same time, the processor reads the FPGA preprocessing results and processes the nth image in parallel, where n≥0.

Images