Multi-mode image acquisition processing method for visible light sensor

Abstract

The invention discloses a multi-mode image acquisition processing method for a visible light sensor, which comprises the following steps that: an FPGA (Field Programmable Gate Array) receives a function mode and an image window center coordinate set by a processor, determines the size of an image window and calculates corresponding row and column information; the FPGA generates periodic exposure starting and stopping signals to expose all rows where the window is located, and after exposure is finished, each row of image data in the image sensor is split into N channels to be output in parallel according to a fixed format; the FPGA receives image data from N channels of the sensor at the same time and stores the image data in N on-chip RAMs; the FPGA determines the number of RAMs needing to be read and the RAM address reading range according to the column range of the window area, and single-row window pixel data is read; and the FPGA and the processor process window pixel data according to an image processing method set by an internal function mode flag bit. According to the invention, the requirements of two function modes of target identification and pose measurement on the image size and the update rate can be met, and the mass, the power consumption and the volume of the whole machine can be reduced.

Description

Technical Field

[0001] The invention relates to a multi-mode image acquisition and processing method for a visible light sensor, and belongs to the field of image acquisition and processing. Background Art

[0002] With the rapid development of aerospace technology, the space field has become a new battlefield for strategic games among major powers. Visible light sensors are deployed on-orbit aircraft to capture and track targets, warn, drive away, or even damage high-orbit combat aircraft that maliciously approach and harass our high-orbit satellites, and protect the normal use of our high-value satellites. Star sensors are a high-precision attitude measurement device for aircraft. They collect star images through image sensors and obtain three-axis inertial attitudes through star map processing. Visible light sensors with multiple functional modes such as target recognition and attitude measurement can reduce the number of payloads and effectively reduce the mass, power consumption, and volume of aircraft.

[0003] In attitude measurement mode, in order to ensure measurement accuracy, it is necessary to use full image or large window image for processing, so as to ensure that enough stars can be extracted for attitude solution; in target recognition mode, small window image can be used to locate the target, but a high image update rate is required to achieve fast and accurate target tracking. Different functional modes have different requirements for image update rate and image processing method.

[0004] In recent years, relevant research progress has been made in multi-mode image acquisition and processing methods for sensors, but the existing technologies all have certain application limitations and shortcomings. CN103968845A discloses a processor and FPGA parallel multi-mode star map processing method for star sensors. By dividing and optimizing the star map processing algorithm, the pose measurement function in the two working modes of capture and tracking is realized. However, this scheme cannot meet the requirements of high frame rate window images in the target recognition function mode; CN107478218A discloses a variable update rate image acquisition of a star sensor by changing the underlying hardware driver timing of the image sensor, but this method is only applicable to a specific type of image sensor and is not universal; CN104202513A discloses an image acquisition and processing method for full image mode and window image mode of a star sensor based on FPGA. When intercepting the window image, it is necessary to read out the complete single-row images of multiple on-chip RAM caches and store them in the on-chip RAM again, and then intercept the window image according to the starting column position and the ending column position, which increases the processing time and resource occupancy rate. Summary of the invention

[0005] The technical problem solved by the present invention is: to overcome the shortcomings of the prior art, to provide a multi-mode image acquisition and processing method for a visible light sensor, to determine the image window size according to different functional modes, and to enable the visible light sensor to have both target recognition and star-sensing attitude solution functions by changing the image update rate and the image processing method.

[0006] The technical solution of the present invention is: a multi-mode image acquisition and processing method for a visible light sensor, the visible light sensor includes: an image sensor, a processor, and an FPGA, and the multi-mode image acquisition and processing method includes:

[0007] The FPGA receives the function mode and the image window center coordinates set by the processor, determines the image window size according to the requirements of the image update rate and image size in different function modes, and calculates the row and column information corresponding to the window area;

[0008] The FPGA generates periodic exposure start and end signals to control the image sensor to expose all rows where the window is located. After the exposure is completed, the image sensor splits each row of image data into N channels and outputs them in parallel in a fixed format.

[0009] FPGA simultaneously receives image data from N channels of the sensor and stores it in N on-chip RAMs. After waiting for the data in the N RAMs to form a complete single-row image, FPGA determines the number of RAMs to be read and the RAM address reading range based on the column range of the window area, and reads out the single-row window pixel data.

[0010] The FPGA and the processor process the window pixel data according to the image processing method set by the internal function mode flag, and the processing method includes: target recognition, capturing mode to extract star points for posture measurement, and tracking mode to extract star points for posture measurement.

[0011] Preferably, the row and column information corresponding to the image window area is calculated as follows:

[0012] Calculate the window's starting row number, starting column number, ending row number, and ending column number based on the window size and center coordinates;

[0013] The time for FPGA to read out a single row of data from the window image is calculated based on the starting column number and the ending column number. At the same time, the number and index address of the output channel where the starting column and the ending column of the window are located are calculated based on the number of output channels of the single-row sensor image and the number of image data in each channel.

[0014] Preferably, during the exposure idle time, the interval time for updating each row of data of the image sensor, the start and end row numbers of the window are written into the image sensor register through the SPI interface, and the image sensor will only expose and output data for the pixels from the start row to the end row of the window in the full image.

[0015] Preferably, when performing target recognition on the window pixel data:

[0016] When receiving pixel data, FPGA clusters the rows and columns of pixels above the background threshold. Pixels belonging to the same target are clustered into a pixel block. Pseudo targets are eliminated based on the size and energy of the pixel block. Effective target feature information is calculated and stored. The processor reads the target feature information stored in FPGA for target recognition.

[0017] Target feature information includes: size, energy and center of mass coordinates of each target.

[0018] Preferably, when the window pixel data is captured in a capture mode to extract star points for posture measurement:

[0019] When receiving pixel data, FPGA uses a first-order low-pass filter to filter the image, compares the grayscale value of the filtered pixel with the background threshold, clusters the rows and columns of filtered pixels that are higher than the background threshold, and clusters the pixels belonging to the same star point into a pixel block. Pseudo star points are removed according to the size and energy of the pixel block, and the information of valid star points is calculated and stored; the processor reads the star point information stored in FPGA and performs attitude solution;

[0020] The star point information includes the energy and center of mass coordinates of each star point.

[0021] Preferably, when the window pixel data is subjected to tracking mode to extract star points for posture measurement:

[0022] When receiving pixel data, FPGA intercepts the star point window according to the multiple star point window coordinates and star point window sizes set by the processor. FPGA calculates the background threshold according to the boundary pixels of each window image, traverses the pixel blocks that are higher than the background threshold and belong to the same star point through the connected domain algorithm, removes pseudo star points according to the size and energy of the pixel blocks, calculates and stores the information of valid star points; the processor reads the star point information stored in FPGA and performs attitude solution.

[0023] Preferably, the star point information is the energy and center of mass coordinates of each star point.

[0024] Preferably, the visible light sensor image exposure, acquisition and processing process adopts a three-level pipeline parallel architecture. When the image sensor performs N-channel output of the n+1th image data and exposure integration of the n+2th image, the FPGA performs reception, window capture and image preprocessing of the n+1th image in parallel, and the processor reads the FPGA preprocessing results and processes the nth image in parallel, where n≥0.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] (1) Before image exposure, the row interval and number of exposure rows of the sensor image data output are configured. During image acquisition, only the data of the column where the window is located is read out. The image window size determines the image output and readout time. A high image update rate is achieved under a small window, which can meet the needs of various functional modes.

[0027] (2) Based on the calculated output channel numbers and index addresses of the window start and end columns, the FPGA can directly read out only the window image from the complete single-row image cached in the N on-chip RAMs. This process does not require reading out and storing the complete single-row image from the N on-chip RAMs and then intercepting the window image based on the start and end column numbers, thereby saving image acquisition time and hardware resources.

[0028] (3) According to different functional modes, the processor and FPGA switch the window image size and update rate, and select different image algorithms for parallel processing. This not only ensures the image processing speed, but also enables the visible light sensor to have both target recognition and star-sensing attitude solution functions, which is beneficial to reducing the overall weight, power consumption, and volume of the aircraft. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a basic hardware circuit block diagram of the visible light sensor of the present invention;

[0030] Figure 2 It is the overall flow chart of the present invention;

[0031] Figure 3 It is a schematic diagram of sensor image output and FPGA image reception of the present invention;

[0032] Figure 4 A parallel architecture diagram of visible light sensor image exposure, acquisition and processing of the present invention. DETAILED DESCRIPTION

[0033] Therefore, a multi-mode image acquisition and processing method for visible light sensors is proposed. According to the functional mode set by the processor, window images of different sizes are captured at a variable update rate to meet the requirements of the two functional modes of target recognition and attitude solution on image size and update rate. According to different functional modes, FPGA and processor adopt corresponding image processing methods to enable the visible light sensor to have both target recognition and star-sensitive attitude solution functions.

[0034] The technical solution of the present invention is: a multi-mode image acquisition and processing method for a visible light sensor. The steps are as follows:

[0035] (1) The FPGA receives the function mode and the center coordinates of the image window set by the processor, wherein a small window is used in the target recognition mode and a large window is used in the posture measurement mode. The window start row number, start column number, end row number, and end column number are calculated according to the window size and the center coordinates, and then go to step (2);

[0036] (2) Calculate the time for FPGA to read out a single row of window image data according to the starting column number and the ending column number, and use it as the row interval time for updating the sensor image; at the same time, calculate the output channel number and index address of the window starting column and the ending column respectively according to the number of single-row sensor image output channels and the number of image data in each channel, and then go to step (3);

[0037] (3) At the idle time of exposure, the row interval time of image update and the start and end row numbers of the window are written into the image sensor register through the SPI interface. The image sensor will only expose and output data for the pixels from the start row to the end row, and then go to step (4);

[0038] (4) FPGA generates periodic exposure start and end signals to control the image sensor to expose all rows where the window is located. After the exposure is completed, the image sensor internally splits each row of image data into N channels and outputs them in parallel according to a fixed format, and then goes to step (5);

[0039] (5) FPGA simultaneously receives image data from N channels of the sensor, stores it in N on-chip RAMs, waits for the data in the N on-chip RAMs to form a complete row of images, and then goes to step (6);

[0040] (6) The FPGA determines the number of RAMs to be read and the RAM address reading range according to the output channel number and index address of the window start column and end column in step (2), directly reads out the single-row window pixel data, and then executes step (7); this process does not require reading out the complete single-row image from N on-chip RAMs and storing them in the on-chip RAM, and then intercepting the window image according to the start column number and the end column number;

[0041] (7) FPGA selects an image processing method for the window image obtained in step (6) according to the function mode flag. If it is a target recognition function, step (8) is executed; if the capture mode is used to extract star points for attitude measurement, step (9) is executed; if the tracking mode is used to extract star points for attitude measurement, step (10) is executed;

[0042] (8) In the process of receiving pixel data, the FPGA compares the pixel grayscale value with the background threshold, extracts the pixels above the background threshold, and stores the obtained target feature information, and then executes step (12); the target feature information includes: the size, energy and center of mass coordinates of each target;

[0043] (9) In the process of receiving pixel data, FPGA uses a first-order low-pass filter to filter the image, compares the grayscale value of the filtered pixel with the background threshold, extracts star points from pixels above the background threshold, and stores the obtained star point information and then executes step (13); the star point information is the energy and center coordinates of each star point;

[0044] (10) In the process of receiving pixel data, the FPGA performs star point window capture, and after a single window capture is completed, step (11) is executed, wherein the star point window capture refers to the FPGA capturing star point pixels according to the coordinates and sizes of multiple star point windows set by the processor and storing them in the on-chip RAM;

[0045] (11) FPGA extracts star points from the captured star point window image one by one, stores the obtained star point information, and then executes step (13);

[0046] (12) The processor reads multiple target feature information stored in the FPGA and identifies the target to be tracked and its centroid;

[0047] (13) The processor reads the star point information stored in the FPGA and performs attitude calculation.

[0048] The present invention is described in detail below with reference to the accompanying drawings.

[0049] Figure 1 This is a basic hardware circuit diagram of the visible light sensor of the present invention, including an image sensor, a processor, and an FPGA. The FPGA outputs an exposure signal to control the image sensor to image targets and stars in space. The row address of the image sensor exposure and the interval time for updating each row of data can be configured through the SPI interface; the FPGA collects the pixel data of multiple channels output by the image sensor, and intercepts window images of different sizes at different update rates according to the function mode set by the processor, and selects different image preprocessing methods. The FPGA has a storage unit inside to cache the collected images and processing results. The processor reads the processing result data from the storage unit inside the FPGA chip to perform target recognition and posture resolution.

[0050] like Figure 2 As shown, the present invention proposes a multi-mode image acquisition and processing method for a visible light sensor, the steps are as follows:

[0051] (1) FPGA receives the function mode and image window center coordinates set by the processor, wherein a small window is used in the target recognition mode and a large window is used in the posture measurement mode, and the window start row number row_saddr, end row number row_eaddr, start column number col_saddr, and end column number col_eaddr are calculated according to the window size and the center coordinates, and then goes to step (2); the function mode flag is located inside the FPGA, and the processor sets the mode flag while switching the mode;

[0052] (2) Calculate the time Trd for FPGA to read out a single row of data in the window diagram according to the starting column number col_saddr and the ending column number col_eaddr. Trd is equal to the product of (col_eaddr-col_saddr) and the FPGA main clock cycle, and then go to step (3);

[0053] (3) Based on Figure 3 The sensor image data output format shown in FIG. 1 is used to calculate the output channel number ram_schan and index address ram_schan_addr of the window start column, and the output channel number ram_echan and index address ram_echan_addr of the window end column, as described below, and then go to step (4):

[0054] ram_schan=col_saddr / chan_Len

[0055] ram_echan=col_eaddr / chan_Len

[0056] ram_schan_addr=col_saddr%chan_Len

[0057] ram_echan_addr=col_eaddr%chan_Len

[0058] in:

[0059] chan_Len=Line_Len / N

[0060] Where Line_Len and chan_Len are the number of valid pixels in each line and each channel respectively, N is the number of sensor image channels, and % represents the remainder operation.

[0061] (4) At the idle time of exposure, the row interval time of image update and the start and end row numbers of the window are written into the image sensor register through the SPI interface. The image sensor will only expose and output data for the pixels in the rows from row_saddr to row_eaddr, and then go to step (5). The row interval time uses the time Trd of the FPGA reading out the single row data of the window image calculated in step (2);

[0062] (5) FPGA generates periodic exposure start and end signals to control the exposure of the image sensor. After the exposure is completed, the image data of the sensor is Figure 3 The sensor outputs the image data in the format shown in the figure. The sensor internally splits each row of image data into N channels for parallel output, and then goes to step (6).

[0063] (6) Figure 3 As shown, the FPGA simultaneously receives image data from N channels of the sensor, stores them in N on-chip RAMs, waits for the data in the N RAMs to form a complete row of images, and then goes to step (7);

[0064] (7) Figure 3 As shown, the FPGA determines the number of RAMs to be read and the reading range of each RAM address according to ram_schan, ram_schan_addr, ram_schan and ram_schan_addr, directly reads out the single-row window pixel data, and then executes step (8), wherein the ram_schan, ram_schan_addr, ram_schan and ram_schan_addr are the channel number and index address calculated in step (3). This process does not require the complete single-row image to be read out and stored in the on-chip RAM, and then the window image is intercepted according to the starting column number col_saddr and the ending column number col_eaddr;

[0065] (8) Figure 2 As shown, the FPGA selects an image preprocessing method for the window image obtained in step (7) according to the function mode flag. If it is a target recognition function, step (9) is executed; if the capture mode is used to extract star points for posture measurement, step (10) is executed; if the tracking mode is used to extract star points for posture measurement, step (11) is executed;

[0066] (9) In the process of receiving pixel data, FPGA compares the pixel grayscale value with the background threshold, extracts the pixel whose grayscale is higher than the background threshold, and stores the obtained target feature information, and then executes step (14). The specific process of target extraction is as follows:

[0067] When receiving pixel data, FPGA clusters the rows and columns of pixels above the background threshold. Pixels belonging to the same target are clustered into a pixel block. Pseudo targets are eliminated based on the size and energy of the pixel block, and the feature information of the valid target is calculated.

[0068] (10) Using the capture mode to extract star points for posture measurement: In the process of receiving pixel data, the FPGA uses a first-order low-pass filter to filter the image, compares the grayscale value of the filtered pixel with the background threshold, and then executes step (12);

[0069] (11) Extracting star points in tracking mode for pose measurement: In the process of receiving pixel data, FPGA performs star point window capture. After a single window capture is completed, step (13) is executed. The window coordinates and size are written into FPGA after the processor receives the window interrupt. The window interrupt is generated by FPGA before the end of exposure. The star point window capture refers to FPGA capturing star point pixels according to multiple star point window coordinates and sizes set by the processor and storing them in on-chip RAM.

[0070] (12) FPGA extracts star points from pixels above the background threshold, stores the obtained star point information, and executes step (15). The specific processing of star point extraction is as follows:

[0071] When receiving pixel data, FPGA uses a first-order low-pass filter to filter the image, compares the grayscale value of the filtered pixel with the background threshold, clusters the rows and columns of the filtered pixels that are higher than the background threshold, and clusters the pixels belonging to the same star point into a pixel block. Pseudo-star points are removed according to the size and energy of the pixel block, and the information of valid star points is calculated and stored.

[0072] (13) FPGA extracts star points from the captured star point window image one by one, stores the obtained star point information, and then executes step (16); the star point window image is the image data captured in step (11), and the specific processing is as follows:

[0073] FPGA calculates the background threshold based on the boundary pixels of each window image, traverses the pixel blocks that are higher than the background threshold and belong to the same star point through the connected domain algorithm, removes pseudo star points based on the size and energy of the pixel blocks, and calculates the information of valid star points;

[0074] (14) The processor reads multiple target feature information stored in the FPGA, and identifies the target to be tracked and its centroid, wherein the target feature information is the size, energy, and center of mass coordinates of each target extracted in step (9);

[0075] (15) The processor reads the star point information in the capture mode stored in the FPGA and performs attitude calculation, wherein the star point information is the energy and center of mass coordinates of each star point obtained in step (12);

[0076] (16) The processor reads the star point information in the tracking mode stored in the FPGA and performs attitude calculation, wherein the star point information is the energy and center of mass coordinates of each star point obtained in step (13);

[0077] The target feature information data and star point data obtained in step (14), step (15) and step (16) are directly stored in the FPGA on-chip RAM without the need for an external memory;

[0078] The visible light sensor image exposure, acquisition and processing process adopts a three-stage pipeline parallel architecture, such as Figure 4 As shown, when the image sensor performs N-channel output of the n+1th image and exposure integration of the n+2th image, the FPGA performs reception, window interception and image preprocessing of the n+1th image in parallel, and the processor reads the FPGA preprocessing result and performs processing of the nth image in parallel, where n≥0;

[0079] Before image exposure, the row interval and number of exposure rows of the sensor image data output are configured. During image acquisition, the FPGA determines the number of channel RAMs to be read and the RAM address reading range based on the output channel number and index address of the window start column and end column in step (2), and directly reads out the single-row window image data. This process does not require reading out the complete single-row image from N on-chip RAMs and storing them in the on-chip RAM, and then intercepting the window image based on the start column number and end column number.

[0080] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in this field.

Claims

1. A multi-mode image acquisition and processing method for a visible light sensor, the visible light sensor comprising: An image sensor, a processor, and an FPGA are characterized in that: the multi-mode image acquisition and processing method includes: The FPGA receives the function mode and the image window center coordinates set by the processor, determines the image window size according to the requirements of the image update rate and image size in different function modes, and calculates the row and column information corresponding to the window area; The FPGA generates periodic exposure start and end signals to control the image sensor to expose all rows where the window is located. After the exposure is completed, the image sensor splits each row of image data into N channels and outputs them in parallel in a fixed format. FPGA simultaneously receives image data from N channels of the sensor and stores it in N on-chip RAMs. After waiting for the data in the N RAMs to form a complete single-row image, FPGA determines the number of RAMs to be read and the RAM address reading range based on the column range of the window area, and reads out the single-row window pixel data. The FPGA and the processor process the window pixel data according to the image processing method set by the internal function mode flag, and the processing method includes: target recognition, capturing mode to extract star points for posture measurement, and tracking mode to extract star points for posture measurement.

2. The multi-mode image acquisition and processing method for a visible light sensor according to claim 1, characterized in that: The row and column information corresponding to the calculated image window area is as follows: Calculate the window's starting row number, starting column number, ending row number, and ending column number based on the window size and center coordinates; The time for FPGA to read out a single row of data from the window image is calculated based on the starting column number and the ending column number. At the same time, the number and index address of the output channel where the starting column and the ending column of the window are located are calculated based on the number of output channels of the single-row sensor image and the number of image data in each channel.

3. The multi-mode image acquisition and processing method for a visible light sensor according to claim 1, characterized in that: During the idle time of exposure, the interval time for updating each row of image sensor data and the start and end row numbers of the window are written into the image sensor register through the SPI interface. The image sensor will only expose and output data for the pixels from the start row to the end row of the window in the whole image.

4. The multi-mode image acquisition and processing method for a visible light sensor according to claim 1, characterized in that: When performing target recognition on window pixel data: When receiving pixel data, FPGA clusters the rows and columns of pixels above the background threshold. Pixels belonging to the same target are clustered into a pixel block. Pseudo targets are eliminated based on the size and energy of the pixel block, and effective target feature information is calculated and stored. The processor reads the target feature information stored in the FPGA to perform target recognition; Target feature information includes: size, energy and center of mass coordinates of each target.

5. The multi-mode image acquisition and processing method for a visible light sensor according to claim 1, characterized in that: When the capture mode is used to extract star points from the window pixel data for pose measurement: When receiving pixel data, FPGA uses a first-order low-pass filter to filter the image, compares the grayscale value of the filtered pixel with the background threshold, clusters the rows and columns of filtered pixels that are higher than the background threshold, and clusters the pixels belonging to the same star point into a pixel block. Pseudo star points are removed according to the size and energy of the pixel block, and the information of valid star points is calculated and stored; the processor reads the star point information stored in FPGA and performs attitude solution; The star point information includes the energy and center of mass coordinates of each star point.

6. The multi-mode image acquisition and processing method for a visible light sensor according to claim 1, characterized in that: When the window pixel data is tracked and star points are extracted for pose measurement: In the process of receiving pixel data, FPGA intercepts the star point window according to the multiple star point window coordinates and star point window sizes set by the processor. FPGA calculates the background threshold according to the boundary pixels of each window image, traverses the pixel blocks that are higher than the background threshold and belong to the same star point through the connected domain algorithm, removes pseudo star points according to the size and energy of the pixel blocks, calculates and stores the information of valid star points; the processor reads the star point information stored in FPGA and performs attitude solution.

7. A multi-mode image acquisition and processing method for a visible light sensor according to any one of claims 5 and 6, characterized in that: The star point information is the energy and center of mass coordinates of each star point.

8. The multi-mode image acquisition and processing method for a visible light sensor according to claim 1, characterized in that: The visible light sensor image exposure, acquisition and processing process adopts a three-level pipeline parallel architecture. When the image sensor performs N-channel output of the n+1th image data and exposure integration of the n+2th image, the FPGA performs reception, window capture and image preprocessing of the n+1th image in parallel. At the same time, the processor reads the FPGA preprocessing results and processes the nth image in parallel, where n≥0.

Images