Target detection method based on FPGA (Field Programmable Gate Array) in low-light scene

Abstract

The invention provides an FPGA-based target detection method in a low-light scene, and the method comprises the following steps: obtaining an ExDark data set, taking the ExDark data set as a basic data set, and selecting a training set, a verification set and a test set from the basic data set; acquiring a low-light image as an extended data set, and performing contrast adjustment, target amplification and super-resolution enhancement processing on the extended data set and the training set to obtain a final training data set; the method comprises the following steps: constructing a Y0L0v3 target detection model based on PyTorch, and deploying the Y0L0v3 target detection model on an FPGA MPSOC; using the final training data set to train a PyTorch-based Y0L0v3 target detection model, and then using the verification data set to verify the PyTorch-based Y0L0v3 target detection model; and performing target detection on the test set by using the verified PyTorch-based Y0L0v3 target detection model. According to the invention, the problem that the existing low-light image enhancement method is limited in improving the target detection performance is solved, the edge target detection based on FPGA MPSoC deployment is realized, and the detection efficiency of the edge end is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and image processing, and in particular relates to a target detection method based on FPGA in a low-light scene. Background Art

[0002] Object detection is a basic task and research frontier of computer vision, and is an important foundation for other visual tasks such as instance segmentation and object tracking. However, in some complex low-light scenes, the accuracy of object detection methods is often affected by image quality issues, so many practical applications based on object detection have encountered difficulties. For example, due to the photosensitivity of the sensor, images captured in low-light scenes usually contain a lot of noise. Therefore, the problem of object detection in low-light scenes has important practical significance.

[0003] The low-light object detection problem can be divided into two sub-problems: low-light image enhancement and general object detection, where low-light image enhancement is used to recover high-quality normal-light images from low-quality low-light images, which is a key step in performing downstream object detection tasks. Traditional low-light image enhancement methods mainly start from basic image signal processing theories and methods, and manually design filters or grayscale mapping methods for image filtering or light estimation to obtain high-quality normal-light images. However, these methods require a lot of prior knowledge and expert experience for manual tuning, and it is difficult to balance multiple image quality degradation problems in complex scenes. In recent years, low-light image enhancement methods based on deep learning have achieved better enhancement results than traditional methods, because deep neural networks can effectively learn the mapping from low-light to normal-light distribution through end-to-end training on paired low-light / normal-light image datasets. However, paired data is very expensive to collect, and synthetic data cannot adapt to complex real-world scenes. Therefore, how to use unpaired data for training and how to effectively improve the performance of enhanced object detection tasks is a valuable research topic.

[0004] Existing general-purpose object detection algorithms work well in clear images under normal lighting, but the performance of the algorithms is severely affected if complex low-light scenes are not modified. However, most methods are still designed for human vision and focus on improving the quality of human visual perception rather than improving the performance of object detection tasks. In addition, most current applications need to be implemented at the edge, where power consumption and performance are important limitations. Therefore, how to combine flicker image enhancement with object detection algorithms and apply them in MPSoC implementation is a key research topic to solve the problem of object detection in flickering scenes. Summary of the invention

[0005] An object of the present invention is to provide a method for object detection based on FPGA in a low-light scene and to provide at least the advantages to be described below.

[0006] Another object of the present invention is to provide an FPGA-based target detection method in low-light scenes. On the one hand, the low-light data set is expanded, and the YOLOv3 target detection model is trained based on low-order image enhancement technology to improve the detection performance of the model in low-light scenes; on the other hand, the model is deployed on the MPSoC board to realize the edge target detection system and improve the detection efficiency. Finally, verification experiments were carried out on the public low-light target detection data set and the ZU3EG-AXU3EGB MPSoC board. The results show that this method can effectively improve the detection accuracy and efficiency.

[0007] The technical solution of the present invention is as follows:

[0008] The target detection method based on FPGA in low-light scene includes the following steps:

[0009] Obtaining an ExDark dataset and using it as a basic dataset, and selecting a training set, a validation set, and a test set from the basic dataset;

[0010] Acquire a low-light image as an extended data set, and perform contrast adjustment, target magnification, and super-resolution enhancement on the extended data set and the training set to obtain the final training data set;

[0011] Build the Y0L0v3 object detection model based on PyTorch and deploy it on FPGA MPSOC;

[0012] The final training data set is used to train the PyTorch-based Y0L0v3 target detection model and then the validation data set is used to validate the model;

[0013] and performing target detection on the test set using the verified PyTorch-based Y0L0v3 target detection model;

[0014] in;

[0015] The contrast adjustment, target enlargement and super-resolution enhancement processing include:

[0016] The histogram equalization method is used to adjust the contrast;

[0017] The seam carving algorithm is used to multiply the gradient map with the visual saliency map to obtain the final importance energy map of the entire image, calculate the cumulative energy map of the image, and then search for the lowest energy point, return to find the lowest energy line, delete or insert the lowest energy line, and obtain the final target image;

[0018] The ultra-high resolution images are reconstructed using the transformer-based SwinIR network model, which consists of three modules: shallow feature extraction, deep feature extraction, and reconstruction module.

[0019] Preferably, in the FPGA-based target detection method in the low-light scene, the ExDark dataset includes: low-light images without light sources; low-light images surrounded by darkness; low-light images with a single visible light; low-light images with multiple but weaker light sources; low-light images with multiple but stronger light sources; low-light images with a visible screen; low-light images of indoor rooms with bright windows; low-light images of indoor rooms with bright screens; low-light images of indoor rooms; low-light images of targets inspected in shadows; low-light images of targets inspected in the dark; low-light images of targets inspected in a dark room with bright windows.

[0020] Preferably, in the target detection method based on FPGA in a low-light scene,

[0021] The ExDark dataset includes 7363 low-light images;

[0022] The training set contains 3000 images, 250 images per category;

[0023] The validation set contains 1800 images, 150 images per category;

[0024] The test set contains 2568 images.

[0025] Preferably, in the FPGA-based target detection method in the low-light scene, a histogram equalization method is used to adjust the contrast, including:

[0026] Assume that the grayscale distribution of the first image is f, g represents the grayscale distribution of the original image after the HE method. Let the grayscale in the grayscale distribution be r and s, and their corresponding pixel numbers be fr and gs, respectively. Then the probability of grayscale distribution is given by formula (1), and D represents the total grayscale distribution.

[0027]

[0028] Formula (2) is derived from formula (1):

[0029] ∫p f (r)dr=∫p g (s)ds=1 (2)

[0030] In order to make the histogram evenly distributed, the distribution probability of each gray level needs to be equal, and formula (3) is derived:

[0031]

[0032] Substituting equation (3) into equation (2), we get equation (4):

[0033]

[0034] This gives the transformation equation (5) for the HE method, where pf(r) is the distribution of the original map.

[0035] g=(g max -g min ) r (r)+g min (5).

[0036] Preferably, in the target detection method based on FPGA in a low-light scene,

[0037] The seam carving algorithm is used to multiply the gradient map with the visual saliency map to obtain the final importance energy map of the entire image, calculate the cumulative energy map of the image, and then search for the lowest energy point, return to find the lowest energy line, delete or insert the lowest energy line, and obtain the final target image, including:

[0038] Assume that it is an n*m image, and the vertical pixel lines are defined as follows:

[0039]

[0040] Here, x(i) is a mapping: x: [1, 2, 3, ...., n] → [1, 2, 3, ...., m]. From the above formula, we can see that a vertical pixel line is an 8-connected path consisting of pixels from the first row to the last row of the image, with one or only one pixel removed from each row;

[0041] Likewise, a horizontal pixel line is defined as

[0042]

[0043] Where x(j) is a mapping: y: [1, ..., m] → [1, ..., n]. A horizontal pixel line is composed of 8 connected paths from the first column to the last column of the image, and one and only one pixel is deleted from each column;

[0044] Therefore, the path of the vertical pixel line is Then the path of the horizontal pixel line is similar. If a row or column of an image is to be deleted, then processing the image will cause all the pixels below or to the right of that row to shift upwards or left to supplement the deleted pixel line;

[0045] The energy function of each pixel in the image is e, then the total energy of the pixel line is defined as

[0046]

[0047] Find the vertical line of pixels with the lowest total energy:

[0048]

[0049] Horizontal pixel lines are found in a similar way. ;

[0050] For a pixel point (i, j) in image I, remember that the cumulative energy of the point is M(i, j),

[0051]

[0052] Therefore, we only need to traverse the second to the last row in image I and calculate the above equation to get the cumulative energy; the pixel with the smallest cumulative energy in the last row is the end point of the vertical pixel line with the smallest energy;

[0053] Go back from this point, each time looking for the pixel with the minimum cumulative energy in the field, the known minimum cumulative energy point before, and so on, until you find the vertical pixel line with the minimum energy;

[0054] For finding horizontal pixel lines, the method is similar, and the accumulated energy is M(i, j), as shown below:

[0055]

[0056] From the formula of cumulative energy, it can be seen that the minimum value of the last row of M must be the end of the vertical minimum energy line, so the vertical minimum energy line can be found by tracing back from this point;

[0057] The method for finding the horizontal minimum energy line is similar.

[0058] Preferably, in the FPGA-based target detection method in the low-light scene, the ultra-high-resolution image is reconstructed using a transformer-based SwinIR network model, including:

[0059] Given an input of size W×H×C, the Swin transformer first converts the input into (HW / M2)×M2×C and divides it into non-overlapping windows of size M×M, where HW / M2 ​​is the total number of windows;

[0060] For a local window feature X∈RM2×C, the query vector Q, key vector K, and value vector V are calculated as follows:

[0061] Q = XP Q ,K=XP K ,V=XP V (12)

[0062] pq, pk and PV are respectively the learnable matrices of query vector, key vector, value vector and shared weights between windows, which are generally Q, K, V∈RM2×d and calculated through a self-attention mechanism within a local window;

[0063] Note that the matrix is ​​shown in formula (13):

[0064]

[0065] where B is a learnable relative position encoding; d = C / Nhead, Nhead represents the amount of attention from multiple heads; the attention function is then executed h times in parallel and the results of the multi-head self-attention mechanism are spliced ​​together;

[0066] Then use a multi-layer perception with two fully connected layers and use the Gelu nonlinear function between the fully connected layers;

[0067] Before the two modules of multi-head attention and multi-layer perception using residual connections, multi-head attention and multi-layer hierarchical norm layers using residual connections are added. The whole process is expressed by equations (14) and (15):

[0068] X=MSA(LN(X))+X (14)

[0069] X=MLP(LN(X))+X (15).

[0070] The present invention has the following beneficial effects:

[0071] 1. Improve image enhancement algorithm: Optimize the image enhancement algorithm suitable for FPGA implementation so that it can perform well in resource-constrained, power-sensitive and real-time applications. At the same time, it is necessary to comprehensively consider hardware design, algorithm optimization and application requirements to achieve a better FPGA image enhancement solution.

[0072] 2. Real-time performance optimization: Utilize the parallel computing power and hardware accelerators of FPGA to achieve low-latency low-light imaging enhancement. This can be used in applications that require fast response, such as surveillance, military night vision, and medical imaging.

[0073] 3. Detection performance in low-light scenes: Object detection in low-light scenes and related issues, a low-level low-light object detection image enhancement method based on MPSoC. In view of the current problems of insufficient and poor quality low-light image datasets, three low-order image enhancement methods are used to expand the dataset and improve the detection performance of Y0LO in low-light scenes.

[0074] 4. Improve edge detection efficiency: To address the problem that existing low-light image enhancement methods have limited improvement in target detection performance, an edge target detection system based on MPSoC deployment has been implemented, which effectively improves edge detection efficiency.

[0075] Other advantages, objectives and features of the present invention will be embodied in part through the following description, and in part will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 The overall research framework of the FPGA-based target detection method in low-light scenes provided by the present invention;

[0077] Figure 2 A low-cost image processing technology for the FPGA-based target detection method in low-light scenes provided by the present invention - image expansion processing diagram;

[0078] Figure 3 The implementation process of model deployment on MPSoC of the FPGA-based target detection method in low-light scenes provided by the present invention;

[0079] Figure 4 This is an object detection process based on MPSoC in the FPGA-based target detection method in a low-light scene provided by the present invention. DETAILED DESCRIPTION

[0080] The present invention is further described in detail below in conjunction with embodiments so that those skilled in the art can implement the invention with reference to the description.

[0081] It should be understood that terms such as “having”, “including” and “comprising” used herein do not specify the existence or addition of one or more other elements or combinations thereof.

[0082] Object detection, as the most basic and challenging task in computer vision, has broad application prospects. However, in low-light scenes, image quality problems such as low brightness, low contrast, and high noise cause the performance of object detection to degrade significantly.

[0083] Based on this, we explored the target detection algorithm in low-light scenes, and proposed a low-level image enhancement method for low-light target detection based on the MPSoC method. For low-light image enhancement, we used three low-order image enhancement techniques to improve image quality, enhance existing low-light image data and actual data acquisition data, expand the training data set, and strengthen the training foundation of downstream target detection models; for general target detection, we trained the Y0L0v3 target detection model based on the above data set and deployed it on FPGA MPSOC to achieve high-performance detection. The method proposed in this paper organically combines low-light image enhancement and target detection models. The overall research framework is as follows: Figure 1 shown.

[0084] 1. Three low-level data augmentation techniques are used to enhance ExDark and the training dataset is expanded using actual low-light image data.

[0085] 2. The PyTorch-based YOLOv3 target detection model is deployed on FPGA MPSoC to improve the target detection performance.

[0086] 3. The low-light image enhancement model and the target detection model are combined to solve the target detection problem in low-light scenes.

[0087] 1. In practice, the quality of low-light images taken under poor lighting conditions is usually very poor. Traditional methods mainly include two categories: histogram-based methods and correction-based methods. The histogram-based method, namely adaptive histogram equalization (AHE), can map the histogram of the entire image to a new distribution pixel by pixel. The purpose of reflectance theory is to separate illumination and reflectance, and reflectance-based methods usually use illumination maps to improve image quality. However, due to the large noise in flickering images, the performance of traditional methods is poor. Unlike methods that aim to improve image quality for better human visual perception, the present invention adopts flickering image enhancement for edge computing scenarios, aiming to improve the performance of downstream visual tasks. Object detection performance is used to quantitatively evaluate the effectiveness of our method, rather than from the perspective of perceptual quality.

[0088] 2. In terms of hardware implementation, although GPUs have been proven to provide extremely high throughput and are widely used for hardware acceleration of DNNs, they are generally not suitable for energy and power-constrained applications such as edge computing scenarios due to their high power consumption. FPGAs allow us to implement irregular parallelism, custom data types, and application-specific hardware architectures, providing great flexibility to adapt to new models with higher sparsity and compact network structures.

[0089] Operation process

[0090] 1. Low-light image training dataset

[0091] The present invention selects the public dataset ExDark consisting of all low-light images with object-level annotations as the basic dataset. The ExDark dataset contains 7363 low-light images, including 12 different target classes. Among them, the training set contains 3000 images, 250 images per category, the validation set contains 1800 images, 150 images per category, and the test set contains 2568 images. Most of the images in the dataset are downloaded from search engines, and a few images are selected from the above-mentioned large public datasets (Microsoft C0C0, etc.). The resolution and aspect ratio of these images vary greatly, and the quality of the captured images is variable. In addition to the target type, ExDark also includes information on 10 different low-light scene categories, ranging from very low light to complex low light. It is worth noting that the dataset does not contain positive exposure images paired with low-light images, which makes it difficult to apply supervised low-light image enhancement algorithms.

[0092] The ExDark dataset contains a variety of scenes, including common low-light scene types: (a) very low-light images: (b) low-light images with no light source: (c) low-light images surrounded by darkness: (d) low-light images with a single visible light: (e) low-light images with multiple, but weaker light sources; (f) low-light images with multiple but stronger light sources; (g) low-light images with a visible screen: (h) low-light images of indoor rooms with bright windows: (i) low-light images of indoor rooms with bright screens: (j) low-light images of indoor rooms: (k) low-light images of targets being inspected in the shadow: (1) low-light images of targets being inspected in the dark: and (m) low-light images of targets being inspected in a dark room with bright windows. The complexity of the scene types contained in this dataset greatly increases the difficulty of recognition and detection, and it is difficult for unimproved general object detection methods to work well enough on this dataset. At the same time, the ExDark training dataset is too small, which affects the performance of the model. Therefore, the ExDark dataset needs to be expanded to improve the generalization ability of the model.

[0093] 2. Low-level image enhancement

[0094] Most of the currently available low-light image datasets are from the field of image enhancement, where images do not have detection annotation information and their quantity is difficult to support the training of flicker image detection models. To enhance the available dataset, we will expand the dataset by collecting 5,000 low-light images from the real world and generate more low-light images by combining low-order image enhancement techniques. Three low-cost image enhancement methods will be used to process ExDark and real-world images, namely contrast adjustment, enlarged objects, and super-resolution.

[0095] Contrast adjustment: This paper uses the histogram equalization method (HE) to adjust the contrast of the image. This method makes the grayscale distribution in the histogram more uniform, thereby expanding the dynamic range and improving the contrast of the entire image.

[0096] Assume that the grayscale distribution of the first image is f, g represents the grayscale distribution of the original image after the HE method. Let the grayscale in the grayscale distribution be r and s, and their corresponding pixel numbers be fr and gs, respectively. Then the probability of grayscale distribution is given by formula (1), and D represents the total grayscale distribution.

[0097]

[0098] Formula (2) is derived from formula (1):

[0099] ∫p f (r)dr=∫p g (s)ds=1 (2)

[0100] In order to make the histogram evenly distributed, the distribution probability of each gray level needs to be equal, and formula (3) is derived:

[0101]

[0102] Substituting equation (3) into equation (2), we get equation (4):

[0103]

[0104] This gives the transformation equation (5) for the HE method, where pf(r) is the distribution of the original map.

[0105] g=(g max -g min ) r (r)+g min (5)

[0106] Zooming in on the subject: To better extract subject features in low-light images, we use an improved seam carving algorithm to zoom in on the subject. The main feature of the seam carving algorithm is that it takes into account the importance of image content, defines the importance of different scenes in the image through an energy function, and then, by continuously deleting or copying the pixel lines with the lowest energy, while maintaining the original appearance of the visual subject to the greatest extent. The original appearance of the visual subject is also completed with an unequal height ratio zoom.

[0107] Assume that it is an n*m image, and the vertical pixel lines are defined as follows:

[0108]

[0109] Here, x(i) is the mapping: x:[1,2,3,....,n]→[1,2,3,....,m]. From the above formula, we can see that the vertical pixel line is an 8-connected path consisting of pixels from the first row to the last row of the image, with one or only one pixel removed from each row. Similarly, the horizontal pixel line is defined as

[0110]

[0111] Where x(j) is the mapping: y: [1, ..., m] → [1, ..., n]. The horizontal pixel line is composed of 8 connected paths from the first column to the last column of the image, and one and only one pixel is deleted from each column. Therefore, the path of the vertical pixel line is A similar situation applies to the path of horizontal lines of pixels. If a row or column of an image is deleted, processing the image causes all the pixels below or to the right of that row to shift upwards or leftwards to complement the deleted line of pixels.

[0112] Line clipping has limited effect on the visual appearance of the image; it only affects the area around the line being removed or added, and has no effect on the rest of the pixels in the image. Therefore, finding the right line of pixels is crucial.

[0113] That is, the energy function of each pixel in the image is e, and the total energy of the pixel line is defined as

[0114]

[0115] Find the vertical line of pixels with the lowest total energy:

[0116]

[0117] Horizontal pixel lines are also found in a similar way. The problem of finding the lowest energy pixel line is solved using the dynamic programming method. For a pixel point (i, j) in image I, remember that the cumulative energy of the point is M(i, j), we have

[0118]

[0119] Therefore, we only need to traverse the second to the last row in image I and calculate the above equation to get the cumulative energy. The pixel with the minimum cumulative energy in the last row is the end point of the vertical pixel line with the minimum energy. Then, we go back from this point, each time looking for the minimum cumulative energy point before the pixel with the minimum cumulative energy in the field, and so on, until we find the vertical pixel line with the minimum energy.

[0120] For finding horizontal pixel lines, the method is similar, and the accumulated energy is M(i, j), as shown below:

[0121]

[0122] From the formula of cumulative energy, we can see that the minimum value of the last row of M must be the end of the vertical minimum energy line, so we can find the vertical minimum energy line by tracing back from this point. The method for finding the horizontal minimum energy line is similar.

[0123] In order to make the pixel importance calculation more reasonable, the improvement process of the original algorithm is: multiply the gradient map with the visual saliency map to obtain the final importance energy map of the entire image, calculate the cumulative energy map of the image, then search for the lowest energy point, return to find the lowest energy line, delete or insert the lowest energy line, and get the final target image. The improved algorithm can better maintain the subject effect, especially for the visual subject under dark lighting conditions.

[0124] Super-resolution: This paper uses the transformer-based SwinIR network model to reconstruct super-resolution images. The model consists of three modules: shallow feature extraction, deep feature extraction, and reconstruction module. The shallow feature extraction module uses convolutional layers to extract shallow features and directly passes them to the reconstruction module to retain low-frequency information. The deep feature extraction module mainly consists of residual Swin transformer blocks (RSTBs) containing Swin transformers. Each RSTB uses multiple Swin transformer layers for local attention and cross-window interaction. In addition, SwinIR adds a convolutional layer at the end of the block for enhanced feature enhancement and uses residual links to provide a shortcut for feature aggregation. Finally, shallow and deep features are fused in the reconstruction module to achieve high-quality image reconstruction. The Swin transformer layer is based on the standard multi-head self-attention of the original transformer layer. The main difference lies in the self-attention and transfer window mechanism. Given an input of size W×H×C, the Swin transformer first transforms the input into (HW / M2)×M2×C and divides it into a non-overlapping window of size M×M, where HW / M2 ​​is the total number of windows.

[0125] The size of each window must be calculated separately. For a local window feature X∈R M2×C , the query vector Q, key vector K, and value vector V are calculated as follows:

[0126] Q = XP Q ,K=XP K ,V=XP V (12)

[0127] pq, pk and PV are respectively the learnable matrices of query vector, key vector, value vector and shared weights between windows, generally Q, K, V ∈ R M2×d The calculation is performed through a self-attention mechanism within a local window, as shown in the following formula. The attention matrix is ​​shown in formula (13):

[0128]

[0129] Where B is a learnable relative position code. d = C / N h ead,N h ead represents the amount of attention from multiple heads. We then execute the attention function h times in parallel and concatenate the results of the multi-head self-attention mechanism together.

[0130] Then a multi-layer perception with two fully connected layers is used, and the Gelu nonlinear function is used between the fully connected layers, which increases the nonlinear expressiveness of the model. In addition, before the two modules of multi-head attention and multi-layer perception using residual connection, multi-head attention and multi-layer hierarchical norm layers using residual connection are added. The whole process is expressed by equations (14) and (15):

[0131] X=MSA(LN(X))+X (14)

[0132] X=MLP(LN(X))+X (15)

[0133] In summary, we augmented the images using three low-level image processing techniques with the goal of expanding the training dataset to improve the performance of high-level vision models. Figure 2 Some processed images are shown.

[0134] 3. Implementing model deployment on MPSoC

[0135] In this section, we use the Vitis-AI development environment to implement the deployment of the target detection model on the MPSoC. The implementation process is as follows: Figure 3 As shown. The Vitis-AI development environment is mainly composed of the Vitis-AI Development Kit (DNNDK), which is used for AI reasoning on the Xilinx ZYNQ series hardware platform, including an optimized DPU IP core, a tool library, a runtime library, and a deep neural network (DNN) model library. DNNDK consists of an optimized DPU IP core, a tool library, a runtime library, and a DNN model library. Vitis-AI provides legal tools to call the DPU IP core on the PL side, which enables the development of the TensorFolw framework and the Caffe framework based on the C++ language on the object board. In the present invention, the object establishes a recognition model based on the DNN of the tensor flow, and the object board model is ZU3EG-AXU3EGB.

[0136] In the above process, DNN parameter quantization calibration and DPU kernel compilation are the key to realize the deployment of the model to MPSoC. Quantization calibration refers to the process of converting the parameters of the model from high-precision quantities to low-precision quantities, thereby reducing the storage space and memory occupation of the model. The present invention quantizes and calibrates the target detection model to speed up the calculation speed, reduce the power consumption of the device, and adapt it to the deployment of MPSoC. Once the neural network is trained in the mainstream framework, the network model can be exported as a .pb file containing the model parameter information. Before the DPU imports this parameter information, it must first be quantized, that is, the fixed point. The fixed-point process requires the input of an unlabeled test set. Post-quantization calibration requires test data to restore or optimize the accuracy degradation caused by information loss during the quantization process and improve the generalization of the model. Once quantized and the Pb model is obtained, the DPU kernel compilation stage will be carried out. The DPU kernel is compiled using the Vitis AI compiler. During the compilation process, the VitisAI compiler converts the preprocessed deep learning model IR into binary code and stores it in a .elf file. The DPU can load and execute this file to perform efficient computing and reasoning tasks.

[0137] After obtaining the .elf file, the next step is to cross-compile it to generate a dynamic link library .so file. It contains the Y0L0v3 kernel and only requires a Python application driver to perform Y0L0v3 extrapolation on the target board. The 64-bit ARM GOC cross-compilation toolchain in the Xilinx Runtime Documenter is used to generate the dynamic link library file. Then it is to set up the target board to start and initialize the Petalimux system image and configure the runtime environment. At this point, the entire process of deploying the target instrumentation type to the MPSoC implementation is in place.

[0138] The ARM side of the target board uses the DNNDK API (Python) to input the pre-processed image into the DPU, and obtains the data after performing the DPU convolution operation through the API. Figure 4 The flow of the object board for edge computing tasks is shown. The compiled dynamic link library. So file is placed in the same directory as the application, the DNNDK API is used to parse the .so file, the DPU references the library file to run in a predetermined manner, and then the API is used to retrieve data from the memory allocated by the DPU. Finally, the coordinates and category information of the object obtained after processing are displayed.

[0139] The present invention solves the problem that existing low-light image enhancement methods have limited improvement in target detection performance, realizes edge target detection based on MPSoC deployment, and effectively improves the detection efficiency at the edge.

[0140] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to specific details.

Claims

1. The FPGA-based target detection method in low-light scenes is characterized by: The following steps are involved: Obtaining an ExDark dataset and using it as a basic dataset, and selecting a training set, a validation set, and a test set from the basic dataset; Acquire a low-light image as an extended data set, and perform contrast adjustment, target magnification, and super-resolution enhancement on the extended data set and the training set to obtain the final training data set; Build the Y0L0v3 object detection model based on PyTorch and deploy it on FPGA MPSOC; The final training data set is used to train the PyTorch-based Y0L0v3 target detection model and then the validation data set is used to validate the model; and performing target detection on the test set using the verified PyTorch-based Y0L0v3 target detection model; in; The contrast adjustment, target enlargement and super-resolution enhancement processing include: The histogram equalization method is used to adjust the contrast; The seam carving algorithm is used to multiply the gradient map with the visual saliency map to obtain the final importance energy map of the entire image, calculate the cumulative energy map of the image, and then search for the lowest energy point, return to find the lowest energy line, delete or insert the lowest energy line, and obtain the final target image; The ultra-high resolution images are reconstructed using the transformer-based SwinIR network model, which consists of three modules: shallow feature extraction, deep feature extraction, and reconstruction module.

2. The FPGA-based target detection method in a low-light scene as claimed in claim 1, characterized in that: The ExDark dataset includes: low-light images with no light source; low-light images surrounded by darkness; low-light images with a single visible light; low-light images with multiple, but weaker light sources; low-light images with multiple, but stronger light sources; low-light images with a visible screen; low-light images of indoor rooms with bright windows; low-light images of indoor rooms with bright screens; low-light images of indoor rooms; low-light images of targets being inspected in shadows; low-light images of targets being inspected in the dark; low-light images of targets being inspected in a dark room with bright windows.

3. The FPGA-based target detection method in a low-light scene as claimed in claim 2, characterized in that: The ExDark dataset includes 7363 low-light images; The training set contains 3000 images, 250 images per category; The validation set contains 1800 images, 150 images per category; The test set contains 2568 images.

4. The FPGA-based target detection method in a low-light scene as claimed in claim 3, characterized in that: The contrast is adjusted using histogram equalization, including: Assume that the grayscale distribution of the first image is f, g represents the grayscale distribution of the original image after the HE method. Let the grayscale in the grayscale distribution be r and s, and their corresponding pixel numbers be fr and gs, respectively. Then the probability of grayscale distribution is given by formula (1), and D represents the total grayscale distribution. Formula (2) is derived from formula (1): ∫p f (r)dr=∫p g (s)ds=1 (2) In order to make the histogram evenly distributed, the distribution probability of each gray level needs to be equal, and formula (3) is derived: Substituting equation (3) into equation (2), we get equation (4): This gives the transformation equation (5) for the HE method, where pf(r) is the distribution of the original map. g=(g max -g min )p r (r)+g min (5)。 5. The FPGA-based target detection method in a low-light scene as claimed in claim 4, characterized in that: The seam carving algorithm is used to multiply the gradient map with the visual saliency map to obtain the final importance energy map of the entire image, calculate the cumulative energy map of the image, and then search for the lowest energy point, return to find the lowest energy line, delete or insert the lowest energy line, and obtain the final target image, including: Assume that it is an n*m image, and the vertical pixel lines are defined as follows: Here, x(i) is a mapping: x: [1, 2, 3, ...., n] → [1, 2, 3, ...., m]. From the above formula, we can see that a vertical pixel line is an 8-connected path consisting of pixels from the first row to the last row of the image, with one or only one pixel removed from each row; Likewise, a horizontal pixel line is defined as Where x(j) is a mapping: y: [1, ..., m] → [1, ..., n]. A horizontal pixel line is composed of 8 connected paths from the first column to the last column of the image, and one and only one pixel is deleted from each column; Therefore, the path of the vertical pixel line is Then the path of the horizontal pixel line is similar. If a row or column of an image is to be deleted, then processing the image will cause all the pixels below or to the right of that row to shift upwards or left to supplement the deleted pixel line; The energy function of each pixel in the image is e, then the total energy of the pixel line is defined as Find the vertical line of pixels with the lowest total energy: Horizontal pixel lines are found in a similar way. ; For a pixel point (i, j) in image I, remember that the cumulative energy of the point is M(i, j), Therefore, we only need to traverse the second to the last row in image I and calculate the above equation to get the cumulative energy; the pixel with the smallest cumulative energy in the last row is the end point of the vertical pixel line with the smallest energy; Go back from this point, each time looking for the pixel with the minimum cumulative energy in the field, the known minimum cumulative energy point before, and so on, until you find the vertical pixel line with the minimum energy; For finding horizontal pixel lines, the method is similar, and the accumulated energy is M(i, j), as shown below: From the formula of cumulative energy, it can be seen that the minimum value of the last row of M must be the end of the vertical minimum energy line, so the vertical minimum energy line can be found by tracing back from this point; The method for finding the horizontal minimum energy line is similar.

6. The method for target detection based on FPGA in a low-light scene as claimed in claim 5, characterized in that: The ultra-high resolution image is reconstructed using the transformer-based SwinIR network model, including: Given an input of size W×H×C, the Swin transformer first converts the input into (HW / M2)×M2×C and divides it into non-overlapping windows of size M×M, where HW / M2 ​​is the total number of windows; For a local window feature X∈RM2×C, the query vector Q, key vector K, and value vector V are calculated as follows: Q= XP Q ,K=XPK,V=XP V (12) pq, pk and PV are respectively the learnable matrices of query vector, key vector, value vector and shared weights between windows, which are generally Q, K, V∈RM2×d and calculated through a self-attention mechanism within a local window; Note that the matrix is ​​shown in formula (13): where B is a learnable relative position encoding; d = C / Nhead, Nhead represents the amount of attention from multiple heads; the attention function is then executed h times in parallel and the results of the multi-head self-attention mechanism are spliced ​​together; Then use a multi-layer perception with two fully connected layers and use the Gelu nonlinear function between the fully connected layers; Before the two modules of multi-head attention and multi-layer perception using residual connections, multi-head attention and multi-layer hierarchical norm layers using residual connections are added. The whole process is expressed by equations (14) and (15): X=MSA(LN(X))+X (14) X=MLP(LN(X))+X (15).

Images