A method and system for multi-channel image acquisition and processing based on embedded systems

Abstract

This invention relates to an embedded multi-channel image acquisition and processing method. It acquires multiple image data streams, stitches them together to form a wide-angle image, and after calibration, extracts image feature points using a feature detection algorithm. The stitched image undergoes image fusion processing. When searching for the optimal stitching line, the texture feature changes around a pixel are used as a judgment factor. Based on the stitched wide-angle image, an improved YOLOv3 target detection algorithm is employed to achieve finer-grained target detection. The method provided by this invention, through a joint algorithm, first obtains the type and location information of all targets in the initial frame using the improved YOLOv3 algorithm for target detection. After selecting the target bounding box, the tracking template is initialized using the improved KCF algorithm for continuous tracking.

Description

Technical Field

[0001] This invention belongs to the field of image acquisition and processing technology, specifically relating to an embedded multi-channel image acquisition and processing method and system. Background Technology

[0002] With the improvement of computing power, target detection and target tracking are widely used in various scenarios, such as autonomous driving, smart factories, intelligent video surveillance, sports video analysis, and launch vehicle tracking and separation of satellites. However, despite the widespread attention paid to the field of target detection and tracking, there are still many problems in the widespread application of target tracking systems in industry.

[0003] The most widely used target detection algorithm in industry is the YOLO series, which has a fast processing speed and high average detection accuracy and cross-union ratio. However, the YOLO series is poor at processing small targets and often misses detections.

[0004] Real-time target tracking typically employs correlation filtering-based target tracking algorithms, which offer relatively fast processing speeds. Kernel correlation filtering (KCF) is the most widely used algorithm. However, KCF has several drawbacks: 1. Fixed scale, resulting in low tracking success rate when the target scale changes; 2. Difficulty in handling occlusion and complex scenes; 3. Low success rate over long periods. For image sources, using a single camera for target tracking results in a limited field of view, making it prone to target occlusion and out-of-view issues, leading to tracking failures. Replacing a single camera with multiple cameras represents a technological development direction for target detection and tracking. However, multi-camera tracking is susceptible to issues such as asynchronous acquisition and inaccurate consistency judgment, also causing tracking failures, which urgently need to be addressed. Summary of the Invention

[0005] To address the shortcomings of existing methods, the present invention aims to provide an embedded multi-channel image acquisition and processing method and system that can simultaneously acquire multiple images and perform image stitching, and perform target detection and target tracking based on the stitched image. First, it improves the detection capability of the YOLO v3 algorithm for small and occluded targets; second, it improves the tracking capability of the KCF algorithm in situations with scale changes, occlusion, and complex scenes; and third, it improves the long-term tracking capability.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: An embedded multi-channel image acquisition and processing method includes the following steps: Acquire multiple image data streams and stitch them together to form a wide-angle image, specifically including... After calibration, the multi-channel image data is stitched together using the following steps: Feature detection algorithms are used to extract image feature points, overlapping regions are segmented, mismatched points are removed, and homography matrix is ​​calculated to achieve feature point registration. After stitching, the images are fused together. A rectangular region is determined by the optimal stitching line algorithm. Its left and right boundaries are used as the boundary values ​​of the stitching region and recorded. In the overlapping region, the two sides of the stitching region are used as the transition region. When searching for the optimal seam line, the texture feature changes around a pixel are used as a criterion. After feature point detection, matching, and purification, the feature points corresponding to the overlapping areas have the highest correlation. The presence of moving objects in the overlapping images leads to differences in the texture features of surrounding pixels. The formula for calculating the texture difference energy function is as follows: in, and for The average gray level of the surrounding area.

[0007] Increase texture difference factor E texture Then, the energy function is redefined as follows: in, Indicates the color difference in the overlapping areas. Indicating gradient differences in overlapping regions, the search for the optimal stitching line incorporates a texture difference factor in addition to color and structural difference intensities. Following the dynamic programming search method described above, the optimal stitching line is found. This approach effectively eliminates ghosting in overlapping areas, ultimately reducing the impact of ghosting on target detection and tracking.

[0008] By recording the boundary values ​​of the suture line ( Figure 6 The two vertical lines in the "stitching area" are used to determine the overlapping part based on the threshold. Figure 6 The image is divided into regions ("overlapping region Ivo"), and then the pixels in different regions are fused according to the set weights. The specific process is as follows: When a point is located in either the left or right image region, the pixel value remains unchanged. When the point is within the suture area, the fusion formula is: ; When the point is in the transition region, if , ,otherwise, Similarly, the calculation rules on the left side are the same; among them, These are the pixel values ​​of the overlapping points. To determine the threshold; Based on the stitched wide-angle image, an improved YOLOv3 object detection algorithm is used to achieve finer-grained object detection, specifically including: Yolo v3 uses the K-means clustering algorithm to determine the aspect ratio of target candidate boxes. The Euclidean distance criterion from the ground truth to the cluster center in the K-means clustering algorithm has been changed to the Intersection over Union (IoU) value of the ground truth boxes (the ratio of the intersection to the union of two bounding boxes, used to calculate the degree of overlap between the two boxes). The calculation method has been modified as follows: This means converting the Euclidean distance into 1 minus the IoU value between the ground truth bounding box and the cluster center.

[0009] Yolo v3 acquires feature maps through multi-scale downsampling, adds a receptive field module (RFB module), and performs dilated convolution through different convolution kernels, so that each scale of detection can obtain multiple different receptive fields, simulating the human visual system, and merging multiple receptive fields.

[0010] Furthermore, it also includes improvements to the KCF algorithm based on the stitched wide-angle image, adding scale adaptation and multi-feature fusion, specifically including: To adapt to large scale changes, a one-dimensional correlation filter is added to estimate the scale. Compared to scale adaptation through scale pooling, adding a one-dimensional filter for scale estimation refines the granularity and expands the scope of scale changes.

[0011] First, let's assume the target size obtained from the initial frame is... If we start extracting training samples from the current frame, and the center of the extracted sample is taken as the coordinates of the target center, then the scale of this sample can be expressed as: Here Let n be the scaling factor and n be the exponential range, as shown below: Here, S represents the number of scales selected. Increasing S can improve the accuracy of positioning. However, as can be seen from the above scale calculation formula, the speed of lossy tracking increases exponentially. This means that the scale estimation is from fine to coarse from the inside out, which can locate the target more accurately.

[0012] The cost function of the one-dimensional filtered tracker for scaling estimation is expressed as follows: In the formula, This represents a one-dimensional Gaussian function, where f is the scale eigenvector, h represents the filter basis samples, and i represents the dimension of the eigenvector. Here are the regularization parameters. The filter needs to be trained using ridge regression to minimize the cost function. The final scale-related filter is calculated as follows: In the above formula This represents the sample input of frame t. Let S represent the scale pool of size S established for frame t, i=1,2…….S (A and B are filter parameters obtained from ridge regression training); the maximum response of the scale filter is obtained by inverse Fourier transform, which is the scale estimate for this frame; To optimize the KCF algorithm, after extracting the FHOG and dimensionality-reduced CN features, they need to be fused to obtain the final kernel correlation filter response. The filter response values ​​of the FHOG and CN features are as follows: In the above formula These represent the HOG and CN features within the target location in the current frame, respectively. These represent the filter template from the previous frame and the features after the corresponding parameter updates, respectively. This represents the value of the filter template parameters from the previous frame after performing a discrete Fourier transform.

[0013] Different features cannot be directly added together. The weights of various features vary depending on the situation. Therefore, a dynamic weighting method is chosen to complete the fusion. The calculation method is as follows: In the above formula The peak-to-sidelobe ratio is determined by both the FHOG and CN characteristics of the filter response. Peak-to-Side-Lobe Ratio (PSR) is an evaluation of the degree of matching between the target and the template; the higher the value, the more reliable the tracking results. The PSR calculation method is as follows: Ideally, the response of a correlation filter should follow a normal distribution. The formula represents the maximum value of the correlation response, and the region surrounding the peak is defined as the sidelobe. u and These represent the mean and standard deviation of the sidelobe regions, respectively. The fused correlation response is obtained, and the peak position of the correlation response is the new location of the target.

[0014] Furthermore, it also includes an improved re-detection strategy in the KCF algorithm based on the stitched wide-angle image, specifically including: A template adaptive update strategy is adopted to enable it to handle such situations.

[0015] The update strategy of the KCF algorithm is shown in the following formula: in A visual model representing the target. Represents filter parameters, This represents the learning rate parameter. Here, we also use the PSR (Physical Sequence Rate) introduced above as a way to adaptively change the update strategy. The update rate parameter is adjusted by using the ratio of the average PSR of historical frames to the PSR of the current frame. An update rate adjustment parameter is defined as follows: here This represents the adjustment coefficient, and a threshold T is set for the update rate adjustment parameter. When the threshold is lower than the threshold, the update rate parameter needs to be adjusted; otherwise, the original parameter can be used. Therefore, the adaptive update rate parameter can be expressed as: Furthermore, the improved KCF target tracking algorithm flow... include First, features are extracted from the tracking region obtained by detection in the initial frame. Then, the system is trained using samples densely sampled by a cyclic matrix to obtain a correlation filter and a scale filter. For subsequent frames, the region surrounding the target in the previous frame is used as a candidate region to calculate its correlation with the correlation filter. The region with the highest correlation is selected as the predicted region. Subsequently, multi-scale traversal is performed, and the scale filter with the largest response is the current target scale, thus determining the target bounding box. If it is not the last frame, the parameters of the correlation filter and the scale filter are updated, and target tracking is performed in the next frame. The algorithm terminates when the current frame is the last frame.

[0016] Furthermore, it also includes criteria for evaluating the degree of matching between the target and the filter template: The matching degree between the target and the filter template is evaluated, and a confidence level is set. When the matching degree is lower than the confidence level, re-detection is adopted. Based on the need for long-term tracking, a threshold M is set for the peak-to-sidelobe ratio. When the peak-to-sidelobe ratio is less than M, it is proven that the filter template has drifted, the target has been lost, training is stopped, and the improved YOLO v3 detector is used to re-detect the target and initialize the tracking model.

[0017] Furthermore, the overall process of the joint algorithm of the improved YOLO v3 target detection algorithm and the improved KCF target tracking algorithm is as follows: include First, the improved YOLO v3 algorithm is used to detect all targets in the initial frame to obtain their types and locations. After selecting the target bounding box, the improved KCF algorithm is used to initialize the tracking template and perform continuous tracking. When the PSR is lower than the threshold, the detection algorithm is used to re-detect the target and match all detected targets with the initial tracking target template. If the match is successful, the tracking template is re-initialized; otherwise, the tracking fails.

[0018] The present invention also provides a readable storage medium comprising a stored program, wherein the program executes the above-described embedded multi-channel image acquisition and processing method when it is run.

[0019] This invention also provides an embedded multi-channel image acquisition and processing system, comprising an embedded development platform and a host computer that communicate with each other. The embedded development platform includes an image acquisition unit, and the host computer includes an image processing unit. The image acquisition unit includes a camera that acquires multiple image data streams; The image processing unit includes a program that executes a multi-channel image acquisition and processing method based on an embedded method at runtime. The program includes an image stitching module for stitching multi-channel image data into a wide-angle image; it also includes an improved YOLO v3 target detection algorithm module for achieving finer-grained target detection.

[0020] Furthermore, the program also includes an improved KCF algorithm module, which includes adding a scale adaptation algorithm module to the original KCF algorithm module to add a one-dimensional correlation filter to complete the scale adaptation estimation; it also adds a multi-feature fusion algorithm module to perform multi-feature fusion, which fuses the FHOG and dimensionality-reduced CN features after extraction to obtain the final kernel correlation filter response, and the peak position of the correlation response is the new position of the target.

[0021] Furthermore, the improved KCF algorithm module adds a re-detection strategy module, which is used for the re-detection strategy, i.e., the template adaptive update strategy, to enhance the long-term target tracking capability. Specifically, the update rate parameter is adjusted by using the ratio of the average PSR of historical frames to the PSR of the current frame. When the PSR is lower than a threshold, re-detection is performed using the detection algorithm, and all detected targets are matched with the initial tracking target template. If a match is successful, the tracking template is re-initialized; otherwise, tracking fails. In this tracking failure, the tracking needs to start from the target detection stage again for the next tracking attempt.

[0022] Compared with the prior art, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: The present invention provides an embedded multi-channel image acquisition and processing method and system, which uses a multi-channel image acquisition card to simultaneously acquire multiple images and ensure good synchronization, thus preparing for image stitching.

[0023] Image stitching is achieved by combining the optimal seam fusion algorithm and the fade-in / fade-out fusion algorithm. Specifically, the search for the optimal seam adds a texture difference factor on the basis of color intensity and structural difference intensity. Then, the optimal seam is found according to the dynamic programming search method, which can better eliminate the "ghosting" in the overlapping area and ultimately reduce the impact of "ghosting" on target detection and tracking. By using stitched wide-angle images for target detection and tracking, the accuracy of detection and tracking is improved. The YOLO v3 algorithm is improved. Specifically, the Euclidean distance criterion from the real target to the cluster center in the K-means clustering algorithm is changed to the IoU value of the real bounding box, reducing the possibility of target bounding box scale errors caused by scale changes and increasing the detection capability of smaller targets. The KCF algorithm is improved by adding a one-dimensional filter for scale estimation instead of scale pooling, which refines the granularity and expands the range of scale changes. After extracting FHOG and dimensionality-reduced CN features, they need to be fused to obtain the final kernel correlation filter response. A template adaptive update strategy is adopted, using PSR as a way to adaptively change the update strategy. The ratio of the average PSR of historical frames to the PSR of the current frame is used to determine the correction of the update rate parameter, and an update rate adjustment parameter is defined. These measures enhance the tracking capability under target scale changes, occlusion, and complex backgrounds. A re-detection mechanism is added. Based on the need for long-term tracking, a threshold M is set for the peak-to-sidelobe ratio. When the peak-to-sidelobe ratio is less than M, it is determined that the filter template has drifted, the target has been lost, training is stopped, and the aforementioned improved YOLO v3 detector is used to re-detect the target, thereby enhancing the long-term target tracking capability.

[0024] In summary, the joint algorithm first obtains the type and location information of all targets in the initial frame through target detection using the improved YOLO v3 algorithm. After selecting the target bounding box, the tracking template is initialized through the improved KCF algorithm and continuous tracking is performed. When the PSR is lower than the threshold, the detection algorithm is used for re-detection, and all detected targets are matched with the initial tracking target template. If the match is successful, the tracking template is re-initialized; if the match fails, the tracking fails. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the system structure of the present invention; Figure 2 This is a flowchart of the image acquisition and processing process of the system of the present invention; Figure 3 This is a schematic diagram of the image frame buffer state switching of the system of the present invention; Figure 4 This is a schematic diagram of the image transmission method of the system of the present invention; Figure 5 This is a schematic diagram of the image transmission process of the system of the present invention; Figure 6 This is a schematic diagram of an improved fade-in / fade-out fusion image stitching method according to the present invention. Figure 7 A schematic diagram illustrating the addition of a one-dimensional (scale) filter to the improved KCF algorithm of this invention for scale estimation; Figure 8 The flowchart shows the improved KCF target tracking algorithm of the present invention. Figure 9 This is a flowchart of the KCF target tracking process with added re-detection in the method of the present invention; Figure 10 This is a flowchart of the software program of the system of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Embodiments of the present invention provide an embedded multi-channel image acquisition and processing method, which is described in conjunction with an embedded multi-channel image acquisition and processing system. The system includes an embedded development platform and a host computer that communicate with each other. The embedded development platform includes an image acquisition unit, and the host computer includes an image processing unit. The image acquisition unit includes a camera that acquires multiple image data streams. The image processing unit includes a readable storage medium containing a stored program that executes an embedded multi-channel image acquisition and processing method. This program includes an image stitching module for stitching the acquired multi-channel image data into a wide-angle image. It also includes an improved YOLOv3 target detection algorithm module for achieving finer-grained target detection. The program further includes an improved KCF algorithm module, which adds a scale adaptation algorithm module to the original KCF algorithm module to add a one-dimensional correlation filter for scale adaptation estimation. It also adds a multi-feature fusion algorithm module for multi-feature fusion, fusing FHOG and dimensionality-reduced CN features after extraction to obtain the final kernel correlation filter response. The peak position of the correlation response is the new position of the target. The improved KCF algorithm module of this program also adds a re-detection strategy module, which is a template adaptive update strategy to enhance long-term target tracking capabilities. Specifically, the update rate parameter is adjusted by using the ratio of the average PSR of historical frames to the PSR of the current frame. When the PSR is below a threshold, re-detection is performed using the detection algorithm, and all detected targets are matched with the initial tracking target template. If a match is successful, the tracking template is re-initialized; otherwise, tracking fails. In this case of tracking failure, the tracking process needs to restart from the target detection stage for the next iteration.

[0028] An embodiment of the present invention provides a multi-channel image acquisition and processing method based on embedded systems, comprising the following steps: Step S1: Acquire multiple image data streams and stitch them together into a wide-angle image, specifically including... To acquire more effective information and reduce blind spots, a four-channel (limited by the acquisition card; one channel can also use the improved algorithm of this invention without image stitching; ideally, two or more channels require image stitching; four channels provide a 360-degree panoramic view of a sphere, at the center of a regular tetrahedron; three or two channels are also possible) PAL-format analog signal fisheye camera (ordinary cameras have insufficient field of view but can still be used) is used to acquire image data. The TW6865 acquisition card decodes the analog signals and encodes them into digital signals. The acquisition card connects to the embedded platform via a PCIe data interface. When the embedded Linux operating system calls the V4L2 framework to request data, the acquisition card transmits the acquired image data to the operating system. After simple preprocessing of the image information, it is transmitted to the host computer via the network, where image stitching is performed. The system structure is as follows: Figure 1As shown. The embedded development platform requires fast image processing speed, so the hardware platform selected is the NXP i.MX6Q processor chip. This processor integrates a media processing engine, multiple image processing units, and a GPU processing unit, providing strong image processing capabilities. Deploying the Linux operating system on the embedded device requires reconstruction and corresponding functional expansion based on different hardware platforms and functional requirements. Image acquisition and preprocessing require calling the OpenCV vision library, while video stream encoding and decoding require FFmpeg, involving numerous cross-compilation issues. To simplify the system debugging process, the Yocto project is used to build the operating system. During system construction, the graphical interface is specified as fsl-img-wayland, the processor as imx6qsabreauto, and the corresponding software development kit is built using fsl-image-qt5. After building the complete operating system, the MFGTool provided by NXP is used to burn the system into the embedded development board's memory. Using the TFTP protocol, the embedded development platform directly loads the embedded system kernel from the host computer via the network after booting from UBoot. Additionally, the host computer's NFS service provides root file system mounting services for the embedded platform. V4L2 is a driver framework in the Linux kernel that shields users from the hardware features of various video capture devices and provides a unified interface. As a video framework, V4L2 does not handle all driver work; its primary goal is to simplify video device control and data reading / writing. This framework is widely used in various embedded devices. In summary, this example uses the aforementioned camera device to acquire and read / write data. While the V4L2 driver framework supports read / write I / O, this method avoids data conflicts but involves frequent copying of image data between user space and kernel space, resulting in frequent kernel mode entries that consume significant time resources, reducing efficiency and hindering real-time data processing. Since this example requires real-time acquisition of continuous video streams, memory mapping (mmap) is used instead of read / write I / O, directly mapping memory regions to the user to avoid data copying and improve data transfer rates. The video acquisition and processing flow is as follows: Figure 2 As shown, the video device is a TW6865 capture card, containing four video inputs to ensure the synchronization of multi-channel video capture. Multiple frame buffers are requested through the V4L2 driver framework API and mapped to user space. Each frame buffer has three states: driver-accessible, video output queue, and application processing. Video cyclic capture is achieved by controlling the frame buffer states (data from the previous cycle is replaced by data from the current cycle). Frame buffer state switching is as follows: Figure 3As shown. Images captured by non-professional camera devices inevitably produce noise, which may lead to the loss of feature points or image distortion. Therefore, filtering processing is required before the image information is transmitted to the host computer. Common filtering methods include Gaussian filtering, mean filtering, and median filtering. Gaussian filtering easily loses image edge information, which may make it difficult to blend image edges. Mean filtering reduces the clarity of the image. Therefore, median filtering is chosen to reduce image noise while avoiding image distortion and loss of feature points. The application that acquires image data obtains the frame buffer in the acquisition output queue and calls the median filtering related interfaces in the OpenCV vision library to implement image noise reduction preprocessing. The video data acquired by embedded devices can usually only be accessed by other devices within the local area network. To break this limitation, a port mapping for the embedded device is provided through a public network server, allowing the host computer to obtain video streams from different local area networks, such as... Figure 4 As shown; port mapping allows a computer on a local area network (LAN) to provide services to the public network. In this case, a remote computer can access a video stream and parse it by accessing a specific port of a public server. This example uses Frp to implement port mapping, achieving network-to-network traversal. Frp, as a high-performance reverse proxy application, supports multiple communication protocols such as TCP, UDP, HTTP, and HTTPS, and can encrypt login to the public server via the SSH protocol. Nginx is an open-source reverse proxy server deployed on an embedded platform. After the embedded platform completes image acquisition and preprocessing, it pushes an RTMP stream to the Nginx server. The RTMP video stream transmission protocol, processing, and pushing process are as follows... Figure 5 As shown; in order to cooperate with multithreading, the V4L2 framework is used to perform multithreaded (specifically four threads) data synchronization processing.

[0029] To ensure that the multiple video streams are synchronized as much as possible after being transmitted over the network, each camera image processing and output requires a separate thread. FFmpeg is used to compress the video frames after median filtering and then push the stream to the Nginx server to achieve cross-LAN video transmission.

[0030] Fisheye lenses differ from ordinary cameras in their imaging model and produce distortion. The calibration method for fisheye lenses includes the following steps: Based on the coordinate information of the distorted image and the real image of the fisheye lens, a fisheye lens distortion model is established. This model needs to consider not only barrel distortion but also eccentric distortion, and then further correct these distortions. Assume the coordinates of a point in the distorted image are... The corrected point coordinates are The relationship between the two is: Radial distortion model: in .

[0031] Eccentric distortion model: Actual imaging model: These are the pixel coordinates of the points after correction.

[0032] in Let be the distortion parameter matrix we are looking for.

[0033] The calibration algorithm steps are implemented as follows: The program implementation process of the fisheye lens calibration algorithm is as follows: Read the chessboard image to determine the number of chessboard squares and the number of corner points in each row and column on the calibration board; Extract detection information from the read images, cache and save the detected corner points on each image, and then perform sub-pixel refinement and save it; Calibration begins. Based on the precise corner points obtained in the previous step, the 3D coordinates of the corner points are initialized. First, it is necessary to assume that the calibration plate is located in the world coordinate system. z On the plane where =0, then start the calibration, and save the camera intrinsic parameters, distortion parameters, rotation and translation vectors obtained from the calibration; The calibration results are evaluated. Using the camera parameters calibrated in the previous step, the 3D points in space are reprojected and recalculated to obtain their corresponding new projection points. The error between the original projection points and the new projection points is calculated. The average error for each image and the sum of the errors for all images are also calculated. Save the calibration results and display the corrected image.

[0034] Refining corner points first involves setting the precise corner point as an unknown, and then selecting points around the coarse corner point based on their distance from the coarse corner point. p Points, add weights, p The greater the distance from a point with a large corner, the smaller its weight. The black and white squares on the chessboard are squares. p The vector sum formed with the precise corner points p The gradient is perpendicular, thus obtaining the intersection of the black and white squares. p Point. Based on the perpendicularity condition, equations can be established, and the precise point can be solved using the least squares method. This process can be iterated until the precise point that meets the accuracy requirements is obtained.

[0035] Step S2: After the multi-channel image data is calibrated, the stitching process is as follows: The calibrated fisheye lens images need to be stitched together. The image stitching algorithm steps are as follows: Image feature point extraction is based on the ORB feature detection algorithm. However, the obtained feature points are prone to mismatches, leading to inaccurate homography matrix calculation. To further refine the obtained feature points, the Random Sample Consensus (RANSAC) algorithm is used. Overlapping regions are segmented to remove mismatched points, improving the efficiency of iteration. The homography matrix is ​​then calculated to achieve feature point registration. The homography matrix is ​​used to describe the spatial relationship between matching points.

[0036] The stitched image exhibits ghosting in the overlapping fields of view of each shot. To eliminate ghosting, image fusion processing is required. A rectangular region is determined using the optimal stitching line algorithm, and its left and right boundaries are used as the boundary values ​​of the stitching region and recorded. Within the overlapping region, the two sides of the stitching region are used as transition regions.

[0037] When searching for the optimal seam line, the texture feature changes around a pixel are used as a criterion. After feature point detection, matching, and purification, the feature points corresponding to the overlapping region have the highest correlation. The presence of moving objects in the overlapping image leads to differences in the texture features of surrounding pixels. Here, "surrounding" refers to the area around the moving object contained within the overlapping region. The formula for calculating the texture difference energy function is as follows: in, and for The mean gray level of the surrounding area, where "surrounding area" represents the neighborhood of the pixel; i and j represent the pixel coordinates. and for The average gray value of the pixel neighborhood (8-neighborhood) in the two grayscale images that need to be stitched together.

[0038] Increase texture difference factor E texture Then, the energy function is redefined as follows: in, Indicates the color difference in the overlapping areas. To represent the gradient difference in the overlapping areas, the search for the optimal stitching line adds a texture difference factor to the existing color intensity and structural difference intensity. The optimal stitching line is found using a dynamic programming search method. Five sets of samples are collected, and the image stitching effects are compared between commonly used multi-band fusion algorithms and the algorithm of this invention. Compared with the multi-band fusion algorithm, the algorithm of this invention has a lower mean square error (MSE), a higher peak signal-to-noise ratio (PSNR), and a lower cross-entropy (CE). In other words, "ghosting in overlapping areas is eliminated better." An improved fade-in / fade-out fusion diagram is shown. Figure 6 .

[0039] By recording the boundary values ​​of the suture line ( Figure 6 The two vertical lines in the "stitching area" are used to determine the overlapping part based on the threshold. Figure 6 The image is divided into regions ("overlapping region Ivo"), and then the pixels in different regions are fused according to the set weights. The specific process is as follows: When a point is located in either the left or right image region, the pixel value remains unchanged. When the point is within the suture area, the fusion formula is: ; When the point is in the transition region, if , ,otherwise, Similarly, the calculation rules on the left side are the same. Among them, These are the pixel values ​​of the overlapping points. To determine the threshold. I r This represents the pixel value in the image on the right, i.e., I. right , Figure 6 There is, I l This represents the pixel value in the left-hand image, i.e., I. left The formula for calculating the grayscale value of pixels in the overlapping region is as follows: like Figure 6 a, A schematic diagram of fade-in / fade-out blending; area A is composed of images. The pixel value of the interior point determines region B, which is determined by the image. The pixel value of the interior point determines the value of the overlapping region of the image, if it is C. The maximum value in region C. Let x be the x-coordinate of any point in the overlapping region of the image, and let C be the smallest value. Then, we can calculate the weight of image A relative to that point: The pixel values ​​of points within the overlapping region C are calculated as follows: and These are the pixel values ​​of points in the two images, respectively.

[0040] Step S3: Based on the stitched wide-angle image, improve the YOLO v3 object detection algorithm to achieve finer-grained object detection, specifically including: Yolo v3 uses the K-means clustering algorithm to determine the aspect ratio of candidate bounding boxes. The algorithm judges their similarity by the Euclidean distance between manually labeled bounding boxes in the prior information of the training set, and finally classifies them into k pre-selected target objects. When there are large-scale targets in the training set, obtaining the pre-selected box proportions using the K-means algorithm can lead to large errors. Therefore, the Euclidean distance criterion from the ground truth to the cluster center in the K-means clustering algorithm is changed to the IoU value (Intersection over Union, the ratio of the intersection to the union of two bounding boxes, used to calculate the degree of overlap between the two boxes). The calculation method is modified as follows: This is equivalent to using a normalization-like method, which converts the Euclidean distance into 1 minus the IoU value between the ground truth box and the cluster center. The smaller this value, the smaller the Euclidean distance between the ground truth box and the cluster center, thereby reducing the possibility of incorrect target box proportions caused by scale changes.

[0041] Yolo v3 acquires feature maps through multi-scale downsampling. In this example, the minimum downsampling factor is 8. When the feature size is smaller than... Sometimes, the corresponding features may not be acquired, leading to missed detections and false detections. The detection accuracy of lightweight networks cannot compare with that of deep CNN networks. Methods to improve accuracy include feature fusion and receptive field enhancement. Therefore, possible improvement strategies include ASPP and receptive field modules. However, this system requires a real-time detector, so while improving accuracy, network speed must also be considered. Compared to ASPP, the receptive field block (RFB) in RFBnet has a much smaller computational cost, which can increase the receptive field of the lightweight network and improve detection accuracy while maintaining speed. Therefore, this method requires the addition of an RFB module. Thus, after multi-scale downsampling to obtain the feature map of the image, a receptive field module (RFB module) is added, through (… Dilated convolutions are performed using different convolution kernels (e.g., attenuated convolutions) to enable detection at each scale to obtain multiple different receptive fields, simulating the human visual system. By merging multiple receptive fields, the detection capability of features is enhanced.

[0042] As a better example, the method of the present invention also includes an improved KCF algorithm based on the stitched wide-angle image, adding scale adaptation and multi-feature fusion to better represent the target and enhance the tracking ability for target scale changes, occlusion, and complex backgrounds, specifically including: To adapt to large scale changes, a one-dimensional correlation filter is added to estimate the scale. Compared to scale adaptation through scale pooling, replacing scale pooling with a one-dimensional filter for scale estimation refines the granularity and expands the scope of scale changes. Figure 7 As shown; compared with the traditional KCF algorithm, the improved KCF algorithm can better represent the color features of objects by adding CN feature extraction and dimensionality reduction; compared with HOG features, FHOG features add orientation-sensitive features while increasing computational efficiency, and have stronger robustness.

[0043] First, let's assume the target size obtained from the initial frame is... If we start extracting training samples from the current frame, and the center of the extracted sample is taken as the coordinates of the target center, then the scale of this sample can be expressed as: Here Here, n is the scaling factor, set to 1.02. The range of n is shown below: Here, S represents the number of scales selected. Increasing S can improve the accuracy of positioning, but it will reduce the tracking speed, so a balance needs to be sought. After testing, the value of S selected is 33. As can be seen from the above scale calculation formula, the scale size grows exponentially, indicating that the scale estimation is from fine to coarse from the inside out, which can accurately locate the target.

[0044] The cost function of the one-dimensional filtered tracker for scaling estimation is expressed as follows: In the formula, This represents a one-dimensional Gaussian function, where f is the scale eigenvector, h represents the filter basis samples, and i represents the dimension of the eigenvector. Here, the regularization parameter is chosen. =0.01; the filter needs to be trained using ridge regression to minimize the cost function. The final calculated scale-related filter results are as follows: In the above formula This represents the sample input of frame t. Let S represent the scale pool of size S established for frame t, i=1,2…….S; the maximum response of the scale filter is obtained by inverse Fourier transform, which is the scale estimate for this frame; To optimize the KCF algorithm, after extracting the FHOG and dimensionality-reduced CN features, they need to be fused to obtain the final kernel correlation filter response. The filter response values ​​of the FHOG and CN features are as follows: In the above formula These represent the HOG and CN features within the target location in the current frame, respectively. These represent the filter template from the previous frame and the features after the corresponding parameter updates, respectively. This represents the value of the filter template parameters from the previous frame after performing a discrete Fourier transform.

[0045] Different features cannot be directly added together. The weights of various features vary depending on the situation. Therefore, a dynamic weighting method is chosen to complete the fusion. The calculation method is as follows: In the above formula The peak-to-sidelobe ratio is determined by both the FHOG and CN characteristics of the filter response. Peak-to-Side-Lobe Ratio (PSR) is an evaluation of the degree of matching between the target and the template; the higher the value, the more reliable the tracking results. The PSR calculation method is as follows: Ideally, the response of a correlation filter should follow a normal distribution. The formula represents the maximum value of the correlation response, and the region surrounding the peak is defined as the sidelobe. u and These represent the mean and standard deviation of the sidelobe regions, respectively. The fused correlation response is obtained, and the peak position of the correlation response is the target's current position, thus achieving tracking.

[0046] As a better example, the method of the present invention also includes an improved re-detection strategy added to the KCF algorithm based on the stitched wide-angle image to enhance long-term target tracking capability, specifically including: The original KCF algorithm updates the appearance model and filter coefficients through interpolation, which means that the update frequency is constant. However, during the tracking process, the target may undergo drastic deformation, rotation, occlusion, or other situations in a short period of time. If the original update strategy is still adopted, the template may drift or even become incorrect, leading to tracking failure. Therefore, an adaptive template update strategy is adopted to cope with such situations.

[0047] The update strategy of the KCF algorithm is shown in the following formula: in A visual model representing the target. Represents filter parameters, This represents the learning rate parameter. Here, we also use the PSR (Physical Sequence Rate) introduced above as a way to adaptively change the update strategy. The update rate parameter is adjusted by using the ratio of the average PSR of historical frames to the PSR of the current frame. An update rate adjustment parameter is defined as follows: here This represents the adjustment coefficient, and a threshold T is set for the update rate adjustment parameter. When the threshold is lower than the threshold, the update rate parameter needs to be adjusted; otherwise, the original parameter can be used. Therefore, the adaptive update rate parameter can be expressed as: .

[0048] As an example of the method of this invention, the improved KCF target tracking algorithm flow is as follows: Figure 8 As shown, the algorithm first extracts features from the tracking region obtained by detection in the initial frame, then trains the system using samples densely sampled by a cyclic matrix to obtain a correlation filter and a scale filter. For subsequent frames, the region surrounding the target in the previous frame is used as a candidate region to calculate its correlation with the correlation filter. The region with the highest correlation is selected as the predicted region. Multi-scale traversal is then performed; the scale filter with the largest response is the current target scale, thus determining the target bounding box. If it is not the last frame, the parameters of the correlation filter and scale filter are updated, and target tracking begins in the next frame. The algorithm terminates when the current frame is the last frame.

[0049] As a better example of the method of the present invention, a standard method for evaluating the degree of matching between the target and the filter template is also included: During long-term tracking, the filter template can shift due to target deformation, partial target occlusion, and motion blur caused by high-speed target movement. Over time, this shift accumulates, leading to template errors. Tracking results are obtained by densely sampling the current candidate region and calculating the response value between the template and the target. If the template is incorrect, target loss will occur, resulting in tracking failure. Therefore, we need to establish a standard. The matching degree between the target and the filter template is evaluated, and a confidence level is set. When the matching degree is lower than the confidence level, re-detection is adopted. Based on the requirements of long-term tracking, a threshold M is set for the peak-to-sidelobe ratio (PSR). When the PSR is less than M, it indicates that the filter template has drifted, the target has been lost, training is stopped, and the improved YOLO v3 detector is used to re-detect the target and initialize the tracking model. Under normal tracking conditions, the PSR value is between 20 and 60; when the PSR value is lower than 7, it means that the target is occluded or lost, therefore M is set to 7.

[0050] As an example of the method of this invention, the overall process of the joint algorithm of the improved YOLO v3 target detection algorithm and the improved KCF target tracking algorithm is described. Figure 9 As shown, include First, the improved YOLO v3 algorithm is used to detect all target types and locations in the initial frame. After selecting the target bounding box, the improved KCF algorithm is used to initialize the tracking template and perform continuous tracking. When the PSR (Target Sequence Rate) falls below a threshold, the detection algorithm is used for re-detection, and all detected targets are matched against the initial tracking template. If a match is successful, the tracking template is re-initialized; otherwise, tracking fails. In this tracking failure, a new target needs to be selected for tracking (if the target has been lost, tracking stops). This algorithm can only improve the success rate of long-term tracking; it does not guarantee long-term tracking. The present invention also provides a readable storage medium, which includes a stored program, wherein the program executes the above-described embedded multi-channel image acquisition and processing method when it runs. Figure 10 As shown.

[0051] This invention also provides an embedded multi-channel image acquisition and processing system, including an embedded development platform and a host computer that communicate with each other. The embedded development platform includes an image acquisition unit, and the host computer includes an image processing unit. The image acquisition unit includes a camera that acquires multiple image data streams. The image processing unit includes the aforementioned program, which includes an image stitching module for stitching multiple acquired image data into a wide-angle image; it also includes an improved YOLO v3 target detection algorithm module for achieving finer-grained target detection.

[0052] The program also includes an improved KCF algorithm module, which adds a scale adaptation algorithm module to the original KCF algorithm module to add a one-dimensional correlation filter to complete the scale adaptation estimation; it also adds a multi-feature fusion algorithm module to perform multi-feature fusion, after extracting FHOG and dimensionality-reduced CN features, and then fuses them to obtain the final kernel correlation filter response. The peak position of the correlation response is the current position of the target, thus realizing tracking.

[0053] The improved KCF algorithm module adds a re-detection strategy module, which is a template adaptive update strategy to enhance the long-term target tracking capability. Specifically, the update rate parameter is adjusted by the ratio of the average PSR of historical frames to the PSR of the current frame. When the PSR is lower than the threshold, the detection algorithm is used to re-detect, and all detected targets are matched with the initial tracking target template. If the match is successful, the tracking template is re-initialized; if the match fails, the tracking fails.

Claims

1. A method for multi-channel image acquisition and processing based on embedded systems, characterized in that... Includes the following steps: Acquire multiple image data streams and stitch them together to form a wide-angle image, specifically including... After calibration, the multi-channel image data is stitched together using the following steps: Feature detection algorithms are used to extract image feature points, overlapping regions are segmented, mismatched points are removed, and homography matrix is ​​calculated to achieve feature point registration. After stitching, the images are fused together. A rectangular region is determined by the optimal stitching line algorithm. Its left and right boundaries are used as the boundary values ​​of the stitching region and recorded. In the overlapping region, the two sides of the stitching region are used as the transition region. When searching for the optimal seam line, the texture feature changes around a pixel are used as a criterion. After feature point detection, the feature points corresponding to the overlapping regions have the highest correlation. Since the pixel texture features of the overlapping images are different, the formula for calculating the texture difference energy function is as follows: ， in, and for The average gray level of the surrounding area; Increase texture difference factor E texture Then, the energy function is redefined as follows: ， in, Indicates the color difference in the overlapping areas. The gradient difference in the overlapping part is represented by the texture difference factor. The search for the best seam line adds the texture difference factor to the color intensity and structural difference intensity. The best seam line is found by using the dynamic programming search method described above. By recording the boundary values ​​of the seam line, the overlapping area is divided into regions according to a threshold, and then the pixels in different regions are fused according to the set weights. The specific process is as follows: When a point is located in either the left or right image region, the pixel value remains unchanged. When the point is within the suture area, the fusion formula is: ; When the point is in the transition region, if , ,otherwise, Similarly, the calculation rules on the left side are the same; among them, These are the pixel values ​​of the overlapping points. To determine the threshold; Based on the stitched wide-angle image, the YOLO v3 object detection algorithm is improved to achieve finer-grained object detection, specifically including: The Yolo v3 object detection algorithm needs to divide the image into equal parts after compressing the image size. The grid is set with S value of s1 by default. To achieve a balance between detection speed and accuracy, S is set to s2, which improves the average detection accuracy while still meeting the real-time requirements of the detection rate. Yolo v3 uses the K-means clustering algorithm to determine the aspect ratio of the target candidate boxes. The Euclidean distance criterion from the ground truth to the cluster center in the K-means clustering algorithm has been changed to the IoU value of the ground truth boxes. The calculation method has been modified as follows: That is, converting the Euclidean distance into 1 minus the IoU value between the ground truth bounding box and the cluster center; Yolo v3 acquires feature maps through multi-scale downsampling, adds a receptive field module, and performs dilated convolution using different convolution kernels, enabling detection at each scale to obtain multiple different receptive fields, simulating the human visual system. It also merges multiple receptive fields to enhance feature detection capabilities.

2. The embedded multi-channel image acquisition and processing method according to claim 1, characterized in that... It also includes improvements to the KCF algorithm based on the stitched wide-angle image, adding scale adaptation and multi-feature fusion, specifically including: To adapt to large scale changes, a one-dimensional correlation filter is added to estimate the scale. Compared to scale adaptation through scale pooling, adding a one-dimensional filter for scale estimation refines the granularity and expands the scope of scale changes. First, let's assume the target size obtained from the initial frame is... If we start extracting training samples from the current frame, and the center of the extracted sample is taken as the coordinates of the target center, then the scale of this sample can be expressed as: ， Here As a scaling factor, the range of n is represented as follows: ， Here, S represents the number of scales selected. Increasing S can improve the accuracy of positioning, but it will reduce the tracking speed. Therefore, a balance needs to be sought. After testing, the value of S is selected as a certain value. As can be seen from the above scale calculation formula, the scale size grows exponentially, which means that the scale estimation is from fine to coarse from the inside out, which can locate the target more accurately. The cost function of the one-dimensional filtered tracker for scaling estimation is expressed as follows: ， In the formula, This represents a one-dimensional Gaussian function, where f is the scale eigenvector, h represents the filter basis samples, and i represents the dimension of the eigenvector; here, the regularization parameter is used. The value is constant; the filter needs to be trained using ridge regression to minimize the cost function; the final calculated scale-related filter results are as follows: ， In the above formula This represents the sample input of frame t. Let S represent the scale pool of size S established for frame t, i=1,2…….S; the maximum response of the scale filter is obtained by inverse Fourier transform, which is the scale estimate for this frame; To optimize the KCF algorithm, after extracting the FHOG and dimensionality-reduced CN features, they need to be fused to obtain the final kernel correlation filter response. The filter response values ​​of the FHOG and CN features are as follows: ， In the above formula These represent the HOG and CN features within the target location in the current frame, respectively. These represent the filter template from the previous frame and the features after the corresponding parameter updates, respectively. This represents the value of the filter template parameters from the previous frame after performing a discrete Fourier transform. Different features cannot be directly added together. The weights of various features vary depending on the situation. Therefore, a dynamic weighting method is chosen to complete the fusion, and the calculation method is as follows: ， In the above formula The peak-to-sidelobe ratio is determined by both the FHOG and CN characteristics of the filter response. ， Peak to Side Lobe Ratio (PSR) is an evaluation of the degree of matching between the target and the template. The higher the value, the more reliable the tracking results. The PSR calculation method is as follows: ， Ideally, the response of a correlation filter should follow a normal distribution. The formula represents the maximum value of the correlation response, and the region surrounding the peak is defined as the sidelobe. u and These represent the mean and standard deviation of the sidelobe region, respectively; the fused correlation response is obtained, and the peak position of the correlation response is the current position of the target.

3. The embedded multi-channel image acquisition and processing method according to claim 2, characterized in that... This also includes improvements to the KCF algorithm based on stitched wide-angle images, such as a re-detection strategy to enhance long-term target tracking capabilities. Specifically, this includes: The original KCF algorithm updates the appearance model and filter coefficients through interpolation, which means that the update frequency is constant. However, during the tracking process, the target may undergo drastic deformation, rotation, or occlusion in a short period of time. If the original update strategy is still adopted, the template may drift or even become incorrect, leading to tracking failure. Therefore, an adaptive template update strategy is adopted here to cope with such situations. The update strategy of the KCF algorithm is shown in the following formula: ， in A visual model representing the target. Represents filter parameters, This represents the learning rate parameter; here, PSR is also used as a way to adaptively change the update strategy. The update rate parameter is adjusted by using the ratio of the average PSR of historical frames to the PSR of the current frame. An update rate adjustment parameter is defined as follows: ， here This represents the adjustment coefficient, and a threshold T is set for the update rate adjustment parameter. When the threshold is lower than the threshold, the update rate parameter needs to be adjusted; otherwise, the original parameter can be used. Therefore, the adaptive update rate parameter can be expressed as: 。 4. The embedded multi-channel image acquisition and processing method according to claim 3, characterized in that... The improved KCF target tracking algorithm flow. include: First, features are extracted from the tracking region obtained by detection in the initial frame. Then, the region is trained using samples densely sampled by a cyclic matrix to obtain a correlation filter and a scale filter. For subsequent frames, the region surrounding the target in the previous frame is used as a candidate region to calculate the correlation with the correlation filter. The region with the highest correlation is selected as the predicted region. Then, multi-scale traversal is performed. The scale filter with the largest response is the current target scale, thus determining the target box. If it is not the last frame, the parameters of the correlation filter and the scale filter are updated, and target tracking is performed in the next frame. When the current frame is the last frame, the algorithm ends.

5. The embedded multi-channel image acquisition and processing method according to claim 4, characterized in that... It also includes criteria for evaluating the degree of matching between the target and the filter template: The matching degree between the target and the filter template is evaluated, and a confidence level is set. When the matching degree is lower than the confidence level, re-detection is adopted. Based on the need for long-term tracking, a threshold M is set for the peak-to-sidelobe ratio. When the peak-to-sidelobe ratio is less than M, it is proven that the filter template has drifted, the target has been lost, training is stopped, and the improved YOLOv3 detector is used to re-detect the target and initialize the tracking model.

6. The embedded multi-channel image acquisition and processing method according to claim 5, characterized in that... The overall process of the joint algorithm of the improved YOLO v3 target detection algorithm and the improved KCF target tracking algorithm. include: First, the improved YOLO v3 algorithm is used to detect all targets and obtain their types and locations in the initial frame. After selecting the target bounding box, the improved KCF algorithm is used to initialize the tracking template and perform continuous tracking. When the PSR is lower than the threshold, the detection algorithm is used to re-detect the target and match all detected targets with the initial tracking target template. If the match is successful, the tracking template is re-initialized; otherwise, the tracking fails.

7. A computer-readable storage medium, characterized in that, The readable storage medium includes a stored program, wherein the program executes the embedded multi-channel image acquisition and processing method as described in any one of claims 1 to 6.

8. An embedded multi-channel image acquisition and processing system, comprising an embedded development platform and a host computer that communicate with each other, wherein the embedded development platform includes an image acquisition unit, and the host computer includes an image processing unit, characterized in that, The image acquisition unit includes a camera that acquires multiple image data streams; The image processing unit includes the program of claim 7, the program including an image stitching module for stitching multiple acquired image data into a wide-angle image; and also includes an improved YOLO v3 target detection algorithm module for achieving finer-grained target detection.

9. The embedded multi-channel image acquisition and processing system according to claim 8, characterized in that... The program also includes an improved KCF algorithm module, which adds a scale adaptation algorithm module to the original KCF algorithm module to add a one-dimensional correlation filter to complete the scale adaptation estimation; it also adds a multi-feature fusion algorithm module to perform multi-feature fusion, which fuses the FHOG and dimensionality-reduced CN features after extraction to obtain the final kernel correlation filter response, and the peak position of the correlation response is the new position of the target.

10. The embedded multi-channel image acquisition and processing system according to claim 9, characterized in that... The improved KCF algorithm module adds a re-detection strategy module, which is a template adaptive update strategy to enhance the long-term target tracking capability. Specifically, the update rate parameter is corrected by the ratio of the average PSR of historical frames to the PSR of the current frame. When the PSR is lower than the threshold, re-detection is performed by the detection algorithm, and all detected targets are matched with the initial tracking target template. If the match is successful, the tracking template is re-initialized; if the match fails, the tracking fails.

Images