Unmanned ship sensing system detection method

Abstract

The application provides an unmanned ship sensing system detection method, comprising obtaining an initial background image according to an initial video image captured by a sensor; comparing a current input frame image captured by the sensor with the initial background image to obtain a moving target or update the initial background image; wherein the obtaining of the initial background image according to the video image captured by the sensor comprises the following steps: obtaining the average value Mean(x, y) and the average difference MD(x, y) of N frames of images in the initial video image; replacing the pixel value of the N+1th frame of image with the pixel value of the Nth frame of image which changes more greatly; and calculating the average value of the N frames of images after replacement, and the average value is the pixel value of the initial background image. The application has the beneficial effect of providing an unmanned ship sensing system detection method with high precision of the obtained initial background image.

Description

Technical Field

[0001] This invention relates to a detection method for an unmanned surface vessel (USV) sensing system. Background Technology

[0002] Unmanned surface vessels (USVs) are another important unmanned platform following drones, unmanned land vehicles, and unmanned underwater vehicles. Compared to drones, unmanned underwater vehicles, mobile robots, and unmanned land vehicles, USVs are widely recognized as a crucial means of conducting surface warfare in future conflicts. Furthermore, USVs possess advantages such as small size, high speed, and intelligence, and their research and development are of great significance for the development and utilization of marine resources, the protection of maritime rights, the assurance of maritime navigation safety, and the enhancement of national influence.

[0003] Therefore, how to detect targets on the water surface is a very important issue. According to existing technology, foreground detection algorithms are generally used to identify regions of interest or moving targets (foreground) and static parts (background) in the video in order to detect moving targets from the monitoring video and thus identify unmanned surface vessels.

[0004] However, because the video footage may contain trembling leaves, flowing water, and other static objects (background), these objects are easily misidentified as moving objects (foreground). This results in a large number of motion ghosting images in the initial background image, leading to low image accuracy and large errors. Summary of the Invention

[0005] The purpose of this invention is to provide a detection method for an unmanned surface vessel (USV) sensing system.

[0006] A detection method for an unmanned surface vessel (USV) perception system, including

[0007] The initial background image is obtained from the initial video image captured by the sensor;

[0008] The current input frame image captured by the sensor is compared with the initial background image to obtain a moving target or to update the initial background image;

[0009] The method for obtaining the initial background image based on the video image captured by the sensor includes the following steps:

[0010] Obtain the average value Mean(x,y) and average difference MD(x,y) of N frames in the initial video image;

[0011] Replace the pixel values ​​with larger changes in the Nth frame image with the pixel values ​​of the (N+1)th frame image. If |I i(x,y)-Mean(x,y)|>μMD(x,y), then;

[0012] Calculate the average value of the N frames after replacement; this average value is the pixel value of the original background image.

[0013] Where i = 0, 1, 2, ..., N; μ = 2.5.

[0014] Furthermore, the formula for calculating the average value Mean(x,y) is as follows:

[0015] Where I represents the pixel value.

[0016] Furthermore, the formula for calculating the average difference MD(x,y) is as follows:

[0017]

[0018] MD(x,y)=D(x,y) / N.

[0019] Furthermore, the average value of the replaced N frames is calculated, which is the pixel value of the initial background image. The formula for calculating the average value of the replaced N frames is as follows:

[0020] Furthermore, comparing the current input frame image captured by the sensor with the initial background image to obtain a moving target or to update the initial background image includes the following steps:

[0021] The initial background image and the current input frame image are sparsely represented:

[0022] The obtained sparse coefficients are measured to obtain the compressed measurement values ​​of the initial background image and the current input frame image: y bn =Φθ bn y tn =Φθ tn Where n represents time, n = 0, 1, 2, 3, ...

[0023] Obtain the compressed measurement value of the background image at time n+1:

[0024]

[0025] Among them, I n B represents the current input frame image. n Represents the background image, y tn It is the current input frame image I n The compression measurement value, y bnBackground image B n The compression measurement value, y bn+1 The background image B at time n+1. n+1 The compressed measurement value represents the model update rate, which is a constant satisfying 0 ≤ 1. Its reciprocal represents the time constant during the decay process, which is generally an empirical value; M represents the number of compressed measurements; i represents the position of the corresponding compressed measurement value; y b0 It is a compressed measurement of the initial background image;

[0026] At position i, if the following condition is met, it indicates that there is a moving target at this position:

[0027] |y tn+1 (i)-y tn (i)|>T n+1 (i)

[0028] Among them, T n+1 (i) is a threshold that is updated in real time, and its update strategy is expressed as follows:

[0029]

[0030] Where a is an integer close to 1, and the threshold T n+1 The size of (i) can be adjusted by changing the value of a; the difference between the measurements of the current input image and the background image is defined as:

[0031] r tn =||y tn (i)-y bn (i)||2

[0032] The difference is determined based on a threshold, and the corresponding measurement value in the difference image is assigned to either the foreground or background. Let the objective function be:

[0033]

[0034] If the difference between the measured values ​​of the current input image and the background image is less than the set threshold, it is determined to be the measured value of the background; otherwise, it is determined to be the measured value of the foreground target.

[0035] The advantage of this application is that it provides a method for detecting unmanned surface vessels using a perception system that obtains an initial background image with high accuracy. Attached Figure Description

[0036] Figure 1 This is the main interface of the system software of the present invention;

[0037] Figure 2 This is the main interface of the parameter setting module of the present invention;

[0038] Figure 3This is a schematic diagram of the target detection result interface of the present invention;

[0039] Figure 4 The image shown is the original image of the input image preprocessing submodule in this invention.

[0040] Figure 5 The image with added salt and pepper noise in this invention;

[0041] Figure 6 The image with Gaussian noise added in this invention;

[0042] Figure 7 This is the image of the median filtering and smoothing result in this invention;

[0043] Figure 8 This is a schematic diagram showing the coordination of the teaching platform structure, the water tank environment structure, and the data acquisition structure in this invention.

[0044] Figure 9 This is a schematic diagram showing the layout of the teaching platform structure, the water tank environment structure, and the data acquisition structure in this invention.

[0045] Figure 10 This is a schematic diagram of the single Gaussian background modeling method in this invention;

[0046] Figure 11 This is a flowchart of the Gaussian mixture background model in this invention;

[0047] Figure 12 This is a schematic diagram of the YOLO3 deep learning model in this invention;

[0048] Figure 13 This is a schematic diagram of the NanoDet model architecture in this invention;

[0049] Figure 14 This is a schematic diagram of the FCOS architecture in this invention;

[0050] Figure 15 This is a schematic diagram of the lightweight detection process in this invention;

[0051] Figure 16 This is a schematic diagram of the separating hyperplane in the SVM algorithm of this invention;

[0052] Figure 17 This is a flowchart of the detection method of the unmanned surface vessel perception system in this invention;

[0053] Figure 18 This is a flowchart of the method for obtaining the initial background image in this invention. Detailed Implementation

[0054] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0055] like Figure 1-18 As shown, an unmanned surface vessel (USV) perception experimental platform system includes a teaching platform structure, a water tank environment structure, and a data acquisition structure. The teaching platform structure includes a server deploying a target detection experimental software system, a display component connected to the server, and an operation component. The water tank environment structure can be used to simulate natural aquatic environments, including but not limited to oceans, lakes, and rivers. The data acquisition structure is used to collect information about the natural aquatic environment simulated by the water tank environment structure and transmit the collected information to the server for processing.

[0056] Specifically, such as Figure 8-11 The illustrated pool environment structure includes a pool body, environmental interference components, and a water surface target. In this embodiment, the pool is made of transparent acrylic material, approximately 3m long, 2m wide, and 0.4m high, with a water volume of approximately 2.4m³. In other embodiments, it can be made of other materials or made to other sizes as needed. The environmental interference components include wave generators and / or mist generators, used to simulate the interference environment such as waves and water mist in a natural water body, thereby verifying and demonstrating the data acquisition capabilities and effects of the data acquisition structure, as well as the processing capabilities and effects of the target detection experimental software system on the acquired data. It can also verify and compare the processing capabilities and effects of subsequently developed processing algorithms. The environmental interference components are located on the four walls or around the pool body. In this embodiment, the mist generator uses an industrial-grade ultrasonic humidifying sprayer, and the wave generator uses an ultra-quiet circulating frequency pump with ten adjustable wave generation modes, including constant flow mode, gap mode, turbulent flow mode, and tidal mode. One or more can be installed depending on the pool area. The surface target can float on the water surface in the pool. In this embodiment, the surface target is simulated by a set of ship models of different sizes.

[0057] Specifically, the data acquisition structure includes a mounting bracket and sensors. In this embodiment, the sensors include a vision sensor and a ranging sensor. The vision sensor is planned to be a Hikvision DS-2TD5136 / DS-2TD5137 series low-power thermal imaging dual-spectrum PTZ camera, and the laser ranging sensor is planned to be a Maxtor L2 laser ranging sensor. The sensors and the mounting bracket are detachably connected, allowing for the rapid addition (or replacement) of target detection sensors (such as binocular vision, ultrasonic, lidar, etc.), facilitating subsequent teaching and research into new target detection methods.

[0058] Specifically, the target detection experimental software system includes a user management module, a parameter setting module, an image processing module, and a video processing module. The user management module is used for setting user account permissions, allowing for different account permissions based on the needs of different users (e.g., ordinary accounts for students, administrator accounts for instructors). The parameter setting module is used to edit the parameters of the system algorithm, such as image translation distance, rotation angle, morphological erosion size, etc. Figure 2 As shown, depending on the target situation of each analysis, the parameters can be adjusted to improve the processing effect of the image or video. At the same time, different processing results can be obtained by adjusting different parameters, which makes it easier for students to learn and understand the impact of different parameters on subsequent processing, and also facilitates subsequent research on unmanned surface vessel detection technology or algorithms.

[0059] Specifically, the image processing module includes a noise simulation submodule, an image preprocessing submodule, an image segmentation submodule, an image smoothing submodule, an image sharpening submodule, an image geometric transformation submodule, an image arithmetic transformation submodule, an image logical operation submodule, a morphology submodule, and a frequency domain analysis submodule.

[0060] In some embodiments, the noise simulation submodule includes salt-and-pepper noise addition and Gaussian noise addition functions, and the image smoothing submodule includes mean filtering, median filtering, and bilateral filtering functions. The noise addition function artificially adds noise signals of a certain intensity to the target image, obfuscating some of the original image information. This not only allows students to understand the image state and characteristics after being subjected to noise interference, facilitating their learning and memorization, but also allows for subsequent filtering of images affected by different noise interferences using different filtering functions. This facilitates students' intuitive understanding and learning of the image state and characteristics after filtering by different noise interferences, making it easier to grasp the characteristics of different filtering functions and enabling students to learn and apply them. Furthermore, in subsequent research on unmanned surface vessel detection technology, algorithms, and other related technologies, it can be used to verify the noise resistance of research results and facilitate comparison with existing filtering functions. The specific principles of the noise addition and filtering functions are existing technologies and will not be elaborated here.

[0061] In some embodiments, the image preprocessing submodule includes functions for converting color images to grayscale, displaying image histograms, histogram equalization, brightness adjustment, contrast adjustment, saturation adjustment, image pseudo-color, image dehazing, image deraining, and Hough line detection.

[0062] Specifically, the function of converting color images to grayscale is existing technology and will not be elaborated here.

[0063] Specifically, in the image histogram display function, the expression for the image's grayscale histogram function is:

[0064] h(k) = n k k = 0, 1, ..., L-1

[0065] Where L is the image gray level, and k is the pixel value P i gray levels, n k Where N is the number of pixels, and N is the total number of pixels in the image.

[0066] Based on histograms, a normalized histogram is further defined as the relative frequency of gray levels, i.e.

[0067] P r (k)=n k / N.

[0068] Specifically, the calculation process of histogram equalization is as follows: First, calculate the gray-level histogram n of the original image. k The second step is to calculate the total number of pixels N in the original image; the third step is to calculate the gray-level distribution frequency P of the original image. r (k); Fourth step, calculate the cumulative gray-level distribution frequency S of the original image. k ;in, The fifth step is to normalize S k Multiply by L-1 and round to ensure that the gray levels of the equalized image are consistent with the original image before normalization; in the sixth step, based on the above mapping relationship and referring to the pixels in the original image, the image after histogram equalization can be obtained.

[0069] Specifically, in the image pseudo-color function, the pseudo-color conversion of a grayscale image requires the following steps:

[0070] 1) Based on the color scheme, convert the scheme's color space to RGB space, and map the color information to the R, G, and B coordinates of the space one by one;

[0071] 2) Design a color scheme and establish an RGB color mapping table to correlate grayscale values ​​with their corresponding R, G, and B color coordinates;

[0072] 3) Read the image, calculate the color information of each point based on the RGB color mapping table, and obtain the color image data;

[0073] 4) Output the image, encapsulate the color image into a standard bitmap format, and save it.

[0074] Specifically, brightness adjustment function, contrast adjustment function, saturation adjustment function, image deraining function, and image dehazing function are all existing technologies and are not the focus of this application, so they will not be described in detail here.

[0075] Specifically, the steps for detecting lines using the Hough transform in the Hough line detection function are as follows:

[0076] 1) Convert a color image to a grayscale image;

[0077] 2) Use Gaussian kernels to denoise and filter out noise information in the image to remove interference;

[0078] 3) Extract image edges using edge operators;

[0079] 4) Binarize the edge grayscale image;

[0080] 5) Map to Hough space. Prepare two containers: one to display the overview of Hough space and one array to store the voting values. Because the voting process often has a maximum value that exceeds the threshold, which can be as high as several thousand, grayscale images cannot be used directly to record voting information.

[0081] 6) Find local maxima, set a threshold, and filter out interfering lines;

[0082] 7) Draw straight lines and mark corner points.

[0083] In some embodiments, the image segmentation submodule includes threshold binarization, region growing segmentation, and watershed algorithm functions; the segmentation process of the threshold binarization function is as follows:

[0084] Where I(u,v) is the gray level of the element at coordinates (u,v), and k is the optimal threshold used for threshold segmentation. The optimal threshold k is calculated using the following formula:

[0085] When obtained When the threshold value is maximized, the threshold value k is the optimal threshold value.

[0086] in, Let P1(k) represent the maximum inter-class variance of pixel regions Z1 and Z2, where Z1 represents a set of pixel regions with gray levels [0,1,2,3,...,k], and Z2 represents a set of pixel regions with gray levels [k+1,...,L-1]. P1(k) represents the probability of pixel region Z1 occurring, and P2(k) represents the probability of pixel region Z2 occurring. P2(k) = 1 - P1(k); and since the grayscale mean of pixel region Z1 is The grayscale mean of pixel region Z2 is Therefore, m is the cumulative mean up to gray level k, expressed as... And m g The average gray level of the entire image is expressed as...

[0087] The above formula is more convenient and faster in actual calculations because m g The calculation only needs to be performed once, and for all possible k, only two parameters, P1(k) and m, need to be calculated. The threshold k belongs to the range [0, L-1], so it can be obtained by continuously iterating. The threshold k that maximizes the threshold value is the optimal threshold. Of course, if the threshold k is not unique, its average value can be used as a substitute for the optimal threshold.

[0088] Specifically, the region growing segmentation function performs the following steps when processing images:

[0089] 1. Scan the image sequentially to find the first unassigned pixel, and let this pixel be (x0, y0);

[0090] 2. Taking (x0,y0) as the center, consider the 4 neighboring pixels (x,y) of (x0,y0). If (x0,y0) satisfies the growth criterion, merge (x,y) and (x0,y0) in the same region, and push (x,y) onto the stack.

[0091] 3. Take a pixel from the stack, treat it as (x0, y0), and return to step 2;

[0092] 4. If the stack is empty, return to step 1;

[0093] 5. Repeat steps 1-4 until every point in the image has a assigned location. The growth process is now complete.

[0094] Specifically, the watershed algorithm processes images as follows:

[0095] 1) Classify all pixels in the gradient image according to their grayscale values ​​and set a geodesic distance threshold.

[0096] 2) Find the pixel with the smallest gray value (marked by default as the lowest gray value point), and let the threshold increase from the minimum value. These points are the starting points.

[0097] 3) As the horizontal plane grows, it will encounter surrounding neighboring pixels. The geodesic distance from these pixels to the starting point (the point with the lowest gray value) is measured. If it is less than a set threshold, these pixels are submerged; otherwise, dams are set on these pixels. This classifies these neighboring pixels.

[0098] 4) As the horizontal plane rises higher, more and higher dams will be set up until the maximum gray value is reached. All areas meet on the watershed line, and these dams partition the entire image pixels.

[0099] Using the above algorithm to perform watershed operations on an image may result in densely packed small regions due to noise points or other interference, meaning the image is divided too finely (over-segmented). This is because there are many local minima in the image, and each point forms its own small region.

[0100] When over-segmentation occurs, the solution is:

[0101] 1) Perform Gaussian smoothing on the image to remove many small minimum values, and these small partitions will be merged.

[0102] 2) Instead of starting from the minimum value, you can use relatively high grayscale pixels as the starting point (which needs to be manually marked by the user) and start flooding from the marked point. This will merge many small areas into one area.

[0103] Specifically, the image sharpening submodule includes gradient sharpening, Roberts operator, Sobel operator, and Laplacian operator; where gradient represents the rate of change of grayscale values, and the gradients of the image at point (x,y) in the x and y directions are shown below:

[0104]

[0105]

[0106] As can be seen from the expression above, the gradient of an image is equivalent to the difference between two adjacent pixels.

[0107] The gradients in the x and y directions can be expressed together as the combined gradient of the image using the following formula:

[0108]

[0109] Specifically, the template for the Roberts operator is shown below:

[0110] Specifically, the template for the Sobel operator is shown below:

[0111] Specifically, the basic process of the Laplace operator includes:

[0112] 1) Determine the gray value of the center pixel of the image and the gray values ​​of other pixels around it. If the gray value of the center pixel is higher, increase the gray value of the center pixel; otherwise, decrease the gray value of the center pixel, thereby achieving image sharpening.

[0113] 2) In the algorithm implementation process, the Laplacian operator calculates the gradients of the four or eight directions of the neighborhood center pixel, and then adds the gradients together to determine the relationship between the gray level of the center pixel and the gray levels of other pixels in the neighborhood.

[0114] 3) Finally, the pixel grayscale is adjusted based on the results of gradient calculation.

[0115] The Laplacian operator is divided into four-neighbor and eight-neighbor operators. The four-neighbor operator calculates the gradient in four directions of the center pixel of the neighborhood, while the eight-neighbor operator calculates the gradient in eight directions.

[0116] The four-neighbor template of the Laplacian operator is shown below:

[0117] The Laplacian operator's eight-neighbor template is shown below:

[0118] Specifically, the image geometric transformation submodule includes image translation, image rotation, image mirroring, and image scaling functions. These functions are all existing technologies and will not be described in detail here.

[0119] Specifically, the image arithmetic transformation submodule includes addition and subtraction functions. Assuming that image x1 and image x2 are added together, and the output image is x, then we have...

[0120] If we subtract the two images, we have

[0121] Specifically, the image logic operation submodule mainly deals with binary images, performing operations between two or more images on a pixel-by-pixel basis. Commonly used logical operations include AND, OR, NOT, and XOR.

[0122] (a) AND operation: Defined as the common part of image A and image B. Convert the binary image to 0 and 1 values; loop through the pixels of both images; perform an AND operation on the pixels and save the result.

[0123] (b) OR operation: Defined as the sum of parts of image A and image B, resulting in a set composed of both images A and B. Similar to the above, but the OR operation is performed on the pixels and the result is retained.

[0124] (c) NOT operation: Defined as the region in the image area excluding image A. The specific operation is to invert the image pixels, that is, set the pixel value of 1 to 0 and the pixel value of 0 to 1.

[0125] (d) XOR operation: Defined as adding a portion of image B to image A, and then removing the overlapping portion. Specifically, if two pixels have the same value, set it to 0; otherwise, set it to 1.

[0126] Specifically, the morphology submodule is mainly used to extract image components from the image that are meaningful for expressing and describing the shape of the region, so that subsequent recognition work can capture the most essential (most discriminative) shape features of the target object, such as boundaries and connected regions. The basic morphological operations of binary images, including dilation, erosion, opening and closing, are existing technologies and will not be elaborated here.

[0127] Specifically, the frequency domain analysis submodule includes Fourier transform spectrum plotting, high / low pass filtering, and homomorphic filtering. In different research fields, the Fourier transform has many different variants, such as continuous Fourier transform and discrete Fourier transform.

[0128] For a two-dimensional image f(x,y) with M rows and N columns, the Fourier transform result can be obtained by first performing a one-dimensional discrete Fourier transform of length N on the row variable y, and then performing a Fourier transform of length M on the column variable x, as shown in the equation:

[0129]

[0130] Decomposing the above equation into two parts, we first obtain F(x,v), and then obtain F(u,v) from F(x,v):

[0131]

[0132]

[0133] A high-pass filter allows high-frequency information to pass through while filtering out low-frequency information; a low-pass filter does the opposite.

[0134] The ideal low-pass filter template is:

[0135] Where D0 represents the passband radius, and D(u, v) is the distance to the center of the spectrum (Euclidean distance), calculated as follows:

[0136]

[0137] M and N represent the size of the spectrum image, and (M / 2, N / 2) is the center of the spectrum.

[0138] An ideal high-pass filter is the opposite of this; it is simply 1 minus the low-pass filter template.

[0139] When applying homomorphic filtering, an image can be viewed as consisting of two parts, i.e., f(x,y) = f i (x,y)f r (x,y)

[0140] Among them, fi This represents the illumination component that varies with spatial location, characterized by slow changes and concentration in the low-frequency part of the image. r This represents the reflection component of a scene as it is reflected to the human eye. It contains a wide range of information about the scene and is rich in high-frequency components.

[0141] The homomorphic filtering process consists of the following 5 basic steps:

[0142] 1) Perform a logarithmic transformation on the original graph to obtain two additive components;

[0143] 2) Perform a Fourier transform on the logarithmic graph to obtain its corresponding frequency domain representation:

[0144] DFT[lnf(x,y)]=DFT[lnf i (x,y)]+DFT[lnf r (x,y)]

[0145] 3) Design a frequency domain filter H(u,v) to perform frequency domain filtering on the logarithmic image;

[0146] 4) Inverse Fourier transform, returning the logarithmic image in the spatial domain;

[0147] 5) Take the exponent to obtain the spatial domain filtering result.

[0148] In some embodiments, the video processing module includes a video stabilization submodule, a video enhancement submodule, a video super-resolution submodule, a background modeling submodule, and an object detection submodule; wherein the video stabilization submodule, video enhancement submodule, and video super-resolution submodule employ existing technologies and will not be described in detail here. The background modeling submodule includes single Gaussian background modeling, Gaussian mixture background modeling, principal component background modeling, and compressed sensing background modeling functions; the object detection submodule includes support vector machine functions, single-stage object detection (YOLO) functions, and anchorless object detection functions.

[0149] Specifically, the single Gaussian background modeling function applies a single Gaussian background model, and the principle of the single Gaussian background modeling method is as follows: Figure 10 As shown, the single Gaussian background modeling method assumes that the pixel value of each pixel in an image changes randomly over time, and that the probability of that pixel value occurring follows a Gaussian distribution. The probability density function of the target pixel is:

[0150]

[0151] Where (x,y) are the coordinates of the target pixel, I(x,y,r) is the pixel value of the target pixel at time t, and μ tLet σ be the mean of the target pixel (x, y) at time t. t Let be the standard deviation of the target pixel (x, y) at time t.

[0152] The specific steps of the single Gaussian background modeling method include:

[0153] Step 1: Initialize the background model; select the first N frames of the target video, and take the mean of their pixel values ​​as the mean of the model. The initial standard deviation is selected to be between 20 and 30; in this embodiment, the initial standard deviation is 25. The initialization is as follows:

[0154]

[0155]

[0156] Step 2: Moving Target Detection; After the background model initialization is complete, the moving target detection phase begins, using the Gaussian distribution principle for foreground detection. The detection formulas for foreground and background are defined as follows:

[0157] |I(x, y, t)-μ t-1 (x, y)|<λσ t-1

[0158] |I(x ， y ， t)-μ t-1 (x, y)|≥λσt-1

[0159] Where I(x, y, r) is the pixel value of pixel z(x, y) at time t, and μ t-1 It is the mean of the background model at time t-1, σ t-1 λ is the standard deviation of the background model at time t-1, and λ is the foreground judgment coefficient, which ranges from 2.5 to 3.0.

[0160] Step 3: Background Model Update; Finally, there is the background model update stage. As time progresses, the background will change somewhat. To accurately detect moving targets in real time, the background model needs to be updated accordingly to respond to these changes. The general principle for updating is: when the current pixel is detected as foreground, its background model remains unchanged; when the current pixel is detected as background, its background model is updated as follows:

[0161] μ t (x, y) = (1-α)μ t-1 (x, y) + αI(x, y, t)

[0162]

[0163] Where α is the background learning rate, and 0 < α < 1.

[0164] As can be seen from the background model update principle described above, only pixels identified as background are updated, while the background model of pixels identified as foreground remains unchanged. The single Gaussian background model is suitable for target detection in single-modal scenes.

[0165] Specifically, the Gaussian mixture background modeling function applies the Gaussian mixture background modeling algorithm. The Gaussian mixture background model is to create multiple single Gaussian background models for each pixel during the background modeling process.

[0166] When processing three-channel video, it is assumed that the r, g, and b color channels of an image pixel are independent and have the same variance. Assume the observation dataset for a pixel X at a certain location in the image is {X1, X2, ..., X...}. t}, X t ={r t g t b t Let} be a sample of a pixel at time t, where rt, gt, and bt are the color values ​​of the r, g, and b channels of the sample pixel, respectively. Then, the probability density function of a single sample point Xt follows a Gaussian mixture distribution:

[0167]

[0168]

[0169]

[0170] Where, at time t, w i,t The weights are the weights of the i-th Gaussian distribution, η(X). t ,μ i,t ,∑ i,t ) is the probability of the i-th Gaussian distribution, μ i , t For its mean, ∑ i,t Its covariance matrix is Let I be the variance, k be the number of Gaussian distributions, and n be the number of pixel channels in the image. For a single-channel image, n is 1. The sum of the weights in the Gaussian mixture background model is 1, which is...

[0171]

[0172] The specific steps of the Gaussian mixture background modeling algorithm include:

[0173] Step 1: Input the video sequence;

[0174] Step 2: Model initialization; Obtain the pixel information of the first N frames of the video sequence to build the model. The model is defined as follows:

[0175]

[0176]

[0177] Foreground detection; after model initialization, at time t, each new pixel sample Xt is compared with the current k Gaussian models one by one, in the following comparison method: |x, -μ i，t-， |≤2.5σ i，t-1 i = 1, 2, ..., k

[0178] If the mathematical relationship between the new pixel sample and any of the Gaussian models satisfies the above formula, then the sample pixel is considered to match it, and the background model corresponding to the sample pixel is updated. If none of the Gaussian models match the new sample pixel, then the pixel value of the sample pixel replaces the mean of the model with the smallest weight among the k Gaussian models, and the new background model retains the weight of the replaced Gaussian model.

[0179] Among the k Gaussian models, the Gaussian model with large weights and small variance is used to describe the background, while the Gaussian model with small weights and large variance is used to describe the foreground. This is because the background occupies a large proportion of the field of view in a video, while the foreground occupies a smaller proportion. Each model is selected based on this. The models are arranged in descending order, with larger weights and smaller standard deviations placed first. If the weights of the first M Gaussian models satisfy formula (2-17), then the first M Gaussian models of that pixel are considered to be used to describe the background, and the remaining Gaussian models are used to describe moving objects in the foreground.

[0180]

[0181] If the value of T is too small, M may be 1, and the model will become a single Gaussian model; if the value of T is too large, the model may over-define the background, causing some foreground to be detected as background, resulting in missed detections.

[0182] Step 3: Model Update; As shown in Step 2, when a new pixel sample satisfies the formula condition, it is considered a background pixel and its background model needs to be updated. The weight update formula is as follows: ω i，t =(1-α)ω i，t-1 +αP i，t

[0183] For the matching model, P i,t =1, for unmatched models, P i,t =0, and then normalize the weights of each model. α is the weight update rate, which is generally 0 < α < 1.

[0184] After the weights are updated, the model parameters are updated as follows:

[0185] ρ=α / ω i，t

[0186] μ i，t =(1-ρ)μ i，t-1 +ρX t

[0187]

[0188] Where ρ is the parameter update rate.

[0189] Gaussian mixture background modeling avoids the instability of single-model detection by building multiple models for each pixel, and improves the background model's adaptability to dynamic backgrounds, such as shaking leaves and flowing water ripples, which are not real moving targets.

[0190] Specifically, in the principal component background modeling function, the input data is generally a sequence of images captured by a fixed camera. Each frame is rearranged into a column vector, and then the sequence of all image column vectors is arranged in chronological order to obtain a matrix D. This matrix D consists of two parts: the relatively stable background portion of the scene, corresponding to a low-rank matrix A; and the foreground portion consisting of moving targets, corresponding to another matrix E. That is: D = A + E

[0191] In this matrix, matrix D is known, while matrices A and E are unknown. A is a low-rank matrix.

[0192] When all elements in matrix E follow an independent and identically distributed Gaussian distribution, the optimal matrix A is obtained by optimization using the following formula: min A,E ||E|| F strank(A)≤r, D=A+E

[0193] The optimal solution to the above optimization problem can be obtained by performing singular value decomposition on matrix D. Since the assumption that all elements in E follow a Gaussian distribution may not hold true in background modeling, we replace it with another assumption that better reflects the practical application scenario: matrix E is a sparse matrix with non-zero elements. In this case, the problem of solving the low-rank matrix A is transformed into a bi-objective optimization problem: min A,E (rank(A),||E||0)stD=A+E

[0194] Introducing a tradeoff factor constant λ>0, the above bi-objective optimization problem is transformed into the following single-objective optimization problem:

[0195] min A,E rank(A)+λ||E||0s.tD=A+E

[0196] Relaxing the objective function of the above single-objective optimization problem (since the nuclear norm of a matrix is ​​the envelope of its rank; and the (1,1) norm of a matrix is ​​the convex hull of its zero norm), we obtain the following optimization problem:

[0197] min A,E ||A|| * +λ·||E|| 1,1 stD = A + E

[0198] The above optimization problem can be solved using an iterative thresholding algorithm or an augmented Lagrange multiplier method. Since the iterative thresholding algorithm or the augmented Lagrange multiplier method are existing technologies, they will not be described in detail here.

[0199] To achieve automatic segmentation of video backgrounds, a video needs to be automatically divided into several parts that each satisfy the low-rank assumption. One frame is selected as a reference frame, and two distance functions are defined. Their weighted average is used to measure the distance between two frames. If the distance between a frame and the reference frame is less than a certain threshold, the frame is considered to belong to the same part as the reference frame; otherwise, it belongs to another part.

[0200] In images satisfying the low-rank hypothesis, the background of the same portion of the image is largely the same, while the foreground may differ. These differ from the background and foreground of the other portion of the image. Therefore, the differences between image sequences satisfying the low-rank hypothesis are relatively small. The difference between normalized inter-frame subtractions is used to measure the inter-frame distance fd between two frames fa and fb in n frames:

[0201]

[0202] Based on the actual situation of video background modeling, the sudden change in the background divides the video into two parts in time. Each part is usually continuous in time, so the normalized time difference T is used. d To measure the time distance between two frames:

[0203]

[0204] The distance between two frames is calculated by multiplying the inter-frame distance D and the temporal distance of each frame by a coefficient and then summing the results, as shown below. When this distance is less than a threshold Th, the two frames are considered to belong to the same part; otherwise, they belong to different parts. Because the two coefficients and the threshold are relative, the coefficient for the inter-frame distance can be fixed at 1, the coefficient for the temporal distance at λ, and the threshold at Th.

[0205] D(f a ,f b )=f d (fa ,f b )+λ·T d (f a ,f b )

[0206] D(f a ,f b )≤Th

[0207] Next, several video segments that met the local low-rank hypothesis and could be divided into two parts were found. These videos were manually divided, and then the values ​​of λ and Th were adjusted so that the video segmentation results were consistent with the manual segmentation results. In this way, suitable values ​​of λ and Th were found.

[0208] In some embodiments, the compressed sensing background modeling function applies a background modeling algorithm based on compressed sensing and background subtraction, which can accurately and robustly detect foreground moving targets, effectively reducing computation and lowering the algorithm's time complexity. The background subtraction method requires the established background model to be sufficiently robust and necessitates setting an appropriate binarization threshold in post-processing to accurately detect foreground targets. Since the accuracy of the initial background image affects the update time of the background image in subsequent processes, obtaining a high-precision initial background image is necessary. Therefore, this embodiment provides an improved initial background modeling method. The modeling idea of ​​this method is as follows: First, the average value of several frame images is obtained using the averaging method. Then, the average difference between these frame images is calculated, and this average difference is used to remove pixel values ​​with large variations in these frame images. Finally, these frame images with large variations removed are averaged again, and the average value obtained this time is used as the pixel value of the initial background image.

[0209] Specifically, the improved initial background modeling method described above includes the following specific modeling steps:

[0210] Step 1: Calculate the average value Mean(x,y) of the N frames used to obtain the initial background image. The calculation formula is as follows:

[0211]

[0212] Step 2: Calculate the average difference MD(x,y) between N frames of images. The calculation formula is as follows:

[0213]

[0214] MD(x,y)=D(x,y) / N

[0215] Step 3: Replace the pixel values ​​that change significantly in the Nth frame image with the pixel values ​​of the (N+1)th frame image; that is, replace the pixel values ​​that change significantly in the current frame image with the pixel values ​​of the next frame image.

[0216] If |I i (x,y)-Mean(x,y)|>μMD(x,y), then I i (x, y) = I i+1 (x, y)

[0217] Where i = 0, 1, 2, ..., N, and μ = 2.5 (empirical value).

[0218] Step 4: Calculate the average value of the N replaced frames, and use this average value as the pixel value of the initial background image:

[0219] After obtaining the initial background image, in order to reduce the amount of image data transmitted, compressed sensing technology is used to first perform sparsification processing on the obtained initial background image and the current input frame image, and then the compressed measurement values ​​are obtained using the measurement matrix. The process is as follows:

[0220] Step 1: Perform sparse representation of the obtained initial background image and the current input frame image:

[0221] Step 2: Measure the obtained sparsity coefficients to obtain the compressed measurement value of the image: y bn =Φθ bn y tn =Φθ tn

[0222] Where n represents time, n = 0, 1, 2, 3, ... As can be seen from the two processes above, after processing with compressed sensing technology, the system no longer transmits the pixel values ​​of the image, but a small number of compressed measurement values ​​of the image, thus greatly reducing the amount of image data transmitted.

[0223] Traditional background image update strategies are designed for pixel values, while compressed sensing technology yields only a small number of compressed measurement values, not pixel values. Therefore, when designing an update strategy, the focus should shift to the compressed measurement values. Since a difference image can be reconstructed using the compressed measurement difference between the current frame and the background image, background updates can also be performed using the compressed measurement values ​​of the current frame and the background image. Now, assume I... n B n Representing the current input frame image and the background image respectively, y tn isI n The compression measurement value, y bn It is B n The compressed measurement value, then the background B at time n+1 n+1 Compression measurement value y bn+1It is calculated by the following formula:

[0224]

[0225] Where α represents the model update rate, a constant satisfying 0 < α < 1, and its reciprocal represents the time constant during the decay process, generally an empirical value; M represents the number of compressed measurements; i represents the position of the corresponding compressed measurement; y b0 It is a compressed measurement of the initial background image.

[0226] At position i, if the following condition is met, it indicates that there is a moving target at this position.

[0227] |y tn+1 (i)-y tn (i)|>T n+1 (i)

[0228] Among them, T n+1 (i) is a threshold that is updated in real time, and its update strategy is expressed as follows:

[0229]

[0230] Where a is an integer close to 1, and the threshold T n+1 The value of (i) can be adjusted by changing the value of a.

[0231] Once a video image is acquired through a camera, it is transmitted frame by frame to the target detection experimental software system. Subsequently, the target detection experimental software system automatically updates the background image based on the target's movement, minimizing the impact of changes in the external environment.

[0232] The difference between the measurements of the current input image and the background image is defined as: r tn =||y tn (i)-y bn (i)||2

[0233] The difference is determined based on a threshold, and the corresponding measurement value in the difference image is assigned to either the foreground or background. Let the objective function be:

[0234] The determination is based on the following criteria: if the difference between the measured values ​​of the current input image and the background image is less than a set threshold, it is determined to be a measurement of the background; otherwise, it is determined to be a measurement of the foreground target. Generally, setting the determination threshold is crucial because only with an appropriate threshold can moving foreground targets be accurately segmented. If the threshold is too large, the foreground may be mistaken for the background, making target detection ineffective. If the threshold is too small, the algorithm will be sensitive to interference factors in the image, misdetecting minor changes in the background as foreground targets. There are generally two methods for setting the threshold: one is based on empirical values, and the other is adaptively learned within the algorithm. Setting a fixed threshold based on experience is relatively simple but less flexible and may reduce the robustness of the detection algorithm in some scenarios. After obtaining the difference result through the set threshold, further processing is needed to obtain an accurate and complete background. For example, under changing lighting conditions, the shadows of moving targets may be misdetected as foreground targets, so shadow interference needs to be removed; or a small amount of noise generated in the foreground before and after difference can be processed using certain denoising methods to obtain a complete background, and so on.

[0235] Specifically, Support Vector Machines (SVM) is a binary classification model. Its basic model is a linear classifier with the largest margin defined in the feature space. The largest margin distinguishes it from the perceptron. SVM also incorporates kernel tricks, making it a de facto non-linear classifier. The learning strategy of SVM is margin maximization, which can be formalized as solving a convex quadratic programming problem, equivalent to minimizing a regularized hinge loss function. The SVM learning algorithm is an optimization algorithm for solving convex quadratic programming problems.

[0236] The basic idea behind SVM learning is to find a separating hyperplane that correctly partitions the training dataset and maximizes the geometric margin. For example... Figure 16 As shown, wx+b=0 is the separating hyperplane. For linearly separable datasets, there are infinitely many such hyperplanes (i.e., perceptrons), but the separating hyperplane with the largest geometric margin is unique.

[0237] Before proceeding with the derivation, let's define some terms. Suppose we have a training dataset T = {(x1, y1), (x2, y2), ..., (x...} on a feature space. N ,y N )}. Where x i ∈R n y i ∈{+1,-1},i=1,2,...,N,x i Let y be the i-th eigenvector. iThe class label is +1 for positive examples and -1 for negative examples. We further assume the training dataset is linearly separable.

[0238] Geometric margin: For a given dataset T and a hyperplane wx + b = 0, the geometric margin of the hyperplane with respect to the sample points (xi, yi) is defined as:

[0239] The minimum geometric margin of the hyperplane with respect to all sample points is: r = minr i

[0240] In fact, this distance is the so-called distance from the support vector to the hyperplane.

[0241] Based on the above definition, the problem of finding the maximum splitting hyperplane in the SVM model can be expressed as the following constrained optimization problem:

[0242]

[0243]

[0244] After simplification, the problem of finding the maximum split hyperplane in the SVM model can be expressed as the following constrained optimization problem:

[0245]

[0246] sty i (wx i +b)≥1,i=1.2,...N

[0247] This is a convex quadratic programming problem with inequality constraints, and its dual problem can be obtained by using the Lagrange multiplier method.

[0248] A key property of Support Vector Machines (SVMs) is that after training, most of the training samples do not need to be retained, and the final model only depends on the support vectors. Since perfectly linearly separable data is almost nonexistent in reality, the concept of "soft margin" is introduced to address this issue. This allows some points to not satisfy the constraint, i.e., yj(wxj+b)≥1. Using hinge loss, the original optimization problem is rewritten as:

[0249] sty i (wx i +b)≥1-ξ i

[0250] ξ i ≥0, i=1.2,...N

[0251] Where ξ iLet ξ be a "slack variable". i =max(0,1-y) i (wx i +b)) is a hinge loss function. Each sample has a corresponding slack variable, representing the degree to which the sample does not meet the constraints. C is called the penalty parameter; the larger the value of C, the greater the penalty for classification. Similar to the approach for solving linearly separable problems, we first use the Lagrange multiplier method to obtain the Lagrange function, and then solve its dual problem.

[0252] In summary, the linear support vector machine learning algorithm can be obtained as follows:

[0253] Input: Training dataset T = {(x1,y1),(x2,y2),...,(x N ,y N )}. Where x i ∈R n y i ∈{+1,-1}, i=1,2,...,N.

[0254] Output: Separating hyperplane and classification decision function

[0255] 1) Choose a penalty parameter C > 0, construct and solve the convex quadratic programming problem:

[0256] 0≤α i ≤C, i=1.2,...N

[0257] Obtain the optimal solution

[0258] 2) Calculation:

[0259] Choose α * A component Meet the conditions calculate:

[0260] 3) Find the separating hyperplane: w * ·x+b * =0

[0261] Classification decision function: f(x) = sign(w) * ·x+b * ).

[0262] Specifically, regarding the YOLO single-stage object detection function, object detection is an important branch of computer vision. Its basic function is to locate and classify objects in an image using the smallest possible bounding box. The main improvements of the YOLO3 deep learning model include: adjusting the network structure; utilizing multi-scale features for object detection; and replacing softmax with Logistic regression for object classification. The Darknet-53 structure built in the model borrows ideas from ResNet, adding residual modules to the network. This helps solve the gradient problem in deep networks. Each residual module consists of two convolutional layers and one shortcut connection. 1, 2, 8, 8, 4 represent the number of repeated residual modules. The entire v3 structure lacks pooling and fully connected layers. Downsampling is achieved by setting the stride of the convolution to 2, reducing the image size by half after each convolutional layer. Each convolutional layer is implemented using convolution + Batch Normalization + Leaky ReLU, and zero padding is added after each residual module.

[0263] Given a 416*416*3 image as input, the Darknet network generates predictions at three different scales, each corresponding to N channels containing prediction information; the prediction results are for anchors of each size in each grid. YOLOv3 generates a total of 13*13*3 + 26*26*3 + 52*52*3 predictions. Each prediction corresponds to 85 dimensions: 4 (coordinate values), 1 (confidence score), and 80 (number of COCO classes).

[0264] For multi-scale detection, multiple scales are used for prediction, specifically by upsampling and concatenating data in the final layers of the network prediction process. The impact of resolution on prediction is explained below:

[0265] Resolution information directly reflects the number of pixels that make up an object. The more pixels an object has, the richer and more detailed its representation of details; in other words, the richer the resolution information. This is why large-scale feature maps provide resolution information. Semantic information in object detection refers to the information that distinguishes an object from its background; that is, semantic information tells you that this is an object, and the rest is background. In different categories, semantic information does not require a lot of detailed information. Large resolution information can actually reduce semantic information. Therefore, small-scale feature maps provide better semantic information while providing the necessary resolution information. (However, for small objects, small-scale feature maps cannot provide the necessary resolution information, so they must be combined with large-scale feature maps.)

[0266] YOLOv3 goes a step further by employing three feature maps at three different scales for object detection, enabling the detection of more fine-grained features. The results of these three detections are not identical; roughly speaking, different scales detect objects of different sizes. The network's final output has three scales: 1 / 32, 1 / 16, and 1 / 8. After several convolutional operations following layer 79, a prediction of 1 / 32 (13*13) is obtained. This high downsampling factor results in a large receptive field, making it suitable for detecting larger objects in the image. This result is then upsampled and concatenated with the result from layer 61, followed by several convolutional operations to obtain a prediction of 1 / 16; this has a medium-scale receptive field, suitable for detecting medium-sized objects. The result from layer 91, after upsampling, is concatenated with the result from layer 36, and after several convolutional operations, a result of 1 / 8 is obtained. This has the smallest receptive field, suitable for detecting small objects.

[0267] `concat`: Tensor concatenation. It concatenates the upsampled data from an intermediate layer in Darknet with that of a subsequent layer. This concatenation operation differs from the `add` operation in residual layers; concatenation expands the tensor's dimensions, while `add` simply adds the data without changing the tensor's dimensions.

[0268] YOLO2 began using K-means clustering to obtain the size of the prior boxes, and YOLO3 continued this approach, setting 3 prior boxes for each downsampling scale, resulting in a total of 9 prior box sizes.

[0269] The nine prior boxes in the COCO dataset are: (10x13), (16x30), (33x23), (30x61), (62x45), (59x119), (116x90), (156x198), and (373x326).

[0270] In terms of allocation, larger prior boxes (116x90), (156x198), and (373x326) are applied to the smallest 13*13 feature map (which has the largest receptive field), suitable for detecting larger objects. Medium-sized prior boxes (30x61), (62x45), and (59x119) are applied to the medium-sized 26*26 feature map (which has a medium receptive field), suitable for detecting medium-sized objects. Smaller prior boxes (10x13), (16x30), and (33x23) are applied to the largest 52*52 feature map (which has a smaller receptive field), suitable for detecting smaller objects.

[0271] When predicting bounding boxes, Yolo v3 uses logistic regression. Each time Yolo v3 predicts a bounding box, the output is the same as in v2: (tx, ty, tw, th, to), and then the absolute values ​​(x, y, w, h, c) are calculated using Equation 1.

[0272] Logistic regression is used to give an objectness score (for NMS) to the area enclosed by the anchor, that is, how likely this location is to be the target.

[0273] Yolo v3 operates on only one prior, the optimal one. Logistic regression, on the other hand, is used to find the objectness score (the probability of an object's existence) from the nine anchor priors. Instead of using softmax to predict object categories, it uses the output of logistic regression. This allows for multi-label objects (e.g., a person with labels "Woman" and "Person").

[0274] The training strategy for YOLOv3 is as follows:

[0275] 1) The prediction box is divided into three cases: positive, negative, and ignore.

[0276] 2) Positive Examples: Randomly select a ground truth box and calculate the Intersection over Union (IoU) with all 4032 boxes. The predicted box with the highest IoU is the positive example. A predicted box can only be assigned to one ground truth box. For example, if the first ground truth box has already matched a positive example, the next ground truth box will be the one with the highest IoU among the remaining 4031 boxes. The order of the ground truth boxes is irrelevant. Positive examples generate confidence loss, detection box loss, and class loss. The predicted box is the label of the corresponding ground truth box (requires reverse encoding, using the actual x, y, w, h to calculate tx, ty, tw, th; the class label is 1 for the corresponding class and 0 for the rest; the confidence label is 1).

[0277] 3) Ignore examples: Excluding positive examples, any example whose IOU with any ground truth is greater than a threshold (empirical value is 0.5) is ignored. Ignoring examples does not produce any loss.

[0278] 4) Negative examples: excluding positive examples (the bounding box with the highest IOU with the ground truth, but whose IOU is less than the threshold, is still considered a positive example), if the IOU with all ground truths is less than the threshold (0.5), then it is a negative example. Negative examples only incur loss based on confidence, and the confidence label is 0.

[0279] feature Figure 1 The abstract expression for the loss function of Yolov3 is as follows:

[0280]

[0281]

[0282] The YOLOv3 Loss is the sum of the Loss values ​​of the three feature maps:

[0283] Loss = loss N1 +loss N2 +loss N3

[0284] 1) λ is a weight constant that controls the ratio between the detection box loss, obj confidence loss, and noobj confidence loss. Typically, the number of negative examples is several tens of times greater than the number of positive examples. The detection effect can be controlled through the weight hyperparameter.

[0285] 2) If it is a positive example, output 1; otherwise, output 0. If the sample is negative, output 1; otherwise, output 0. Ignore all samples and output 0.

[0286] 3) For x, y, w, and h, MSE can be used as the loss function, or smooth L1 loss (from Faster R-CNN) can be used. Smooth L1 can make training smoother. Since confidence and class labels are binary classifications of 0 and 1, cross-entropy is used as the loss function.

[0287] YOLOv3 has a slightly better accuracy than SSD, is almost on par with Faster R-CNN, and is worse than RetinaNet. However, its speed is at least twice that of SSD, RetinaNet, and Faster R-CNN. YOLOv3 with an input size of 320*320 takes only 22ms to process a single image, and the simplified YOLOv3 Tiny is even faster.

[0288] Specifically, in the anchorless object detection function, NanoDet is an ultra-fast and lightweight mobile anchor-free object detection model. NanoDet is a single-stage anchor-free object detection model similar to FCOS, using ATSS for object sampling and the Generalized Focal Loss function for classification and box regression. The final NanoDet model architecture is as follows... Figure 13 As shown.

[0289] Loss Function: NanoDet uses the Generalized Focal Loss function proposed by Li Xiang et al. This function can remove the Centerness branch of FCOS, saving a large number of convolutions on this branch, thereby reducing the computational overhead of the detection head, making it very suitable for lightweight deployment on mobile devices.

[0290] Lightweight detection head: The FCOS series uses a shared weight detection head, which means that the same set of convolutions is used to predict detection boxes for the multi-scale feature maps generated by FPN, and then a learnable scale value is used as a coefficient for each layer to scale the predicted boxes.

[0291] The advantage is that it reduces the number of parameters in the detector head to 1 / 5 of that without weight sharing. This is very useful for large models with hundreds of channels in the detector head alone, but for lightweight models, shared weight detector heads are not very meaningful. Since mobile models perform inference on the CPU, shared weights do not speed up the inference process. Moreover, with a very lightweight detector head, shared weights further reduce its detection capability. Therefore, it is more appropriate to use a set of convolutions for each feature layer.

[0292] Meanwhile, the FCOS series uses Group Normalization (GN) as the normalization method in its detector head. GN has many advantages over BN, but it has one drawback: BN can directly integrate its normalized parameters into the convolution during inference, saving this computational step, while GN cannot. To save time on the normalization operation, GN was replaced with BN.

[0293] FCOS's detection head uses four 256-channel convolutions as one branch, meaning there are a total of eight c=256 convolutions across the bounding box regression and classification branches, resulting in a very high computational cost. To reduce its complexity, we first replaced ordinary convolutions with depthwise separable convolutions and reduced the number of stacked convolutions from four to two groups. Regarding the number of channels, we compressed the 256 dimensions to 96 dimensions. The choice of 96 was made to ensure the number of channels remains a multiple of 8 or 16, allowing us to benefit from the parallel acceleration offered by most inference frameworks.

[0294] Finally, drawing inspiration from the YOLO series, bounding box regression and classification are computed using the same set of convolutions, then split into two parts. The resulting lightweight detection head is shown below. Figure 15 As shown.

[0295] FPN layer improvements: There are many improvements to FPN, such as EfficientDet using BiFPN, YOLO v4 and v5 using PAN, and BalancedFPN, etc. While BiFPN has powerful performance, the stacked feature fusion operations can reduce the running speed. PAN, on the other hand, has only two paths, top-down and bottom-up, which is very simple and makes it a good choice for feature fusion in lightweight models.

[0296] The original PAN and the PAN in the YOLO series both used convolutions with stride=2 for scaling large-scale feature maps to small-scale ones. However, this project, for lightweight considerations, chose to completely remove all convolutions in PAN, retaining only 1x1 convolutions extracted from the backbone network features for feature channel alignment. Upsampling and downsampling are both performed using interpolation. Unlike the concatenation operation used in YOLO, this model chooses to directly add the multi-scale feature maps, making the computational cost of the entire feature fusion module very small.

[0297] Backbone Network: ShuffleNetV2 1.0x was chosen as the backbone network. The last convolutional layer of this network was removed, and features downsampled by 8, 16, and 32 times were extracted and input into PAN for multi-scale feature fusion. The entire backbone model uses code provided by Torchvision, which can directly load the ImageNet pre-trained weights provided by Torchvision, greatly helping to accelerate model convergence.

[0298] After lightweighting the three main modules (Head, Neck, and Backbone) of the one-stage detection model, the currently open-source NanoDet-m model was obtained. With a 320x320 input resolution, the entire model has only 0.72B Flops, while yolov4-tiny has 6.96B, nearly ten times smaller! The model's parameter size is also only 0.95M, and the weight file, after using ncnn optimize for 16-bit storage, is only 1.8MB, making it very suitable for deployment on mobile devices, effectively reducing the size of the app, and also more friendly to lower-end embedded devices.

[0299] In some embodiments, this application also discloses a detection method for an unmanned surface vessel (USV) perception system, including...

[0300] An initial background image is obtained based on the initial video image captured by the sensor. (If the obtained initial video image does not meet the requirements, the image processing module described above can be used to process the image. For example, when it is necessary to verify the recognition capability of the system, the noise simulation submodule can be used to add noise to the image; or when the image is not clear, other modules can be used to process it to make the image clear.)

[0301] Compare the current input frame image captured by the sensor with the initial background image to obtain the moving target or update the initial background image;

[0302] The method for obtaining the initial background image includes the following steps:

[0303] Obtain the mean value Mean(x,y) and mean difference MD(x,y) of N frames in the initial video image;

[0304] Replace the pixel values ​​with larger changes in the Nth frame image with the pixel values ​​of the (N+1)th frame image. If |I i (x,y)-Mean(x,y)|>μMD(x,y), then;

[0305] Calculate the average value of the N frames after replacement; this average value is the pixel value of the original background image.

[0306] Where i = 0, 1, 2, ..., N; μ = 2.5.

[0307] Specifically, the formula for calculating the average value Mean(x,y) is as follows: Where I represents the pixel value. The formula for calculating the average difference MD(x,y) is as follows: MD(x,y)=D(x,y) / N.

[0308] Specifically, the average value of the replaced N frames is calculated. This average value is the pixel value of the original background image. The formula for calculating the average value of the replaced N frames is as follows:

[0309] Specifically, comparing the current input frame image captured by the sensor with the initial background image to obtain a moving target or update the initial background image includes the following steps:

[0310] Sparse representation of the initial background image and the current input frame image:

[0311] The obtained sparse coefficients are measured to obtain compressed measurement values ​​of the initial background image and the current input frame image: y bn =Φθ bn y tn =Φθ tn Where n represents time, n = 0, 1, 2, 3, ...

[0312] Obtain the compressed measurement value of the background image at time n+1:

[0313]

[0314] Among them, I n B represents the current input frame image. n Represents the background image, y tn It is the current input frame image I n The compression measurement value, y bn Background image B n The compression measurement value, y bn+1 The background image B at time n+1. n+1 The compressed measurement value represents the model update rate, which is a constant satisfying 0 and 1. Its reciprocal represents the time constant during the decay process, which is generally an empirical value; M represents the number of compressed measurements; i represents the position of the corresponding compressed measurement value; y b0 It is a compressed measurement of the initial background image;

[0315] At position i, if the following condition is satisfied, it indicates that there is a moving target at this position: |y tn+1 (i)-y tn (i)|>T n+1 (i)

[0316] Among them, T n+1 (i) is a threshold that is updated in real time, and its update strategy is expressed as follows:

[0317]

[0318] Where a is an integer close to 1, and the threshold T n+1 The value of (i) can be adjusted by changing the value of a;

[0319] The difference between the measurements of the current input image and the background image is defined as: r tn =||y tn (i)-y bn (i)||2

[0320] The difference is determined based on a threshold, and the corresponding measurement value in the difference image is assigned to either the foreground or background. Let the objective function be:

[0321] If the difference between the measured values ​​of the current input image and the background image is less than the set threshold, it is determined to be the measured value of the background; otherwise, it is determined to be the measured value of the foreground target.

[0322] It is understood that the embodiments described in this invention are some embodiments and features, and any person skilled in the art can make some changes and variations based on the spirit of the invention described herein. These changes and variations are also considered to fall within the scope of this invention and the scope of the independent and appended claims.

Claims

1. A method for detecting a target by an unmanned surface vehicle (USV) sensing system, comprising obtaining an initial background image from an initial video image captured by a sensor; comparing a current input frame image captured by the sensor with the initial background image to obtain a moving target or update the initial background image; wherein in the step of obtaining the initial background image from the initial video image captured by the sensor, the method comprises the following steps: obtaining a mean value Mean(x, y) and a mean difference MD(x, y) of N frame images in the initial video image; replacing the pixel value of the N+1th frame image with the pixel value of the Nth frame image which changes more greatly, if , then; calculating a mean value of the N frame images after replacement, and the mean value is a pixel value of the initial background image; wherein i = 0, 1, 2, …, N; μ = 2.5; and a calculation formula of the mean value Mean(x, y) is as follows: wherein I is a pixel value; and a calculation formula of the mean difference MD(x, y) is as follows: ;the average value of the N frames of images after replacement, which is the pixel value of the initial background image, the calculation formula of the average value of the N frames of images after replacement is as follows: ; said comparing the current input frame image captured by the sensor with the initial background image to obtain a moving object or to update the initial background image, comprises the steps of: performing sparse representation on the initial background image and the current input frame image; measuring the obtained sparse coefficients to obtain a compressed measurement value of the initial background image and the current input frame image; wherein n represents a time point, and n = 0, 1, 2, 3, …, obtaining a compressed measurement value of a background image at a time point n+1; where I n represents the current input frame image, B n represents the background image, y tn is the compression measure of the current input frame image I n , y bn is the compression measure of the background image B n , y bn+1 is the compression measure of the background image B n+1 at the n+1 time; a represents the updating speed of the model, which is a constant, and its reciprocal represents the time constant in the decay process; M represents the number of compression measures; i represents the position of the corresponding compression measure; y b0 is the compression measure of the initial background image; at a position i, if a condition is satisfied, it is indicated that there is a moving target at the position: where T n+1 (i) is a real-time updated threshold, whose updating strategy is expressed as follows: where a is an adjustable integer close to 1, and threshold T n+1 The size of (i) can be adjusted by changing the value of a. a difference value of the measurement values of the current input image and the background image is defined as: determining the difference value according to a threshold value, and classifying corresponding measurement values in a difference image into a foreground or a background, and setting a determination target function as: if the difference value of the measurement values of the current input image and the background image is less than a set threshold value, the measurement value is determined as a background value, otherwise, the measurement value is determined as a foreground target value.

Images