An adaptive people detection method for complex indoor and outdoor environments

Abstract

The present invention relates to the field of image processing technology, and discloses an adaptive personnel detection method for complex indoor and outdoor environments, comprising: obtaining video stream data at different light intensities, and dividing the image into background frames without pedestrians and foreground frames with pedestrians at different light intensities according to the optical flow field of each frame image in the video stream data; using a pedestrian detection neural network model to obtain pedestrian image blocks representing pedestrian instances in the foreground frame; pasting pedestrian image blocks belonging to the same pedestrian in the foreground frame at different time points into different background frames according to set spatiotemporal constraints, thereby obtaining a plurality of combined image data of different light intensities that meet scale constraints; training the pedestrian detection neural network model; and detecting the combined image data through the trained pedestrian detection neural network model. The present invention avoids generating noise data and effectively reduces the difficulty of training the pedestrian detection neural network model.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an adaptive personnel detection method for complex indoor and outdoor environments. Background Art

[0002] In the oil industry, intelligent monitoring systems have become an important means to ensure safe production, especially in complex operating environments, where automatic detection and monitoring of personnel are crucial. However, due to the changeable weather, complex equipment backgrounds, and differences in lighting and angles in oil operation scenarios, traditional pedestrian detection algorithms are difficult to effectively adapt to these challenges. For example, in oilfield monitoring systems, cameras are usually distributed in different areas, and the interference factors of equipment, background, and environment are diverse. Traditional pedestrian detection models rely on fixed feature extraction algorithms and perform poorly in these scenarios. In addition, due to the massive amount of surveillance videos, the workload of manually annotating pedestrian data is huge and error-prone, and the problem of insufficient training sets becomes more prominent.

[0003] Existing pedestrian detection technologies mainly rely on artificially designed features or simple data augmentation to generate diverse data, but these methods are limited in effectiveness when dealing with diverse monitoring devices and scenarios in complex monitoring environments. The spatiotemporal continuity characteristics in video streams are often ignored, and pedestrian trajectory information cannot be fully utilized to optimize the robustness of the model. Therefore, current pedestrian detection technologies show great limitations in the complex environment of the oil industry, and there is an urgent need for a pedestrian detection method that can adaptively generate labeled data and optimize the model. Summary of the invention

[0004] The present invention aims to solve the problems of adaptability, low annotation efficiency and insufficient data utilization of pedestrian detection models in complex monitoring scenarios in the petroleum industry. An adaptive personnel detection method and system for complex indoor and outdoor environments are proposed. By acquiring background frames and foreground frames in monitoring videos, combining noise reduction algorithms and pedestrian trajectory reconstruction, and integrating pedestrian scale constraints, massive high-quality annotated data samples are adaptively generated. These data samples are used to fine-tune and optimize the pedestrian detection neural network model to improve the accuracy of pedestrian detection.

[0005] The present invention is achieved through the following technical solutions:

[0006] An adaptive person detection method for complex indoor and outdoor environments, comprising:

[0007] S1, acquiring video stream data at different light intensities, and dividing the image into background frames without pedestrians and foreground frames with pedestrians at different light intensities according to the optical flow field of each frame image in the video stream data;

[0008] S2, using a pedestrian detection neural network model combined with a pedestrian trajectory detection algorithm to perform pedestrian target detection on all the foreground frames, and correspondingly obtain pedestrian image blocks representing pedestrian instances in the foreground frames;

[0009] S3, pasting the pedestrian image blocks belonging to the same pedestrian in the foreground frames at different time points into different background frames according to the set spatiotemporal constraints, thereby obtaining a plurality of combined image data of different illumination intensities satisfying the scale constraints, wherein one background frame can be pasted with multiple pedestrians;

[0010] S4, fine-tuning the parameters of the pedestrian detection neural network model based on the LoRA technology, and inputting the plurality of combined image data into the fine-tuned pedestrian detection neural network model, and judging whether the pedestrian detection neural network model satisfies the scene adaptability through the loss function, if so, jumping to S5, otherwise, returning to S2;

[0011] S5. Detect the combined image data using the trained pedestrian detection neural network model.

[0012] As an optimization, the image is divided into a background frame without pedestrians and a foreground frame with pedestrians according to the optical flow field of each frame image in the video stream data. f The specific process is:

[0013] A1. Obtain video stream data under different light intensities, and divide the video stream data into a plurality of frame images according to time points;

[0014] A2, using the neuFlow V2 algorithm model to perform optical flow estimation on the images of the frames to obtain optical flow fields of the images of the frames;

[0015] A3, the optical flow field is divided into H levels according to the intensity, wherein the image with the optical flow field less than or equal to level h1 is defined as the background frame, and the image with the optical flow field greater than or equal to level h2 is defined as the foreground frame, wherein h1 <h2<H。

[0016] As an optimization, a pedestrian detection neural network model is used in combination with a pedestrian trajectory detection algorithm to detect the foreground frame I f Perform pedestrian target detection, and obtain the corresponding foreground frame I f The specific process of the rectangular box representing the pedestrian instance is as follows:

[0017] B1, obtaining a number of foreground frames at consecutive time points in the video stream data;

[0018] B2. Using a pedestrian detection neural network model, respectively perform pedestrian target detection on a plurality of the foreground frames to obtain an initial rectangular frame representing a pedestrian instance;

[0019] B3, cropping the portion corresponding to the initial rectangular frame in the foreground frame to obtain a corresponding pedestrian image block;

[0020] B4. According to the pedestrian's spatiotemporal constraints, the pedestrian image block is subjected to noise reduction processing to obtain a more reliable pedestrian image block.

[0021] As an optimization, the pedestrian detection neural network model is an efficientDet model.

[0022] As an optimization, for one of the foreground frames, the specific process of pasting the pedestrian image block in the foreground frame into the background frame is:

[0023] C1. Randomly selecting M pedestrian image blocks from the N pedestrian image blocks obtained from the foreground frame as pasting objects;

[0024] C2, pasting the M pasted objects into a grayscale image, wherein the size of the grayscale image is consistent with the size of the foreground frame, and the mask in the grayscale image is set to: α=1 at the position with the final rectangular frame, and α=0 at the position without the final rectangular frame;

[0025] C3. Cut out the pixels at the position corresponding to α=1 in the grayscale image, and paste them into the background frame, thereby forming combined image data.

[0026] As an optimization, the time and space constraints are specifically:

[0027] Scaling is performed according to the size of the pedestrian image block in the original foreground frame and the scaling ratio of the corresponding position in the background frame where the pedestrian image block needs to be pasted, wherein the relative position of the pedestrian image block in the original foreground frame is consistent with the relative position in the background frame where the pedestrian image block needs to be pasted.

[0028] As an optimization, the pedestrian image block is scaled proportionally and pasted into the background frame to form a new pedestrian image block. The height and width of the new pedestrian image block are respectively expressed as:

[0029] h new =s×h orig

[0030] w new =s×w orig

[0031] Among them, h new 、w new They represent the height and width of the new pedestrian image block, s is the actual scaling ratio, and h orig 、w origThey represent the height and width of the pedestrian image block in the original foreground frame respectively.

[0032] As an optimization, the pasting position of the pedestrian image block in the background frame to be pasted is specifically expressed as:

[0033]

[0034] Among them, (x new ,y new ) represents the center position coordinates of the pedestrian image block in the background frame, (x orig ,y orig ) represents the median position coordinate of the pedestrian image block in the foreground frame, and Δx and Δy are the offsets of the corresponding positions in the background frame respectively.

[0035] As an optimization, the specific formula of the actual scaling ratio s is:

[0036]

[0037] Where p is the adjustment factor, d img is the depth of the pedestrian image block, d ref is the reference depth, s ref The base scaling factor.

[0038] As an optimization, fine-tuning the parameters of the pedestrian detection neural network model specifically involves fine-tuning the A layer network close to the output layer of the pedestrian detection neural network model, where A is a positive integer. The specific process of training the pedestrian detection neural network model is:

[0039] D1. For each network in the A-layer network, the weight matrix W of each network is i ∈R m×n Decomposed into the product of two low-rank matrices: W i ≈U i U i T

[0040] in, and r is the rank of decomposition, i∈[1,A], m and n represent the input feature dimension and output feature dimension of the network respectively; represents the set of real numbers, i.e. the matrix dimension, Indicates U i is a matrix of dimension m×r Indicates that V i is a matrix of dimension n×r;

[0041] D2. Initialize U i and V iThus, the initialized pedestrian detection neural network model is obtained, and then the combined image data is input as input data into the pedestrian detection neural network model to obtain a predicted pedestrian image block as a predicted label;

[0042] D3, using the pedestrian image block in the combined image data as the true label and the predicted label to calculate the loss function, and adjusting the U in the pedestrian detection neural network model by the loss function i and V i .

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] 1. The present invention proposes a video frame classification method based on optical flow method, which effectively obtains foreground frames and background frames in the video, and then uses them to generate annotation data.

[0045] 2. Based on the spatiotemporal continuity of pedestrians in surveillance videos, the present invention proposes a method based on pedestrian trajectory reconstruction to perform noise reduction and optimization on the results of pedestrian detection, obtain high-confidence pedestrian detection image blocks, and lay the foundation for the subsequent generation of accurate annotation data.

[0046] 3. Different from Google's traditional copy-paste data augmentation method, the solution proposed in this invention adds the conditions of pedestrian temporal and spatial constraints in the surveillance video. At the same time, in order to ensure that the adaptively generated data is more authentic and reliable, this solution introduces the scale constraints of pedestrians. This solution avoids the generation of noise data and effectively reduces the difficulty of model training. Using the labeled data generation method of this solution, it is theoretically possible to obtain unlimited high-quality labeled data. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0048] Figure 1 A flowchart of the method described in the present invention;

[0049] Figure 2 This is an example of the optical flow field of the foreground frame and the background frame;

[0050] Figure 3 This is an example of the pedestrian detection result of the foreground frame of the pedestrian detection neural network model;

[0051] Figure 4 This is a data sample diagram obtained using the traditional copy-paste algorithm;

[0052] Figure 5This is a sample diagram of labeled data obtained using the algorithm of this solution;

[0053] Figure 6 A comparison chart of the effects of using a pedestrian detection neural network model and the effects of the model obtained by the solution of the present invention. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0055] The general process of the present invention is:

[0056] The background and foreground frames in the video stream are obtained through the optical flow algorithm, and the noise detection results are filtered through pedestrian trajectory reconstruction to generate reliable pedestrian image blocks. At the same time, the spatiotemporal constraints that match the monitoring scene and the scale constraints of pedestrians are adopted to further ensure the authenticity of the generated labeled data and ensure that the pedestrian detection model still maintains a high recognition rate under different pedestrian motion trajectories, lighting and viewing angles. By introducing the LoRA fine-tuning mechanism, the present invention can quickly adapt to the pedestrian detection needs in different scenarios and significantly improve the detection performance of the model in various scenarios.

[0057] The present invention aims to provide an adaptive pedestrian detection method based on surveillance video streams. By acquiring background and foreground frames and a noise reduction algorithm, annotated data samples under various surveillance scenarios are adaptively generated, and these data are used to iteratively optimize and train the pedestrian detection neural network model to improve the accuracy, annotation efficiency and data utilization of pedestrian detection in surveillance scenarios.

[0058] Next, the present invention is described in detail.

[0059] This embodiment 1 provides an adaptive person detection method for complex indoor and outdoor environments. Figure 1 As shown, including:

[0060] S1, acquiring video stream data at different light intensities, and dividing the image into background frames without pedestrians and foreground frames with pedestrians at different light intensities according to the optical flow field of each frame image in the video stream data;

[0061] The specific process of S1 is:

[0062] A1. Obtain video stream data under different light intensities, and divide the video stream data into a plurality of frame images according to time points;

[0063] A2, using the neuFlow V2 algorithm model to perform optical flow estimation on the images of the frames to obtain optical flow fields of the images of the frames;

[0064] A3, the optical flow field is divided into H levels according to the intensity, wherein the image with the optical flow field less than or equal to level h1 is defined as the background frame, and the image with the optical flow field greater than or equal to level h2 is defined as the foreground frame, wherein h1 <h2<H。

[0065] Background frame and foreground frame acquisition:

[0066] In order to make the annotated data contain as many scenes as possible to cope with the situation where the model effect is not good due to reasons such as lighting changes and field of view angle adjustments, the same camera can take frames from multiple camera angles (switching different camera angles corresponds to different monitoring coverage areas in actual monitoring scenarios) to cover different scenes, and the same scene can be taken from multiple cameras to cover different angles of the same scene. In addition, frames can be taken under different lighting conditions.

[0067] The purpose of obtaining background frames (image frames without pedestrians) and foreground frames (image frames with pedestrians) is to provide background and pedestrian instances for the subsequent production of annotation data. In actual monitoring scenarios, images can be roughly divided into two categories, namely, static images of pure scene information (without any running objects) and dynamic images containing running objects (this article focuses on pedestrians appearing in the scene), so the solution in this article will be static images of pure scene information w';

[0068] As the background frame, the dynamic picture of pedestrians is used as the foreground frame (in the actual process of implementing the solution, a frame under the dynamic picture is taken). To achieve this goal, the present invention adopts the optical flow method to obtain the background frame and the foreground frame. Considering the efficiency of calculation, this paper adopts the neuFlow V2 algorithm model to estimate the optical flow of the video. The core logic of optical flow calculation is expressed as follows:

[0069] F(x,y)=R(B(I t ,I t-1 ))

[0070] Among them, F(x,y) represents the optical flow field of each arbitrary pixel position (x,y), B(I t ,I t-1 ) indicates that the algorithm is based on two consecutive frames I t and I t-1 The output of the lightweight backbone network is used to extract the initial optical flow features. R(·) represents the fast fine-tuning module, which receives the initial flow features and refines them to produce the final flow estimate. After obtaining the optical flow estimate of each pixel, the optical flow field result of the entire image frame can be obtained.

[0071] For the convenience of description, the plurality of video stream data in the present invention are collected from the same angle of the same camera under different light intensities.

[0072] In the specific implementation of this solution, images from the same angle of the same camera are used to generate annotation data. Considering the influence of different lighting conditions, images from three time points of the same day, namely morning, noon and evening, are taken during the implementation process. At the same time, in order to further generate enough training data, this solution divides the entire optical flow field of the image into 10 levels according to the field intensity, with the higher intensity being level 1, 2, ..., 10, and image frames less than or equal to level 3 are used as background frames, and image frames greater than or equal to level 7 are used as foreground frames. Figure 2 The figure shows the optical flow field of the background frame and the optical flow field of the foreground frame. It can be seen that the optical flow field of the foreground frame reflects a large amount of motion information compared to the background frame. The darker the color, the stronger the motion in the picture. It can be clearly seen that the optical flow field value of the background frame (basically no pedestrian motion) is small and the intensity is low, while the optical flow field value of the foreground frame (with a lot of pedestrian motion) is large and the intensity is high.

[0073] S2, using a pedestrian detection neural network model combined with a pedestrian trajectory detection algorithm to perform pedestrian target detection on all the foreground frames, and correspondingly obtain pedestrian image blocks representing pedestrian instances in the foreground frames;

[0074] The first step is to obtain the pedestrian image block:

[0075] Use the pedestrian detection neural network model to detect the foreground frame I f Perform target detection to obtain the initial rectangular frame representing the pedestrian instance in the image, and crop the pedestrian instance frame from the foreground frame to obtain N pedestrian image blocks, which can be expressed as (p1, p2, ..., p N ).

[0076] Specifically, the specific process is:

[0077] B1, obtaining a number of foreground frames at consecutive time points in the video stream data;

[0078] B2. Using a pedestrian detection neural network model, respectively perform pedestrian target detection on a plurality of the foreground frames to obtain an initial rectangular frame representing a pedestrian instance;

[0079] B3. Crop the portion corresponding to the initial rectangular frame in the foreground frame to obtain a corresponding pedestrian image block.

[0080] The general target detection model efficientDet is used to complete the pedestrian detection in the foreground frame. In the process of generating data annotation, this scheme only retains the pedestrian boxes with a confidence level greater than 0.95. The detection effect is as follows: Figure 3As shown, the pedestrian detection neural network model is used to detect pedestrians in the foreground frame, and the pedestrians are marked by the green boxes in the figure.

[0081] Secondly, the pedestrian trajectory noise reduction is performed.

[0082] The general pedestrian detection basic model often does not work well when detecting pedestrians in open monitoring scenes. If the annotation data is further generated based on these erroneous detection results, a large amount of noise data will be introduced. In fact, in the video stream data, pedestrians themselves show the characteristics of spatiotemporal continuity. In order to optimize the detection effect and obtain high-confidence pedestrian detection results, this scheme uses a multi-target tracking algorithm to reconstruct pedestrian trajectories and minimize abnormal instantaneous false detections and missed detections to obtain high-confidence pedestrian samples and ensure the reliability of the subsequent generated annotation data.

[0083] That is, B4, according to the pedestrian's spatiotemporal constraints, the pedestrian image block is subjected to noise reduction processing, so as to obtain a more reliable pedestrian image block.

[0084] When using a general pedestrian detection framework, the effect is often not ideal and there will be a relatively large number of false detections. Here, by introducing a pedestrian tracking algorithm, the false detection rate can be reduced and the confidence of the pedestrian image blocks used for copying and pasting can be improved. The specific principle can be understood as a concept of voting smoothing of multi-frame image detection results. The credibility of the pedestrian detection result (a pedestrian frame) in a certain frame may not be high, but if the person is detected in multiple frames at the same time, it can be considered that such a pedestrian detection frame is often more credible, and will be more credible when used for subsequent copying and pasting, without introducing unnecessary noise.

[0085] According to the natural spatiotemporal constraints of pedestrians in the monitoring scene, the pedestrian detection frames obtained in B1-B3 are denoised to obtain more reliable pedestrian image blocks.

[0086] S3, pasting the pedestrian image blocks belonging to the same pedestrian in the foreground frames at different time points into different background frames according to the set spatiotemporal constraints, thereby obtaining a plurality of combined image data of different illumination intensities satisfying the scale constraints, wherein one background frame can be pasted with multiple pedestrians;

[0087] In this solution, foreground frames will not be pasted to background frames across scenes. That is, foreground frames in the same time period, such as foreground frames in the morning, will only be pasted to background frames in that time period.

[0088] What this means is that, assuming that three pedestrians are determined to be pasted into the background frame, then the pedestrian image blocks of these three pedestrians are pasted into several background frames according to the constraints in several foreground frames collected at consecutive time points. This pedestrian detection scheme does not pay attention to the video stream. When generating samples, the pedestrian tracking and denoising part needs to consider tracking based on the video stream, and does not pay attention to the video stream when pasting.

[0089] The specific process of S3 is:

[0090] C1. Randomly select M pedestrian image blocks from the N pedestrian image blocks obtained from the foreground frame as pasting objects; it should be noted that the random selection here is for the first foreground frame, and the pedestrian image blocks selected in the subsequent foreground frames are the same as the pedestrian image blocks selected in the first foreground frame.

[0091] C2. Randomly paste the M objects into a grayscale image, wherein the size of the grayscale image is consistent with the size of the foreground frame, and the mask in the grayscale image is set to: α=1 for the position with the final rectangular frame, and α=0 for the position without the final rectangular frame; in the specific implementation process of this scheme, it is not completely random, but the position of the pedestrian image block in the background frame map is 50 to 100 pixels near its corresponding position in the foreground frame, which has been stated in the subsequent part.

[0092] C3. Cut out the pixels at the position corresponding to α=1 in the grayscale image, and paste them into the background frame, thereby forming combined image data.

[0093] In fact, the above process can be summarized as follows:

[0094] According to the formula

[0095] I labeled =α×I g +(1-α)×I b

[0096] Mix the contents of two images (foreground frame and background frame) together; in the formula, I g is the image where the object is pasted, and is generated by: f We obtain N pedestrian image blocks and randomly select M of them, which can be expressed as (p1, p2, ..., p M ), and then randomly place (after random placement, the M image blocks may overlap each other) into a grayscale image whose RGB values ​​are all (122, 122, 122) and whose size is the same as I f Keep it consistent, expressed as I0, where M≤N. I bis the target image (the background frame image in this scheme), α is the mask, and its size is the same as I g Consistent, the value of α is 1 where there is a pedestrian frame, and 0 where there is no pedestrian frame; that is, I g Cut out the pixels in the mask part and paste them into I b Since the image blocks pasted into the target image are all pedestrian image blocks, the target image I labeled These are image samples that already have instance information of pedestrian box annotations, which can be used for further model iterative optimization training.

[0097] The constraint condition is a scale constraint, specifically:

[0098] Scaling is performed according to the size of the pedestrian image block in the original foreground frame and the scaling ratio of the corresponding position in the background frame where the pedestrian image block needs to be pasted, wherein the relative position of the pedestrian image block in the original foreground frame is consistent with the relative position in the background frame where the pedestrian image block needs to be pasted.

[0099] The semantic information of the real scene should be considered when pasting the pedestrian image block into the new image frame, especially the relationship between the size of the pedestrian and the distance from the camera, as well as the correspondence between the correct pasting position of the pedestrian image block in the new image and the position after scaling. Before pasting the pedestrian image block into the new image frame, it is necessary to scale it according to the size of the pedestrian image block in the original image (i.e., its proportion of the original image or its size relative to a certain reference scale) and the scaling ratio of the corresponding position in the new image frame. Assume that the height of the pedestrian image block in the original image is h orig , with a width of w orig , the scaling ratio of the corresponding position in the new image frame is s, then the height and width of the scaled pedestrian image block are h new and w new It can be expressed as:

[0100] h new =s×h orig

[0101] w new =s×w orid

[0102] The pasting position of the pedestrian image block in the new image frame should be consistent with its relative position in the original image, while taking into account the scaled size. Assume that the center position of the pedestrian image block in the original image is (x orig ,y orig ), the offset of the corresponding position in the new image frame is (Δx, Δy) (the offset may be determined by factors such as the size and layout requirements of the new image frame), then the pasting position (x new ,y new) can be expressed as:

[0103]

[0104] Note that the scaling factor s is used here to inversely transform the original position, and half of the scaled pedestrian image block size is added to obtain the center coordinates of the pasting position.

[0105] The pedestrian detection result is represented as a rectangular box [upper left corner x coordinate, upper left corner y coordinate, lower right corner x coordinate, lower right corner y coordinate], and the center position is the intersection of the two diagonal lines of the rectangular box.

[0106] The offset is a random value, see the subsequent detailed introduction for details.

[0107] In real camera imaging scenes, pedestrians closer to the camera tend to be larger, while pedestrians farther away tend to be smaller. To simulate this relationship, when pasting pedestrian image blocks, the scaling ratio s can be adjusted according to the depth information in the original image (such as the depth value obtained by camera calibration and 3D reconstruction, or simplified to the depth estimated according to the relative size in the image). Specifically, a reference depth d can be set. ref and the corresponding scaling factor s ref , and then according to the depth d of the pedestrian image block img Calculate the actual scaling factor s:

[0108]

[0109] Among them, p is an adjustment factor used to control the impact of depth change on the scaling ratio, d img is the depth of the pedestrian image block, d ref is the reference depth, s ref The base scaling factor.

[0110] The depth of the pedestrian image block is actually how far the pedestrian is from the camera when the image plane is restored to the three-dimensional space. This solution is also set to a fixed value during implementation, because the position of the image pasted in the background frame is not much different from that of the pedestrian image block in the foreground frame.

[0111] Under the premise that the video stream data are collected from the same camera at the same angle under different light intensities, this embodiment uses the same camera at the same angle to complete the mapping when generating the annotation data, so the reference depth d in the mapping process is ref and the corresponding scaling factor s refIt can be directly set to 1, the adjustment factor p is 1, the position of the pedestrian image block in the background frame map is 50 to 100 pixels near its corresponding position in the foreground frame (i.e. Δx∈[50,100], Δy∈[50,100]), then d img is 1, so the scaling factor s is also 1. Figure 4 As shown in the figure, directly using Google's traditional copy-paste algorithm will result in a large number of unreal data samples. Such data is often noise data, which will inevitably increase the difficulty of model training. After using the algorithm optimized by this solution, we can get the following Figure 5 The example shown, Figure 5 The three pictures in the figure actually only use the image of one person. By extracting the pedestrian image blocks at the time nodes (corresponding to the foreground frames at different moments in different videos), the image blocks of the pedestrian at multiple angles can be obtained. Combined with the scale constraints of this scheme, it is possible to adaptively generate unlimited annotated data samples, and then train a scene-adaptive pedestrian detection model.

[0112] If it is not done under the same camera, then these parameters (d ref 、s ref , adjustment factor p) can be obtained through camera three-dimensional calibration and is set as a fixed value during the implementation of this solution.

[0113] If you copy and paste across cameras to generate labeled samples, you also need to consider the imaging effects of different cameras. These involve the underlying physical parameters of the camera, which are difficult to obtain during the actual implementation process. This solution emphasizes the same camera or cameras with the same model parameters.

[0114] S4, fine-tuning the parameters of the pedestrian detection neural network model based on the LoRA technology, and inputting the plurality of combined image data into the fine-tuned pedestrian detection neural network model, and judging whether the pedestrian detection neural network model satisfies the scene adaptability through the loss function, if so, jumping to S5, otherwise, returning to S2;

[0115] Fine-tuning the parameters of the pedestrian detection neural network model specifically involves fine-tuning the A layer network close to the output layer of the pedestrian detection neural network model, where A is a positive integer. The specific process of training the pedestrian detection neural network model is as follows:

[0116] D1. For each network in the A-layer network, the weight matrix W of each network is i ∈R m×n Decomposed into the product of two low-rank matrices: W i ≈U i V i T

[0117] in, and r is the rank of decomposition, i∈[1,A], m and n represent the input feature dimension and output feature dimension of the network respectively; represents the set of real numbers, i.e. the matrix dimension; Indicates U i is a matrix of dimension m×r Indicates that V i is a matrix of dimension n×r;

[0118] D2. Initialize U i and V i Thus, the initialized pedestrian detection neural network model is obtained, and then the combined image data is input as input data into the pedestrian detection neural network model to obtain a predicted pedestrian image block as a predicted label;

[0119] D3, using the pedestrian image block in the combined image data as the true label and the predicted label to calculate the loss function, and adjusting the U in the pedestrian detection neural network model by the loss function i and V i .

[0120] That is, the solution in the present invention fine-tunes the general pedestrian detection neural network model based on the low-rank adaptation (LoRA) method, and achieves rapid model iteration optimization and deployment by reducing the model parameter updates while maintaining or improving the model performance. The specific process of LoRA fine-tuning training pedestrian detection neural network model is as follows: Since the several layers of the neural network model close to the output layer are more sensitive to specific tasks, the solution of the present invention selects several layers of the model close to the output layer for fine-tuning. Specifically, for the selected weight matrix W∈R m×n , (where m is the input feature dimension and n is the output feature dimension), decompose it into the product of two low-rank matrices:

[0121] W≈UV T

[0122] where U∈R m×r and V∈R n×r , and r<<min(m,n), where r is the rank of decomposition, which determines the number of parameters after decomposition and the complexity of the model.

[0123] In the actual training process, different methods can be used to initialize U and V, such as random initialization, SVD (singular value decomposition) initialization based on the original W, etc. During the training process, the present invention fixes all parameters except U and V in the pre-trained model, and only optimizes U and V. The training goal is to minimize the loss function L, which usually includes the classification error and possible regularization terms:

[0124]

[0125] Where L is the loss function (such as cross entropy loss), X is the input data, y is the label, and W init is the initial value of W, ‖·‖ F is the Frobenius norm and λ is the regularization strength.

[0126] The pedestrian detection neural network model is fine-tuned and trained based on LoRA to obtain model A1; if the effect of pedestrian detection by model A1 in actual scenes does not meet expectations, a new model can be obtained by repeating steps S2-S4 based on model A1.

[0127] S5. Detect the combined image data using the trained pedestrian detection neural network model.

[0128] Figure 6 The effect of the pedestrian detection neural network model is compared with the effect of the model obtained by the solution of the present invention (the detected pedestrian is marked with a green box). Obviously, the model based on the solution of the present invention has better effect.

[0129] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An adaptive person detection method for complex indoor and outdoor environments, characterized in that: include: S1, acquiring video stream data at different light intensities, and dividing the image into background frames without pedestrians and foreground frames with pedestrians at different light intensities according to the optical flow field of each frame image in the video stream data; S2, using a pedestrian detection neural network model combined with a pedestrian trajectory detection algorithm to perform pedestrian target detection on all the foreground frames, and correspondingly obtain pedestrian image blocks representing pedestrian instances in the foreground frames; S3, pasting the pedestrian image blocks belonging to the same pedestrian in the foreground frames at different time points into different background frames according to the set spatiotemporal constraints, thereby obtaining a plurality of combined image data of different illumination intensities satisfying the scale constraints, wherein one background frame can be pasted with multiple pedestrians; S4, fine-tuning the parameters of the pedestrian detection neural network model based on the LoRA technology, and inputting the plurality of combined image data into the fine-tuned pedestrian detection neural network model, and judging whether the pedestrian detection neural network model satisfies the scene adaptability through the loss function, if so, jumping to S5, otherwise, returning to S2; S5. Detect the combined image data using the trained pedestrian detection neural network model.

2. The adaptive personnel detection method for complex indoor and outdoor environments according to claim 1, characterized in that: According to the optical flow field of each frame image in the video stream data, the image is divided into a background frame without pedestrians and a foreground frame with pedestrians. f The specific process is: A1. Obtain video stream data under different light intensities, and divide the video stream data into a plurality of frame images according to time points; A2, using the neuFlow V2 algorithm model to perform optical flow estimation on the images of the frames to obtain optical flow fields of the images of the frames; A3, the optical flow field is divided into H levels according to the intensity, wherein the image with the optical flow field less than or equal to level h1 is defined as the background frame, and the image with the optical flow field greater than or equal to level h2 is defined as the foreground frame, wherein h1 <h2<H。 3. The adaptive personnel detection method for complex indoor and outdoor environments according to claim 1, characterized in that: The pedestrian detection neural network model is combined with the pedestrian trajectory detection algorithm to detect the foreground frame I f Perform pedestrian target detection, and obtain the corresponding foreground frame I f The specific process of the rectangular box representing the pedestrian instance is as follows: B1, obtaining a number of foreground frames at consecutive time points in the video stream data; B2. Using a pedestrian detection neural network model, respectively perform pedestrian target detection on a plurality of the foreground frames to obtain an initial rectangular frame representing a pedestrian instance; B3, cropping the portion corresponding to the initial rectangular frame in the foreground frame to obtain a corresponding pedestrian image block; B4. According to the pedestrian's spatiotemporal constraints, the pedestrian image block is subjected to noise reduction processing to obtain a more reliable pedestrian image block.

4. The adaptive personnel detection method for complex indoor and outdoor environments according to claim 1 or 3, characterized in that: The pedestrian detection neural network model is the efficientDet model.

5. The adaptive personnel detection method for complex indoor and outdoor environments according to claim 1, characterized in that: For one of the foreground frames, the specific process of pasting the pedestrian image block in the foreground frame into the background frame is: C1. Randomly selecting M pedestrian image blocks from the N pedestrian image blocks obtained from the foreground frame as pasting objects; C2, pasting the M pasted objects into a grayscale image, wherein the size of the grayscale image is consistent with the size of the foreground frame, and the mask in the grayscale image is set to: α=1 at the position with the final rectangular frame, and α=0 at the position without the final rectangular frame; C3. Cut out the pixels at the position corresponding to α=1 in the grayscale image, and paste them into the background frame, thereby forming combined image data.

6. The adaptive personnel detection method for complex indoor and outdoor environments according to claim 1, characterized in that: The time and space constraints are specifically: Scaling is performed according to the size of the pedestrian image block in the original foreground frame and the scaling ratio of the corresponding position in the background frame where the pedestrian image block needs to be pasted, wherein the relative position of the pedestrian image block in the original foreground frame is consistent with the relative position in the background frame where the pedestrian image block needs to be pasted.

7. The adaptive personnel detection method for complex indoor and outdoor environments according to claim 6, characterized in that: The pedestrian image block is scaled in proportion and pasted into the background frame to form a new pedestrian image block. The height and width of the new pedestrian image block are respectively expressed as: h new =s×h orig In new =s×w orig Among them, h new 、w new They represent the height and width of the new pedestrian image block, s is the actual scaling ratio, and h orig 、w orig They represent the height and width of the pedestrian image block in the original foreground frame respectively.

8. The adaptive personnel detection method for complex indoor and outdoor environments according to claim 7, characterized in that: The pasting position of the pedestrian image block in the background frame to be pasted is specifically expressed as: Among them, (x new ,y new ) represents the center position coordinates of the pedestrian image block in the background frame, (x orig ,y orig ) represents the median position coordinate of the pedestrian image block in the foreground frame, and Δx and Δy are the offsets of the corresponding positions in the background frame respectively.

9. The adaptive personnel detection method for complex indoor and outdoor environments according to claim 7, characterized in that: The specific formula of the actual scaling ratio s is: Where p is the adjustment factor, d img is the depth of the pedestrian image block, d ref is the reference depth, s ref The base scaling factor.

10. The adaptive personnel detection method for complex indoor and outdoor environments according to claim 1, characterized in that: Fine-tuning the parameters of the pedestrian detection neural network model specifically involves fine-tuning the A layer network close to the output layer of the pedestrian detection neural network model, where A is a positive integer. The specific process of training the pedestrian detection neural network model is as follows: D1. For each network in the A-layer network, the weight matrix W of each network is i ∈R n×b Decomposed into the product of two low-rank matrices: in, and r is the rank of decomposition, i∈[1,A], m and n represent the input feature dimension and output feature dimension of the network respectively; represents the set of real numbers, i.e. the matrix dimension; This means that U i is a matrix of dimension m×r Indicates that V i is a matrix of dimension n×r; D2. Initialize U i and V i Thus, the initialized pedestrian detection neural network model is obtained, and then the combined image data is input as input data into the pedestrian detection neural network model to obtain a predicted pedestrian image block as a predicted label; D3, using the pedestrian image block in the combined image data as the true label and the predicted label to calculate the loss function, and adjusting the U in the pedestrian detection neural network model by the loss function i and V i .

Images