System for improving applicability of intelligent camera

Abstract

The invention discloses a system for improving the applicability of an intelligent camera, and relates to the technical field of intelligent camera shooting. RGB, infrared and depth sensors are used for cooperative sensing, a dynamic inter-frame interpolation algorithm is used for complementing video data, target trajectory prediction is carried out based on a target motion vector analysis algorithm in combination with a recurrent neural network LSTM + Kalman filtering, and the applicability of the intelligent camera is improved. Meanwhile, three-dimensional scene modeling and shielding recovery are introduced, and environment information complementation of a dead angle area is optimized; through a cross-camera target identity re-identification ReID algorithm, seamless tracking is carried out among multiple cameras by utilizing appearance characteristics, gait characteristics and biological characteristics of a target, and meanwhile, PTZ camera control is optimized in combination with reinforcement learning, and the angles of the cameras are dynamically adjusted; a generative adversarial network GAN based on space-time consistency is adopted, historical video data and environment modeling are combined, missing pictures in visual dead angles are intelligently complemented, and the continuity and authenticity of monitoring pictures are improved.

Description

Technical Field

[0001] The invention relates to the technical field of intelligent camera technology, and in particular to a system for improving the applicability of intelligent cameras. Background Art

[0002] Smart cameras are widely used in security, traffic monitoring, smart home and other fields. Their core goal is to achieve all-round monitoring, improve target recognition accuracy, and promptly warn of abnormal events.

[0003] To this end, traditional smart cameras usually improve monitoring capabilities by increasing camera pixels, increasing camera installation density, introducing PTZ (pan-tilt rotation) cameras, and combining AI algorithms. However, in actual applications, these solutions still have obvious limitations when dealing with blind spots in monitoring. In situations such as building corners, vehicle shadows, dense crowds, and low-light environments, the monitored target can easily temporarily leave the field of vision, causing picture breakpoints, thereby affecting the continuity of monitoring and the integrity of event analysis.

[0004] In scenes such as building corners, aisles between shelves, and blind spots of subway station escalators, the fixed viewing angle of the camera cannot fully cover the target. Once the target enters the blocked area, the monitoring system cannot continue to track it. In crowded places such as airports, shopping malls, stadiums, or fast-moving targets, smart cameras may miss key frames due to frame rate limitations and limited computing resources, causing the target to disappear temporarily, making subsequent tracking difficult. In addition, in the edge areas of the monitoring screen or long-distance targets, such as when high-altitude cameras monitor pedestrians on the ground, the AI ​​recognition accuracy decreases due to the decrease in resolution, making it difficult to accurately identify and complete the target's motion trajectory. Therefore, there is an urgent need for a more efficient system to complete visual blind spot data and improve the applicability of smart cameras. Summary of the invention

[0005] In view of the above existing problems, the present invention is proposed.

[0006] The present invention provides a system solution to improve the applicability of smart cameras. Traditional solutions rely on multiple cameras, PTZ control and AI recognition, but still have limitations in blind spot tracking, target completion and low-resolution recognition.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] The embodiment of the present invention provides a system for improving the applicability of a smart camera, which includes:

[0009] The front-end data acquisition module is used to acquire video data of the monitoring area through the collaboration of multiple cameras;

[0010] Intelligent processing and analysis module, which is used to analyze the fused video data based on the target motion vector analysis algorithm, extract the motion characteristics of the monitored target, and combine the recursive neural network LSTM and Kalman filter to predict the most likely trajectory of the target after it enters the visual blind spot;

[0011] The intelligent processing and analysis module further includes an environment modeling and occlusion recovery unit, which is used to perform structured modeling of the physical environment within the camera monitoring range based on three-dimensional scene reconstruction technology;

[0012] Real-time target tracking module, which is used to map target identities between cameras based on the cross-camera target identity re-identification (ReID) algorithm, combining the target's appearance features, gait features, and biometric features;

[0013] The real-time target tracking module also controls the PTZ camera in conjunction with a reinforcement learning algorithm to dynamically adjust the camera angle based on the predicted trajectory of the target;

[0014] The data completion optimization module is used to intelligently complete the missing images of the target in the visual blind spot using the generative adversarial network (GAN) based on spatiotemporal consistency;

[0015] The storage and backtracking module adopts an event-driven storage mechanism, which automatically triggers storage when the target enters a visual blind spot or reappears.

[0016] As a preferred solution of the system for improving the applicability of the intelligent camera described in the present invention, wherein: in the front-end data acquisition module, an RGB sensor, an infrared sensor and a depth sensor are additionally deployed, and the video data is additionally fused with the data of the RGB sensor, the infrared sensor and the depth sensor;

[0017] The front-end data acquisition module also uses a dynamic inter-frame interpolation algorithm to complete the missing frames in the video stream.

[0018] As a preferred solution of the system for improving the applicability of smart cameras described in the present invention, the application method of the system includes:

[0019] Step S1, using multiple cameras to collaboratively acquire multi-angle video data of the monitoring area, and fusing the data of the RGB sensor, infrared sensor, and depth sensor, and using a dynamic inter-frame interpolation algorithm to complete the missing frames in the video stream;

[0020] Step S2, using the target motion vector analysis algorithm to perform real-time analysis on the video data fused in step S1, extracting the motion features of the monitored target, and combining the recursive neural network LSTM and Kalman filter to predict the most likely trajectory of the target after it enters the visual blind spot;

[0021] Perform environmental modeling and occlusion recovery, and structurally model the physical environment within the camera's monitoring range based on 3D scene reconstruction technology;

[0022] Step S3, using a cross-camera target identity re-identification (ReID) algorithm to map target identities between cameras based on the target's appearance features, gait features, and biometric features, and combining a reinforcement learning algorithm to control the PTZ camera and dynamically adjust the camera angle according to the predicted trajectory of the target in step S2;

[0023] The method of controlling the PTZ camera in combination with the reinforcement learning algorithm and dynamically adjusting the camera angle according to the predicted trajectory of the target includes:

[0024] Based on the DeepQLearning reinforcement learning algorithm, the camera angle is automatically adjusted.

[0025] Combined with the predicted trajectory, the optimal rotation path of the camera is calculated.

[0026] In a multi-camera environment, coordinate the adjustment of PTZ cameras;

[0027] Step S4, using a generative adversarial network (GAN) based on spatiotemporal consistency to intelligently complete the missing images of the target in the visual blind spot;

[0028] In step S4, the generative adversarial network GAN adopts a dual-stream adversarial structure, in which one generation path performs time series completion based on historical video data, and the other generation path performs background consistency optimization based on environmental modeling;

[0029] Step S5, using an event-driven storage mechanism, automatically triggering the storage mode when the target enters the visual blind spot or reappears.

[0030] As a preferred solution of the system for improving the applicability of smart cameras described in the present invention, in the data fusion process of step S1:

[0031] The RGB sensor is used to collect target color and texture information.

[0032] Infrared sensors are used for object detection in low light and at night,

[0033] Depth sensors are used to estimate the position of objects in three-dimensional space.

[0034] A spatiotemporal feature fusion algorithm is used to jointly calculate RGB, infrared and depth data.

[0035] As a preferred solution of the system for improving the applicability of smart cameras described in the present invention, in step S1, the dynamic inter-frame interpolation algorithm includes:

[0036] Based on the interpolation method of optical flow estimation, the motion trajectory between the previous and next frames is calculated.

[0037] The deep learning-based temporal frame completion model FLAVR predicts the content of missing frames.

[0038] Adopt an adaptive interpolation strategy to dynamically adjust the complement frame rate in high-motion scenes.

[0039] As a preferred solution of the system for improving the applicability of smart cameras described in the present invention, the step of using a dynamic inter-frame interpolation algorithm to complete the missing frames in the video stream is as follows:

[0040] Set the video stream frame sequence to F:

[0041] F={I 1 ,I 2 ,…,I n},

[0042] Among them, I t Represents the frame of the video stream at time t, n is the total number of video frames,

[0043] If a missing frame is detected, m , then through the adjacent frame I m-1 and I m+1 Perform interpolation completion, the completion formula is:

[0044]

[0045] in, represents the interpolated frame, f interp Represents the interpolation function, the input is the adjacent frame I m-1 and I m+1 ,

[0046] The RAFT optical flow model is used to calculate the inter-frame motion information. m At , the RAFT optical flow network is used to calculate the inter-frame motion information, and the calculation formula is:

[0047] V m =f RAFT (I m-1 ,I m+1 ),

[0048] Among them, V m Missing frame I m The optical flow field at f RAFT Optical flow network for recursive feature extraction,

[0049] Combined with the FLAVR interpolation network for frame completion, the missing frames are calculated based on the FLAVR interpolation network. The calculation formula is:

[0050]

[0051] Among them, f FLAVR Represents the FLAVR interpolation network based on 3D convolution, with the input being adjacent frames I m-1 ,I m+1 And the optical flow information V m ,

[0052] In high-motion scenes, the interpolation strategy is dynamically adjusted, and the adjustment formula is:

[0053] r=αΣ p |V m (p)|,

[0054] Among them, r is the interpolation rate adjustment factor, α is the adjustment parameter, p is the pixel point, |V m (p)| is the optical flow amplitude at pixel p.

[0055] As a preferred solution of the system for improving the applicability of smart cameras described in the present invention, the method of using the target motion vector analysis algorithm to perform real-time analysis on video data includes:

[0056] The long short-term memory (LSTM) neural network is used to predict the target trajectory.

[0057] Combined with Kalman filter algorithm to correct the error of possible target trajectory;

[0058] The method of performing environment modeling and occlusion recovery is as follows:

[0059] Build a physical environment model based on 3D point cloud,

[0060] Neural radiation field (NeRF) technology is used to reconstruct the monitoring images.

[0061] Combined with the historical movement pattern of the target, the probability distribution is used to infer its possible movement direction.

[0062] As a preferred solution of the system for improving the applicability of the intelligent camera described in the present invention, the step of using the target motion vector analysis algorithm to perform real-time analysis on the video data fused in step S1, extracting the motion characteristics of the monitored target, and combining the recursive neural network LSTM and Kalman filter to predict the most likely trajectory of the target after entering the visual blind spot is as follows:

[0063] Define the target motion vector, and assume that the motion state of the target at time t is M t :

[0064]

[0065] Among them, xt ,y t Indicates the coordinates of the target in the monitoring area, Indicates the speed of the target in the x,y direction,

[0066] A two-layer LSTM network is used to model the target motion trajectory. The two-layer LSTM structure includes:

[0067]

[0068] in, is the hidden state of the first layer of LSTM, is the hidden state of the second layer LSTM, and are the previous hidden states of the first and second layers, respectively, W h ,b h is the neural network weight parameter, represents the predicted target state, f LSTM1 represents the state update function of the first layer LSTM, M t-1 represents the target motion state at the previous moment, f LSTM2 Represents the state update function of the second layer LSTM;

[0069] The Kalman filter is used to correct the predicted trajectory. The Kalman filter state update formula is:

[0070] M t+1 =AM t +w t ,

[0071] z t+1 =HM t+1 +v t ,

[0072] Among them, A is the state transfer matrix, H is the observation matrix, and w t ,v t is process noise and observation noise, z t+1 is the observed value,

[0073] Calculate the Kalman gain K t , the calculation formula is:

[0074] K t =P t H T (HP t H T +R) -1 ,

[0075] Correct the trajectory, the correction formula is:

[0076]

[0077] Among them, P t is the prediction covariance matrix, R is the measurement noise covariance matrix, K t is the Kalman gain.

[0078] As a preferred solution of the system for improving the applicability of smart cameras described in the present invention, the steps of using a cross-camera target identity re-identification ReID algorithm to map target identities between cameras based on the target's appearance features, gait features and biometric features, and combining a reinforcement learning algorithm to control the PTZ camera are as follows:

[0079] Construct a feature vector based on ResNet-50, expressed as:

[0080] F t =f ResNet50 (I t ),

[0081] Among them, I t is the input image frame, F t is the feature descriptor of the target, f ResNet50 is the ResNet-50 deep neural network,

[0082] Metric learning is used to calculate feature similarity S(F t ,F j ), the calculation formula is:

[0083]

[0084] Among them, S is the cosine similarity, F j is the target feature vector in the database, |F t | and |F j | respectively represent F t and F j The L2 norm of

[0085] Optimize PTZ camera control based on DeepQ-Learning. The optimization process includes:

[0086] Define the learning state as s t :

[0087] Define the action space as a t :a t ∈{rotation, tilt, zoom},

[0088] Perform Q-Learning update, the update formula is:

[0089]

[0090] Among them, Q(st ,a t ) is state s t and action a t Q value, r t is the reward value, γ is the discount factor, a' represents the best action among all possible actions that may be performed at the next moment, s t+1 Indicates the target state at the next moment.

[0091] As a preferred solution of the system for improving the applicability of smart cameras described in the present invention, the step of using a generative adversarial network GAN based on spatiotemporal consistency to intelligently complete the missing images of the target in the visual blind spot is:

[0092] Using a two-stream generative adversarial network GAN, the GAN loss function is:

[0093]

[0094] in, represents the GAN loss function, Represents the distribution of real data p data The expectation of the sampled data x in (x), represents the expected value of the probability that a real sample is judged as real, D(x) represents the probability that the discriminator D predicts the authenticity of sample x, represents the noise distribution p z (z) is the expectation of the sampling noise z, log(1-D(G(z))) represents the logarithm of the probability that a false sample generated by the generator is judged as false by the discriminator,

[0095] GAN completes the path, and the completion formula is:

[0096]

[0097] in, represents the missing frame generated by GAN, I m-1 ,I m+1 Represent the previous frame and the next frame respectively, G is the generator, D is the discriminator, M t Represents the motion state of the target at time t.

[0098] The beneficial effects of the present invention are as follows: the present invention utilizes RGB, infrared and depth sensors for collaborative perception, and adopts a dynamic inter-frame interpolation algorithm to complete video data. Based on the target motion vector analysis algorithm, it combines the recursive neural network LSTM+Kalman filter to predict the target trajectory, and introduces three-dimensional scene modeling and occlusion recovery to optimize the environmental information completion of blind spots. Through the cross-camera target identity re-identification ReID algorithm, the target's appearance features, gait features and biological features are utilized to perform seamless tracking between multiple cameras. At the same time, it combines reinforcement learning to optimize PTZ camera control, dynamically adjust the camera angle, and reduce the target loss rate.

[0099] The present invention adopts a generative adversarial network (GAN) based on spatiotemporal consistency, combines historical video data and environmental modeling, intelligently completes missing images in visual blind spots, and improves the continuity and authenticity of monitoring images; and adopts an event-driven storage mechanism to automatically trigger high-precision storage when a target enters a visual blind spot or reappears.

[0100] In summary, the present invention realizes the full-process accurate monitoring of the target and improves the adaptability and data processing capabilities of the monitoring system in complex environments, providing a more efficient and intelligent monitoring solution for scenarios such as smart security, smart cities, and unmanned driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0102] Figure 1 A schematic diagram of the framework of the system for improving the applicability of smart cameras according to the present invention. DETAILED DESCRIPTION

[0103] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0104] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0105] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0106] Example 1, reference Figure 1 , this embodiment provides a system for improving the applicability of a smart camera, including:

[0107] The front-end data acquisition module is used to acquire video data of the monitoring area through the collaboration of multiple cameras;

[0108] In the front-end data acquisition module, RGB sensors, infrared sensors and depth sensors are additionally deployed, and the video data is additionally fused with the data of RGB sensors, infrared sensors and depth sensors;

[0109] The front-end data acquisition module also uses a dynamic inter-frame interpolation algorithm to complete the missing frames in the video stream;

[0110] Intelligent processing and analysis module, which is used to analyze the fused video data based on the target motion vector analysis algorithm, extract the motion characteristics of the monitored target, and combine the recursive neural network LSTM and Kalman filter to predict the most likely trajectory of the target after it enters the visual blind spot;

[0111] The intelligent processing and analysis module also includes an environment modeling and occlusion recovery unit, which is used to perform structured modeling of the physical environment within the camera monitoring range based on 3D scene reconstruction technology;

[0112] Real-time target tracking module, which is used to map target identities between cameras based on the cross-camera target identity re-identification (ReID) algorithm, combining the target's appearance features, gait features, and biometric features;

[0113] The real-time target tracking module also combines reinforcement learning algorithms to control the PTZ camera, dynamically adjusting the camera angle based on the predicted trajectory of the target;

[0114] The data completion optimization module is used to intelligently complete the missing images of the target in the visual blind spot using the generative adversarial network (GAN) based on spatiotemporal consistency;

[0115] The storage and backtracking module adopts an event-driven storage mechanism, which automatically triggers storage when the target enters a visual blind spot or reappears.

[0116] This embodiment also provides the above-mentioned system application method for improving the applicability of the smart camera, including:

[0117] Step S1, using multiple cameras to collaboratively acquire multi-angle video data of the monitoring area, and fusing the data of the RGB sensor, infrared sensor, and depth sensor, and using a dynamic inter-frame interpolation algorithm to complete the missing frames in the video stream;

[0118] During the data fusion process of step S1:

[0119] The RGB sensor is used to collect target color and texture information.

[0120] Infrared sensors are used for object detection in low light and at night,

[0121] Depth sensors are used to estimate the position of objects in three-dimensional space.

[0122] Adopting spatiotemporal feature fusion algorithm to jointly calculate RGB, infrared and depth data;

[0123] In step S1, the dynamic frame interpolation algorithm includes:

[0124] Based on the interpolation method of optical flow estimation, the motion trajectory between the previous and next frames is calculated.

[0125] The deep learning-based temporal frame completion model FLAVR predicts the content of missing frames.

[0126] Adopting adaptive interpolation strategy to dynamically adjust the frame rate in high motion scenes;

[0127] The steps of using dynamic inter-frame interpolation algorithm to complete the missing frames in the video stream are:

[0128] Set the video stream frame sequence to F:

[0129] F={I 1 ,I 2 ,…,I n},

[0130] Among them, I t Represents the frame of the video stream at time t, n is the total number of video frames,

[0131] If a missing frame is detected, m , then through the adjacent frame I m-1 and I m+1 Perform interpolation completion, the completion formula is:

[0132]

[0133] in, represents the interpolated frame, f interp Represents the interpolation function, the input is the adjacent frame I m-1 and I m+1 ,

[0134] The RAFT optical flow model is used to calculate the inter-frame motion information. m At , the RAFT optical flow network is used to calculate the inter-frame motion information, and the calculation formula is:

[0135] V m =f RAFT (I m-1 ,I m+1 ),

[0136] Among them, V m Missing frame I m The optical flow field at f RAFT Optical flow network for recursive feature extraction,

[0137] Combined with the FLAVR interpolation network for frame completion, the missing frames are calculated based on the FLAVR interpolation network. The calculation formula is:

[0138]

[0139] Among them, f FLAVR Represents the FLAVR interpolation network based on 3D convolution, with the input being adjacent frames I m-1 ,I m+1 And the optical flow information V m ,

[0140] In high-motion scenes, the interpolation strategy is dynamically adjusted, and the adjustment formula is:

[0141] r=α∑ p |V m (p)|,

[0142] Among them, r is the interpolation rate adjustment factor, α is the adjustment parameter, p is the pixel point, |V m (p)| is the optical flow amplitude at pixel p,

[0143] Specifically, in this step, for missing frame completion in dynamic scenes, the RAFT optical flow model is combined to calculate motion information, and the FLAVR interpolation network is used for temporal interpolation to ensure the consistency of the interpolated frames and the video content; in addition, an adaptive interpolation strategy is used to adapt to high-speed motion scenes to enhance the accuracy and robustness of completion;

[0144] Step S2, using the target motion vector analysis algorithm to perform real-time analysis on the video data fused in step S1, extracting the motion features of the monitored target, and combining the recursive neural network LSTM and Kalman filter to predict the most likely trajectory of the target after it enters the visual blind spot;

[0145] Perform environmental modeling and occlusion recovery, and structurally model the physical environment within the camera's monitoring range based on 3D scene reconstruction technology;

[0146] Methods for real-time analysis of video data using target motion vector analysis algorithms include:

[0147] The long short-term memory (LSTM) neural network is used to predict the target trajectory.

[0148] Combined with Kalman filter algorithm to correct the error of possible target trajectory;

[0149] The method for environment modeling and occlusion recovery is:

[0150] Build a physical environment model based on 3D point cloud,

[0151] Neural radiation field (NeRF) technology is used to reconstruct the monitoring images.

[0152] Combine the historical movement pattern of the target and use probability distribution to infer its possible movement direction;

[0153] The target motion vector analysis algorithm is used to analyze the video data fused in step S1 in real time, extract the motion characteristics of the monitored target, and combine the recursive neural network LSTM and Kalman filter to predict the most likely trajectory of the target after entering the visual blind spot.

[0154] Define the target motion vector, and assume that the motion state of the target at time t is M t :

[0155]

[0156] Among them, x t ,y t Indicates the coordinates of the target in the monitoring area, Indicates the speed of the target in the x,y direction,

[0157] A two-layer LSTM network is used to model the target motion trajectory. The two-layer LSTM structure includes:

[0158]

[0159] in, is the hidden state of the first layer of LSTM, is the hidden state of the second layer LSTM, and are the previous hidden states of the first and second layers, respectively, W h ,b h is the neural network weight parameter, represents the predicted target state, f LSTM1 represents the state update function of the first layer LSTM, M t-1 represents the target motion state at the previous moment, fLSTM2 Represents the state update function of the second layer LSTM;

[0160] The Kalman filter is used to correct the predicted trajectory. The Kalman filter state update formula is:

[0161] M t+1 =AM t +w t ,

[0162] z t+1 =HM t+1 +v t ,

[0163] Among them, A is the state transfer matrix, H is the observation matrix, and w t ,v t is process noise and observation noise, z t+1 is the observed value,

[0164] Calculate the Kalman gain K t , the calculation formula is:

[0165] K t =P t HT(HP t H T +R) -1 ,

[0166] Correct the trajectory, the correction formula is:

[0167]

[0168] Among them, P t is the prediction covariance matrix, R is the measurement noise covariance matrix, K t is the Kalman gain,

[0169] Specifically, a two-layer LSTM network is used here for trajectory prediction, and Kalman filtering is combined to correct trajectory deviation; LSTM captures the long-term dependent motion characteristics of the target, and Kalman filtering optimizes short-term errors to improve the accuracy and robustness of trajectory prediction;

[0170] Step S3, using a cross-camera target identity re-identification (ReID) algorithm to map target identities between cameras based on the target's appearance features, gait features, and biometric features, and combining a reinforcement learning algorithm to control the PTZ camera and dynamically adjust the camera angle according to the predicted trajectory of the target in step S2;

[0171] The method of controlling the PTZ camera by combining the reinforcement learning algorithm and dynamically adjusting the camera angle according to the predicted trajectory of the target includes:

[0172] Based on the DeepQLearning reinforcement learning algorithm, the camera angle is automatically adjusted.

[0173] Combined with the predicted trajectory, the optimal rotation path of the camera is calculated.

[0174] In a multi-camera environment, coordinate the adjustment of PTZ cameras;

[0175] The cross-camera target identity re-identification ReID algorithm is used to map the target identity between cameras based on the target's appearance features, gait features, and biometric features. At the same time, the steps of controlling the PTZ camera in combination with the reinforcement learning algorithm are as follows:

[0176] Construct a feature vector based on ResNet-50, expressed as:

[0177] F t =f ResNet50 (I t ),

[0178] Among them, I t is the input image frame, F t is the feature descriptor of the target, f ResNet50 is the ResNet-50 deep neural network,

[0179] Metric learning is used to calculate feature similarity S(F t ,F j ), the calculation formula is:

[0180]

[0181] Among them, S is the cosine similarity, F j is the target feature vector in the database, |F t | and |F j | respectively represent F t and F j The L2 norm of

[0182] Optimize PTZ camera control based on DeepQ-Learning. The optimization process includes:

[0183] Define the learning state as s t :

[0184] Define the action space as a t :a t ∈{rotation, tilt, zoom},

[0185] Perform Q-Learning update, the update formula is:

[0186]

[0187] Among them, Q(s t ,a t ) is state s t and action a t Q value, r t is the reward value, γ is the discount factor, a' represents the best action among all possible actions that may be performed at the next moment, s t+1 Indicates the target state at the next moment;

[0188] Specifically, this step uses ResNet-50 to re-identify the identity of targets across cameras, and uses DeepQ-Learning reinforcement learning to adjust PTZ camera parameters to improve continuous target tracking and camera collaboration capabilities;

[0189] Step S4, using a generative adversarial network (GAN) based on spatiotemporal consistency to intelligently complete the missing images of the target in the visual blind spot;

[0190] In step S4, the generative adversarial network (GAN) adopts a two-stream adversarial structure, in which one generation path performs time series completion based on historical video data, and the other path performs background consistency optimization based on environment modeling;

[0191] The steps of using the generative adversarial network GAN based on spatiotemporal consistency to intelligently complete the missing images of the target in the visual blind spot are as follows:

[0192] Using a two-stream generative adversarial network GAN, the GAN loss function is:

[0193]

[0194] in, represents the GAN loss function, Represents the distribution of real data p data The expectation of the sampled data x in (x), represents the expected value of the probability that a real sample is judged as real, D(x) represents the probability that the discriminator D predicts the authenticity of sample x, represents the noise distribution p z (z) is the expectation of the sampling noise z, log(1-D(G(z))) represents the logarithm of the probability that a false sample generated by the generator is judged as false by the discriminator,

[0195] GAN completes the path, and the completion formula is:

[0196]

[0197] in, represents the missing frame generated by GAN, I m-1 ,I m+1Represent the previous frame and the next frame respectively, G is the generator, D is the discriminator, M t represents the motion state of the target at time t;

[0198] Specifically, a two-stream GAN structure is used here to combine historical video data and environmental modeling information to improve the realism and spatial consistency of missing frame completion;

[0199] Step S5, using an event-driven storage mechanism, automatically triggering the storage mode when the target enters the visual blind spot or reappears.

[0200] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A system for improving the applicability of smart cameras, characterized by: include, The front-end data acquisition module is used to acquire video data of the monitoring area through the collaboration of multiple cameras; Intelligent processing and analysis module, which is used to analyze the fused video data based on the target motion vector analysis algorithm, extract the motion characteristics of the monitored target, and combine the recursive neural network LSTM and Kalman filter to predict the most likely trajectory of the target after it enters the visual blind spot; The intelligent processing and analysis module further includes an environment modeling and occlusion recovery unit, which is used to perform structured modeling of the physical environment within the camera monitoring range based on three-dimensional scene reconstruction technology; Real-time target tracking module, which is used to map target identities between cameras based on the cross-camera target identity re-identification (ReID) algorithm, combining the target's appearance features, gait features, and biometric features; The real-time target tracking module also controls the PTZ camera in conjunction with a reinforcement learning algorithm to dynamically adjust the camera angle based on the predicted trajectory of the target; The data completion optimization module is used to intelligently complete the missing images of the target in the visual blind spot using the generative adversarial network (GAN) based on spatiotemporal consistency; The storage and backtracking module adopts an event-driven storage mechanism, which automatically triggers storage when the target enters a visual blind spot or reappears.

2. A system for improving the applicability of a smart camera as claimed in claim 1, characterized in that: In the front-end data acquisition module, an RGB sensor, an infrared sensor and a depth sensor are additionally deployed, and the video data is additionally fused with the data of the RGB sensor, the infrared sensor and the depth sensor; The front-end data acquisition module also uses a dynamic inter-frame interpolation algorithm to complete the missing frames in the video stream.

3. A system for improving the applicability of smart cameras as claimed in claim 2, characterized in that: The application methods of this system include: Step S1, using multiple cameras to collaboratively acquire multi-angle video data of the monitoring area, and fusing the data of the RGB sensor, infrared sensor, and depth sensor, and using a dynamic inter-frame interpolation algorithm to complete the missing frames in the video stream; Step S2, using the target motion vector analysis algorithm to perform real-time analysis on the video data fused in step S1, extracting the motion features of the monitored target, and combining the recursive neural network LSTM and Kalman filter to predict the most likely trajectory of the target after it enters the visual blind spot; Perform environmental modeling and occlusion recovery, and structurally model the physical environment within the camera's monitoring range based on 3D scene reconstruction technology; Step S3, using a cross-camera target identity re-identification (ReID) algorithm to map target identities between cameras based on the target's appearance features, gait features, and biometric features, and combining a reinforcement learning algorithm to control the PTZ camera and dynamically adjust the camera angle according to the predicted trajectory of the target in step S2; The method of controlling the PTZ camera in combination with a reinforcement learning algorithm and dynamically adjusting the camera angle according to the predicted trajectory of the target includes: Based on the DeepQLearning reinforcement learning algorithm, the camera angle is automatically adjusted. Combined with the predicted trajectory, the optimal rotation path of the camera is calculated. In a multi-camera environment, coordinate the adjustment of PTZ cameras; Step S4, using a generative adversarial network (GAN) based on spatiotemporal consistency to intelligently complete the missing images of the target in the visual blind spot; In step S4, the generative adversarial network GAN adopts a dual-stream adversarial structure, in which one generation path performs time series completion based on historical video data, and the other generation path performs background consistency optimization based on environmental modeling; Step S5, using an event-driven storage mechanism, automatically triggering the storage mode when the target enters the visual blind spot or reappears.

4. A system for improving the applicability of a smart camera as claimed in claim 3, characterized in that: During the data fusion process of step S1: The RGB sensor is used to collect target color and texture information. Infrared sensors are used for object detection in low light and at night, Depth sensors are used to estimate the position of objects in three-dimensional space. A spatiotemporal feature fusion algorithm is used to jointly calculate RGB, infrared and depth data.

5. A system for improving the applicability of a smart camera as claimed in claim 4, characterized in that: In step S1, the dynamic inter-frame interpolation algorithm includes: Based on the interpolation method of optical flow estimation, the motion trajectory between the previous and next frames is calculated. The deep learning-based temporal frame completion model FLAVR predicts the content of missing frames. Adopt an adaptive interpolation strategy to dynamically adjust the complement frame rate in high-motion scenes.

6. A system for improving the applicability of a smart camera as claimed in claim 5, characterized in that: The step of using a dynamic inter-frame interpolation algorithm to complete the missing frames in the video stream is: Set the video stream frame sequence to F: F={I1,I2,...,I n }, Among them, I t Represents the frame of the video stream at time t, n is the total number of video frames, If a missing frame is detected, m , then through the adjacent frame I m-1 and I m+1 Perform interpolation completion, the completion formula is: in, represents the interpolated frame, f interp Represents the interpolation function, the input is the adjacent frame I m-1 and I m+1 , The RAFT optical flow model is used to calculate the inter-frame motion information. m At , the RAFT optical flow network is used to calculate the inter-frame motion information, and the calculation formula is: V m =f RAFT (I m-1 ,I m+1 ), Among them, V m Missing frame I m The optical flow field at f RAFT Optical flow network for recursive feature extraction, Combined with the FLAVR interpolation network for frame completion, the missing frames are calculated based on the FLAVR interpolation network. The calculation formula is: Among them, f FLAVR Represents the FLAVR interpolation network based on 3D convolution, with the input being adjacent frames I m-1 ,I m+1 And the optical flow information V m , In high-motion scenes, the interpolation strategy is dynamically adjusted, and the adjustment formula is: r=α∑p| V m (p)|, Among them, r is the interpolation rate adjustment factor, α is the adjustment parameter, p is the pixel point, | V m (p)| is the optical flow amplitude at pixel p.

7. A system for improving the applicability of a smart camera as claimed in claim 6, characterized in that: The method of using the target motion vector analysis algorithm to perform real-time analysis on video data includes: The long short-term memory (LSTM) neural network is used to predict the target trajectory. Combined with Kalman filter algorithm to correct the error of possible target trajectory; The method of performing environment modeling and occlusion recovery is as follows: Build a physical environment model based on 3D point cloud, Neural radiation field (NeRF) technology is used to reconstruct the monitoring images. Combined with the historical movement pattern of the target, the probability distribution is used to infer its possible movement direction.

8. A system for improving the applicability of a smart camera as claimed in claim 7, characterized in that: The step of using the target motion vector analysis algorithm to perform real-time analysis on the video data fused in step S1, extracting the motion features of the monitored target, and combining the recursive neural network LSTM and Kalman filter to predict the most likely trajectory of the target after entering the visual blind spot is as follows: Define the target motion vector, and assume that the motion state of the target at time t is M t : Among them, x t ,y t Indicates the coordinates of the target in the monitoring area, Indicates the speed of the target in the x and y directions. A two-layer LSTM network is used to model the target motion trajectory. The two-layer LSTM structure includes: in, is the hidden state of the first layer of LSTM, is the hidden state of the second layer LSTM, and are the previous hidden states of the first and second layers, respectively, W h , b h is the neural network weight parameter, represents the predicted target state, f LSTM1 represents the state update function of the first layer LSTM, M t-1 represents the target motion state at the previous moment, f LSTM2 Represents the state update function of the second layer LSTM; The Kalman filter is used to correct the predicted trajectory. The Kalman filter state update formula is: M t+1 =AM t +w t , With t+1 =HM t+1 +v t , Among them, A is the state transfer matrix, H is the observation matrix, and w t , v t is process noise and observation noise, z t+1 is the observed value, Calculate the Kalman gain K t , the calculation formula is: K t =P t H T (HP t H T +R) -1 , Correct the trajectory, the correction formula is: Among them, P t is the prediction covariance matrix, R is the measurement noise covariance matrix, K t is the Kalman gain.

9. A system for improving the applicability of a smart camera as claimed in claim 8, characterized in that: The steps of using a cross-camera target identity re-identification ReID algorithm to map target identities between cameras based on the target's appearance features, gait features, and biometric features, and combining a reinforcement learning algorithm to control the PTZ camera are as follows: Construct a feature vector based on ResNet-50, expressed as: F t =f ResNet50 (I t ), Among them, I t is the input image frame, F t is the feature descriptor of the target, f ResNet50 is the ResNet-50 deep neural network, Metric learning is used to calculate feature similarity S(F t , F j ), the calculation formula is: Among them, S is the cosine similarity, F j is the target feature vector in the database, |F t | and |F j | respectively represent F t and F j The L2 norm of Optimize PTZ camera control based on DeepQ-Learning. The optimization process includes: Define the learning state as s t : Define the action space as a t :a t ∈{rotation, tilt, zoom}, Perform Q-Learning update, the update formula is: Among them, Q(s t , a t ) is state s t and action a t Q value, r t is the reward value, γ is the discount factor, a′ represents the best action among all possible actions that may be performed at the next moment, S t+1 Indicates the target state at the next moment.

10. A system for improving the applicability of a smart camera as claimed in claim 9, characterized in that: The steps of using a generative adversarial network (GAN) based on spatiotemporal consistency to intelligently complete the missing images of the target in the visual blind spot are: Using a two-stream generative adversarial network GAN, the GAN loss function is: in, represents the GAN loss function, Represents the distribution of real data p data The expectation of the sampled data x in (x), represents the expected value of the probability that a real sample is judged as real, D(x) represents the probability that the discriminator D predicts the authenticity of sample x, represents the noise distribution p z (z) is the expectation of the sampling noise z, log(1-D(G(z))) represents the logarithm of the probability that a false sample generated by the generator is judged as false by the discriminator, GAN completes the path, and the completion formula is: in, represents the missing frame generated by GAN, I m-1 , I m+1 Represent the previous frame and the next frame respectively, G is the generator, D is the discriminator, M t Represents the motion state of the target at time t.

Images