A multi-target visual recognition method and system in a dynamic scene

Abstract

The application belongs to the technical field of visual identification, and provides a multi-target visual identification method and system in a dynamic scene, which comprises the following steps: collecting multi-target identification images in real time in a dynamic scene, arranging the multi-target identification images into a multi-target time sequence image sequence, constructing a background model of the multi-target identification images, combining dynamic background interference compensation based on optical flow field calculation, generating a moving target mask matrix, and generating a target candidate frame, intercepting an image in the target candidate frame, marking the image as a moving target image, constructing and training a multi-target visual identification model with a channel space double-attention module, identifying the moving target image, generating an information list of moving targets in the moving target image according to the identification result, taking the moving targets as nodes, constructing a space-time graph model containing the nodes and edges, realizing time sequence moving target correlation and judging whether a broken node appears, and if a broken node appears, constructing a conditional diffusion model to generate a virtual sample for matching connected nodes of the broken node.

Description

Technical Field

[0001] The present invention belongs to the field of visual recognition technology, and in particular to a multi-target visual recognition method and system in a dynamic scene. Background Art

[0002] In many fields such as modern intelligent monitoring, autonomous driving, and drone inspection, multi-target visual recognition technology in dynamic scenes is crucial. With the growing demand for intelligence and automation in various industries, accurately and efficiently identifying multiple targets in dynamic scenes has become a key issue that needs to be solved urgently.

[0003] In dynamic scenes, there are many complex factors that make multi-target visual recognition face severe challenges. Existing multi-target visual recognition methods have exposed many shortcomings when dealing with these complex situations. Traditional background difference-based methods often have relatively simple background model construction and are difficult to accurately adapt to the complex changes in the background in dynamic scenes. As a result, moving targets are easily misjudged as background during the background update process, or the background cannot be updated in time to accurately detect the target, resulting in target loss or misdetection. For example, in some traffic monitoring videos, when there are dynamic interferences such as shaking leaves in the background, existing multi-target visual recognition methods often misjudge them as moving targets.

[0004] In terms of target detection, the performance of traditional methods is significantly reduced when dealing with complex scenarios such as occlusion, uneven lighting, and dynamic background changes. For the occlusion problem, once the target is partially obscured, detection algorithms based on feature matching or region segmentation are easily unable to accurately identify the target due to missing features or incomplete regions. In the case of uneven lighting, such as in the transition area between indoors and outdoors, the rapid changes in light intensity and color will make it difficult to extract the target's features, resulting in reduced detection accuracy. When the background changes dynamically, such as when the camera moves or there is a large area of ​​background object movement in the scene, traditional detection algorithms are likely to misjudge background changes as target movement, resulting in a large number of false detections.

[0005] In the target recognition link, existing methods have shortcomings in feature extraction and classification. They have poor adaptability to the diversity and changes of targets in complex scenes, and it is difficult to extract effective features that are sufficient to distinguish different targets. There is a lack of effective enhancement of key target features and suppression mechanisms for background interference. In complex dynamic scenes, its recognition accuracy and robustness are still unsatisfactory.

[0006] In response to the above problems, the present invention proposes a multi-target visual recognition method and system in dynamic scenes. Summary of the Invention

[0007] In order to make up for the deficiencies of the prior art, at least one technical problem raised in the background technology is solved.

[0008] The technical scheme adopted by the present application to solve the technical problem is: a multi-target visual recognition method in a dynamic scene, comprising:

[0009] Real-time collection of multi-target recognition images in a dynamic scene, and arrangement into a multi-target time sequence image sequence;

[0010] Based on the multi-target time sequence image sequence, a background model of the multi-target recognition image is constructed, and a dynamic background interference compensation based on optical flow field calculation is combined to generate a moving target mask matrix;

[0011] According to the moving target mask matrix, a target candidate box is generated, the image in the target candidate box is intercepted, and is marked as a moving target image, a multi-target visual recognition model introducing a channel space double attention module is constructed and trained, the moving target image is recognized, and an information list of the moving target in the moving target image is generated according to the recognition result;

[0012] The moving target recognized in the multi-target time sequence image sequence is taken as a node, a space-time graph model containing nodes and edges is constructed based on the information list of the moving target, the time sequence moving target is associated, and it is judged whether a broken node appears, if a broken node appears, a conditional diffusion model is constructed to generate a virtual sample to match the connected nodes of the broken node;

[0013] The generation method of the moving target mask matrix is:

[0014] The multi-target time sequence image sequence is obtained, an improved Gaussian mixture model is used to construct a background model of the multi-target recognition image, and the background model contains a preset number of Gaussian distributions;

[0015] Based on the background model, preliminary target detection is performed on the current multi-target recognition image, if the pixel value of the image pixel coordinate in the multi-target recognition image does not match all Gaussian distributions in the background model, the image pixel coordinate is judged as a moving target candidate point, otherwise it is judged as a background pixel, all image pixel coordinates in the multi-target recognition image are traversed, a preliminary foreground mask is generated based on the judgment result, and the preliminary foreground mask is dynamically compensated for background interference to obtain a moving target mask matrix;

[0016] The construction method of the background model is:

[0017] For each image pixel coordinate in the multi-target recognition image, a preset K Gaussian distribution is used to fit the pixel value change of the image pixel coordinate in the multi-target time sequence image sequence, and the Gaussian distribution parameters are calculated;

[0018] The multi-target time sequence image sequence is divided into an initialization subsequence and an analysis subsequence, the initialization subsequence is used to initialize Gaussian distribution parameters, and for pixel values of image pixel coordinates in multi-target recognition images in the analysis subsequence, matching conditions of the pixel values and each Gaussian distribution are judged, if the pixel values match any Gaussian distribution, Gaussian distribution parameters are updated, and if the Gaussian distribution is not matched, only the weight in the Gaussian distribution parameters is updated;

[0019] All multi-target recognition images in the analysis subsequence are traversed, Gaussian distributions are sorted based on the Gaussian distribution parameters, and a preset number of Gaussian distributions are selected as the background model based on the sorting;

[0020] The manner of dynamically compensating the preliminary foreground mask for background interference is:

[0021] Optical flow field calculation is performed on the current multi-target recognition image and the adjacent multi-target recognition image to obtain the optical flow field of the multi-target recognition image, an affine transformation matrix is estimated from the optical flow field by using a random sample consensus algorithm, and the background model is reversely transformed by using the obtained affine transformation matrix to obtain a compensated background image;

[0022] Difference is performed on the current multi-target recognition image and the compensated background image, and a difference value of each image pixel coordinate is calculated, for a moving target candidate point in the preliminary foreground mask, if the difference value of the image pixel coordinate corresponding to the moving target candidate point is less than a preset difference threshold, the image pixel coordinate is corrected as a background pixel, and dynamic background interference compensation is realized on the preliminary foreground mask;

[0023] The manner of obtaining the information list of the moving target is:

[0024] The information list of the moving target includes a target category, a feature vector, a bounding box center coordinate and a corresponding multi-target recognition image timestamp of the moving target;

[0025] The multi-target visual recognition model is used to recognize the moving target image to obtain the target category and the feature vector of the moving target, and an image pixel coordinate of a midpoint in a target candidate box corresponding to the moving target image in the multi-target recognition image is obtained and marked as the bounding box center coordinate of the moving target;

[0026] The specific manner of obtaining the feature vector is:

[0027] The multi-target visual recognition model includes an image processing submodel and a visual recognition submodel, the moving target image is input into the image processing submodel to output a recognizable image, the recognizable image is input into the visual recognition submodel, the visual recognition submodel includes a basic network and a channel-spatial dual attention module, the channel-spatial dual attention module includes a channel attention submodule and a spatial attention submodule;

[0028] The identifiable image is processed by the basis network to obtain a basis feature map, the channel attention submodule processes the basis feature map, calculates a channel attention weight, and processes the basis feature map by weighting to obtain a channel weighted feature map, the spatial attention submodule processes the basis feature map, calculates a spatial attention weight, and processes the channel weighted feature map by weighting to obtain a reinforced feature map, and the reinforced feature map is globally averaged pooled to compress the reinforced feature map to obtain a feature vector;

[0029] The spatio-temporal graph model is constructed in the following manner:

[0030] The spatio-temporal graph model is constructed by taking the moving targets recognized in the multi-target time-series image sequence as nodes, and the node attribute of each node contains a list of information of the corresponding moving target, the list of information of the moving target including the target category, the feature vector, the bounding box center coordinate and the corresponding multi-target recognition image timestamp, an edge is established between nodes that are time-series adjacent, and the attribute of each edge contains an edge weight, and if the edge weight is less than a preset weight standard, the edge is removed;

[0031] The edge weight is calculated in the following manner:

[0032] The nodes at both ends of the edge are respectively marked as a front node and a rear node according to the time sequence, the Kalman filtering algorithm is used to predict the corresponding moving target of the front node, the predicted bounding box center coordinate at the next timestamp is obtained, and the predicted bounding box center coordinate and the bounding box center coordinate of the corresponding moving target of the rear node are processed to calculate the predicted similarity of the edge;

[0033] The feature vectors of the corresponding moving targets of the nodes at both ends of the edge are obtained, and the cosine similarity between the feature vectors is calculated, and the cosine similarity obtained is the feature similarity of the edge, and the data is fused to obtain the edge weight in combination with the feature similarity of the edge;

[0034] The virtual sample is obtained in the following manner:

[0035] A large number of moving target sample images with occlusion or motion blur and corresponding normal sample images are collected, and a diffusion training set is obtained by sorting, and a conditional training model is constructed and trained using the diffusion training set;

[0036] The feature vector of the corresponding moving target of the broken node is obtained, the predicted bounding box center coordinate of the corresponding moving target at the current timestamp is obtained, the feature vector and the predicted bounding box center coordinate are input into the trained conditional diffusion model as conditions to perform conditional diffusion, and a virtual sample is generated.

[0037] A multi-target visual recognition system in a dynamic scene includes the following modules:

[0038] The acquisition module: real-time acquisition of multi-target recognition images in a dynamic scene, and arrangement into a multi-target time sequence image sequence;

[0039] The background separation module: based on the multi-target time sequence image sequence, a background model of the multi-target recognition image is constructed, a dynamic background interference compensation based on a light flow field calculation is combined, and a moving target mask matrix is generated;

[0040] The target recognition module: a target candidate frame is generated according to the moving target mask matrix, an image in the target candidate frame is intercepted, and is marked as a moving target image; a multi-target visual recognition model introducing a channel space double attention module is constructed and trained, the moving target image is recognized, and an information list of the moving target in the moving target image is generated according to the recognition result;

[0041] The correlation recognition module: the moving target recognized in the multi-target time sequence image sequence is taken as a node, a space-time graph model containing nodes and edges is constructed based on the information list of the moving target, time sequence moving target correlation is realized, and whether a broken node appears is judged; if a broken node appears, a conditional diffusion model is constructed to generate a virtual sample to match a connected node.

[0042] The beneficial effects of the present application are as follows:

[0043] 1. The present application can accurately generate a moving target mask matrix by real-time acquisition of multi-target recognition images and arrangement into a time sequence, dynamic background interference compensation by light flow field calculation, effectively eliminate background interference, improve target detection accuracy, lay a foundation for subsequent accurate recognition of moving targets, make the recognition process more targeted, greatly improve the efficiency and precision of multi-target recognition in a complex dynamic scene.

[0044] 2. The present application constructs and trains a multi-target visual recognition model introducing a channel space double attention module, can strengthen the model's ability to capture key features, further optimize the recognition effect, generate an accurate moving target information list, at the same time, construct a space-time graph model to realize time sequence moving target correlation, can clearly present the target motion trajectory, when a broken node appears, generate a virtual sample to match a connected node through a conditional diffusion model, guarantee the integrity of target correlation, provide a reliable guarantee for continuous and stable recognition of multi-target in a dynamic scene. BRIEF DESCRIPTION OF DRAWINGS

[0045] The present application will be further described below with reference to the accompanying drawings.

[0046] Figure 1 is a step flow chart of a multi-target visual recognition method in a dynamic scene according to an embodiment of the present application;

[0047] Figure 2A module architecture diagram of a multi-target visual recognition system in a dynamic scene according to an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the technical means, creative features, purposes and effects of the present application easy to understand, the present application is further described below in conjunction with specific embodiments.

[0049] Embodiment 1

[0050] Please refer to Figure 1 A multi-target visual recognition method in a dynamic scene according to an embodiment of the present application, comprising the following steps:

[0051] S1: Real-time collection of multi-target recognition images in a dynamic scene, and arrangement into a multi-target time sequence image sequence;

[0052] An industrial-grade high-speed camera is selected, core parameters are set for the industrial-grade high-speed camera, the core parameters include resolution, pixel size, frame rate and exposure time, three-dimensional calibration is performed on the industrial-grade high-speed camera, a checkerboard calibration method is adopted, a plurality of calibration board images in different postures are collected, image pixel coordinates of each pixel point of the calibration board in the calibration board image are obtained, data processing is performed in combination with corresponding world coordinate system coordinates, an intrinsic matrix of the industrial-grade high-speed camera is calculated, and a distortion coefficient is obtained by solving in combination with the intrinsic matrix through minimization of re-projection error;

[0053] It should be noted that the intrinsic matrix and the distortion coefficient are inherent parameters of the camera and belong to the intrinsic properties of the camera. The intrinsic matrix describes the characteristics of the optical system of the camera, including focal length and image principal point coordinates. The distortion coefficient reflects the optical distortion characteristics of the camera lens, including radial distortion coefficient and tangential distortion coefficient, which are caused by hardware factors such as lens manufacturing process;

[0054] The multi-target recognition images are collected in real time by the industrial-grade high-speed camera in the dynamic scene, the original image data collected is transmitted to the industrial computer through a gigabit Ethernet interface, a frame synchronization triggering mechanism is adopted to ensure stable time interval of adjacent frames;

[0055] The collected original image data is preprocessed in real time. Specifically, the distortion coefficient is used to correct the distortion of the original image data, the distorted image pixel coordinates in the original image data are corrected to ideal non-distorted image pixel coordinates, the bilateral filtering is used to remove the salt and pepper noise of the original image data, and the adaptive histogram equalization is used to enhance the gray difference between the multi-target and the background, thereby avoiding the loss of multi-target caused by uneven illumination;

[0056] The original image data preprocessed in real time is marked as a multi-target recognition image, a tracking recognition period ending at the current timestamp is set, and the multi-target recognition images in the tracking recognition period are arranged in ascending order based on the collection timestamp to form a multi-target time sequence image sequence S;

[0057] ;

[0058] wherein T represents the total number of timestamps in the tracking recognition period, T also represents the current timestamp since the current timestamp is the end point of the tracking recognition period, and t represents a timestamp in the tracking recognition period, , the corresponding actual time is , denotes an adjacent frame time interval, denotes a multi-target recognition image corresponding to the timestamp t in the tracking recognition period, denotes a multi-target recognition image corresponding to the current timestamp;

[0059] It should be noted that the purpose of this step is to complete image collection and preprocessing to provide high-quality multi-target time sequence image sequences for subsequent recognition. The intrinsic matrix and distortion coefficient are obtained through camera calibration, the image distortion is corrected, the image quality is improved through filtering, enhancement and other preprocessing, and finally the multi-target time sequence image sequence sorted by time is formed. The image distortion caused by lens hardware defects is eliminated, the noise and uneven illumination effects are reduced, and the target information in the image is ensured to be accurate and complete, thereby laying a reliable data foundation for subsequent target detection and recognition;

[0060] S2: Based on the multi-target time sequence image sequence, a background model of the multi-target recognition image is constructed, and a dynamic background interference compensation based on the calculation of the optical flow field is combined to generate a motion target mask matrix;

[0061] Based on the multi-target time sequence image sequence, an improved Gaussian Mixture Model (GMM) is used to construct a background model of the multi-target recognition image;

[0062] Specifically, for each image pixel coordinate (x, y) in the multi-target recognition image , K=5 Gaussian distributions are used to fit the pixel value changes of the image pixel coordinates in the multi-target time sequence image sequence, and the Gaussian distribution parameters are as follows:

[0063] ;

[0064] wherein is the weight of the kth Gaussian distribution, is the mean value of the kth Gaussian distribution at the image pixel coordinate (x, y), is the variance of the kth Gaussian distribution at the image pixel coordinate (x, y);

[0065] Taking a starting point of the multi-target time sequence image sequence as a starting point, an initialization sub-sequence is intercepted in the multi-target time sequence image sequence, a part of the multi-target time sequence image sequence excluding the initialization sub-sequence is marked as an analysis sub-sequence, Gaussian distribution parameters are initialized by using multi-target recognition images in the initialization sub-sequence, and for multi-target recognition images in the analysis sub-sequence , a pixel value of an image pixel coordinate (x, y) in the multi-target recognition image is calculated , a matching degree of the pixel value with each Gaussian distribution is calculated, and if the matching degree satisfies

[0066] ;

[0067] , it is considered that the pixel value matches the kth Gaussian distribution, wherein denotes a standard deviation of the kth Gaussian distribution at the image pixel coordinate (x, y);

[0068] if the pixel value matches the kth Gaussian distribution, Gaussian distribution parameters are updated as

[0069] ;

[0070] ;

[0071] ;

[0072] , wherein is a learning rate, is a weight update coefficient;

[0073] if the pixel value matches the kth Gaussian distribution, the weight is updated as

[0074] ;

[0075] all Gaussian distributions are sorted in descending order according to sizes of the weights, and the first m Gaussian distributions with weights greater than a preset weight standard are selected as the background model;

[0076] , a multi-target recognition image corresponding to a current timestamp is preliminarily detected based on the background model, if a pixel value of an image pixel coordinate (x, y) in the multi-target recognition image does not match all Gaussian distributions in the background model, the image pixel coordinate is determined as a moving target candidate point, and is recorded as , otherwise, the image pixel coordinate is determined as a background pixel, and is recorded as ; ;

[0077] the multi-target recognition image ​​all image pixel coordinates to obtain a preliminary foreground mask;

[0078] It should be noted that the preliminary foreground mask can contain dynamic background interference, and therefore the dynamic background interference needs to be eliminated;

[0079] The pyramid Lucas-Kanade optical flow algorithm is used to calculate the optical flow field of the multi-target recognition image corresponding to the current timestamp and the adjacent multi-target recognition image , estimate the affine transformation matrix M from the optical flow field by using the random sample consensus algorithm, and perform reverse transformation on the background model by using the obtained affine transformation matrix M to obtain a compensated background image ;

[0080] The multi-target recognition image and the compensated background image are differentiated to calculate the difference value of each image pixel coordinate ;

[0081] ;

[0082] The preliminary foreground mask is compensated for dynamic background interference based on the calculated difference value. For the image pixel coordinates in the preliminary foreground mask , if the difference value of the image pixel coordinates is less than a preset difference threshold, the image pixel coordinates are modified to background pixels, and the modification ;

[0083] All image pixel coordinates are traversed to compensate the preliminary foreground mask for dynamic background interference to obtain a motion target mask matrix ;

[0084] ;

[0085] wherein, represents the resolution of the multi-target recognition image;

[0086] It should be noted that the purpose of this step is to construct a background model and separate the motion target from the dynamic background. The pixel value change is fitted by using the improved Gaussian mixture model to generate a preliminary foreground mask, and then the motion compensation is performed by combining the optical flow field and the affine transformation matrix to obtain an accurate motion target mask matrix, which effectively distinguishes the motion target from the dynamic background, reduces the interference of the dynamic background on target detection, improves the accuracy and robustness of motion target detection, adopts K=5 Gaussian distributions to construct the background model, processes in stages by using the initialization subsequence and the analysis subsequence, so that the model can better adapt to the background change, combines the optical flow field calculation with the affine transformation matrix estimation for motion compensation, specifically eliminates the dynamic background interference, and improves the foreground mask accuracy.

[0087] S3: generating a target candidate box according to the motion target mask matrix, intercepting an image in the target candidate box, marking the image as a motion target image, constructing and training a multi-target visual recognition model introducing a channel-spatial double attention module, recognizing the motion target image, and generating an information list of the motion target in the motion target image according to a recognition result;

[0088] Based on the obtained motion target mask matrix, an improved Selective Search algorithm is used to generate a plurality of target candidate boxes, which represent the bounding rectangles of each motion target in the multi-target recognition image. The image pixel coordinates of the center point of the target candidate box in the multi-target recognition image are obtained, and the center coordinates of the bounding box of the motion target are marked. In the multi-target recognition image, the image in the target candidate box is intercepted and marked as a motion target image.

[0089] The channel-spatial double attention module is introduced, and a multi-target visual recognition model is constructed and trained to recognize the motion target image. The multi-target visual recognition model includes an image processing sub-model and a visual recognition sub-model.

[0090] Specifically, the image processing sub-model includes an input layer, a preprocessing layer, and an output layer. The input layer receives an image. In the preprocessing layer, a bicubic interpolation method is used to standardize the size of the received image, and data augmentation including random horizontal flipping, brightness adjustment, and contrast adjustment is performed. The image after size standardization and data augmentation is normalized by normalizing the pixel values of the RGB three channels of the image to the range of [-1, 1], and an identifiable image is output.

[0091] The input of the visual recognition sub-model is the identifiable image, and the output is a feature vector and a target class. ResNet-50 is selected as the base network, which contains 5 convolutional blocks. The residual unit connection is used to solve the gradient vanishing problem during the training of the deep network. The identifiable image is subjected to convolution operation to obtain a base feature map. A channel-spatial double attention module is added after the last convolutional block of the base network. The channel-spatial double attention module includes a channel attention submodule and a spatial attention submodule.

[0092] The channel attention submodule performs global average pooling operation on the base feature map to obtain a channel description vector. The channel description vector is input into a neural network containing two fully connected layers. The first fully connected layer uses a ReLU activation function, and the output dimension is C / r, where C represents the number of channels and r represents the compression ratio. The second fully connected layer restores the dimension to C and uses a sigmoid activation function to obtain a channel attention weight. The base feature map is weighted based on the channel attention weight to obtain a channel weighted feature map.

[0093] The spatial attention sub-module respectively performs maximum pooling and average pooling on the basis feature map along the channel dimension to obtain two feature maps, splices the two feature maps along the channel dimension, and inputs the spliced feature map into a convolution layer for convolution operation. After the convolution operation, the number of channels is 1, and a sigmoid activation function is used to obtain a spatial attention weight. The channel weighted feature map is weighted based on the spatial attention weight to obtain a reinforced feature map;

[0094] It should be noted that the reinforced feature map has the effect of reinforcing the target key features and suppressing background interference.

[0095] The reinforced feature map is globally averaged pooled to compress the reinforced feature map into a feature vector.

[0096] A large number of multi-target sample images of a marked target class are collected in an actual dynamic scene, covering scenes of different weather, different light, different time period, and different target state. A multi-target sample set is obtained by sorting. The multi-target sample set is input into the image processing sub-model to obtain a recognizable image set of the marked target class. The recognizable image set of the marked target class is divided into a training set and a verification set according to a ratio of 8:2. The training set is used to train the visual recognition sub-model. The total number of training rounds is set. A cross-entropy loss function is used as a loss function. A stochastic gradient descent is used as an optimizer. The performance of the visual recognition sub-model is evaluated using the verification set after each training round. When the accuracy of the verification set no longer improves between consecutive rounds, or when the number of training rounds reaches the preset total number of training rounds, it is determined that the training of the visual recognition sub-model is completed.

[0097] The moving target image is input into the multi-target visual recognition model. The moving target image is output as a recognizable image by the visual recognition sub-model. The feature vector and the target class of the moving target image are output by the visual recognition sub-model.

[0098] For each moving target, an information list including the target class, the feature vector, the bounding box center coordinates, and the corresponding multi-target recognition image timestamp is generated.

[0099] It should be noted that this step has the effect of generating a target candidate box and completing target recognition. The list containing the target class, the feature vector, and other information is output. The target candidate box is extracted based on the moving target mask matrix. The multi-target visual recognition model with the channel spatial double attention module is used to recognize the target image to obtain the feature vector and the target class. The moving target is accurately positioned and its class is recognized. The extracted feature vector can effectively represent the target appearance feature, providing a reliable basis for subsequent cross-frame target association, improving the accuracy of target recognition and the feature representation capability. The channel spatial double attention module is introduced to strengthen the target key features and suppress the background interference, enhance the discriminability of the feature map, and improve the adaptability of the model to different scenes by using the preprocessing method combining bicubic interpolation, multi-dimensional data enhancement, and normalization.

[0100] S4: taking the moving targets identified in the multi-target time sequence image sequence as nodes, constructing a space-time graph model containing the nodes and edges, realizing time sequence moving target association and judging whether a broken node appears, if so, constructing a conditional diffusion model to generate a virtual sample for the broken node to match the connected nodes;

[0101] Taking the moving targets identified in the multi-target time sequence image sequence as nodes, constructing a space-time graph model, the node attribute of each node containing the information list of the corresponding moving target, establishing an edge between the nodes adjacent in time sequence, the attribute of each edge containing an edge weight, the edge weight being used to measure the possibility of the nodes at both ends of the edge belonging to the same moving target, if the edge weight is greater than or equal to a preset weight standard, judging that the nodes at both ends of the edge belong to the same moving target, otherwise judging that the nodes at both ends of the edge do not belong to the same moving target, and removing the edge;

[0102] The edge weight is calculated by weighted fusion of the predicted similarity of the edge and the feature similarity of the edge;

[0103] The calculation method of the predicted similarity of the edge is as follows:

[0104] For the nodes at both ends of the edge, the node earlier in time sequence is marked as a front node, and the node later in time sequence is marked as a rear node, the Kalman filtering algorithm is used to predict the corresponding moving target of the front node, the predicted bounding box center coordinates at the next time stamp are obtained, the Euclidean distance d between the predicted bounding box center coordinates and the bounding box center coordinates of the corresponding moving target of the rear node is calculated, and the predicted similarity of the edge is calculated based on the Euclidean distance d ;

[0105] ;

[0106] The smaller the Euclidean distance is, the greater the predicted similarity is;

[0107] The feature similarity of the edge is calculated by obtaining the feature vectors of the corresponding moving targets of the nodes at both ends of the edge, and calculating the cosine similarity between the feature vectors, and the calculated cosine similarity is the feature similarity of the edge;

[0108] The time sequence moving target association is realized based on the space-time graph model, the nodes connected by the edges are classified into a unified node group, and the node group is sorted according to time sequence to obtain a time sequence node sequence;

[0109] If there is no edge connected between the node in the current time stamp and any node in the previous time stamp, the node is marked as a to-be-analyzed rear node;

[0110] If there is no edge between the node in the previous timestamp and any node in the current timestamp, the node is marked as a broken node, the time sequence node sequence of the broken node is obtained, and the Kalman filtering algorithm is used to predict the motion target corresponding to the time sequence node sequence to obtain the predicted bounding box center coordinates of the motion target corresponding to the current timestamp;

[0111] The feature vector and the bounding box center coordinates of the motion target are taken as conditions to construct and train a conditional diffusion model;

[0112] Specifically, a large number of motion target sample images with occlusion or motion blur are collected in an actual dynamic scene, and normal sample images corresponding to the motion target without occlusion or motion blur are collected, and a diffusion training set is obtained by sorting. The diffusion training set is used to train the constructed conditional training model, so that the conditional diffusion model learns the noise parameters when the motion target appears occlusion and motion blur;

[0113] Based on any broken node, a virtual sample is generated for the broken node to match the connected nodes using the conditional diffusion model;

[0114] Specifically, the feature vector of the motion target corresponding to the broken node is obtained, the predicted bounding box center coordinates of the motion target corresponding to the current timestamp are obtained, the feature vector and the predicted bounding box center coordinates are taken as conditions to input the trained conditional diffusion model for conditional diffusion, and a plurality of virtual samples are generated. The virtual sample includes a virtual feature vector and a virtual bounding box center coordinate. For any virtual sample, the virtual feature vector and the virtual bounding box center coordinate are sorted and merged into a virtual similarity analysis vector;

[0115] All nodes to be analyzed after the current timestamp are obtained. For any node to be analyzed after, the feature vector and the bounding box center coordinates of the motion target corresponding to the node to be analyzed after are obtained, the feature vector and the bounding box center coordinates are sorted and merged into an actual similarity analysis vector, the cosine similarity between the actual similarity analysis vector and any virtual similarity analysis vector is calculated, if the cosine similarity is greater than a preset similarity threshold, it is determined that the node to be analyzed after and the broken node belong to the same motion target, and an edge is established between the node to be analyzed after and the broken node to connect them;

[0116] All nodes to be analyzed are traversed. If none of the nodes to be analyzed after and the broken nodes belong to the same motion target, it is determined that the motion target is lost, and the time sequence node sequence of the broken node is marked as the target motion trajectory corresponding to the motion target is stored;

[0117] For the node to be analyzed, all broken nodes are traversed. If there is no node to be analyzed and any broken node that belongs to the same motion target, the node to be analyzed is marked as a starting node, and it is determined that the motion target corresponding to the starting node is a new motion target;

[0118] It should be noted that the role of this step is to build a spatio-temporal graph model to realize the correlation and trajectory breakage repair of the time sequence motion target, to build a spatio-temporal graph with the motion target as the node, to realize the target correlation through the edge weight calculation, to generate a virtual sample for matching by using a conditional diffusion model for the trajectory breakage, to perfect the target motion trajectory, to realize the correlation of the same target between different frames, to solve the problem of trajectory breakage caused by occlusion, motion blur and the like, to ensure the continuity of the target and the integrity of the trajectory, to provide complete trajectory data for subsequent tasks such as target behavior analysis, to calculate the edge weight by weighted fusion of the motion prediction similarity and the feature similarity, to comprehensively consider the target motion continuity and the appearance consistency, to improve the correlation accuracy, and to generate a virtual sample for trajectory breakage repair by using a conditional diffusion model, to enhance the adaptability to complex scenes by combining double matching of virtual and real samples.

[0119] The technical scheme of the embodiment of the application is: real-time acquisition of multi-target recognition images in a dynamic scene, arrangement into a multi-target time sequence image sequence, construction of a background model of the multi-target recognition images based on the multi-target time sequence image sequence, generation of a motion target mask matrix in combination with dynamic background interference compensation based on optical flow field calculation, generation of a target candidate frame according to the motion target mask matrix, interception of images in the target candidate frame, marking as motion target images, construction and training of a multi-target visual recognition model introducing a channel space double attention module, recognition of the motion target images, generation of an information list of the motion targets in the motion target images according to the recognition result, construction of a spatio-temporal graph model containing nodes and edges with the motion targets recognized in the multi-target time sequence image sequence as the nodes, realization of the correlation of the time sequence motion targets and judgment of whether a broken node appears, and if so, generation of a virtual sample for matching the connected nodes for the broken node by constructing a conditional diffusion model.

[0120] Embodiment 2

[0121] As shown in Figure 2 A multi-target visual recognition system in a dynamic scene according to the embodiment of the application includes the following modules:

[0122] The acquisition module: real-time acquisition of multi-target recognition images in a dynamic scene, arrangement into a multi-target time sequence image sequence;

[0123] The background separation module: construction of a background model of the multi-target recognition images based on the multi-target time sequence image sequence, generation of a motion target mask matrix in combination with dynamic background interference compensation based on optical flow field calculation;

[0124] The target recognition module: generation of a target candidate frame according to the motion target mask matrix, interception of images in the target candidate frame, marking as motion target images, construction and training of a multi-target visual recognition model introducing a channel space double attention module, recognition of the motion target images, generation of an information list of the motion targets in the motion target images according to the recognition result;

[0125] Correlation recognition module: the moving target identified in the multi-target time sequence image sequence as a node, based on the information list of the moving target, a space-time graph model containing nodes and edges is constructed, the correlation of the time sequence moving target is realized and it is judged whether a broken node appears, if it appears, a conditional diffusion model is constructed to generate a virtual sample to match the connected nodes for the broken node.

[0126] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A multi-target visual recognition method in a dynamic scene, characterized by: include: Collect multi-target recognition images in real time in dynamic scenes and organize them into multi-target time-series image sequences; Based on the multi-target time-series image sequence, a background model of the multi-target recognition image is constructed, and the moving target mask matrix is ​​generated by combining the dynamic background interference compensation based on the optical flow field calculation; The moving target mask matrix is ​​generated as follows: Acquire a multi-target time-series image sequence, and use an improved Gaussian mixture model to construct a background model for the multi-target recognition image. The background model includes a preset number of Gaussian distributions. Perform preliminary target detection on the current multi-target recognition image based on the background model. If the pixel value of the image pixel coordinate in the multi-target recognition image does not match all Gaussian distributions in the background model, the image pixel coordinate is judged to be a moving target candidate point. Otherwise, it is judged to be a background pixel. All image pixel coordinates in the multi-target recognition image are traversed, and a preliminary foreground mask is generated based on the judgment result. The preliminary foreground mask is then compensated for dynamic background interference to obtain a moving target mask matrix. Generate target candidate frames based on the moving target mask matrix, capture the image within the target candidate frame, mark it as a moving target image, build and train a multi-target visual recognition model that introduces a channel-space dual-attention module, recognize the moving target image, and generate a list of moving target information in the moving target image based on the recognition results; Taking the moving targets identified in the multi-target time-series image sequence as nodes, a spatiotemporal graph model containing nodes and edges is constructed based on the information list of the moving targets to realize the association of time-series moving targets and determine whether there are broken nodes. If so, a conditional diffusion model is constructed to generate virtual samples to match the connected nodes for the broken nodes.

2. The multi-target visual recognition method in a dynamic scene according to claim 1, characterized in that: The background model is constructed as follows: For each image pixel coordinate in the multi-target recognition image, the preset K Gaussian distributions are used to fit the pixel value changes of the image pixel coordinates in the multi-target time series image sequence, and the Gaussian distribution parameters are calculated; The multi-target time-series image sequence is divided into an initialization subsequence and an analysis subsequence. The initialization subsequence is used to initialize the Gaussian distribution parameters. In the analysis subsequence, for the pixel value of the image pixel coordinate in the multi-target recognition image, the matching status of the pixel value and each Gaussian distribution is judged. If the pixel value matches any Gaussian distribution, the Gaussian distribution parameters are updated. If it does not match the Gaussian distribution, only the weights in the Gaussian distribution parameters are updated. All multi-target recognition images in the subsequence are traversed and analyzed, Gaussian distributions are sorted based on Gaussian distribution parameters, and a preset number of Gaussian distributions are selected as background models based on the sorting.

3. The multi-target visual recognition method in a dynamic scene according to claim 1, characterized in that: The method of performing dynamic background interference compensation on the preliminary foreground mask is: The optical flow field of the current multi-target recognition image and the adjacent multi-target recognition images is calculated to obtain the optical flow field of the multi-target recognition image. The affine transformation matrix is ​​estimated from the optical flow field using the random sampling consistency algorithm. The obtained affine transformation matrix is ​​used to perform inverse transformation on the background model to obtain the compensated background image. The current multi-target recognition image is differentiated from the compensated background image, and the differential value of each image pixel coordinate is calculated. For the moving target candidate point in the preliminary foreground mask, if the differential value of the image pixel coordinate corresponding to the moving target candidate point is less than the preset differential threshold, the image pixel coordinate is corrected to the background pixel to achieve dynamic background interference compensation for the preliminary foreground mask.

4. The multi-target visual recognition method in a dynamic scene according to claim 1, characterized in that: The method for obtaining the information list of the moving target is: The information list of moving targets includes the target category, feature vector, bounding box center coordinates and corresponding multi-target recognition image timestamp of the moving target; The multi-target visual recognition model is used to identify the moving target image, and the target category and feature vector of the moving target are obtained. The image pixel coordinates of the midpoint of the target candidate box corresponding to the moving target image in the multi-target recognition image are obtained and marked as the center coordinates of the bounding box of the moving target.

5. The multi-target visual recognition method in a dynamic scene according to claim 4, characterized in that: The specific method of obtaining the feature vector is: The multi-target visual recognition model includes an image processing sub-model and a visual recognition sub-model. The moving target image is input into the image processing sub-model, which outputs a recognizable image. The recognizable image is then input into the visual recognition sub-model. The visual recognition sub-model includes a basic network and a channel-space dual attention module. The channel-space dual attention module includes a channel attention sub-module and a spatial attention sub-module. After the recognizable image is processed by the basic network, a basic feature map is obtained. The channel attention submodule processes the basic feature map, calculates the channel attention weight, and performs weighted processing on the basic feature map to obtain a channel weighted feature map. The spatial attention submodule processes the basic feature map, calculates the spatial attention weight, and performs weighted processing on the channel weighted feature map to obtain an enhanced feature map. The enhanced feature map is globally average pooled and compressed to obtain a feature vector.

6. The multi-target visual recognition method in a dynamic scene according to claim 1, characterized in that: The spatiotemporal graph model is constructed as follows: A spatiotemporal graph model is constructed using the moving targets identified in a multi-target time-series image sequence as nodes. The node attributes of each node contain an information list of the corresponding moving targets. The information list of the moving targets includes the target category, feature vector, bounding box center coordinates and the corresponding multi-target recognition image timestamp. Edges are established between adjacent nodes in the time sequence. The attributes of each edge include the edge weight. If the edge weight is less than the preset weight standard, the edge will be cleared.

7. The multi-target visual recognition method in a dynamic scene according to claim 6, characterized in that: The edge weight is calculated as follows: The nodes at both ends of the edge are marked as the front node and the back node according to the time sequence. The Kalman filter algorithm is used to predict the moving target corresponding to the front node. The center coordinates of the predicted bounding box at the next timestamp are obtained. The center coordinates of the predicted bounding box and the center coordinates of the bounding box of the moving target corresponding to the back node are processed to calculate the predicted similarity of the edge. The feature vectors of the moving targets corresponding to the nodes at both ends of the edge are obtained, and the cosine similarity between the feature vectors is calculated. The calculated cosine similarity is the feature similarity of the edge. The edge weight is obtained by data fusion combined with the feature similarity of the edge.

8. The multi-target visual recognition method in a dynamic scene according to claim 7, characterized in that: The virtual sample is obtained as follows: Collect a large number of moving target sample images with occlusion or motion blur and the corresponding normal sample images, organize them to obtain a diffusion training set, build a conditional training model and use the diffusion training set for training; The feature vector of the moving target corresponding to the fracture node is obtained, and the center coordinates of the predicted bounding box corresponding to the moving target at the current timestamp are obtained. The feature vector and the center coordinates of the predicted bounding box are used as conditional inputs to the trained conditional diffusion model for conditional diffusion to generate virtual samples.

9. A multi-target visual recognition system in a dynamic scene, characterized by: Includes the following modules: Acquisition module: collects multi-target recognition images in real time in dynamic scenes and organizes them into multi-target time-series image sequences; Background separation module: Based on the multi-target time-series image sequence, it builds a background model for multi-target recognition images, combines dynamic background interference compensation based on optical flow field calculation, and generates a moving target mask matrix; The moving target mask matrix is ​​generated as follows: Acquire a multi-target time-series image sequence, and use an improved Gaussian mixture model to construct a background model for the multi-target recognition image. The background model includes a preset number of Gaussian distributions. Perform preliminary target detection on the current multi-target recognition image based on the background model. If the pixel value of the image pixel coordinate in the multi-target recognition image does not match all Gaussian distributions in the background model, the image pixel coordinate is judged to be a moving target candidate point. Otherwise, it is judged to be a background pixel. All image pixel coordinates in the multi-target recognition image are traversed, and a preliminary foreground mask is generated based on the judgment result. The preliminary foreground mask is then compensated for dynamic background interference to obtain a moving target mask matrix. Target recognition module: Generates target candidate frames based on the moving target mask matrix, captures the image within the target candidate frame, marks it as a moving target image, builds and trains a multi-target visual recognition model that introduces a channel-space dual-attention module, recognizes the moving target image, and generates a list of moving target information in the moving target image based on the recognition results; Association recognition module: Using the moving targets identified in the multi-target time-series image sequence as nodes, based on the information list of the moving targets, a spatiotemporal graph model containing nodes and edges is constructed to realize the association of time-series moving targets and determine whether broken nodes appear. If so, a conditional diffusion model is constructed to generate virtual samples to match the broken nodes with connected nodes.

Images