Park intelligent behavior recognition image processing method based on large model

Abstract

The invention discloses a park intelligent behavior recognition image processing method based on a large model, and the method comprises the steps: obtaining a video stream containing a plurality of image frames, constructing a high-dimensional attribute matrix to represent the video stream, and enabling each dimension to represent different features extracted from a video; the method comprises the following steps of: executing park behavior identification by utilizing a large model of a plurality of cross-time resolution modules, automatically judging a splitting requirement of a high-dimensional attribute matrix, and splitting the high-dimensional attribute matrix into a first attribute matrix and a second attribute matrix which respectively comprise different types of feature dimensions; performing time downsampling on the second attribute matrix when necessary by intelligently adjusting the number of time pooling layers; the two attribute matrixes are processed in parallel, and spatial features and time features are extracted respectively; and outputting a prediction result about the park behavior. According to the invention, personnel behaviors, vehicle behaviors and abnormal behaviors can be accurately and efficiently identified, intelligent support is provided for park safety management, resource scheduling and abnormity monitoring, and park management efficiency and intelligent level are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method for park intelligent behavior recognition based on a large model. Background Art

[0002] With the rapid development of information technology, artificial intelligence and image processing technology, intelligent park management has gradually become an important part of modern urban management. The demand for intelligent behavior recognition, monitoring and management in parks is increasing, especially in security monitoring, resource scheduling, personnel management and environmental monitoring, which have become the focus of park managers. Image processing technology is one of the core supporting technologies for intelligent behavior recognition in parks, and its application areas cover personnel behavior recognition, vehicle monitoring and environmental detection. In order to improve the intelligence and automation level of park management, intelligent behavior recognition technology based on video image streams has received widespread attention and has been gradually applied to actual systems. However, existing methods for intelligent behavior recognition in parks still face many technical challenges.

[0003] Traditional image processing technology mainly relies on feature extraction, classification and recognition algorithms, such as traditional computer vision methods based on hand-designed features, including HOG (Histogram of Oriented Gradients) features, SIFT (Scale-Invariant Feature Transform), SURF

[0004] (Speeded-Up Robust Features), and template matching-based technologies. The notable features of these methods are that they require complex manual feature design, low computational efficiency during algorithm operation, and the recognition effect is easily affected by environmental changes, such as changes in light intensity, viewing angle, and occlusion. With the rapid rise of deep learning technology, especially the widespread application of convolutional neural networks (CNNs) and recurrent neural networks (RNNs), image recognition technology has made significant progress in recognition accuracy and speed, which is particularly prominent in behavior recognition and dynamic scene analysis.

[0005] However, despite the excellent performance of deep learning technology in image recognition, it still faces the problems of huge computational complexity and strong dependence on data in practical applications. In the actual application of intelligent behavior recognition in parks, video image streams are usually continuous and high-dimensional. Extracting effective features from them and performing efficient recognition while ensuring computational efficiency are still the difficulties and challenges of current technology.

[0006] The implementation of behavior recognition within the park requires real-time processing of a large amount of continuous video stream data, especially in large parks, where the amount of video data involved is huge and the structure is complex. The key to achieving real-time behavior recognition is to ensure recognition accuracy while avoiding calculation delays caused by excessive data volume. The extraction of spatiotemporal features of video streams is particularly important in this process. Spatiotemporal feature extraction involves a comprehensive analysis of spatial information (such as object shape, color, position, etc.) and temporal information (such as dynamic changes of objects, motion trajectories, etc.) within the video frame. Most existing methods use a single spatiotemporal feature extraction technology, which makes it difficult to simultaneously take into account efficient extraction of multi-dimensional spatiotemporal features and accurate behavior recognition.

[0007] In addition, as people, objects, and scenes in the video stream change, traditional methods are insufficient in dealing with changes in different scales and speeds. For example, in a park, the temporal and hierarchical characteristics of human behavior and vehicle motion are relatively complex, and traditional recognition methods have difficulty capturing multi-scale, multi-temporal resolution information, resulting in poor recognition of fine-grained behaviors. For example, in park monitoring, there are obvious behavioral differences between fast-moving targets and slow-moving targets (such as walking pedestrians and fast-moving vehicles), and existing methods have limited performance in distinguishing these behaviors. Summary of the invention

[0008] In order to solve the above problems in the prior art, the present invention proposes an image processing method for park intelligent behavior recognition based on a large model, the method comprising:

[0009] Get a video stream containing multiple image frames;

[0010] Constructing a high-dimensional attribute matrix to represent the received video stream, wherein the high-dimensional attribute matrix has multiple dimensions, wherein each dimension represents a different feature extracted from the campus video stream;

[0011] A large model with multiple cross-temporal resolution modules is used to perform park behavior recognition. The large model automatically determines the splitting requirements of the high-dimensional attribute matrix, and splits the high-dimensional attribute matrix according to the judgment result to obtain two independent attribute matrices, where:

[0012] The first attribute matrix contains a first set of dimensions;

[0013] The second attribute matrix contains a second set of dimensions;

[0014] The large model intelligently adjusts the number of time pooling layers of the second attribute matrix, and performs time downsampling only when necessary according to actual needs;

[0015] The large model processes the two attribute matrices in parallel, respectively processing the first attribute matrix and the second attribute matrix that has been time-downsampled;

[0016] Output prediction results about campus behavior in the received video stream.

[0017] The video stream is a dynamic video stream collected in real time by multiple cameras in the park, and the video stream is transmitted at a predetermined frame rate.

[0018] Each dimension of the high-dimensional attribute matrix represents a spatial feature, a temporal feature, or a combination of the two extracted from the video stream. The spatial features include the appearance, posture, color, and texture of the character, and the temporal features include the movement trajectory, speed, and behavior changes of the character.

[0019] The large model uses a deep convolutional neural network (CNN) to extract spatial features and a recurrent neural network (RNN) or a 3D convolutional neural network to process temporal features.

[0020] The split first attribute matrix contains spatial features related to human behavior, and the second attribute matrix contains motion features related to vehicle behavior.

[0021] The temporal pooling layer intelligently adjusts the number of poolings to flexibly adjust the compression degree of the temporal dimension according to the actual needs of the video content.

[0022] The parallel processing includes using a convolutional neural network (CNN) to process spatial features in the first attribute matrix and using a recurrent neural network (RNN) to process time series features in the second attribute matrix.

[0023] The output prediction results of the park behavior include the recognition results of the behavior of people in the park, the recognition results of the behavior of vehicles, and the detection results of abnormal behavior.

[0024] The behavior recognition results are used for decision support of the park management system, including security warning, resource scheduling, abnormal behavior monitoring, and personnel and equipment management.

[0025] The behavior recognition system further supports self-learning and optimization.

[0026] Beneficial effects:

[0027] The present invention realizes accurate behavior recognition of video streams by efficiently constructing a high-dimensional attribute matrix combined with the automatic judgment and dynamic adjustment capabilities of a large model. Through intelligent splitting and parallel processing of the high-dimensional attribute matrix, spatial features and temporal features can be extracted separately to improve recognition accuracy and processing efficiency. The number of time pooling layers is intelligently adjusted, and time downsampling is dynamically optimized according to the complexity of the behavior to ensure effective retention of information and reduction of computational burden. Ultimately, the present invention can quickly output park behavior prediction results, provide intelligent support for security monitoring, resource scheduling, and abnormal behavior detection, and significantly improve the level of intelligence in park management. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present application, but do not constitute an improper limitation of the present invention. In the drawings:

[0029] Figure 1 A diagram of the method steps of the present invention is shown. DETAILED DESCRIPTION

[0030] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments, wherein the illustrative embodiments and descriptions are only used to explain the present invention but are not intended to limit the present invention.

[0031] The present invention is described in detail below in conjunction with an embodiment. The image stream used in the embodiment is a dynamic video stream of the park captured by a video surveillance system in the park. Through an intelligent behavior recognition method based on video streams, the behavior characteristics of people, vehicles and other objects in the park can be effectively extracted from the video streams, and behavior recognition and prediction can be performed.

[0032] As attached Figure 1 As shown, this embodiment introduces in detail a method for image processing of intelligent behavior recognition in a park based on a high-dimensional attribute matrix and a cross-time resolution module. The method can accurately extract spatiotemporal features from the video stream of the park and perform behavior recognition through an intelligent behavior recognition model. It is suitable for a variety of scenarios such as park monitoring, personnel management, vehicle dispatching, and security monitoring.

[0033] Step 1: Get a video stream containing multiple image frames

[0034] In this embodiment, multiple high-resolution surveillance cameras are deployed in the park to capture the dynamic video stream in the park in real time. The shooting range of each camera covers different areas of the park to ensure comprehensive monitoring of all activities in the park. Each video stream consists of multiple continuous image frames, each frame of which corresponds to a specific time point and shows static or dynamic information of the park scene. The frame rate of the video stream is usually set to 30 frames per second (FPS), but it can be adjusted according to actual needs to ensure smooth and delay-free images to adapt to different park monitoring scenarios.

[0035] Each frame of the image may contain multiple moving targets, including but not limited to pedestrians, vehicles, object handling equipment, robots, etc. The behavior and movement of these targets in the park are the focus of the intelligent behavior recognition system of the present invention. In addition, the video stream may also contain additional information related to the environment, such as temperature and humidity, light intensity, noise level, etc. This information is important for understanding the overall environmental status and behavior patterns in the park. For example, at night or in a dimly lit environment, the image quality of the camera may be affected, which in turn affects the effect of target detection and behavior recognition. Therefore, the system needs to have powerful image enhancement and compensation technology to ensure accurate target recognition under various environmental conditions.

[0036] In practical applications, the video stream captured by the camera becomes the input data source of the intelligent behavior recognition system of the present invention after certain processing. The video stream provides rich spatiotemporal information for subsequent behavior analysis, covering the target's motion trajectory, speed, behavior changes and other characteristics, and is the basis for various behavior recognition and prediction tasks in the park. By acquiring these high-quality video stream data in real time, the system can respond to various activities in the park in a timely manner, thereby providing accurate and real-time decision support for park managers.

[0037] Step 2: Construct a high-dimensional attribute matrix to represent the video stream

[0038] After preprocessing, the acquired video stream is used as input data to construct a high-dimensional attribute matrix. This process aims to effectively organize the spatiotemporal information into a high-dimensional matrix by extracting the multi-dimensional features of each frame of the image for further behavior analysis and prediction.

[0039] First, each frame in the video stream needs to go through a preprocessing step, which usually includes denoising, image enhancement, and normalization. The purpose of these preprocessing steps is to improve the quality of the image and provide clearer and more representative image data for the subsequent feature extraction process. Image enhancement technology can be used to improve the image quality in low-light environments, while denoising can eliminate interference caused by poor shooting conditions (such as high-noise environments).

[0040] On the preprocessed image data, a deep convolutional neural network (CNN) is first used to extract the spatial features of the image. A convolutional neural network is a deep learning model based on local receptive field and weight sharing, which can effectively and automatically learn key spatial features from the original image. By stacking multiple convolutional layers, CNN can abstract low-level features (such as edges, corners, textures, etc.) and high-level features (such as the shape, appearance and category of objects) in the image layer by layer. In the campus video stream, the deep CNN can extract important spatial information in each frame of the image, such as:

[0041] Appearance: The appearance, outline, and clothing characteristics of each target.

[0042] Posture: The posture and movement of the target.

[0043] Color and Texture: The color distribution and texture details of the target, which are especially important for distinguishing different types of targets.

[0044] In this way, deep CNN can not only extract the static spatial features of the image, but also recognize the shape, movement and relative position of different targets.

[0045] Next, we use a recurrent neural network (RNN) or a 3D convolutional network to extract timing information based on the time dimension of the video stream. Timing information includes changes in the target on the time axis, such as the target's motion trajectory, speed, acceleration, and changes in behavior patterns. In the park scene, timing information is crucial for identifying dynamic behaviors (such as people walking, vehicles driving, objects moving, etc.).

[0046] Recurrent Neural Network (RNN): Mainly used to capture the temporal dependency in sequence data, suitable for modeling the before-and-after relationship of each time point in the video stream. RNN can predict the temporal characteristics of the current frame based on the input of historical frames. For park behavior recognition, RNN can effectively capture the changes in the movement trajectory of people or objects, as well as the transition of behavior states.

[0047] 3D convolutional network: For video streams, 3D convolutional networks can extract spatiotemporal features simultaneously by adding time dimension processing to the convolution operation. The advantage of this method is that it can directly perform convolution operations on the spatial and temporal dimensions of video data, which is suitable for processing continuous action recognition in dynamic scenes.

[0048] In the process of extracting temporal information, the output of RNN or 3D convolutional network usually includes the target's motion trajectory, speed, acceleration and other information over a period of time. This information plays an important role in tasks such as judging behavior type and predicting future actions.

[0049] Through the above feature extraction steps, the spatial information and temporal information of each frame of the image are converted into a set of feature vectors. Then, these feature vectors are constructed into a high-dimensional attribute matrix according to their different dimensions. Specifically, each dimension of the matrix represents different feature information in the video stream, including:

[0050] The first dimension: represents the spatial information of the image, such as shape, edges, and contours, and is usually extracted by convolutional neural networks.

[0051] The second dimension: represents the color and texture information of the image, which is used to describe the color distribution and surface characteristics of the target.

[0052] The third dimension: represents the motion trajectory and speed information of the target, which is usually extracted by a temporal modeling network (such as RNN or 3D convolutional network).

[0053] The fourth dimension: represents the temporal characteristics of the target’s behavioral changes, including the target’s acceleration, behavioral changes, etc.

[0054] In this way, the spatial and temporal features of all image frames are effectively organized into a high-dimensional matrix, and the feature vectors of each dimension carry different types of information. For example, the high-dimensional attribute matrix of a frame of image may contain the color information, texture features, target position changes, motion direction, etc. of the frame, and these features combined can fully represent the performance of the frame of image in both spatial and temporal dimensions.

[0055] The size of the high-dimensional attribute matrix usually varies with the length of the video stream and the complexity of the image. For example, for a scene containing a 10-second video stream (30 frames per second), the image size of each frame may be 224x224 pixels, and the feature vector dimension of each image may be as high as 2048 dimensions. If a high-dimensional attribute matrix is ​​used to represent all features, the total size of the matrix may reach hundreds of GB. Therefore, in order to process large-scale video data, efficient data storage and processing methods must be adopted.

[0056] In this embodiment, in order to optimize storage and computing efficiency, distributed computing and GPU acceleration technology are used. All feature matrix data are first stored in a distributed database and loaded on demand for processing. The feature extraction process of each frame of image is processed in parallel by GPU, which can significantly improve computing efficiency and reduce processing time.

[0057] The constructed high-dimensional attribute matrix provides rich feature information for subsequent behavior recognition, target detection and video analysis. In the park intelligent behavior recognition system, based on these high-dimensional matrices, the system can perform multi-level feature fusion and behavior analysis to identify different behavior patterns in the park (such as gathering of people, abnormal movement, illegal parking of vehicles, etc.). In addition, using the spatiotemporal features in the high-dimensional attribute matrix, the system can also predict behavior, identify potential abnormal behaviors in advance and trigger alarms, further improving the safety and intelligent management level of the park.

[0058] Through this efficient spatiotemporal feature construction method, the present invention can extract richer and more accurate behavioral information from the park video stream, solving the problem of insufficient spatiotemporal feature extraction capabilities in traditional video analysis methods, and providing strong technical support for the intelligent management and behavior prediction of the park.

[0059] Step 3: The large model automatically determines the need for splitting high-dimensional attribute matrices

[0060] After constructing the high-dimensional attribute matrix, the next step is to split the matrix. Since park behavior recognition involves various types of behaviors (such as walking, running, cycling, parking, etc.), the spatiotemporal characteristics of these behaviors may be significantly different. Therefore, based on the spatiotemporal characteristics required for different types of behaviors, the high-dimensional attribute matrix must be reasonably split to optimize the subsequent behavior recognition effect. To this end, the present invention adopts an intelligent large model, which is based on a deep learning method and can automatically determine whether the high-dimensional attribute matrix needs to be split and decide how to split it.

[0061] The core of the intelligent big model is a behavior recognition model based on deep learning, which can automatically analyze the spatiotemporal features in the high-dimensional attribute matrix, determine which features are important for the current behavior recognition task, and make splitting decisions based on this information. Specifically, the working principle of the intelligent big model can be summarized in the following steps:

[0062] Feature importance analysis: The model first analyzes the spatial and temporal features of each dimension in the high-dimensional attribute matrix to evaluate the contribution of each feature to behavior recognition. In park behavior recognition, some behaviors (such as walking and running) are more dependent on spatial features (such as the appearance and posture of the person), while other behaviors (such as fast driving of vehicles) may be more dependent on temporal features (such as speed and acceleration). Therefore, the model assigns a weight to each dimension based on the importance of the feature in recognition.

[0063] Behavior type recognition: Based on the input video stream, the model automatically determines which features are most critical according to the different types of behaviors (such as pedestrians, vehicles, objects, etc.). For example, for walking behavior, spatial features may be more important, while for fast-moving behaviors (such as vehicle driving), temporal features may dominate. The model automatically and selectively splits specific dimensions in the matrix based on the currently recognized behavior type.

[0064] Split decision generation: After analyzing the importance of features and the needs of behaviors, the model generates split decisions. If the model recognizes that a certain type of behavior requires processing multiple dimensions at the same time (such as requiring both spatial and temporal features), it will automatically decide whether to split the high-dimensional attribute matrix and split the matrix into multiple sub-matrices based on specific needs. The split sub-matrices will focus on different feature dimensions so that they can be processed independently, avoiding interference between different features and improving recognition accuracy.

[0065] Under the control of the intelligent big model, the entire splitting process will be automatically carried out according to the different characteristic requirements of the behavior. The specific splitting process is as follows:

[0066] First, the model determines whether the high-dimensional attribute matrix needs to be split. If splitting is required, the model automatically generates a splitting strategy based on the following conditions:

[0067] Behavior type: For complex campus behaviors (such as fast-moving vehicles and slow-moving people), the model will determine which features need to be separated based on the current behavior type. For example, pedestrian behavior features usually include relatively stable spatial information (such as appearance and posture) and relatively gentle temporal information (such as pace and speed), while vehicle behavior features include relatively dynamic temporal information (such as acceleration and driving trajectory). The model will decide whether to split the matrix based on these differences in order to process these features separately.

[0068] Differences in spatiotemporal features: The model analyzes the differences in spatiotemporal features. For example, there are large differences in the speed, acceleration and other temporal information of fast-moving vehicles and slow-moving pedestrians in the park. In this case, the model decides to split the original matrix into two parts: one part is dedicated to processing features with faster temporal changes, and the other part is dedicated to processing features with slower temporal changes.

[0069] After determining that splitting is necessary, the model will determine the dimension allocation of the sub-matrix after splitting based on the feature type and importance of each dimension. Specifically, the high-dimensional attribute matrix after splitting will be divided into two independent sub-matrices, each of which represents different types of feature information. The matrix structure after splitting is as follows:

[0070] The first sub-matrix: This sub-matrix specializes in processing spatial features and some temporal features. In park behavior recognition, spatial features (such as the shape, color, and posture of the target) are crucial for the recognition of most behaviors. For behaviors such as walking, spatial features can usually characterize the behavior pattern more accurately, so this sub-matrix will focus on processing spatial features and some low-frequency temporal information.

[0071] Second sub-matrix: This sub-matrix is ​​dedicated to processing time series features, especially time series features with high frequency changes. For example, for fast-moving vehicle behaviors, time series features (such as speed, acceleration, trajectory, etc.) change rapidly and are closely related to behavior recognition. The split second sub-matrix will focus on processing these fast-changing time series information in order to accurately identify dynamic behaviors.

[0072] The intelligent large model adopts an adaptive dimension selection and allocation strategy. During the splitting process, the model not only considers the behavior type, but also dynamically adjusts the allocation strategy of each dimension. For specific scenarios or behaviors, the model may increase the weight of some dimensions or reduce the weight of some dimensions. For example, when identifying walking behavior in a park, the model may increase the weight of the spatial feature dimension while reducing the weight of the temporal feature; for fast-moving targets (such as vehicles), the weight of the temporal feature will be increased, and the weight of the spatial feature may be appropriately reduced.

[0073] The two sub-matrices after splitting will be processed separately. The dimensions of each sub-matrix focus on different types of features, so that the model can perform deep learning and behavior analysis specifically for a certain type of feature, avoiding unnecessary feature interference, thereby improving the accuracy and efficiency of recognition.

[0074] The first sub-matrix: will be fed into the deep learning network for further spatial feature processing. This process may involve multiple convolutional layers and pooling layers to extract more abstract spatial information and use it for behavior classification and prediction.

[0075] The second sub-matrix: will be sent to the time series analysis network (such as RNN or 3D convolutional network) for processing. Through time series modeling, the system can analyze the target's motion trajectory and behavior changes, thereby accurately identifying dynamic behaviors.

[0076] Through this automated splitting process, the intelligent large model can flexibly adjust the processing method of the high-dimensional attribute matrix according to the actual scene and behavior type, so as to achieve the best results in different types of behavior recognition. This process not only improves the accuracy of the park behavior recognition system, but also optimizes the processing efficiency, especially in the face of large-scale video streams and complex behavior scenes, it can effectively reduce the consumption of computing resources and improve the real-time and intelligent level of the system.

[0077] Step 4: Split the high-dimensional attribute matrix into two independent attribute matrices

[0078] In step 3, the intelligent big model has automatically determined whether the high-dimensional attribute matrix needs to be split based on the analysis of the behavior type and spatiotemporal characteristics, and determined the dimension of the split based on the judgment result. Next, the system splits the high-dimensional attribute matrix into two independent attribute matrices, which are used to process different categories of feature information respectively. This splitting process can effectively separate the features of different behavior types, thereby improving the recognition accuracy of various behaviors in the park. The following is a detailed technical implementation plan for this step.

[0079] The high-dimensional attribute matrix contains information from different feature dimensions, which can include spatial features (such as the shape, position, posture, etc. of the target) and temporal features (such as the speed, acceleration, motion trajectory, etc. of the target). The goal of campus behavior recognition is to recognize and classify various behavior types in real time from video streams, such as walking, running, cycling, vehicle driving, etc. Different behavior types may rely on different feature categories.

[0080] In this context, the system splits the high-dimensional attribute matrix into two independent matrices, processing the features related to different behavior categories separately, ensuring that each feature category can be processed and analyzed independently. In this way, features of different categories will not interfere with each other, and the model can more accurately identify different types of behaviors.

[0081] Based on the judgment results of the intelligent big model in step 3, the system has identified which feature dimensions are closely related to the current behavior type. Therefore, the split dimensions will be selected based on the following rules:

[0082] Spatial feature dimension: For the characteristics of behavior types (such as pedestrians, robots, etc.) that rely on spatial information, the system will use these spatial features as part of the first attribute matrix. These spatial features include but are not limited to the target's appearance, posture, gait, position change, etc. For example, the movement trajectory, pace, head posture, etc. of pedestrians are all spatial features, which are closely related to the static appearance and movement pattern of the target.

[0083] Time series feature dimension: For behavior types (such as vehicles, fast-moving objects, etc.) that rely on time series information, the system will use these time series features as part of the second attribute matrix. Time series features include the speed, acceleration, direction change, movement trajectory, etc. of the target. For example, the speed, acceleration, steering angle, etc. of a vehicle are time series features, which are key factors in distinguishing fast-moving targets.

[0084] The high-dimensional attribute matrix after splitting will be divided into two independent attribute matrices, each of which will contain dimensions related to a feature category. The structure of the attribute matrix after splitting can be described as follows:

[0085] First attribute matrix: This matrix contains the dimensions of spatial features, which are mainly used to represent the static and dynamic spatial information of the target. For example, for pedestrians, the first attribute matrix may include the following dimensions:

[0086] The physical features of the person (e.g. outline, shape, posture, etc.);

[0087] The character's gait characteristics (such as step length, step frequency, posture changes, etc.);

[0088] The person’s location information (such as coordinate position in the image, relative motion trajectory, etc.).

[0089] The dimension of the first attribute matrix will be based on these spatial features, and feature extraction will be performed through a deep convolutional neural network (CNN) to provide spatial information for the behavior recognition model.

[0090] Second attribute matrix: This matrix contains the dimensions of temporal features, which are mainly used to represent the dynamic behavior information of the target. For example, for a vehicle, the second attribute matrix may include the following dimensions:

[0091] Vehicle speed (e.g., distance traveled per unit time);

[0092] Acceleration (such as vehicle acceleration, deceleration, turning, etc.);

[0093] Movement trajectory (such as the vehicle's driving route, steering angle, etc.).

[0094] The dimension of the second attribute matrix will be based on the time series feature, and the time series information will be processed through a recurrent neural network (RNN), a long short-term memory network (LSTM) or a 3D convolutional neural network. This can capture the dynamic change information of the target on the time axis.

[0095] After splitting the matrix, two independent attribute matrices will process spatial features and temporal features respectively. Ensuring that features of different categories are processed independently has the following advantages:

[0096] Avoid feature interference: By splitting spatial features and temporal features into different matrices, interference between features is avoided. For some behaviors (such as fast-moving vehicles), temporal features may change very quickly, while spatial features change relatively slowly. Independent processing can reduce the computational complexity of the model and improve accuracy.

[0097] Improve recognition accuracy: Specializing different types of features can avoid the influence of some irrelevant features on the behavior recognition results. For example, walking behavior usually relies on stable spatial features, while fast-moving vehicles require more sensitive temporal features. If these features are mixed, the recognition accuracy may be reduced.

[0098] Optimize computational efficiency: The split matrices can be processed using the most suitable network structure according to their respective feature types. For example, spatial features are usually processed using CNN, while temporal features are processed using RNN or LSTM networks. In this way, the system can select the best processing method according to different feature types, further improving computational efficiency.

[0099] The two independent attribute matrices after splitting will be sent to the subsequent behavior recognition module for processing. The specific processing flow is as follows:

[0100] Processing of the first attribute matrix: The first attribute matrix mainly processes spatial features and is processed using a deep convolutional neural network (CNN). The network will extract high-order representations of spatial features through multiple convolutional layers and compress the features through pooling layers. Finally, the spatial features are mapped to specific behavior categories (such as walking, standing, running, etc.) through a fully connected layer.

[0101] Processing of the second attribute matrix: The second attribute matrix mainly processes time series features, using long short-term memory network (LSTM) or 3D convolutional neural network (3D-CNN) to process time series information. These networks can learn the change rules of the target on the time axis, analyze the target's motion trajectory, speed, acceleration and other characteristics, and output the corresponding behavior category.

[0102] By splitting the high-dimensional attribute matrix into two independent attribute matrices, the system can process the features of different behavior types separately, thereby improving the recognition accuracy of various behaviors in the park. This splitting process not only optimizes the feature processing method and avoids interference between different features, but also improves computing efficiency and enhances the system's real-time recognition capabilities in complex scenarios. The split matrices will be processed using the most suitable network respectively, further ensuring the accuracy and reliability of the recognition results.

[0103] Step 5: Intelligently adjust the number of temporal pooling layers and perform temporal downsampling

[0104] In the intelligent behavior recognition in the park, the accurate extraction and processing of time features are crucial to the accuracy of behavior recognition. Especially when it comes to time series features such as motion trajectory, gait changes, and object movement, how to effectively process time dimension information and optimize the balance between computational complexity and accuracy has become a core issue. In order to achieve this goal, the present invention adopts an intelligent pooling mechanism in the processing of time series features in the second attribute matrix, dynamically adjusts the number of time pooling layers, and performs time downsampling to adapt to different behavior patterns and video content. The following is a detailed implementation plan for this step.

[0105] In deep learning models, pooling layers are mainly used to reduce the dimensionality of input data while retaining key information, especially in time series data processing. For time series features in video streams (such as the gait changes of people or the driving trajectory of vehicles), the temporal pooling layer can effectively reduce redundant data and improve the computational efficiency of the model.

[0106] There are usually two forms of pooling operations: Max Pooling and Average Pooling. In the process of processing time series features, the pooling layer can extract data within a time window and retain the maximum value (or average value) within the window, thereby achieving dimensionality reduction of time features.

[0107] However, in practical applications, the complexity of time series data may vary depending on the type of behavior. For simple behavior patterns (such as standing or walking slowly), too many pooling layers may lose too many temporal details; while for complex behavior patterns (such as fast-moving vehicles or multi-target interactions), increasing the number of pooling layers can effectively compress redundant data and improve processing efficiency.

[0108] The method for intelligently adjusting the number of pooling layers proposed in this invention dynamically adjusts the number of pooling layers based on the complexity of video content and the requirements of behavior patterns to achieve temporal downsampling and computational optimization. Its core principles are as follows:

[0109] Dynamic behavior complexity assessment: The system first evaluates the changes in temporal features in the video stream through a deep learning model. If the behavior in the video is relatively simple (for example, the gait changes slowly, or the moving target has a low speed), it is considered that the temporal feature changes are small, and the number of pooling layers can be appropriately reduced to retain more detailed information; if the behavior is more complex (for example, fast-moving targets or complex interactions between multiple targets), the number of pooling layers can be increased to compress redundant information in the time dimension and improve the processing efficiency of the model.

[0110] Intelligent adjustment of the number of pooling layers: Based on the evaluation of the complexity of the behavior, the model intelligently adjusts the number of pooling layers. The specific adjustment method is as follows:

[0111] If the behavior to be recognized is simple and the temporal features change slowly (such as static or slowly moving targets), reduce the number of pooling layers to avoid over-compression of temporal features and retain more details;

[0112] If the behavior to be recognized is more complex and the temporal features change rapidly (such as fast-moving vehicles or complex crowd behavior), increase the number of pooling layers to reduce the time dimension and the amount of computation.

[0113] The key to this intelligent mechanism is to evaluate the behavior complexity through real-time analysis of timing features by the model, and dynamically adjust the pooling layer according to demand.

[0114] The first step in intelligently adjusting the number of pooling layers is to perform pattern analysis on the behaviors in the video stream. This analysis is based on the frequency and magnitude of changes in temporal features, as well as the speed and trajectory of the target's motion:

[0115] Speed ​​variation: The speed of objects in the video (such as pedestrians walking, running, and vehicles driving) is an important factor in determining the complexity of the behavior. Higher speed variations are often accompanied by more complex temporal features, so more pooling layers are needed to reduce the dimensionality.

[0116] Behavior pattern recognition: Based on convolutional neural networks (CNN) and long short-term memory networks (LSTM), the model can learn behavior patterns in videos (such as walking, running, riding, parking, etc.). If the behavior changes dramatically, it means that there is a high fluctuation in the temporal features, and the number of pooling layers should be appropriately increased.

[0117] Target interaction: If multiple targets in the video interact frequently (for example, vehicles and pedestrians in traffic, multiple pedestrians in a park), this will lead to more complex temporal features. The model needs to perform effective pooling processing on the time dimension to avoid excessive computation.

[0118] Based on the behavior complexity evaluation results, the system dynamically adjusts the number of pooling layers according to the following strategy:

[0119] Simple behavior patterns: For simple behavior patterns, such as slow walking, standing, etc., the system will reduce the number of pooling layers. For example, in the case of a relatively slow gait, the model will select fewer pooling layers to retain more time information and avoid excessive downsampling.

[0120] Complex behavior patterns: For complex behavior patterns, such as fast running and vehicle driving, the model will increase the number of pooling layers. Each layer of pooling operation extracts the input according to a certain time window, retains the most important time information, and reduces redundant data, thereby achieving time downsampling. In this process, the system can effectively compress unnecessary time dimensions, reduce the computational burden of the model, and retain key timing features.

[0121] The number of pooling layers is adjusted using an adaptive algorithm. The specific adjustment method is: if the timing characteristics change little within a certain time window, the number of pooling layers will be reduced; if the timing characteristics change dramatically, the number of pooling layers will increase.

[0122] By intelligently adjusting the number of pooling layers, the system can flexibly downsample the time dimension according to the complexity of the video content and the needs of the behavior pattern. For complex behavior patterns, more pooling layers will help the system reduce redundant data and optimize computing resources; while for simple behavior patterns, it can retain more time details and improve recognition accuracy.

[0123] The technical means of intelligently adjusting the number of time pooling layers and performing time downsampling can optimize computing efficiency while ensuring the accuracy of behavior recognition by intelligently adjusting the number of pooling layers according to changes in timing characteristics. This flexible pooling layer adjustment mechanism combines the complexity of video content and the diversity of behavior types, and can provide accurate and efficient campus intelligent behavior recognition solutions in different scenarios.

[0124] Step 6: Process the two attribute matrices in parallel

[0125] In this step, by inputting the split first attribute matrix and the second attribute matrix into the parallel processing module of the large model, the system can efficiently process and analyze different types of spatiotemporal features, thereby improving the overall performance of park behavior recognition. Specifically, the first attribute matrix is ​​mainly used for the extraction and analysis of spatial features, while the second attribute matrix focuses on the processing of temporal information. The implementation strategy of this step is to adopt different processing strategies for these two matrices under the parallel computing framework, thereby accelerating data processing and improving the accuracy of behavior recognition.

[0126] The present invention adopts a parallel processing architecture to input the two split attribute matrices into two processing paths respectively. Each path is processed using a specially designed deep learning model, one of which is mainly responsible for the extraction of spatial features, while the other is focused on the processing of time series features. The core advantage of this parallel processing architecture is that it can perform in-depth analysis on two different types of features at the same time, avoiding the information processing bottleneck that may exist in traditional single feature processing methods, thereby improving processing efficiency.

[0127] For the first attribute matrix, it mainly contains spatial features, such as the shape, color, and posture of objects in the park. In order to effectively extract these spatial features, the system uses a convolutional neural network (CNN). CNN is a deep learning architecture specifically used for image processing, which can automatically extract local features of images through convolutional layers.

[0128] Specifically, before entering the CNN module, the first attribute matrix will undergo standardization and preprocessing steps (such as denoising, resizing, etc.) to ensure that the data meets the network input requirements. The CNN network extracts multi-level features of the image through multiple convolutional layers and pooling layers, gradually representing from low-level texture features (such as edges and corners) to high-level semantic features (such as character posture and object classification).

[0129] Convolutional Layer: Each convolutional layer performs convolution operations on the input data through a set of filters (convolution kernels) to generate different feature maps. The convolution operation can capture the characteristics of local areas in spatial information, and then extract more complex spatial information through stacked convolutions.

[0130] Pooling Layer: Pooling operation is used for downsampling, reducing the size of feature maps while retaining the most important information. Usually, max pooling or average pooling is used to reduce the dimension of features and reduce the amount of calculation.

[0131] After multiple processing of convolution and pooling layers, the spatial features extracted by CNN will provide efficient input for subsequent behavior recognition.

[0132] The second attribute matrix contains the time series information in the video stream, such as the dynamic behavior of characters, the movement trajectory of vehicles, etc. In order to effectively process these time series features, the system uses recurrent neural network (RNN) for time series modeling. Unlike CNN, RNN can process and analyze information in the time dimension and capture dynamic changes in time series data.

[0133] RNN structure: RNN uses its internal loop structure to take the output of the previous moment as the input of the current moment, maintaining the time dependency of the time series data. This enables RNN to process the time series features in the video stream well and capture the dynamic changes of the target in time (such as walking gait, vehicle acceleration, deceleration, etc.).

[0134] Long Short-Term Memory (LSTM) Unit: To overcome the vanishing gradient problem encountered by traditional RNNs during long sequence training, the system can use LSTM units. LSTM controls the flow of information through gating mechanisms (such as input gates, forget gates, and output gates), thereby better capturing dependencies over long time spans and is suitable for long-term memory and change patterns of temporal features.

[0135] When processing time series information, the second attribute matrix is ​​first preprocessed (for example, normalizing the time series data) and then input into the RNN or LSTM model. The network processes each time step in the time series in sequence through multiple layers of RNN or LSTM units and captures the time series dependencies.

[0136] The design of the parallel processing structure enables spatial information and temporal information to be processed simultaneously and independently. The two processing paths do not interfere with each other during the processing process, thereby maximizing the efficiency of computing resources. In addition, after completing their respective feature extraction tasks, the two attribute matrices output feature representations processed by multiple convolutional layers (CNN) or recurrent neural network layers (RNN / LSTM). These feature representations will be further passed to the subsequent fusion module for final behavior recognition and prediction.

[0137] After the two attribute matrices are processed separately, the system will perform data fusion. The specific fusion method may vary according to specific needs. The following are several common fusion strategies:

[0138] Feature-level fusion: The spatial features and temporal features extracted from the two paths are concatenated or weighted to form a unified feature vector. This feature vector will be input into the subsequent classifier or behavior recognition module for the final behavior judgment F fused .

[0139] The formula is:

[0140]

[0141] Among them, F spatial represents the spatial eigenvector, F temporal represents the time series feature vector, Represents a concatenation operation.

[0142] Decision-level Fusion: After the spatial feature and temporal feature processing paths output their respective prediction results, the outputs of the two can be weighted averaged or voted to obtain the final behavior recognition result P. final .

[0143] The formula is:

[0144] P final =w1P spatial +w2P temporal

[0145] Among them, P spatial and P temporal They are the output prediction results of the spatial feature path and the temporal feature path respectively, and w1 and w2 are the weight coefficients.

[0146] In some complex application scenarios, more input features may need to be considered, such as environmental features (temperature, humidity, light, etc.). In this case, the system may need to introduce cross-modal fusion technology to integrate feature information from different sources to achieve more accurate behavior recognition.

[0147] Through parallel processing, the system can efficiently extract spatial information and temporal information, improve computing efficiency and reduce time overhead. In traditional single-channel processing systems, the extraction of spatial features and temporal features is often performed serially, which may lead to longer processing time. Under the parallel architecture, the processing tasks of the two feature types can be performed simultaneously, greatly improving the overall processing capability of the system, especially in real-time video monitoring and rapid response scenarios.

[0148] By processing two attribute matrices in parallel, the system can achieve efficient parallel computing in the extraction and analysis of spatial and temporal features. This processing architecture not only improves computing efficiency, but also ensures high accuracy of behavior recognition. The parallel structure of CNN and RNN enables the system to take into account both temporal and spatial features when processing complex behavior recognition tasks, and has strong application adaptability and real-time performance.

[0149] Step 7: Output the park behavior recognition results

[0150] After the aforementioned image processing and behavior analysis process, the system will eventually generate behavior recognition results for each target in the park (such as pedestrians, vehicles, etc.). These recognition results will reflect the specific behavior status of the target at each time point or each time sequence in the video stream, and will be classified and output according to different recognition target types (such as human behavior, vehicle behavior, etc.). Through accurate behavior recognition of targets in the park, the system can not only monitor the dynamic changes of the park in real time, but also make corresponding management decisions based on the recognition results, providing effective support for intelligent management of the park.

[0151] Based on the input video stream, the system can output behavior predictions for each target based on the processing results of each frame of the image or video sequence. These prediction results will be comprehensively judged based on the spatiotemporal features in the video stream and the spatial and temporal features extracted in the previous steps. The output behavior recognition results can include but are not limited to the following common types:

[0152] Personnel behavior recognition: The system can recognize and classify the specific behavior patterns of people in the park, such as walking, running, standing, talking, sitting, etc. By analyzing the movement trajectory of people in the video stream, combined with the recognition of their appearance characteristics and temporal behavior, the system can effectively distinguish different types of behavior.

[0153] Vehicle behavior recognition: For vehicles in the park, the system can recognize and classify their behaviors, such as driving, parking, accelerating, braking, turning, etc. By monitoring the vehicle's movement trajectory, speed changes, and interaction with the surrounding environment, the system can accurately extract the vehicle's behavioral characteristics.

[0154] Abnormal behavior detection: The system can not only identify regular behaviors, but also automatically identify and mark potential abnormal behaviors. For example, if a target suddenly stops moving, changes in speed abnormally, deviates from the planned route, etc., the system can automatically identify and alarm, prompting managers to take further action.

[0155] These behavior recognition results can be further processed to provide detailed analysis of real-time dynamics within the park and provide decision support for the park management system.

[0156] In order to more intuitively display the recognition results, the system usually visualizes the behavior prediction results and presents them to the park managers in the form of graphics, text or other easy-to-understand methods. These visualization results can not only help managers understand the dynamics of the park in real time, but also serve as an important basis for subsequent data analysis and decision support.

[0157] For example, the system can display the recognition results in the following ways:

[0158] Behavior type labeling: Label each target on the video stream, indicating its current behavior type (such as "walking", "driving", "parking"). This labeling is usually displayed dynamically on the video screen to facilitate management personnel to observe and understand.

[0159] Behavior timeline: For each target, the system can generate a behavior timeline chart to show its behavior changes in different time periods. This timeline chart helps analyze the target's behavior pattern and further optimize the park management strategy.

[0160] Abnormal warning notification: The system can also automatically determine whether abnormal behavior occurs during the behavior recognition process, and push abnormal behavior to management personnel in a timely manner. For example, if it is detected that someone in the park enters a restricted area during an abnormal period of time, or a vehicle stays in a prohibited parking area, the system will automatically trigger an early warning and send a notification to the management personnel.

[0161] The final behavior recognition results not only provide real-time monitoring views for park managers, but also directly affect the park's intelligent decision-making system. The following are several typical application scenarios of behavior recognition results:

[0162] Automatically trigger security alerts: When the system detects potential security threats or abnormal behaviors (such as abnormal stay of people, illegal parking of vehicles, etc.), the system will automatically generate security alerts to remind relevant personnel to take necessary actions. For example, when a crowd of people or other abnormal activities occur in the park, the system can automatically issue a security alert.

[0163] Intelligent resource scheduling: The system can monitor the distribution of people and vehicles in the park in real time and intelligently schedule park resources (such as parking spaces, routes, cleaners, etc.). By analyzing the results of behavior recognition, the park management system can predict which areas may have resource shortages and then optimize scheduling.

[0164] Abnormal behavior monitoring and analysis: The system can monitor abnormal behaviors in the park for a long time and accumulate and analyze data. For example, for violations that often occur in the park, the system can generate behavior reports, analyze the occurrence patterns of these behaviors, and help managers formulate effective management measures and improvement plans.

[0165] Personnel and equipment management: Based on the behavior recognition results, the park management system can accurately manage the personnel and equipment in the park. For example, the system can arrange employees' work tasks or optimize the workflow based on personnel behavior data, and can also monitor the equipment usage status to promptly detect equipment failures or demand changes.

[0166] In addition to real-time output of park behavior recognition results, the system also supports further analysis and optimization of behavior recognition results. Through machine learning and data mining technology, the system can learn from historical data and continuously optimize its behavior recognition algorithm to improve the accuracy and intelligence of recognition.

[0167] Data analysis and feedback: The system can regularly summarize and analyze the results of the park's behavior recognition and identify possible problems or potential risks in park management. Based on these data analyses, managers can adjust park management strategies or improve existing management processes.

[0168] Self-learning and optimization: As the amount of data increases, the system can continuously improve its behavior recognition model through a self-learning mechanism. For example, the system can analyze the types of misidentified behaviors and adjust the model parameters according to the actual situation, thereby continuously improving the accuracy and robustness of recognition.

[0169] Through the above steps, the park intelligent behavior recognition system can not only monitor various behaviors in the park in real time, but also automatically respond according to the recognition results and provide real-time decision support. Whether in security monitoring, resource scheduling, abnormal behavior detection, or personnel and equipment management, the behavior recognition results can effectively improve the level of intelligent park management. With the continuous optimization and development of technology, the park behavior recognition system of the present invention will demonstrate its powerful value in a wider range of application scenarios.

[0170] The park intelligent behavior recognition image processing method of the present invention can significantly improve the processing efficiency while ensuring high-precision recognition. The use of high-dimensional attribute matrix and its splitting processing technology can effectively extract the spatiotemporal characteristics of different targets in the park, and adopt personalized processing strategies for different targets. At the same time, the dynamic adjustment of the number of intelligent pooling layers effectively avoids invalid calculations and improves the response speed of the system. The parallel processing method enables the behavior recognition of multiple targets to be carried out simultaneously, thereby further improving the efficiency and real-time performance of the system.

[0171] The technical solution described in this embodiment can be widely used in security monitoring, behavior analysis, resource scheduling and other fields within the park, greatly improving the level of intelligent park management and providing park managers with accurate and real-time behavior recognition data.

[0172] The above description is only a preferred embodiment of the present invention, so all equivalent changes or modifications made according to the structure, characteristics and principles described in the scope of the patent application of the present invention are included in the scope of the patent application of the present invention.

Claims

1. A large-scale model-based park intelligent behavior recognition image processing method, characterized by: The method comprises: Get a video stream containing multiple image frames; Constructing a high-dimensional attribute matrix to represent the received video stream, wherein the high-dimensional attribute matrix has multiple dimensions, wherein each dimension represents a different feature extracted from the campus video stream; A large model with multiple cross-temporal resolution modules is used to perform park behavior recognition. The large model automatically determines the splitting requirements of the high-dimensional attribute matrix, and splits the high-dimensional attribute matrix according to the judgment result to obtain two independent attribute matrices, where: The first attribute matrix contains a first set of dimensions; The second attribute matrix contains a second set of dimensions; The large model intelligently adjusts the number of time pooling layers of the second attribute matrix, and performs time downsampling only when necessary according to actual needs; The large model processes the two attribute matrices in parallel, respectively processing the first attribute matrix and the second attribute matrix that has been time-downsampled; Output prediction results about campus behavior in the received video stream.

2. The method for image processing of park intelligent behavior recognition based on a large model as claimed in claim 1, characterized in that: The video stream is a dynamic video stream collected in real time by multiple cameras in the park, and the video stream is transmitted at a predetermined frame rate.

3. The method for image processing of park intelligent behavior recognition based on a large model as claimed in claim 1, characterized in that: Each dimension of the high-dimensional attribute matrix represents a spatial feature, a temporal feature, or a combination of the two extracted from the video stream. The spatial features include the appearance, posture, color, and texture of the character, and the temporal features include the movement trajectory, speed, and behavior changes of the character.

4. The method for image processing of park intelligent behavior recognition based on a large model as claimed in claim 1, characterized in that: The large model uses a deep convolutional neural network (CNN) to extract spatial features and a recurrent neural network (RNN) or a 3D convolutional neural network to process temporal features.

5. The method for image processing of park intelligent behavior recognition based on a large model as claimed in claim 1, characterized in that: The split first attribute matrix contains spatial features related to human behavior, and the second attribute matrix contains motion features related to vehicle behavior.

6. The method for image processing of park intelligent behavior recognition based on a large model as claimed in claim 1, characterized in that: The temporal pooling layer intelligently adjusts the number of poolings to flexibly adjust the compression degree of the temporal dimension according to the actual needs of the video content.

7. The method for image processing of park intelligent behavior recognition based on a large model as claimed in claim 1, characterized in that: The parallel processing includes using a convolutional neural network (CNN) to process spatial features in the first attribute matrix and using a recurrent neural network (RNN) to process time series features in the second attribute matrix.

8. The method for image processing of park intelligent behavior recognition based on a large model as claimed in claim 1, characterized in that: The output prediction results of the park behavior include the recognition results of the behavior of people in the park, the recognition results of the behavior of vehicles, and the detection results of abnormal behavior.

9. The method for image processing of park intelligent behavior recognition based on a large model as claimed in claim 1, characterized in that: The behavior recognition results are used for decision support of the park management system, including security warning, resource scheduling, abnormal behavior monitoring, and personnel and equipment management.

10. The method for image processing of park intelligent behavior recognition based on a large model as claimed in claim 1, characterized in that: The behavior recognition system further supports self-learning and optimization.

Images