Big data advanced modeling method for multi-dimensional data fusion

Abstract

Compared with the prior art, the big data advanced modeling method based on multi-dimensional data fusion has the advantages that by fusing audio and image features and comprehensively utilizing various data information sources in a video segment, the recognition precision of behavior features is remarkably improved, the method is suitable for various video monitoring and behavior recognition application scenes, and the modeling efficiency is improved. The method supports the identification demands of different behavior types and complexity and the demands of efficient real-time video behavior identification processing, and meets the application demands of multiple fields such as intelligent monitoring, public safety, behavior analysis and the like. Therefore, the accuracy and correlation of human behaviors in the video segment can be remarkably improved based on audio and image feature fusion recognition, a more comprehensive and deep judgment result is provided, and police officers can make more accurate decisions in the video query process.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a big data advanced modeling method for multi-dimensional data fusion. Background Art

[0002] Conventional technical content of the industry at home and abroad at this stage:

[0003] 1. Data preprocessing:

[0004] Perform pre-processing operations such as extraction, cleaning, and sorting on multi-source data to ensure data quality and consistency.

[0005] 2. Feature extraction:

[0006] Extract representative feature information from the raw data for subsequent data analysis and modeling.

[0007] 3. Data association combination:

[0008] Data from different sources, types and dimensions are associated and combined to form a comprehensive data set.

[0009] 4. Data fusion algorithm:

[0010] According to the purpose and level of fusion, appropriate fusion algorithms are selected, such as feature layer fusion, decision layer fusion and hybrid layer fusion, to process and integrate multimodal data.

[0011] The main disadvantages of current conventional technologies include:

[0012] 1. Data preprocessing overhead is high:

[0013] In the data preprocessing stage, a large number of cleaning, conversion and standardization operations are often time-consuming and labor-intensive, and may introduce new errors. This technical solution reduces the workload of data preprocessing, improves processing efficiency, and ensures data accuracy by introducing automated processing and optimization mechanisms in the data processing process.

[0014] 2. Data redundancy and conflict:

[0015] When multi-source data is fused, redundancy and conflict often occur. If not properly handled, it may affect the accuracy and reliability of the fusion results. To address this problem, this technical solution uses feature extraction and selection technology and noise control mechanism to accurately extract and fuse multimodal data, ensure that redundant and conflicting data are effectively handled, and improve the quality of data fusion.

[0016] 3. Difficulty in selecting fusion algorithms:

[0017] Since different fusion algorithms are suitable for different scenarios and needs, it is difficult to accurately select the best algorithm in practical applications, and it is limited by factors such as data quality and computing resources.

[0018] 4. Insufficient scalability and flexibility:

[0019] Conventional technologies are designed and optimized for specific scenarios, but they require a lot of modifications and adjustments when facing new application scenarios and data sets, which affects scalability and flexibility. This technical solution enhances the versatility of the model through differential privacy noise protection and multimodal feature fusion, and can adapt to diverse behavioral scenarios and data inputs. It also implements a flexible feature extraction and fusion mechanism through deep neural network processing, which improves the scalability of the system. Summary of the invention

[0020] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0021] To this end, one purpose of the present application is to propose a big data advanced modeling method of multi-dimensional data fusion, which significantly improves the accuracy and relevance of human behavior results in video segments, comprehensively considers multiple factors, provides more comprehensive and in-depth search results, reduces manual intervention, improves search efficiency and accuracy, and also reduces operating costs, processes and updates police data in real time, ensures the timeliness and accuracy of search results, finds required information faster, reduces the waste of search time and energy, and provides police officers with more comprehensive and in-depth information support, which helps them better understand the case, analyze trends and formulate strategies, has good scalability and flexibility, and can adapt to police data needs of different scales and types. As the amount of police data continues to increase and change, this technology can be continuously optimized and upgraded to maintain its advancement and competitiveness.

[0022] To achieve the above objectives, the first embodiment of the present application proposes a big data advanced modeling method for multidimensional data fusion, and the big data advanced modeling method comprises the following steps:

[0023] A big data advanced modeling method for multi-dimensional data fusion, characterized in that the big data advanced modeling method comprises the following steps:

[0024] S101: Data cleaning: Clean the video streams captured by the camera, vehicle checkpoints, portrait capture data, and police enforcement data to remove noise data, duplicate data, and abnormal data to ensure data quality and accuracy, and provide a reliable data foundation for subsequent analysis of personnel flow, abnormal behavior identification, etc.

[0025] Data conversion: Convert data from different sources and formats into a unified format for subsequent processing, including video streams captured by cameras, vehicle checkpoints, portrait capture data, and video images of police enforcement.

[0026] Data standardization: normalize the data to ensure the consistency and comparability of various data sources, generate video segments of the same or similar length, and ensure consistency in duration, format, and resolution, so as to meet the multi-dimensional integration needs of traffic monitoring and behavior analysis;

[0027] S102: Feature extraction and selection: Screen and obtain the audio and image data in the video segment, and extract the audio features f in the video segment. audio and the image features f in the video segment pic To obtain;

[0028] S103: Fusing audio and image features to obtain an early fusion feature vector F early ;

[0029] S104: Use weighted voting to fuse the results of the early fusion feature vectors in J classifiers to obtain the final video segment event probability P end ;

[0030] S105: Feature modeling: Using a variety of learning methods to predict event probabilities P in video segments end Modeling is performed, and deep learning technology is used to perform high-level abstraction and representation of behavior categories. A large number of video segments with human behavior labels are used to supervise the training of the deep neural network DNN. The final video segment event probability is output as the probability distribution of each human behavior category.

[0031] Model optimization: During the training process, grid search and random search cross-validation and parameter optimization methods are applied to optimize model performance and improve the prediction accuracy and stability of the model;

[0032] S106: P end As input features, it is input into the deep neural network DNN for classification, and the probability distribution of each human behavior category in the video segment is obtained. The probability distribution of each human behavior category in the video segment is obtained through the Softmax function, so as to judge the most likely human behavior appearing in the video.

[0033] The character behaviors include, but are not limited to, individual existence status, movement behavior, group characteristics, and abnormal activities, including basic standing and / or sitting postures, changes in walking and / or running position and direction, group activities of gathering and / or dispersing, and potential safety risk behaviors such as fighting, falling, wandering, and / or entering restricted areas, to support comprehensive behavior analysis needs in multiple scenarios;

[0034] S107: Data encryption: Encrypt video segments, audio, and images to ensure data security during transmission and storage, and prevent unauthorized users from intercepting or viewing them.

[0035] Access control: Set reasonable access rights and limit data access according to user identity and permission level to prevent unauthorized data operations and ensure that only authorized personnel are allowed to access sensitive data.

[0036] Anonymization: Anonymize information involving individual identification, remove or cover up the direct identification elements of the data to avoid personal privacy leakage.

[0037] Differential privacy protection: Use the Laplace mechanism to add differential privacy noise to the probability distribution of each human behavior category in the video segment, and store the noisy probability distribution data in a database or secure storage system to protect the output behavior category probability and ensure that external visitors cannot infer the individual's true behavior through the probability distribution results.

[0038] In one embodiment of the present application, S103 is implemented by the following steps:

[0039] S1031: directly concatenate audio and image features to generate a fused feature vector F fuSi , where FfuSi = [faudio, fpic],

[0040] S1032: Using a bidirectional recurrent neural network to process temporal dependencies fuSi Perform early fusion to handle temporal dependencies and generate sequence features to generate early fusion feature vector F early :

[0041] F early =BiLSTM(F fuSi ),

[0042] Among them, BiLSTM bidirectional long short-term memory network, F early is the BiLSTM for F fuSi The processed output features are

[0043] Furthermore, the final video segment event probability P end Calculated in the following way:

[0044]

[0045] P j is the early fusion feature vector F early The output probability of the jth classifier, j = {1, 2, 3…J}, C jis the weight of the j-th classifier, and The classifier is an SVM, random forest and / or deep neural network model for identifying human events in image and audio data, which is obtained by a technician in this field by pre-training a data set annotated with human behavior labels.

[0046] In one embodiment of the present application, the deep neural network DNN is obtained by a technician in this field using a large number of video segments with human behavior labels for training supervision, and the deep neural network DNN input is the fused feature vector P end , the output of the deep neural network DNN is the probability distribution of human behavior categories,

[0047] The supervised training process includes:

[0048] S1051: Input data P end Passed from the input layer to the output layer, the predicted output is generated through the calculation of neurons in each layer.

[0049] S1052: Use the cross entropy loss function to calculate the error between the predicted output and the true label to measure the prediction accuracy of the model.

[0050] S1053: Calculate the gradient of the error to each parameter in the network through the back propagation algorithm, and adjust the weights and bias items of each layer of neurons to reduce the error.

[0051] S1054: Use gradient descent and / or Adam optimization algorithm to update weights so that the loss function is gradually minimized.

[0052] S1055: After multiple iterations of training, the model gradually learns the mapping relationship between input features and the probability distribution of human behavior categories, and finally evaluates its performance on the validation set and test set to ensure the accuracy and generalization ability of the model.

[0053] In one embodiment of the present application, the present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a big data advanced modeling method for multidimensional data fusion when executing the program.

[0054] In one embodiment of the present application, the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of a big data advanced modeling method for multidimensional data fusion are implemented.

[0055] In one embodiment of the present application, the present invention also provides a processor, characterized in that the processor is used to run a program, wherein the program executes the steps of the big data advanced modeling method of multidimensional data fusion when running.

[0056] Compared with the prior art, the multidimensional data fusion big data advanced modeling method of the embodiment of the present application has the following advantages:

[0057] 1. Reduce data redundancy through data screening and cleaning, and use conflict resolution strategies to improve the accuracy and reliability of fusion results and ensure the consistency of data fusion.

[0058] 2. The solution of the present invention combines audio and image features, performs early fusion of time series data through a bidirectional recurrent neural network, and makes full use of the key information in the audio and video signals. Compared with the processing of a single modality, this multimodal fusion greatly improves the model's ability to understand complex video events, can maximize the use of data resources, eliminate redundancy and conflict, and provide a more reliable foundation for high-quality data modeling, especially in the accuracy and robustness of behavior recognition.

[0059] 3. Add differential privacy noise to the behavior classification output to protect individual privacy and prevent the outside world from inferring the individual's true behavior through the output data. This privacy protection mechanism is particularly important in public video surveillance and public safety systems to avoid the leakage of sensitive information, effectively protect data privacy, and meet data security and compliance requirements.

[0060] 4. The present invention can flexibly adjust and expand behavior categories to adapt to different application scenarios and needs. At the same time, different classification models, feature extraction algorithms and parameter tuning strategies can be selected according to specific project requirements, and it has good adaptability and scalability.

[0061] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0063] Figure 1 It is a schematic flow chart of the steps of the high-level big data modeling method for multidimensional data fusion of the present invention.

[0064] Figure 2 It is a modular schematic diagram of the high-level big data modeling method of multi-dimensional data fusion of the present invention. DETAILED DESCRIPTION

[0065] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limitations on the present application. On the contrary, the embodiments of the present application include all changes, modifications and equivalents that fall within the spirit and connotation of the appended claims.

[0066] The following describes the big data advanced modeling method of multidimensional data fusion in accordance with an embodiment of the present application in conjunction with the accompanying drawings.

[0067] The high-level big data modeling method of multi-dimensional data fusion provided in the embodiment of the present application can be applied to.

[0068] like Figure 1-2 As shown, the big data advanced modeling method of the embodiment of the present application includes the following steps:

[0069] A big data advanced modeling method for multi-dimensional data fusion, characterized in that the big data advanced modeling method comprises the following steps:

[0070] S101: Data cleaning: Clean the video streams captured by the camera, vehicle checkpoints, portrait capture data, and police enforcement data to remove noise data, duplicate data, and abnormal data to ensure data quality and accuracy, and provide a reliable data foundation for subsequent analysis of personnel flow, abnormal behavior identification, etc.

[0071] Data conversion: converting data from different sources and formats into a unified format for subsequent processing, including video streams captured by cameras, vehicle checkpoints, portrait capture data, and video images of police enforcement;

[0072] It should be noted that in the process of converting data from different sources and formats into a unified format described in this embodiment, standardization and normalization steps need to be taken to provide a consistent data structure and content for subsequent processing and analysis. The specific conversion and processing steps are as follows:

[0073] 1. Data format standardization: Unify image and video data from different sources (such as camera video streams, vehicle checkpoint images, portrait snapshot images, and police enforcement videos) into compatible file formats (such as JPEG, MP4, or AVI) to ensure that all data can be read and parsed normally by subsequent processing algorithms. This step ensures format consistency and reduces compatibility issues caused by multiple formats.

[0074] 2. Frame rate and resolution adjustment: For video stream data, unify its frame rate and resolution to ensure that all video sources have consistent playback rate and image clarity. Unified frame rate facilitates synchronization processing, and unified resolution simplifies image size normalization operations to ensure consistent data scale for model input;

[0075] 3. Integration of timestamp and geographic location information: By combining the metadata of various video and image data, the timestamp and location information are standardized and unified into a fixed format (such as UTC time and longitude and latitude coordinates). This standardization of time and space information helps to achieve event synchronization and spatial association across data sources, facilitating subsequent multi-source data fusion and spatiotemporal analysis.

[0076] 4. Coding consistency of image and video data: For video streams with different encodings (such as H.264, H.265, etc.), encoding conversion is performed to ensure that all videos are stored and transmitted in the same encoding method, which reduces the decoding burden and improves efficiency during processing and transmission;

[0077] 5. Normalization of data labels and metadata: Data from different sources often come with different labels and metadata information (such as camera ID, capture location, law enforcement type, etc.). Unifying the label format, encoding method and field naming will make subsequent data indexing, retrieval and analysis processes more efficient, and will also facilitate the model to use label information for contextual analysis;

[0078] 6. Data storage and management: All standardized data are stored in a unified database or data warehouse, using a hierarchical storage structure (such as stratification by time, location or data type), and implementing scalable access rights management to facilitate distributed processing and efficient call of data;

[0079] Through these standardization steps, all data from different sources can achieve consistent formats and specifications, allowing subsequent feature extraction, behavior analysis, pattern recognition and other processing processes to be carried out on the same data basis, improving the accuracy of data fusion and the analysis efficiency of the overall system.

[0080] Data standardization: normalize the data to ensure the consistency and comparability of various data sources, generate video segments of the same or similar length, and ensure consistency in duration, format, and resolution, so as to meet the multi-dimensional integration needs of traffic monitoring and behavior analysis;

[0081] S102: Feature extraction and selection: Screen and obtain the audio and image data in the video segment, and extract the audio features f in the video segment. audio and the image features f in the video segment pic To obtain;

[0082] S103: Fusing audio and image features to obtain an early fusion feature vector F early ;

[0083] S104: Use weighted voting to fuse the results of the early fusion feature vectors in J classifiers to obtain the final video segment event probability P end ;

[0084] S105: Feature modeling: Using a variety of learning methods to predict event probabilities P in video segments end Modeling is performed, and deep learning technology is used to perform high-level abstraction and representation of behavior categories. A large number of video segments with human behavior labels are used to supervise the training of the deep neural network DNN. The final video segment event probability is output as the probability distribution of each human behavior category;

[0085] It should be noted that the event probability P of the video clip described in this embodiment is obtained by using multiple learning methods. end Modeling is performed to extract and predict the probability distribution of human behavior categories in videos. The specific steps are as follows:

[0086] 1. Feature extraction and high-level abstraction: Deep learning technology processes video frames through architectures such as convolutional neural networks (CNN) or recurrent neural networks (RNN), abstracts and represents human behavior features at a high level, and gradually generates feature vectors representing behavior categories;

[0087] 2. Supervised training: A large amount of video clip data with human behavior labels is used to supervise the deep neural network (DNN) learning. Through repeated training and optimization, the model can learn the characteristic distribution of different behavior categories and enhance the ability to distinguish between categories;

[0088] 3. Event probability output: In the model inference stage, after the video clip is input, DNN calculates the probability distribution of each human behavior category and generates the video clip event probability Pend as the output result. These probabilities indicate the possibility of each behavior category in the current video clip;

[0089] 4. Behavior category prediction and optimization: The output probability distribution can be used to identify the main behavior categories in the video clips. At the same time, the results can be further optimized in combination with other post-processing steps to improve the accuracy and generalization ability of the model.

[0090] Model optimization: During the training process, grid search and random search cross-validation and parameter optimization methods are applied to optimize model performance and improve the prediction accuracy and stability of the model.

[0091] It should be noted that in the model optimization process described in this embodiment, in order to improve prediction accuracy and stability, grid search and random search are used as the main hyperparameter optimization methods, which are combined with the cross-validation strategy and applied to model training. The specific steps are as follows:

[0092] 1. Grid search: Grid search is a systematic hyperparameter search method. By defining a set of possible hyperparameter value combinations, the model will try all combinations and verify the effect of each set of parameters one by one. With the help of cross-validation, grid search can effectively identify the best performing parameter combination, thereby optimizing model performance. However, grid search takes a long time and is suitable for situations with a small number of parameters.

[0093] 2. Random search: Compared with the exhaustive grid search, random search randomly samples from the set parameter space and can find the optimal parameter combination with fewer attempts. Random search is particularly suitable for high-dimensional parameter spaces, which helps to reduce computing costs and improve search efficiency. Combined with cross-validation, random search maintains model accuracy while also enhancing the stability and generalization ability of the model.

[0094] 3. Cross-validation: Cross-validation effectively evaluates the adaptability of the model to different data distributions by dividing the training data into multiple subsets and training and validating each subset. By alternately validating the performance of each parameter combination under different data distributions, it avoids overfitting of the model and improves the generalization ability.

[0095] 4. Combination strategy and adaptive optimization: In order to further improve the optimization effect of the model, a hybrid strategy can be adopted, that is, random search is used to determine a relatively optimal parameter range in the initial screening stage, and then grid search is performed on the range for more precise optimization. In addition, the use of adaptive algorithms (such as Bayesian optimization or ultra-bandwidth optimization) to automatically adjust the parameter search direction helps to find the optimal hyperparameter combination in a shorter time;

[0096] 5. Model evaluation and repeated iteration: After hyperparameter optimization, conduct a comprehensive performance evaluation of the model, such as accuracy, stability, and generalization ability on the validation set and test set. According to the evaluation results, repeatedly adjust the parameter search range and strategy, and continuously iterate to ensure that the model achieves optimal performance.

[0097] S106: P end As input features, it is input into the deep neural network DNN for classification, and the probability distribution of each human behavior category in the video segment is obtained. The probability distribution of each human behavior category in the video segment is obtained through the Softmax function, so as to judge the most likely human behavior appearing in the video.

[0098] The character behaviors include, but are not limited to, individual existence status, movement behavior, group characteristics, and abnormal activities, including basic standing and / or sitting postures, changes in walking and / or running position and direction, group activities of gathering and / or dispersing, and potential safety risk behaviors such as fighting, falling, wandering, and / or entering restricted areas, to support comprehensive behavior analysis needs in multiple scenarios;

[0099] S107: Data encryption: Encrypt video segments, audio, and images to ensure data security during transmission and storage, and prevent unauthorized users from intercepting or viewing them.

[0100] Access control: Set reasonable access rights and limit data access according to user identity and permission level to prevent unauthorized data operations and ensure that only authorized personnel are allowed to access sensitive data.

[0101] Anonymization: Anonymize information involving individual identification, remove or cover up the direct identification elements of the data to avoid personal privacy leakage.

[0102] Differential privacy protection: Use the Laplace mechanism to add differential privacy noise to the probability distribution of each human behavior category in the video segment, and store the noisy probability distribution data in a database or secure storage system to protect the output behavior category probability and ensure that external visitors cannot infer the individual's true behavior through the probability distribution results.

[0103] It should be noted that in the differential privacy protection scheme described in this embodiment, in order to protect the probability distribution of human behavior categories in each video clip from being maliciously exploited by external visitors, we add Laplace mechanism noise to the probability distribution. The detailed steps are as follows:

[0104] 1. Laplace mechanism noise addition: Laplace noise is applied to the behavior category probability distribution output by the model. The noise is adjusted according to the set privacy budget ε (privacy level) to ensure that the accuracy of behavior recognition is retained after perturbation while achieving differential privacy protection;

[0105] 2. Privacy budget setting: The privacy budget ε value controls the noise intensity. A smaller ε value increases the amount of noise, thereby improving the privacy protection level, but may slightly reduce the accuracy of behavior classification. A larger ε value has the opposite effect. In order to balance privacy protection and recognition accuracy, the selection of ε usually requires experiments and optimization.

[0106] 3. Storage and access control: Store the noisy behavior category probability distribution data in a database or secure storage system. The storage system should use encryption mechanisms and strict access rights management to ensure that only authorized users can access the data.

[0107] 4. Query and external access restrictions: By adding differential privacy noise, external visitors can only obtain probabilistic fuzzy information when querying these data, preventing them from inferring the specific behavior of individuals through tiny probability changes. In this way, even under multiple queries, personal behavior patterns will not be revealed, ensuring privacy protection;

[0108] 5. Improved model and system security: Further privacy protection can be achieved by reviewing query logs, limiting frequent access, and combining other privacy technologies (such as homomorphic encryption or federated learning) to prevent data leakage and improve the security and robustness of the entire system.

[0109] It should be noted that the Laplace mechanism described in this embodiment adds noise to the probability distribution of behavior categories to protect the privacy of the output behavior probability data. Before storing the data, the noisy probability distribution is processed so that even if others obtain the probability distribution, they cannot accurately infer the individual's true behavior. This step ensures the security of the data during storage and processing, and meets the requirements for privacy protection in public monitoring.

[0110] It should be noted that the present invention is a big data advanced modeling method for human behavior recognition in video surveillance scenarios. Through multi-dimensional data fusion, feature extraction and privacy protection, it can achieve accurate recognition and security management of complex video events. The main core of the present invention lies in: multimodal fusion of audio and image features, efficient feature extraction, bidirectional recurrent neural network time series processing, deep neural network classification, and differential privacy protection, which ensures accuracy, real-time and privacy in large-scale data.

[0111] It should be noted that the step of filtering and acquiring the audio and image data in the video segment in S102 described in this embodiment is implemented by the following steps:

[0112] S1021: extracting an audio signal from the video, removing background noise irrelevant to the audio signal by using spectrum subtraction and / or filtering methods, and obtaining a denoised audio signal A(t);

[0113] S1022: Use OpenCV to segment the video into static images frame by frame, use Laplace transform to detect blurred images, and use inter-frame difference method to remove redundant image frames, obtain the remaining m frames of static images, pre-process the remaining m frames of static images in sequence, obtain the m-frame processed static image P, and the L-th processed static image in the time sequence of the m-frame processed static image P is represented as PL, where L = {1, 2, 3 ... m}, and the steps of pre-processing the remaining m frames of static images in sequence are as follows:

[0114] S10221: Apply Gaussian smoothing filter to remove high-frequency noise in static images and reduce background interference with human activity monitoring.

[0115] S10222: Perform median filtering to further remove noise while retaining the outline of the person in the static image.

[0116] S10223: Use histogram equalization to enhance the contrast of each static image frame and highlight the details of the characters in the static image.

[0117] S10224: sharpen each still image frame using a Laplacian filter to highlight edges of characters in the still image.

[0118] It should be noted that in the image preprocessing process described in this embodiment, a series of filtering and enhancement techniques (S10221 to S10224) are applied to improve the detectability and feature saliency of human activities in static images, as described in detail as follows:

[0119] S10221: Application of Gaussian smoothing filter. The Gaussian smoothing filter smoothes the image pixel values ​​through convolution operation, mainly removing high-frequency noise in the image. Gaussian filtering can reduce background details and eliminate redundant information, so as to focus the detection on the human activity area. This step helps to eliminate the interference of irrelevant background factors and improve the accuracy of subsequent image processing and human detection.

[0120] S10222: Median filtering: Median filtering uses the median value in the neighborhood to replace each pixel value, effectively removing sudden noises such as salt and pepper noise while maintaining the edge information of the image. This filter can keep the outline of the person clear while removing noise, providing clean data input for further image analysis;

[0121] S10223: Histogram equalization: Histogram equalization enhances the visual effect by adjusting the contrast of the image. This method can significantly improve the uniformity of the brightness distribution in the image, making the details of the person (such as clothing, limbs, etc.) more obvious, which is beneficial to the edge detection and feature extraction in the subsequent steps;

[0122] S10224: Laplacian filter sharpening. The Laplacian filter improves the clarity of the edges of people in the image by detecting and enhancing the edges of the image. This step is particularly effective in highlighting the contours of people in the image, ensuring a clearer representation of the edge features of the person's activities, and providing a higher quality input image for the detection and recognition algorithm;

[0123] Through the combination of these four steps, the characteristics of human activities in static images are enhanced, and background noise and irrelevant information are significantly reduced, ensuring that the model can detect and identify significant behavioral features on clear and clean images.

[0124] In one embodiment of the present application, the audio feature f in the video segment in S102 audio and the image features f in the video segment pic This is accomplished by:

[0125] S2021: Perform a fast Fourier transform on the audio signal A(t), convert A(t) from the time domain to the frequency domain, and obtain the power spectrum of the spectrum |FFT[A(t)]| 2 ,

[0126] S2022: Divide the entire audio signal into T time windows, and calculate the frequency domain features of the audio signal for retrieval by N Mel filter banks in each time window t:

[0127] Smel(n,t)=MelFilterBank{∣FFT[A(t)]∣ 2},

[0128] As a possible case, MelFilterBank is a Mel filter bank, Smel(n,t) is the energy value of the audio signal after being processed by the nth filter in time window t, n is the index of the Mel filter, and each Mel filter corresponds to a different frequency range, n = 1, 2, 3...N, t = 1, 2, 3...T,

[0129] S2023: sort the N Smel(n,t) according to the energy size, and represent the Smel(n,t) sorted from large to small with serial numbers S1, S2, S3...SN, and sort the N Smel(n,t) according to the size and rank the energy value of the nth one,

[0130] S2024: Calculate the characteristic value RE of the audio signal in the time window t t :

[0131]

[0132] S2024: Combine the extracted feature values ​​of each time window in the video to form the audio feature f in the video segment audio :

[0133] f audio =[RE 1 , R.E. 2 , R.E. 3 …RE T ],

[0134] S2025: Extract the edges in PL based on the Prewitt algorithm, detect the contours of the person that may be contained in PL, and then use the morphological closing operation to fill the unclosed area that appears in the edge detection, and extract y closed target contours in PL.

[0135] S2026: Apply the Hu moment matching algorithm to compare the Euclidean distances of the Hu moments of the y closed target contours in PL with the Hu moments of the pre-stored human body shape templates, and use the minimum Euclidean distance obtained as DIS L , where the Hu moment matching algorithm and the Euclidean distance calculation are prior arts, and the specific calculation contents will not be described here.

[0136] S2027: Use a 3x3 sliding window to traverse PL, with a distance of 2 pixels between pixel pairs, and calculate the gray level co-occurrence matrix of the processed image in the directions of 0°, 30°, 60°, 90°, 120° and 150°, respectively, to obtain the contrast values ​​in the directions of 0°, 30°, 60°, 90°, 120° and 150°, and obtain the average value of the contrast value of PL in the directions of 0°, 30°, 60°, 90°, 120° and 150° Para1 L ,

[0137] At the same time, the homogeneity values ​​in the directions of 0°, 30°, 60°, 90°, 120° and 150° are obtained respectively, and the mean value of the homogeneity values ​​in the directions of 0°, 30°, 60°, 90°, 120° and 150° is obtained. L It should be noted that Para1 and Para2 are calculated and obtained by using the existing related functions in the OpenCV and Skimage libraries in the existing Python language, which will not be described in detail here.

[0138] S2028: Extract features from each frame of image and obtain the features of each frame of image in turn:

[0139] FE L =w1×DIs L +w2×Para1 L +w3×Para2 L ,

[0140] Among them, w1+w2+w3=1, and w1>w2+w3, FE L is the image feature value of PL, w1 is the shape matching weight value, w2 is the contrast weight value, w3 is the homogeneity weight value,

[0141] S2029: Based on steps S1025-S1028, image feature values ​​of the static image P after m frames of processing are sequentially generated, and the image feature values ​​of the static image P after m frames of processing are combined to form the image feature f in the video segment. pic :

[0142] f pico =[FE 1 , FE 2 , FE 3 …FE m ].

[0143] It should be noted that step S102 performs fast Fourier transform, Mel filter analysis and eigenvalue extraction on the audio data to obtain audio features. For image data, image features are extracted through edge detection, shape matching, grayscale co-occurrence matrix calculation, etc., and eigenvalues ​​are generated for each frame of the image. Finally, they are combined into an image feature vector to ensure that the audio and image features can accurately represent the behavioral information in the video segment.

[0144] In one embodiment of the present application, S103 is implemented by the following steps:

[0145] S1031: directly concatenate audio and image features to generate a fused feature vector F fuSi , where FfuSi = [faudio, fpic],

[0146] S1032: Using a bidirectional recurrent neural network to process temporal dependencies fuSi Perform early fusion to handle temporal dependencies and generate sequence features to generate early fusion feature vector F early :

[0147] F early =BiLSTM(F fuSi ),

[0148] Among them, F early is the BiLSTM for F fuSi The processed output features,BiLSTM is a bidirectional long short-term memory network of the existing technology.,BiLSTM consists of two independent LSTM networks: one processes backward from the beginning of the sequence, and the other processes forward from the end of the sequence.,At the final output, the features of the two directions are combined into a fused feature vector.

[0149] It should be noted that the fusion process in step S103 retains the key temporal information in the multimodal data in the video segment, making the model more capable of identifying emergencies.

[0150] In one embodiment of the present application, the final video segment event probability P endCalculated in the following way:

[0151]

[0152] P j is the early fusion feature vector F early The output probability of the jth classifier, j = {1, 2, 3…J}, C j is the weight of the j-th classifier, and The classifier is an SVM, random forest and / or deep neural network model for identifying human events in image and audio data, which is obtained by a technician in this field by pre-training a data set annotated with human behavior labels.

[0153] The deep neural network DNN is obtained by a technician in this field using a large number of video segments with human behavior labels for training supervision. The input of the deep neural network DNN is the fused feature vector P end , the output of the deep neural network DNN is the probability distribution of human behavior categories.

[0154] In one embodiment of the present application, the process of supervised training includes:

[0155] S1051: Input data P end Passed from the input layer to the output layer, the predicted output is generated through the calculation of neurons in each layer;

[0156] S1052: Use the cross entropy loss function to calculate the error between the predicted output and the true label to measure the prediction accuracy of the model;

[0157] S1053: Calculate the gradient of the error to each parameter in the network through the back propagation algorithm, and adjust the weights and bias items of neurons in each layer to reduce the error;

[0158] S1054: Use gradient descent and / or Adam optimization algorithm to update weights so that the loss function is gradually minimized;

[0159] S1055: After multiple iterations of training, the model gradually learns the mapping relationship between input features and the probability distribution of human behavior categories, and finally evaluates its performance on the validation set and test set to ensure the accuracy and generalization ability of the model;

[0160] It should be noted that, in the multiple iterative training processes described in this embodiment, the model gradually strengthens the mapping relationship between the input features and the probability distribution of human behavior categories, and continuously adjusts the weight parameters to approach the optimal state. The following is a specific training and evaluation process:

[0161] 1. Gradual learning and weight optimization: Through multiple iterations, the model deeply analyzes the input features under the guidance of the loss function, gradually capturing the complex relationship between features and behavior categories. After each iteration, the weights are adjusted according to error feedback (such as cross entropy loss or mean square error) to gradually improve the accuracy of prediction;

[0162] 2. Validation set monitoring and overfitting control: During the training process, the model is evaluated on the validation set after each round of iteration, and its performance is monitored in real time. If the accuracy on the validation set no longer improves or shows a downward trend, overfitting may occur. To this end, regularization (such as L2 regularization), early stopping mechanism, or data augmentation can be applied to improve generalization ability;

[0163] 3. Model fine-tuning and learning rate adjustment: In the later stage of training, in order to avoid missing the optimal solution, the learning rate can be gradually reduced and the update step of the weight can be refined (called learning rate decay) to more accurately optimize the model and fine-tune some parameters to make the model further fit the data characteristics;

[0164] 4. Final evaluation of the test set: After training, use an independent test set to conduct a final evaluation of the model performance. The test set contains samples that the model has not seen before. The accuracy, precision, recall, and F1 score are evaluated to ensure the generalization performance of the model. This step can verify the performance of the model on new data and is a key step in measuring whether the model is practical.

[0165] 5. Continuous optimization and model updating: As the data in the actual application of the model changes and the behavior categories diversify, continuous optimization can be performed, and the model can be fine-tuned periodically using new data, or re-training and validation evaluation can be performed to maintain the accuracy and robustness of the model.

[0166] The present invention integrates audio and image features and comprehensively utilizes multiple data information sources in video segments, significantly improving the recognition accuracy of behavioral features. It is adaptable to a variety of video surveillance and behavior recognition application scenarios, supports the recognition needs of different behavior types and complexities and the needs of efficient real-time processing of video behavior recognition, and meets the application needs of multiple fields such as intelligent monitoring, public safety, and behavior analysis.

[0167] In summary, the big data advanced modeling method of multi-dimensional data fusion in the embodiment of the present application significantly improves the data processing capability and result accuracy in the video surveillance system. By comprehensively analyzing multimodal video data, the method provides more comprehensive and in-depth monitoring information, effectively reduces manual intervention, improves the efficiency and accuracy of video analysis, and reduces operating costs. This technology supports real-time processing and updating of monitoring data, ensures the timeliness and accuracy of monitoring results, enables monitoring personnel to quickly locate key events, and reduces search time and energy waste. The method provides security personnel with more comprehensive event information and analysis support, helping them to better understand the background of events, analyze trends, and formulate targeted response strategies. In addition, the technology has good scalability and flexibility, and can adapt to video surveillance needs of different scales and types. As the amount of monitoring data continues to grow and change, the method can be continuously optimized and upgraded to maintain its advancement and competitiveness in the field of video surveillance.

[0168] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and deform the above embodiments within the scope of the present application.

Claims

1. A high-level big data modeling method for multi-dimensional data fusion, characterized in that: The big data advanced modeling method comprises the following steps: S101: Data cleaning: Clean the video streams captured by the camera, vehicle checkpoints, portrait capture data, and police enforcement data to remove noise data, duplicate data, and abnormal data to ensure data quality and accuracy, and provide a reliable data foundation for subsequent analysis of personnel flow, abnormal behavior identification, etc. Data conversion: Convert data from different sources and formats into a unified format for subsequent processing, including video streams captured by cameras, vehicle checkpoints, portrait capture data, and video images of police enforcement. Data standardization: normalize the data to ensure the consistency and comparability of various data sources, generate video segments of the same or similar length, and ensure consistency in duration, format, and resolution, so as to meet the multi-dimensional integration needs of traffic monitoring and behavior analysis; S102: Feature extraction and selection: Screen and obtain the audio and image data in the video segment, and extract the audio features f in the video segment. audio and the image features f in the video segment pic To obtain; S103: Fusing audio and image features to obtain an early fusion feature vector F early ; S104: Use weighted voting to fuse the results of the early fusion feature vectors in J classifiers to obtain the final video segment event probability P end ; S105: Feature modeling: Using a variety of learning methods to predict event probabilities P in video segments end Modeling is performed, and deep learning technology is used to perform high-level abstraction and representation of behavior categories. A large number of video segments with human behavior labels are used to supervise the training of the deep neural network DNN. The final video segment event probability is output as the probability distribution of each human behavior category. Model optimization: During the training process, grid search and random search cross-validation and parameter optimization methods are applied to optimize model performance and improve the prediction accuracy and stability of the model; S106: P end As input features, it is input into the deep neural network DNN for classification, and the probability distribution of each human behavior category in the video segment is obtained. The probability distribution of each human behavior category in the video segment is obtained through the Softmax function, so as to judge the most likely human behavior appearing in the video. The character behaviors include, but are not limited to, individual existence status, movement behavior, group characteristics, and abnormal activities, including basic standing and / or sitting postures, changes in walking and / or running position and direction, group activities of gathering and / or dispersing, and potential safety risk behaviors such as fighting, falling, wandering, and / or entering restricted areas, to support comprehensive behavior analysis needs in multiple scenarios; S107: Data encryption: Encrypt video segments, audio, and images to ensure data security during transmission and storage, and prevent unauthorized users from intercepting or viewing them. Access control: Set reasonable access rights and limit data access according to user identity and permission level to prevent unauthorized data operations and ensure that only authorized personnel are allowed to access sensitive data. Anonymization: Anonymize information involving individual identification, remove or cover up the direct identification elements of the data to avoid personal privacy leakage. Differential privacy protection: Use the Laplace mechanism to add differential privacy noise to the probability distribution of each human behavior category in the video segment, and store the noisy probability distribution data in a database or secure storage system to protect the output behavior category probability and ensure that external visitors cannot infer the individual's true behavior through the probability distribution results.

2. The big data advanced modeling method according to claim 1, characterized in that: The S103 is implemented by the following steps: S1031: directly concatenate audio and image features to generate a fused feature vector F fuSi , where FfuSi = [faudio, fpic], S1032: Using a bidirectional recurrent neural network to process temporal dependencies fuSi Perform early fusion to handle temporal dependencies and generate sequence features to generate early fusion feature vector F early : F early =BiLSTM(F fuSi ), Among them, BiLSTM bidirectional long short-term memory network, F early is the BiLSTM for F fuSi The processed output features are Furthermore, the final video segment event probability P end Calculated in the following way: P j is the early fusion feature vector F early The output probability of the jth classifier, j = {1, 2, 3…J}, C j is the weight of the j-th classifier, and The classifier is an SVM, random forest and / or deep neural network model for identifying human events in image and audio data, which is obtained by a technician in this field by pre-training a data set annotated with human behavior labels.

3. The big data advanced modeling method according to claim 2, characterized in that: The deep neural network DNN in S105 is obtained by a technician in this field by using a large number of video segments with human behavior labels for training supervision. The input of the deep neural network DNN is the fused feature vector P end , the output of the deep neural network DNN is the probability distribution of human behavior categories, The supervised training process includes: S1051: Input data P end Passed from the input layer to the output layer, the predicted output is generated through the calculation of neurons in each layer. S1052: Use the cross entropy loss function to calculate the error between the predicted output and the true label to measure the prediction accuracy of the model. S1053: Calculate the gradient of the error to each parameter in the network through the back propagation algorithm, and adjust the weights and bias items of each layer of neurons to reduce the error. S1054: Use gradient descent and / or Adam optimization algorithm to update weights so that the loss function is gradually minimized. S1055: After multiple iterations of training, the model gradually learns the mapping relationship between input features and the probability distribution of human behavior categories, and finally evaluates its performance on the validation set and test set to ensure the accuracy and generalization ability of the model.

4. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 6 are implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

6. A processor, characterized in that: The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 3 when running.

Images