Urban management illegal behavior recognition model optimization method

Through Transformer model and multimodal fusion technology, combined with model compression and expert feedback, the urban management violation identification model is optimized, which solves the shortcomings of CNN in global context and multimodal data processing, and achieves efficient and accurate identification of illegal behaviors. , and reduces the computing resource requirements.

CN118982784APending Publication Date: 2024-11-19CHINA COAL SCI & IND GRP CHONGQING SMART CITY SCI & TECH RES INST CO LTD +1
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411041267.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Traditional convolutional neural networks (CNNs) are difficult to capture global context information and process multimodal data in urban management, resulting in insufficient recognition accuracy and efficiency. The large Transformer model has high computing resources and storage requirements, making it difficult to meet practical application needs.

Method used

The Transformer model is used to build a city management violation identification model. Through data preprocessing, feature library construction, multimodal fusion and model compression technology, combined with expert feedback and supervision fine-tuning, the model is optimized to adapt to complex scenarios and resource-constrained environments.

Benefits of technology

It improves the accuracy and efficiency of identifying illegal behaviors, reduces the computing resource requirements, enhances the robustness and adaptability of the model, and ensures efficient operation in resource-constrained environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982784A_ABST
    Figure CN118982784A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of computer vision and artificial intelligence, in particular to a city management illegal behavior recognition model optimization method, which comprises the following steps: S100, collecting city management illegal behavior sample data; s200, preprocessing the illegal behavior sample data, including data cleaning and format conversion, and converting the data into a model input parameter format; s300, constructing an illegal behavior feature library based on a Transform model, and extracting attribute features and behavior features of illegal behaviors; s400, constructing a ViT-based city management illegal behavior identification large model, and performing training; and S500, optimizing the trained identification large model through a model compression technology, and constructing a small model suitable for micro server deployment. According to the method, training cost and computing resource requirements can be reduced, the precision and efficiency of urban management illegal behavior recognition are improved, and intelligentization and refinement of urban management are realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of computer vision and artificial intelligence, in particular to a method for optimizing a model for identifying illegal behavior in urban management. Background Art

[0002] With the construction of urban management informatization, the scale of urbanization continues to expand, which improves people's happiness index, but also increases the difficulty of urban management. In order to achieve efficient and intelligent urban management, machine learning and computer vision technologies are widely used at home and abroad to promote "non-contact law enforcement" and "smart law enforcement", aiming to improve law enforcement efficiency, reduce law enforcement risks, and expand law enforcement coverage.

[0003] In the existing technology, computer vision technology, as an important means of intelligent monitoring, processes digital images or videos by simulating the human visual system to identify and track target behaviors. Although intelligent video analysis technology can detect cases in a timely manner, reduce labor costs, and effectively utilize video surveillance resources, traditional convolutional neural networks (CNNs) have difficulty capturing global context information due to their inherent inductive preference and translation invariance, and they perform poorly when processing multimodal data. In addition, the limited receptive field of CNN limits its application effect in complex scenarios, making it difficult to meet the requirements for high precision and high efficiency in urban management.

[0004] In recent years, the Transformer model has made significant breakthroughs in the fields of natural language processing and computer vision. Unlike traditional CNN models, the Transformer model has a global receptive field and powerful multimodal fusion capabilities, and performs well in processing large-scale data. The emergence of Visual Transformer (ViT) has enabled the Transformer model to perform well in computer vision tasks such as image classification and object detection. However, existing large models face the problem of high computing resources and storage requirements in practical applications. Summary of the invention

[0005] The purpose of the present invention is to provide a method for optimizing a model for identifying urban management violations, which can reduce training costs and computing resource requirements and improve the accuracy and efficiency of identifying urban management violations.

[0006] The basic solution provided by the present invention: a method for optimizing a model for identifying illegal behaviors in urban management, comprises the following steps: S100, collecting sample data of illegal behaviors in urban management;

[0007] S200, preprocessing the illegal behavior sample data, including data cleaning and format conversion, and converting the data into a model input parameter format;

[0008] S300, build an illegal behavior feature library based on the Transformer model to extract the attribute characteristics and behavior characteristics of illegal behaviors;

[0009] S400, build a large model for identifying urban management violations based on ViT and conduct training;

[0010] S500, optimizes the trained large recognition model through model compression technology and builds a small model suitable for microserver deployment.

[0011] Furthermore, the preprocessing step includes: extracting illegal behavior information from the original video data collected by the camera and sensor; performing data cleaning on the original video data to remove noise and invalid information, black and white noise points and redundant features; and converting the format of the cleaned data to meet the sample input requirements of the Transformer model.

[0012] Furthermore, the construction of the illegal behavior feature library includes: encoding illegal behavior sample data through a Transformer model; extracting attribute characteristics and behavioral characteristics of illegal behaviors, and performing feature analysis and visualization.

[0013] Furthermore, the construction of the ViT-based large model for identifying urban management violations includes: constructing a multimodal fusion large model and using the CLIP method for cross-modal fusion; using the ViT model to train the preprocessed data; and adjusting the model input and output parameters according to the training results.

[0014] Furthermore, the model compression technology includes model distillation, which transfers the knowledge of the trained complex recognition large model to a specialized small model.

[0015] Furthermore, the model compression technology includes converting the matrix representation of the model into a Key-Values ​​representation through a quantization technology.

[0016] Furthermore, the model compression technology also includes using low-rank decomposition technology to decompose the weight matrix of the model into a low-dimensional matrix.

[0017] Furthermore, the model compression technology also includes using unstructured pruning to remove redundant neurons in the model and reduce the model size.

[0018] The principles and beneficial effects of the present invention are:

[0019] 1. Improve recognition accuracy and efficiency: Through data preprocessing, including data cleaning and format conversion, ensure the high quality of input data, thereby improving the accuracy of model training. Based on the Transformer model, build a feature library of illegal behavior, extract key features, and improve the accuracy of model recognition.

[0020] 2. Multimodal data processing: By constructing a large model for urban management violation identification based on ViT and using the CLIP method for cross-modal fusion, the model can process and fuse multiple data such as images and texts, thereby improving the comprehensiveness and robustness of identification.

[0021] 3. Reduce model complexity and computing resource requirements: Through model distillation technology, the knowledge of complex large models is transferred to simplified small models, while maintaining recognition accuracy, reducing computing resource requirements. Use model compression technology (quantization, low-rank decomposition, pruning) to further reduce the storage and computing overhead of the model, making it suitable for resource-constrained environments.

[0022] Furthermore, it also includes the use of RLHF method to adjust the output of the model through expert feedback;

[0023] The beneficial effect of this solution is that by introducing an expert feedback mechanism, the model can receive guidance and correction from professionals when identifying urban management violations, which can effectively solve the problem of misjudgment of the model in specific scenarios and further improve the accuracy and reliability of the model.

[0024] Furthermore, it also includes the use of supervised fine-tuning techniques to adjust and optimize the pre-trained model to adapt to specific tasks and data sets.

[0025] The beneficial effects of this solution are: based on the pre-trained model, retraining for specific tasks and data sets is performed to make the model better adaptable to different application scenarios. The supervised fine-tuning technology can refine the model's feature extraction capabilities and decision-making process, making it perform better in various complex urban management violation scenarios. By adjusting the model parameters and training strategies in a targeted manner, the model can more accurately identify and handle a variety of violations, improving the effect in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Flowchart of the optimization method for the urban management violation identification model. DETAILED DESCRIPTION

[0027] Embodiment 1

[0028] This embodiment describes an optimization method for an urban management illegal behavior identification model, including detailed steps of data collection, preprocessing, feature library construction, model training, model optimization and deployment. This embodiment aims to achieve efficient and accurate urban management illegal behavior identification through computer vision and machine learning technology.

[0029] like Figure 1 The urban management illegal behavior identification model optimization method shown includes the following steps:

[0030] S100: Collect sample data on urban management violations.

[0031] In this embodiment, the collection of sample data is the basis for building an urban management illegal behavior identification model. The sample data mainly includes video and image data from urban surveillance cameras and environmental sensors.

[0032] Install high-resolution surveillance cameras with night vision and wide-angle shooting capabilities at major traffic arteries, squares, parks and public places in the city. Deploy environmental sensors (such as noise sensors and air quality sensors) to assist in obtaining relevant information on illegal activities.

[0033] Cameras and sensors collect video and image data in real time, and the data is transmitted to the city management data center via wireless or wired networks. The data center stores and preliminarily processes the collected raw data to ensure the integrity and security of the data.

[0034] S200: Preprocessing illegal behavior sample data.

[0035] Preprocessing is a key step to ensure data quality and consistency. The preprocessing steps in this embodiment include data cleaning, format conversion, etc.

[0036] Data cleaning: De-noise the original video data and use filtering technology to remove black and white noise in the video. Delete redundant features and invalid information, such as long static images or irrelevant scenes, and retain key frames and clips of important events.

[0037] Format conversion: Convert the cleaned data into a format that meets the sample input requirements of the Transformer model, including unified resolution, frame rate, and encoding format. Perform key frame slicing on the video data to divide the video into fixed-length segments for subsequent feature extraction and analysis.

[0038] S300: Build an illegal behavior feature library based on the Transformer model.

[0039] The construction of the feature library is the key to improving the accuracy of model recognition. In this embodiment, the attribute features and behavior features of illegal behaviors are extracted based on the Transformer model.

[0040] The collected raw video data is preprocessed and converted into segments that conform to the input format of the Transformer model, each of which includes a fixed number of frames. The label information corresponding to each video segment (such as the type of violation, timestamp, etc.) is also prepared as reference data for supervised learning.

[0041] Use the pre-trained Transformer model to encode the pre-processed video clips. The input of the Transformer model is a sequence of video frames, and the output is a high-dimensional feature vector. During the encoding process, each frame of the image is converted into a specific vector representation. Through the self-attention mechanism, the Transformer model can identify the global relationship and temporal dependency between frames.

[0042] For each video clip, a high-dimensional feature vector is extracted. These feature vectors contain the behavioral characteristics and attribute characteristics of the illegal behavior. The behavioral characteristics include the type of action (such as littering, spitting, illegal parking, etc.), the duration of the action, etc. The attribute characteristics include the number of people, the type of vehicle, the time and place of the behavior, etc.

[0043] Perform cluster analysis on the extracted high-dimensional feature vectors and use clustering algorithms to identify common illegal behavior patterns. Through cluster analysis, similar illegal behavior fragments can be classified into one category, thereby identifying the types of illegal behavior that occur frequently. The illegal behavior patterns and their characteristics obtained by cluster analysis are stored in the feature library. The feature library contains information such as feature descriptions, example fragments, and frequency of occurrence of each type of illegal behavior.

[0044] Use visualization tools to reduce the dimensionality of high-dimensional feature vectors and map them into two-dimensional or three-dimensional space for easy understanding and display. By visualizing the distribution of features and behavior patterns, you can intuitively observe the similarities and differences between different violations, helping to analyze and understand the internal structure and relationship of the data.

[0045] S400: Build and train a large model for identifying urban management violations based on ViT.

[0046] In this embodiment, multimodal data is first prepared, including preprocessed image data and corresponding text description information. The image data comes from the video frame of the illegal behavior captured by the camera, and the text description records the type, time, location and other information of the behavior. These data are cross-modally fused through the CLIP (Contrastive Language-Image Pretraining) method. The CLIP method uses contrastive learning to map images and texts to the same feature space, so that the model can simultaneously process and understand multimodal data.

[0047] Specifically, the CLIP model includes a visual encoder and a text encoder. The visual encoder uses the pre-trained ViT model to divide the image data into fixed-size patches. Each patch is linearly embedded and positionally encoded, and then input into the Transformer encoder. The text encoder uses the Transformer architecture to convert text descriptions into embedding vectors. The two encoders share parameters to ensure that the embedding spaces of images and texts are consistent. During training, a contrastive loss function is used to measure the difference between matched and mismatched image-text pairs, allowing the model to learn the semantic associations between images and texts.

[0048] After preparing the multimodal data, it is input into the large model of illegal behavior recognition based on ViT for training. The core of the ViT model is the self-attention mechanism, which captures the global relationship between image blocks through multi-head self-attention layers and feedforward neural network layers to generate high-dimensional feature representations.

[0049] During the training process, data augmentation techniques are used to process image data to improve the generalization ability of the model. Commonly used data augmentation techniques include rotation, cropping, and flipping. These techniques increase the diversity of data and enhance the model's ability to adapt to different directions and perspectives.

[0050] In order to ensure that the model can accurately identify different types of violations, the model is quantitatively evaluated. The performance of the model is verified on the test set, and the model's accuracy, recall rate, F1 score and other indicators are evaluated.

[0051] Accuracy measures the proportion of samples that the model predicts correctly to the total number of samples, recall measures the proportion of actual violations that the model can correctly identify, and the F1 score takes into account the harmonic mean of accuracy and recall to provide an assessment of the overall performance of the model. By analyzing the errors that the model makes in the test set, we can identify the shortcomings of the model in certain types of violations and provide a basis for subsequent improvements.

[0052] S500: Optimize the trained large recognition model through model compression technology and build a small model suitable for microserver deployment.

[0053] In this step, the trained urban management illegal behavior recognition large model is optimized through large model multimodal compression technology to build a small model suitable for microserver deployment and usage environment specialization. This process can not only ensure the recognition accuracy of the model, but also greatly reduce the model's computing resource requirements and storage space, thereby improving the model's actual application performance and deployment flexibility.

[0054] First, the application of model distillation technology. Model distillation technology is a method of transferring knowledge from a complex large model (teacher model) to a specialized small model (student model). In specific operations, the teacher model is first trained to achieve a high level of accuracy. Then, the soft labels generated by the teacher model are used to guide the training of the student model. The soft label contains the predicted probability distribution of the teacher model for each input sample, which can provide more learning information than the hard label. Through the distillation process, the student model can significantly reduce the number of parameters and computational overhead while maintaining recognition accuracy.

[0055] Secondly, it also includes methods such as quantization, pruning and low-rank decomposition:

[0056] Quantization technology: Quantization technology converts the floating-point weights and activation values ​​in the model into low-precision representations (such as 8-bit integers) to reduce the model's storage requirements and computational complexity. During the quantization process, it is necessary to balance the accuracy loss and compression rate to ensure that the quantized model achieves the best balance between performance and efficiency.

[0057] Unstructured pruning technology: Unstructured pruning technology reduces the size and number of neurons in the model by removing unimportant or redundant parameters in the model. Specific methods include pruning connections with small weights and removing neurons and layers that contribute less to the output. The pruned model needs to be retrained to restore and improve model performance.

[0058] Low-rank decomposition technology: Low-rank decomposition technology decomposes the high-dimensional weight matrix of the model into multiple low-dimensional matrix products, thereby reducing the number of model parameters and the amount of calculation. Through low-rank decomposition, the model can maintain high recognition accuracy while reducing resource consumption.

[0059] After the model is distilled and compressed, the optimized small model is deployed on the microserver. Since the small model has greatly reduced requirements for computing resources and storage space, it can run efficiently on edge devices or low-performance servers, meeting the needs of real-time illegal behavior identification in urban management.

[0060] Embodiment 2

[0061] This embodiment also includes adopting the RLHF method to adjust the output results of the model through expert feedback to further improve the accuracy and adaptability of the model.

[0062] The RLHF method is an algorithm based on reinforcement learning that optimizes the learning effect of intelligent systems by combining the knowledge and experience of experts. In RLHF, the model first makes preliminary predictions and provides these predictions to experts for review. Experts will provide feedback on the output of the model based on their own experience and knowledge, indicating which predictions are correct, which are wrong, or which areas need improvement. This feedback information is converted into reward and punishment signals and fed back to the reinforcement learning algorithm to guide further training of the model.

[0063] In the specific implementation, a sample data set containing urban management violations is first constructed, and the initial model is used to predict these data. The prediction results are submitted to experts for review together with the original data. The experts evaluate each prediction result and give corresponding feedback, such as "correct", "wrong" or specific improvement suggestions. Then, the expert feedback is converted into reward and punishment signals in reinforcement learning, and these reward and punishment signals are used to update the parameters of the model.

[0064] By utilizing the expertise and practical experience of experts, the large amount of trial and error required during model training can be effectively reduced. Compared with traditional reinforcement learning methods, the RLHF method enables the model to learn effective strategies more quickly and improve learning efficiency. At the same time, through continuous expert feedback and model optimization, the recognition accuracy of the model can be continuously improved, making the model output results more in line with actual application needs.

[0065] Embodiment 3

[0066] This embodiment also includes using supervised fine-tuning technology to adjust and optimize the pre-trained model to adapt to specific tasks and data sets.

[0067] In this embodiment, the pre-trained model is adjusted and optimized by using supervised fine-tuning technology to adapt to specific urban management illegal behavior identification tasks and data sets. Supervised fine-tuning technology refers to re-training and optimizing the pre-trained model using a labeled specific task data set to improve the performance and adaptability of the model in a specific task.

[0068] First, we collect clearly labeled illegal behavior data from actual business scenarios of urban management, including multimodal data such as images, videos, and text descriptions. After preprocessing, these data form a structured training dataset to ensure data quality and consistency. Then, these labeled data are input into the pre-trained model for supervised fine-tuning.

[0069] Through supervised fine-tuning, the pre-trained model can better adapt to the data characteristics of the urban management illegal behavior identification task, significantly improving the accuracy and robustness of identification. Supervised fine-tuning technology can not only optimize the performance of the model in specific tasks, but also further improve the generalization ability of the model through labeled data, ensuring the stability and reliability of the model in practical applications.

[0070] Embodiment 4

[0071] In this embodiment, prompt engineering technology is also used to generate the required prompts for specific tasks. Prompt engineering technology refers to using the context understanding and natural language generation capabilities of large models to generate the required prompts for specific tasks. Prompt engineering technology mainly involves the fields of natural language processing and machine learning. By designing appropriate prompt words, the model is guided to output results that better meet the specific task requirements.

[0072] In this embodiment, prompt engineering technology is applied to the task of urban management violation identification to leverage the contextual understanding and natural language generation capabilities of the large model to generate the required prompts for specific tasks.

[0073] Collect and organize text and image data related to urban management violations. This data includes descriptions of violations, locations, times, and other information. This information will be used as prompts to help the model better understand the task context and specific requirements.

[0074] Generate appropriate prompt words based on the specific needs of the city management business scenario. For example, when dealing with illegal parking, you can design prompt words such as "Please describe whether there is illegal parking in the image" or "Is there any vehicle violating traffic rules in this image?" These prompt words will be used as input to help the model generate more targeted and accurate results.

[0075] Input the prompt words and related data into the pre-trained large model. The large model uses its powerful context understanding ability to parse the prompt words and generate corresponding output results based on the input data. In this way, the model can quickly generate results that meet the task requirements without requiring a lot of human involvement.

[0076] During the training process, the output quality of the model is further improved by continuously adjusting and optimizing the design of the prompt words. For example, the model output results are analyzed to identify common errors and deficiencies, and then the prompt words are improved to make them more specific and clear, thereby guiding the model to generate more accurate results. At the same time, the language style and expression of the prompt words are adjusted in combination with actual application scenarios to make them closer to actual business needs.

[0077] Prompt engineering technology can quickly generate the prompts required for the task, saving time and labor costs, and the prompts generated by prompt engineering technology can often more accurately reflect the user's intentions and needs, improving the quality and effect of task completion. By applying prompt engineering technology in this embodiment and designing appropriate prompt words in combination with business scenarios, the efficiency and accuracy of urban management illegal behavior identification can be improved.

[0078] The above are only embodiments of the present invention. Common knowledge such as the known specific structures and characteristics in the scheme is not described in detail here. Ordinary technicians in the relevant field know all the common technical knowledge in the technical field to which the invention belongs before the application date or priority date, can obtain all the existing technologies in the field, and have the ability to apply conventional experimental means before that date. Ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the scope of protection of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. The optimization method of the urban management illegal behavior identification model is characterized by: The following steps are involved: S100, collect sample data of urban management violations; S200, preprocessing the illegal behavior sample data, including data cleaning and format conversion, and converting the data into a model input parameter format; S300, build an illegal behavior feature library based on the Transformer model to extract the attribute characteristics and behavior characteristics of illegal behaviors; S400, build a large model for identifying urban management violations based on ViT and conduct training; S500, optimizes the trained large recognition model through model compression technology and builds a small model suitable for microserver deployment.

2. The urban management illegal behavior identification model optimization method according to claim 1 is characterized in that: The preprocessing step includes: extracting illegal behavior information from the original video data collected by the camera and the sensor; cleaning the original video data to remove black and white noise and redundant features; and converting the format of the cleaned data to meet the sample input requirements of the Transformer model.

3. The urban management illegal behavior identification model optimization method according to claim 2 is characterized in that: The construction of the illegal behavior feature library includes: encoding illegal behavior sample data through a Transformer model; extracting attribute characteristics and behavioral characteristics of illegal behaviors, and performing feature analysis and visualization.

4. The urban management illegal behavior identification model optimization method according to claim 3 is characterized in that: The construction of the ViT-based urban management illegal behavior identification large model includes: constructing a multimodal fusion large model and using the CLIP method for cross-modal fusion; using the ViT model to train the pre-processed data; and adjusting the model input and output parameters according to the training results.

5. The urban management illegal behavior identification model optimization method according to claim 4 is characterized in that: The model compression technology includes model distillation, which transfers the knowledge of a well-trained complex recognition large model into a specialized small model.

6. The urban management illegal behavior identification model optimization method according to claim 5 is characterized in that: The model compression technology includes converting the matrix representation of the model into a Key-Values ​​representation through a quantization technology.

7. The urban management illegal behavior identification model optimization method according to claim 6 is characterized in that: The model compression technology also includes using low-rank decomposition technology to decompose the weight matrix of the model into a low-dimensional matrix.

8. The urban management illegal behavior identification model optimization method according to claim 7 is characterized in that: The model compression technology also includes using unstructured pruning to remove redundant neurons in the model and reduce the model size.

9. The urban management illegal behavior identification model optimization method according to claim 1 is characterized in that: It also includes the use of RLHF methods to adjust the model's output parameters and result generation through expert feedback.

10. The urban management illegal behavior identification model optimization method according to claim 1 is characterized in that: It also includes the use of supervised fine-tuning techniques to adjust and optimize pre-trained models to suit specific tasks and datasets.

Citation Information

Patent Citations

  • Training method of image search model and image search method

    CN113590865A

  • Algorithm model for rare tumor category small sample classification based on cost sensitivity

    CN114170426A

  • Feature decoupling small-amount sample pre-training model robustness fine tuning method and device

    CN116994099A

  • Traffic violation model training process anomaly detection method

    CN117648658A

  • Liability cognizance method and device of traffic accident and computer readable storage medium

    WO2020024457A1