Smoking identification method based on multi-modal large model

Through the multimodal large model combined with YOLOv8 and OpenClip ViT-bigG, the problems of high false alarm rate and weak generalization ability of traditional smoking detection methods are solved, and high-precision and efficient smoking behavior recognition are achieved. It is suitable for monitoring systems in complex scenarios. .

CN119027855APending Publication Date: 2024-11-26SHENZHEN VIRTUAL CLUSTERS INFORMATION TECH

Patent Information

Application Number
CN202411123758.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The traditional smoking detection method is based on convolutional neural networks, and the semantic expression ability of the image is insufficient, resulting in high false positive rate and weak generalization ability in complex environments and diverse behavioral patterns, making it difficult to effectively identify smoking behavior.

Method used

Using a smoking recognition method based on multimodal large model, through video stream data acquisition, human object detection, image and text alignment, and multimodal large model construction and training, combined with YOLOv8 object detection model and OpenClip ViT-bigG visual encoder, Incremental fine-tuning is used to form a multimodal large model that can identify smoking behavior.

Benefits of technology

It significantly improves the recognition accuracy and generalization ability of smoking behavior, reduces the computational complexity, enhances the real-time and adaptability of the system, and is suitable for the deployment of large-scale monitoring systems, and reduces the model's computing resource consumption.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027855A_ABST
    Figure CN119027855A_ABST
Patent Text Reader

Abstract

The invention relates to a smoking identification method based on a multi-modal large model, and aims to improve the accuracy and reliability of smoking behavior detection in a complex scene. According to the method, firstly, monitoring video stream data is collected in real time and decomposed into multiple frames of images, an optimized YOLOv8 model is used for human body target detection and labeling, the labeled images and text cues are subjected to alignment training through an OpenClip ViT-bigG visual encoder and a Qwen-7B language model, and a picture-cue alignment model is formed. Then, on the basis of a Qwen-VL model framework, an LoRA fine tuning method is used for carrying out increment fine tuning training on the marked data, and VLM model parameters used for smoking recognition are generated; in practical application, video stream images and prompt words which are input in real time are processed through the trained YOLOv8 and VLM models, and accurate recognition and output of smoking behaviors are achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and computer vision, and in particular to a smoking recognition method based on a multimodal large model. Background Art

[0002] In today's society, smoking not only poses a serious threat to personal health, but is also an important behavior that needs to be strictly controlled in public places, enterprises, institutions, etc. With the rapid development of artificial intelligence and computer vision technology, the application of intelligent monitoring systems in smoking behavior detection has received increasing attention.

[0003] However, traditional smoking detection methods mainly rely on image object detection technology based on convolutional neural networks (CNN). Such methods are usually limited by the weak ability of image semantic expression and are prone to mistaking similar smoking actions or objects for real smoking behaviors, resulting in a high false alarm rate.

[0004] In addition, existing detection systems exhibit weak generalization capabilities when dealing with complex environments and diverse behavioral patterns, and are unable to meet the needs of smoking behavior detection under various changes.

[0005] In recent years, with the development of deep learning technology, especially the emergence of large multimodal models, a new technical approach has been provided for smoking behavior detection.

[0006] By integrating multimodal data such as images and text, the multimodal large model can better understand and express complex behavior patterns, significantly improving the accuracy and robustness of detection.

[0007] Therefore, the smoking behavior detection method based on multimodal large models has broad application prospects and has gradually become the focus and difficulty of research in this field. Summary of the invention

[0008] In order to solve the above problems, the present invention provides a smoking recognition method based on a multimodal large model to improve the accuracy and efficiency of smoking behavior detection and meet application requirements in complex scenarios.

[0009] The present invention is implemented by the following technical solution: a smoking recognition method based on a multimodal large model, the method comprising the following steps:

[0010] a. Video stream data collection and image decomposition:

[0011] Collect video stream data of the monitoring scene in real time, decompose the video stream data into multiple frames of images, and form an image set for subsequent processing;

[0012] b. Human target detection and labeling:

[0013] The human targets in the image set are detected using an optimized YOLOv8 target detection model. The YOLOv8 model improves the accuracy and speed of human target detection by improving the Backbone and Neck structures, including adjusting the convolution kernel size of the first shallow convolution layer from 6×6 to 3×3 and replacing the C3 structure with the C2f structure; the smoking behavior and position of the human targets are annotated according to the detection results;

[0014] c. Alignment between image and prompt word:

[0015] The OpenClip ViT-bigG model is used as a visual encoder to process the annotated images, and combined with the Qwen-7B language model, the image and text prompt words are trained to align and form a picture-prompt word alignment model;

[0016] d. Multimodal large model construction and training:

[0017] Based on the Qwen-VL model framework, multimodal data integration processing is performed, and incremental fine-tuning training of the VLM model is performed on the labeled smoking data. The fine-tuning process is performed using the LoRA (Low-Rank Adaptation) method to obtain the fine-tuned VLM model parameters;

[0018] e. Real-time smoking recognition and result output:

[0019] In actual monitoring scenarios, the real-time collected video stream images and prompt words are input into the trained YOLOv8 model and VLM model to identify the smoking behavior in the image and output the recognition results, including the location and time when the smoking behavior occurred.

[0020] As a preferred technical solution, the Neck part of the YOLOv8 target detection model adopts a combination of FPN and PAN to enhance the fusion capability of shallow features and deep features.

[0021] As a preferred technical solution, the Head stage of the YOLOv8 model adopts an Anchor-free implementation method to predict the grid points on each feature map.

[0022] As a preferred technical solution, the Head stage of the YOLOv8 model adopts an Anchor-free implementation method to predict the grid points on each feature map.

[0023] As a preferred technical solution, in the step of aligning the image and the prompt word, a visual Transformer model pre-trained by CLIP is used to optimize the alignment accuracy of the image and text features.

[0024] As a preferred technical solution, the Qwen-VL model enhances the fusion effect of multimodal features by mapping image features and language features into the same feature space.

[0025] As a preferred technical solution, during the fine-tuning process of the VLM model, the parameters of the visual encoder and the language encoder are frozen, and only the parameters of the picture mapper are adjusted to ensure effective alignment of image features.

[0026] As a preferred technical solution, in the real-time smoking recognition step, by inputting a prompt word containing a preset question, it is determined whether there is smoking behavior in the human body image, and the specific coordinates and occurrence time of the smoking behavior are output.

[0027] As a preferred technical solution, the annotated data file adopts a fixed json format, and the json file includes the corresponding relationship between the image annotation and the prompt word, as well as the corresponding coordinate frame and description text.

[0028] As a preferred technical solution, the fine-tuned VLM model is merged with the main model to improve the recognition accuracy of smoking behavior during reasoning.

[0029] The beneficial effects of the present invention are as follows: the present invention significantly improves the recognition accuracy and generalization ability of smoking behavior by introducing the technical framework of a multimodal large model. Traditional monitoring methods usually rely on convolutional neural networks for image target detection, which is easily limited by the ability to express image semantics, resulting in frequent false alarms. In contrast, the present invention uses the YOLOv8 target detection model for human target detection, and combines the OpenClip ViT-bigG visual encoder and the Qwen-VL multimodal model to align and fine-tune the image and text prompt words, forming a multimodal large model that can accurately identify smoking behavior. By integrating the powerful learning ability of the large model with multimodal data, the present invention effectively improves the reliability and accuracy of smoking behavior detection in complex scenarios, while reducing the computational complexity, enhancing the real-time and adaptability of the system, and is suitable for the deployment and application of large-scale monitoring systems. In addition, the present invention reduces the computing resource consumption of the model by optimizing the model structure and algorithm flow, so that it can adapt to a wider range of application environments while maintaining high performance, and has important practical application value and social significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 The figure is an overall flow chart of the smoking identification method based on the multi-modal large model of the present invention;

[0032] Figure 2 Schematic diagram of the comparison between C3 and C2f structures in the YOLOv8 model of the present invention;

[0033] Figure 3 This is the overall network structure diagram of YOLOv8 for detecting human body in the present invention;

[0034] Figure 4 Schematic diagram of text encoding and image encoding process in the CLIP model of the present invention;

[0035] Figure 5 Schematic diagram of the image-text comparison learning process in the pre-training stage of the CLIP model of the present invention;

[0036] Figure 6 A schematic diagram of comparative learning of image and text feature vectors of the CLIP model of the present invention;

[0037] Figure 7 It is a multi-modal processing flow chart of the present invention;

[0038] Figure 8 It is a fine-tuning flow chart of the present invention. DETAILED DESCRIPTION

[0039] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any manner.

[0040] Any feature disclosed in this specification (including any additional claims, abstract and drawings), unless otherwise stated, may be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

[0041] like Figure 1 As shown, a smoking recognition method based on a multimodal large model of the present invention comprises the following steps:

[0042] a. Video stream data collection and image decomposition:

[0043] Collect video stream data of the monitoring scene in real time, decompose the video stream data into multiple frames of images, and form an image set for subsequent processing;

[0044] b. Human target detection and labeling:

[0045] The human targets in the image set are detected using an optimized YOLOv8 target detection model. The YOLOv8 model improves the accuracy and speed of human target detection by improving the Backbone and Neck structures, including adjusting the convolution kernel size of the first shallow convolution layer from 6×6 to 3×3 and replacing the C3 structure with the C2f structure; the smoking behavior and position of the human targets are annotated according to the detection results;

[0046] c. Alignment between image and prompt word:

[0047] The OpenClip ViT-bigG model is used as a visual encoder to process the annotated images, and combined with the Qwen-7B language model, the image and text prompt words are trained to align and form a picture-prompt word alignment model;

[0048] d. Multimodal large model construction and training:

[0049] Based on the Qwen-VL model framework, multimodal data integration processing is performed, and incremental fine-tuning training of the VLM model is performed on the labeled smoking data. The fine-tuning process is performed using the LoRA (Low-Rank Adaptation) method to obtain the fine-tuned VLM model parameters;

[0050] e. Real-time smoking recognition and result output:

[0051] In actual monitoring scenarios, the real-time collected video stream images and prompt words are input into the trained YOLOv8 model and VLM model to identify the smoking behavior in the image and output the recognition results, including the location and time when the smoking behavior occurred.

[0052] In this embodiment, the Neck part of the YOLOv8 target detection model adopts a combination of FPN and PAN to enhance the fusion capability of shallow features and deep features.

[0053] Among them, the Head stage of the YOLOv8 model adopts an Anchor-free implementation method to predict the grid points on each feature map.

[0054] Among them, the Head stage of the YOLOv8 model adopts an Anchor-free implementation method to predict the grid points on each feature map.

[0055] In the step of aligning the image and the prompt word, a visual Transformer model pre-trained by CLIP is used to optimize the alignment accuracy of image and text features.

[0056] Among them, the Qwen-VL model enhances the fusion effect of multimodal features by mapping image features and language features into the same feature space.

[0057] In this embodiment, during the fine-tuning process of the VLM model, the parameters of the visual encoder and the language encoder are frozen, and only the parameters of the picture mapper are adjusted to ensure effective alignment of image features.

[0058] Among them, in the real-time smoking recognition step, by inputting a prompt word containing a preset question, it is determined whether there is smoking behavior in the human image, and the specific coordinates and occurrence time of the smoking behavior are output.

[0059] In this embodiment, the annotated data file adopts a fixed json format, and the json file includes the correspondence between the image annotation and the prompt word, as well as the corresponding coordinate box and description text. The fine-tuned VLM model is merged with the main model to improve the recognition accuracy of smoking behavior during reasoning.

[0060] like Figure 2 As shown in the figure, the YOLOv8 human detector optimizes the network structure based on the YOLOv5 human detector. The specific optimization is reflected in:

[0061] 1) The convolution kernel size of the first shallow convolutional layer is changed from 6*6 to 3*3, which reduces the receptive field

[0062] 2) C3 in Backbone and Neck is changed to C2f. C2f is different from the multi-channel feature map fusion of C3. It adopts a multi-layer stacked fusion structure, similar to the skip link in the residual block, which enhances the conduction of gradients and reduces the complexity of calculation.

[0063] 3) The SPP module is modified into the SPPF module with the same function to further reduce the computational complexity of the model

[0064] 4) The Neck part uses a combination of FPN and PAN to fuse shallow features with deep features.

[0065] 5) In the Head stage, the implementation method is changed from Ancher base to Anchor free, and each grid point on the feature map is predicted.

[0066] The overall network structure diagram of YOLOv8 for human body detection is as follows Figure 3 shown.

[0067] Specifically, large model data preprocessing and feature vector fusion

[0068] 1) Multimodal data annotation:

[0069] The input of the Qwen-VL model is a picture + a prompt word. After obtaining a picture of a person smoking, the picture is annotated with prompt words to form fine-tuning training data for picture-prompt word-answer pairs. The training data annotation file of Qwen-VL is in a fixed json format. The annotation results are written into the file according to the following rules: [

[0071] {

[0072] "id":"identity_0",

[0073]

[0074] in , <ref>< / ref> , <box>< / box> Are special character tokens

[0075] For content with pictures, it can be represented as: Picture id: img_path\n{my prompt}, where id indicates the image number in the dialog, and "img_path" can be a local image or a network image address.

[0076] The detection box in the conversation can be expressed as <box> (x1,y1),(x2,y2)< / box> , where (x1, y1) and (x2, y2) correspond to the coordinates of the upper left corner and lower right corner respectively, and are normalized to the range of [0, 1000). The text description corresponding to the detection box can also be obtained by <ref> text_caption< / ref> express.

[0077] 2) LLM pre-training

[0078] The Qwen7B model is a model that uses a Transformer network to perform unsupervised learning training on 2.4 trillion tokens of data, including a large amount of online text, professional books, and codes. It supports Chinese, English, and multiple languages, and has a vocabulary of about 150,000 words. Qwen7B has a strong feature expression ability for the text after word segmentation.

[0079] 3) Image Encoder

[0080] When dealing with tasks that combine text and images, since there is no similarity between the features of text and images, the features of the text and images must be aligned and unified before training the text and image model. In multimodal visual encoding, if a classification backbone is used, the effect is very poor. A semantically aligned backbone, such as CLIP's ViT, will have much better results.

[0081] The ViT-bigG model trained using the open source framework OpenCLIP supports input with a resolution of 448*448 and achieves 80.1% Zero-Shot accuracy on the ImageNet dataset. CLIP consists of two models, one is the Transformer encoder for embedding text, and the other is the Visual Transformer (ViT) for embedding images. Figure 4 shown.

[0082] In the pre-training phase, the image-text pairs are grouped in the same batch to bring the output vectors closer together, while separating the image and text vectors that do not belong to a pair, such as Figure 5 shown.

[0083] During prediction, the input image is encoded using the ViT model to obtain the image vector, and the cosine distance is calculated with its category vector as the similarity. If the similarity with T2 is the highest, it belongs to the person category, such as Figure 6 shown.

[0084] Specifically, the smoking behavior recognition module:

[0085] The smoking behavior recognition module is a VLM fine-tuned based on the multimodal Qwen-VL model. The process includes two stages: Qwen-VL pre-training and smoking data fine-tuning. Since the feature vector lengths of CLIP in the Text Encoder and Image Encoder stages are inconsistent, a linear mapper Image Projector is used to map the feature vectors of the two to the same feature dimension space during the process of aligning the vectors of the two. The multimodal processing flow is as follows: Figure 7 As shown:

[0086] 1) In the pre-training stage, multiple image-text pairs are converted into a single-round dialogue. The question in each round of dialogue is to generate a content description corresponding to the image. During the pre-training process, the parameters of the visual encoder and language encoder are frozen, and only the mapper parameters of the image are adjusted. This can maximize the alignment of the image features to the language feature space.

[0087] 2) In the fine-tuning stage, use the labeled smoking picture-prompt word-answer pair data to form multi-round and single-question question-answering instructions, freeze the parameters of the visual encoder, update the parameters of the mapper and LLM, and let the LLM form the ability to have picture input and multi-round question-answering. Fine-tuning is performed in the LoRA way. After fine-tuning, the fine-tuned model is merged with the main model. The merged model is used for reasoning to answer whether the picture of the human body has smoking behavior and the specific coordinates of the smoking action by asking questions. The process of fine-tuning is as follows: Figure 8 shown.

[0088] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that are not conceived through creative work should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope defined in the claims.

Claims

1. A smoking recognition method based on a multimodal large model, characterized in that: The method comprises the following steps: a. Video stream data collection and image decomposition: Collect video stream data of the monitoring scene in real time, decompose the video stream data into multiple frames of images, and form an image set for subsequent processing; b. Human target detection and labeling: The human targets in the image set are detected using an optimized YOLOv8 target detection model. The YOLOv8 model improves the accuracy and speed of human target detection by improving the Backbone and Neck structures, including adjusting the convolution kernel size of the first shallow convolution layer from 6×6 to 3×3 and replacing the C3 structure with the C2f structure; the smoking behavior and position of the human targets are annotated according to the detection results; c. Alignment between image and prompt word: The OpenClip ViT-bigG model is used as a visual encoder to process the annotated images, and combined with the Qwen-7B language model, the image and text prompt words are trained to align and form a picture-prompt word alignment model; d. Multimodal large model construction and training: Based on the Qwen-VL model framework, multimodal data integration processing is performed, and the VLM model is incrementally fine-tuned for the labeled smoking data. The fine-tuning process uses LoRA (Low-Rank Adaptation) method is used to obtain the fine-tuned VLM model parameters; e. Real-time smoking recognition and result output: In actual monitoring scenarios, the real-time collected video stream images and prompt words are input into the trained YOLOv8 model and VLM model to identify the smoking behavior in the image and output the recognition results, including the location and time when the smoking behavior occurred.

2. The method for smoking identification based on a multimodal large model according to claim 1, characterized in that: The Neck part of the YOLOv8 target detection model adopts a combination of FPN and PAN to enhance the fusion capability of shallow features and deep features.

3. The method for smoking identification based on a multimodal large model according to claim 1, characterized in that: The Head stage of the YOLOv8 model adopts an Anchor-free implementation method to predict the gridpoint on each feature map.

4. The method for smoking identification based on a multimodal large model according to claim 1, characterized in that: The Head stage of the YOLOv8 model adopts an Anchor-free implementation method to predict the gridpoint on each feature map.

5. The method for smoking identification based on a multimodal large model according to claim 1, characterized in that: In the step of aligning the image and the prompt word, a visual Transformer model pre-trained by CLIP is used to optimize the alignment accuracy of the image and text features.

6. The method for smoking identification based on a multimodal large model according to claim 1, characterized in that: The Qwen-VL model enhances the fusion effect of multimodal features by mapping image features and language features into the same feature space.

7. The method for smoking identification based on a multimodal large model according to claim 1, characterized in that: During the fine-tuning process of the VLM model, the parameters of the visual encoder and the language encoder are frozen, and only the parameters of the picture mapper are adjusted to ensure effective alignment of image features.

8. The method for smoking identification based on a multimodal large model according to claim 1, characterized in that: In the real-time smoking recognition step, by inputting a prompt word containing a preset question, it is determined whether there is smoking behavior in the human body image, and the specific coordinates and occurrence time of the smoking behavior are output.

9. The method for smoking identification based on a multimodal large model according to claim 1, characterized in that: The annotated data file adopts a fixed json format, and the json file includes the corresponding relationship between the image annotation and the prompt word, as well as the corresponding coordinate frame and description text.

10. The method for smoking identification based on a multimodal large model according to claim 1, characterized in that: The fine-tuned VLM model is merged with the main model to improve the recognition accuracy of smoking behavior during reasoning.

Citation Information

Patent Citations

  • Smoking and calling detection method and system combining action recognition and target detection

    CN116580452A

  • Public place smoking detection method based on improved YOLOv8

    CN116912770A

  • Automatic driving decision planning method and system based on visual language large model

    CN118238848A

  • Behavior recognition method and device based on multi-modal large model and electronic equipment

    CN118314624A

Cited By

  • Vision and priori knowledge co-fusion-based pest image annotation algorithm

    CN119399766A

  • Signal modulation identification and positioning method based on time-frequency graph and visual language model

    CN120763877A

  • A signal modulation recognition and positioning method based on time-frequency graph and visual language model

    CN120763877B