Personnel behavior recognition model training and behavior recognition method and system in electric appliance disassembling process
Through the two-stage training method based on large visual language models and diversified annotation methods, the problem of insufficient accuracy and real-time behavior recognition during product disassembly is solved, and efficient and intelligent behavior analysis is achieved.
Patent Information
- Application Number
- CN202411685573.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-23
- Publication Date
- 2025-05-02
AI Technical Summary
The prior art personnel behavior identification in the process of product disassembly has problems with insufficient accuracy and real-timeness, and the model training process that relies on a large amount of labeled data is cumbersome and costly.
A two-stage training method based on large visual language models is adopted, including the image-text alignment pre-training stage and the instruction fine-tuning stage, and combined with the annotation method of manual and Gemini large models, a personnel behavior recognition model is constructed in the product dismantling process.
It improves the accuracy and real-time nature of behavior recognition, reduces the cost of data annotation, and realizes more intelligent behavioral analysis and judgment.
Smart Images
Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and in particular to a personnel behavior recognition model training method, a behavior recognition method and a system in a product disassembly process. Background Art
[0002] In the field of modern industry and manufacturing, the recognition of human behavior during product disassembly has important practical application value. In the existing technology, behavior recognition mainly relies on manual observation and traditional video surveillance, which have many deficiencies and shortcomings. First, manual observation requires a lot of human resources and is easily affected by human subjective factors, resulting in inaccurate and unstable recognition results. Secondly, although traditional video surveillance can record the product disassembly process, it still relies on manual analysis in behavior recognition, making it difficult to achieve real-time and efficient behavior recognition and management. In addition, existing video surveillance systems usually lack intelligent analysis capabilities, cannot automatically identify and judge human behavior, and cannot provide effective early warning and management measures.
[0003] In recent years, with the development of computer vision and deep learning technologies, behavior recognition technology has made significant progress. However, existing behavior recognition models still face many challenges in dealing with complex behavior scenarios during product disassembly. For example, the actions of personnel during product disassembly are frequent and diverse, the environment is complex, and the light changes frequently. These factors put forward higher requirements on the accuracy and robustness of behavior recognition. In addition, existing behavior recognition methods usually require a large amount of labeled data for model training. The data collection and labeling process is cumbersome and costly, which limits the promotion and application of the technology.
[0004] With the rapid development of large-scale language models, the cost of data annotation has been reduced, and it is possible to batch annotate tens of millions of data. At the same time, large-scale visual language models also have powerful reasoning and understanding capabilities, which can more intelligently analyze and judge the behavior of people during product disassembly.
[0005] Therefore, in view of the deficiencies and shortcomings in the prior art, the present invention proposes a model training method, a behavior recognition method and a system for identifying personnel behavior in the product disassembly process based on a large-scale visual language model, aiming to improve the accuracy, real-time and intelligence level of behavior recognition, thereby effectively solving the technical problems existing in the prior art. Summary of the invention
[0006] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a personnel behavior recognition model training method, a behavior recognition method and a system during product disassembly to solve the problem of lack of real-time supervision of personnel behavior during product disassembly.
[0007] In one aspect, the present invention provides a method for training a personnel behavior recognition model during a product disassembly process, the method comprising the following steps:
[0008] A training data set is obtained, where the training samples include 11 million sample images. Each image is obtained from a surveillance camera in a home appliance disassembly factory, and the image resolution is 2560*1440. Each sample has two types of annotation information defined according to personnel behavior. One type is about image description information, including environmental information, personnel information, work information and time information in the factory. The other type is information based on question-answer pair settings, including 8 types of worker behaviors during product disassembly.
[0009] An initial large visual language model is constructed, which includes a visual feature extraction module and a text generation module. The visual feature extraction module includes a frozen visual encoder for extracting features, a position embedding layer for injecting temporal information into video frames, a video Q-Former for aggregating frame representations, and a linear layer for mapping the output video representations to the large model. The text generation module includes an original LLaMA2-7B model.
[0010] A two-stage training method is designed, which includes an image-text alignment pre-training stage and an instruction fine-tuning stage. The image-text alignment pre-training stage trains the visual feature extraction module. In this part, all parameters of the text generation module are frozen. In the visual feature extraction module, only the pre-trained visual encoder parameters based on BLIP-2 are frozen, and the parameters of the remaining layers will be updated during the training process. The main purpose of this part is to achieve the alignment of image blocks and tokens. The instruction fine-tuning stage fine-tunes the parameters of the last feature mapping layer of the visual feature extraction module and the parameters of the text embedding layer in the text generation module.
[0011] The two-stage training method is used to train the initial large-scale visual language model to obtain a personnel behavior recognition model during the product disassembly process.
[0012] The eight types of personnel behaviors are defined, including: Personnel leaving / returning to the workstation: When the worker is detected to disappear from the camera's field of view, the worker is identified to have left the workstation; when the worker is detected to appear in the camera's field of view, the worker is identified to have returned to the workstation. Personnel hand visible / invisible behavior: According to the provisions of the Hailuyuan factory, the worker's hands can be seen through the camera. When the worker's hands are detected to be visible in the camera's field of view, the hands are identified to be visible; when the hands are detected to be absent in the camera's field of view, the hand is detected to be blocked. Behavior of stacking appliances to be disassembled on the operating table: According to the provisions of the Hailuyuan factory, the maximum number of appliances to be disassembled on a single disassembly station is one. When two or more appliances to be disassembled are detected on the operating table, it is identified that the person has stacked appliances to be disassembled; when it is detected that the number of appliances to be disassembled on the operating table is less than two, it is identified that there is no stacking of appliances to be disassembled. Personnel theft behavior: When it is detected that a worker has privately kept the disassembled products, theft behavior is identified. Violent dismantling behavior: When a worker is detected using a hammer to dismantle items, the worker is identified as engaging in violent dismantling behavior. Smashing behavior: When a worker is detected using a smashing or throwing method to dismantle items, the worker is identified as engaging in smashing behavior. Dangerous behavior: When a situation that is about to endanger the worker's safety is detected, the worker is identified as about to engage in dangerous behavior. Personnel taking off / wearing a helmet behavior: When a worker is detected taking off a helmet, the worker is identified as taking off a helmet. When a worker is detected wearing a helmet, the worker is identified as wearing a helmet.
[0013] The design of image description annotation methods includes annotation based on Gemini large model and manual annotation. The annotation based on Gemini large model includes self-designed prompt word templates and calling Gemini API to generate image descriptions; the manual annotation includes designing annotation management methods, annotation methods and annotation rules. The annotation management method includes, first, a data annotation team is formed by two people (A and B respectively), 20% of the total data set is taken out, and A and B negotiate to jointly annotate this part of the data, aiming to reach a consensus on the annotation standard. Then, the remaining 80% is evenly divided between A and B, and A and B annotate respectively. Finally, A and B exchange the annotated data and check the annotated data with each other. For controversial annotation items, two people need to make a decision together. In the annotation method, first, the annotator needs to be familiar with the definitions of eight types of gestures and body movements, then the annotator watches the video and disassembles the operation instructions twice, and finally gives a text description of the video and saves the text in a csv file. The agreed rules for marking are as follows: use Chinese to organize sentences; if there are no workers in the picture, mark "There are no workers in the picture."; if the workers cannot be clearly observed through the video, mark "The picture is blurry."; if there are workers in the picture, use the workers as the subject and describe it in the third person; try to use the action nouns mentioned in the disassembly operation instruction to describe the workers' actions; use phrases such as "first", "then", "next", and "finally" to organize sentences; if there are multiple workers in the picture, describe the workers' actions in order from left to right.
[0014] The question-answer pair annotation method is designed to include a Gemini-based annotation method, wherein the Gemini-based annotation method includes setting a prompt word template and calling a Gemini API.
[0015] The image-text alignment training method involves supervised training using an image description dataset.
[0016] The training method using instruction fine-tuning involves supervised training on the dataset using question-answering pairs.
[0017] In another aspect, the present invention provides a method for identifying personnel behavior during product disassembly, the method comprising the following steps:
[0018] Obtain camera videos on the factory disassembly station and prompt words corresponding to the identification of personnel behaviors, and input the video data and prompt words into the product disassembly process personnel behavior identification model in the product disassembly process personnel behavior identification model training method as described in any one of claims 1 to 6 to obtain the identification results of the eight types of personnel behaviors.
[0019] On the other hand, the present invention provides a system to realize the technology of personnel behavior recognition in the product disassembly process, including front-end and back-end design, characterized in that the front-end adopts Websocket+timer to receive video metadata, inference results and target frames and video playback progress of synchronized inference results, and the back-end is built in Flask+Websocket mode, relying on millisecond-level frame rate control of the front-end to realize frame synchronization. When executed, the system realizes the steps of the method described in any one of the items mentioned above.
[0020] The beneficial effects of the present invention are at least:
[0021] The product disassembly process personnel behavior recognition model training method, behavior recognition method and system of the present invention include: obtaining camera data of the disassembly station in the factory, annotating image-text description data and question-answer pair data by manual and Gemini-based methods. Constructing an initial large visual language model. Designing a two-stage training method, including an image-text alignment pre-training stage and an instruction fine-tuning stage. The two-stage training method is used to train the initial large visual language model to obtain a product disassembly process personnel behavior recognition model.
[0022] Furthermore, the manual annotation method requires the design of annotation management methods, annotation methods and annotation rules, and the Gemini annotation method requires the design of prompt word templates.
[0023] At the same time, an initial large visual language model is constructed, which includes a visual feature extraction module and a text generation module. The visual feature extraction module includes a frozen visual encoder for extracting features, a position embedding layer for injecting temporal information into video frames, a video Q-Former for aggregating frame representations, and a linear layer for mapping the output video representation to the large model. The text generation module includes an original LLaMA2-7B model.
[0024] Furthermore, a two-stage training method is adopted, including an image-text alignment pre-training stage and an instruction fine-tuning stage. The image-text alignment pre-training stage trains the visual feature extraction module. In this part, all parameters of the text generation module are frozen. In the visual feature extraction module, only the parameters of the pre-trained visual encoder based on BLIP-2 are frozen, and the parameters of the remaining layers will be updated during the training process. The main purpose of this part is to achieve the alignment of the image block and the token. The instruction fine-tuning stage fine-tunes the parameters of the last feature mapping layer of the visual feature extraction module and the parameters of the text embedding layer in the text generation module.
[0025] Additional advantages, purposes, and features of the present invention will be described in part in the following description, will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be realized and obtained by the structures specifically pointed out in the specification and the accompanying drawings.
[0026] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 This is a scene captured by a camera during the disassembly process in one embodiment of the present invention. In the picture, a worker is disassembling the outer shell of a used air conditioner indoor unit.
[0028] Figure 2 This is a structure for obtaining factory camera data in one embodiment of the present invention. The data type is video data. The video data includes multiple folders such as "Air Conditioning Line-Operation Console 11-99_20240124101952", and each folder includes multiple sub-video files.
[0029] Figure 3 This is a video frame extraction program based on FFmpeg in one embodiment of the present invention. The program is based on the Python programming language and extracts frames from a video based on CUDA-accelerated FFmpeg. The frame extraction interval is one frame per second.
[0030] Figure 4 This is a video segmentation program based on FFmpeg in one embodiment of the present invention. The program is based on the Python programming language and segments a video based on CUDA-accelerated FFmpeg. The duration of each segment when segmenting the video is specified to be 10 seconds.
[0031] Figure 5 This is a manual labeling scene in an embodiment of the present invention. The left display in the figure shows the information labeled by the labeler. The upper right display shows the picture to be labeled, and the lower right display shows the disassembly operation instruction, that is, the labeling specification.
[0032] Figure 6 This is a flowchart of image-text data annotation based on Gemini in one embodiment of the present invention, wherein Geminiapi refers to a powerful generative AI model API launched by Google, which has multimodal input and output capabilities and can be widely used in various scenarios.
[0033] Figure 7This is a flow chart of question-answer data annotation based on Gemini in one embodiment of the present invention, wherein the Gemini api refers to a powerful generative AI model API launched by Google, which has multi-modal input and output capabilities and can be widely used in various scenarios.
[0034] Figure 8 It is a text generation module in one embodiment of the present invention, wherein SoftMax refers to an activation function commonly used in machine learning and deep learning, and is widely used in multi-classification problems. Linear refers to a linear neural network layer. Transformer Block refers to a neural network layer built on the basis of Transformer. Input Embedding refers to a network layer that performs embedding conversion on the input, and its function is to convert the input into a sequence represented by Token. SiLU refers to a new type of activation function, also known as the Swish activation function. RMSNorm refers to a normalization method that aims to optimize the performance and efficiency of deep learning models. RMSNorm is an improvement of LayerNorm, which reduces the amount of calculation by discarding the center invariance (i.e., the mean term) while maintaining the normalization effect. RoPE refers to a position encoding strategy for Transformer models, which is widely used in large language models such as LLaMA and ChatGLM. It improves the traditional position encoding method by introducing rotational position encoding, and solves some problems of traditional methods in processing long sequences. In the Transformer model, Q, K, and V stand for Query, Key, and Value, respectively, which are the three key components of the self-attention mechanism. A complete Transformer Block consists of an Attention part and a FeedForward part.
[0035] Fig. 9 This is a visual feature extraction module in one embodiment of the present invention. The input of this module is a video or an image. First, the video or image is converted into a sequence representation embedded with time information through a visual encoder, and then the time series is converted into a one-dimensional sequence representation through a Q-Former neural network layer. The last one-dimensional sequence is mapped into a high-dimensional feature representation through a linear layer.
[0036] Fig.10This is the code of the visual feature extraction module in one embodiment of the present invention. The module is written in Python programming language. The function receives a video and processes it as a time series of frames. The main steps are video frame encoding, query embedding generation, position encoding, video feature integration and projection to a new space. The output is to return inputs_llama (mapped features for downstream models) and atts_llama (corresponding attention masks for guiding the model attention mechanism).
[0037] Fig.11 This is the code for the text generation module in one embodiment of the present invention. The main function of this module is to load the word segmenter of the llama model and splice the image token into the token sequence of the large language model.
[0038] Fig.12 This is a model for identifying the behavior of personnel in the disassembly process in one embodiment of the present invention. In this model, serial number 1 indicates that 2 million pieces of image-text alignment data are used for training on a single GPU for 20 days, and the weights of the Projection layer are updated, while the other weights are not updated. Serial number 2 indicates that 2.1 million instruction fine-tuning data are used for training on a single GPU for 10 days, and the weights in some large language models are updated. Serial number 3 refers to the use of the low-rank adaptive (LoRA) method to update the weights of some large language models using the detection data set to improve the spatial position understanding ability of the entire model.
[0039] Fig.13 The hardware architecture of the disassembly process personnel behavior recognition technology in one embodiment of the present invention is divided into two layers, the upper layer represents the image and video acquisition from the camera. The lower layer represents the deep learning server, the configuration of the server: the CPU is AMD R7 5700G, the GPU is NVIDIA RTX 4090, and two data sets are used for data storage and to improve the response speed.
[0040] Fig.14 This is a software architecture diagram for identifying human behavior during the disassembly process in one embodiment of the present invention. The software architecture is divided into five layers, in which the human behavior identification model works in the data processing layer and the model layer, and the human behavior identification technology works in the response layer, the data processing layer and the model layer. The software version used is also indicated.
[0041] Fig.15 It is a key technology for frame synchronization of a personnel behavior recognition system in the disassembly process in one embodiment of the present invention. This key technology is mainly used to solve the computational bottleneck problem caused by the reasoning of large-scale visual language models in the process of personnel behavior recognition. This key technology adopts an adaptive time window mechanism to enable the front end to dynamically adjust the video frames of the back end real-time reasoning.
Claims
1. A training method for a personnel behavior recognition model in a product disassembly process, characterized by: A training data set is obtained, wherein the training samples include 11 million sample images, each of which is obtained from a surveillance camera in a home appliance disassembly factory, and the image resolution is 2560*1440; each sample has two types of annotation information according to the definition of personnel behavior, one is about image description information, including environmental information, personnel information, work information and time information in the factory, and the other is information based on question-answer pair settings, including 8 types of worker behavior actions during the disassembly process; Constructing an initial large visual language model, the model includes a visual feature extraction module and a text generation module, the visual feature extraction module includes a frozen visual encoder for extracting features, a position embedding layer for injecting temporal information into video frames, a video Q-Former for aggregating frame representations, and a linear layer for mapping the output video representations to the large model, and the text generation module includes an original LLaMA2-7B model; Design a two-stage training method, which includes an image-text alignment pre-training stage and an instruction fine-tuning stage. The image-text alignment pre-training stage trains the visual feature extraction module. In this part, all parameters of the text generation module are frozen. In the visual feature extraction module, only the pre-trained visual encoder parameters based on BLIP-2 are frozen, and the parameters of the remaining layers will be updated during the training process. The main purpose of this part is to achieve the alignment of image blocks and tokens. The instruction fine-tuning stage fine-tunes the parameters of the last feature mapping layer of the visual feature extraction module and the parameters of the text embedding layer in the text generation module; The two-stage training method is used to train the initial large-scale visual language model to obtain a personnel behavior recognition model during product disassembly.
2. The method for training a personnel behavior recognition model during product disassembly according to claim 1, characterized in that: The definition of personnel behavior includes: Personnel leaving / returning to the workstation: When the worker is detected to disappear from the camera's field of view, it is recognized that the worker has left the workstation; when the worker is detected to appear in the camera's field of view, it is recognized that the worker has returned to the workstation. Personnel hand visible / invisible behavior: According to the provisions of the Hailuyuan factory, the worker's hands can be seen through the camera. When the worker's hands are detected to be visible in the camera's field of view, the hands are recognized to be visible; when the hands are detected to be absent in the camera's field of view, it is detected that the hands are blocked. The accumulation of household appliances to be disassembled on the operating table: According to the provisions of the Hailuyuan factory, the maximum number of household appliances to be disassembled on a single disassembly station is one. When two or more household appliances to be disassembled are detected on the operating table, it is recognized that the person has accumulated household appliances to be disassembled; when it is detected that the number of household appliances to be disassembled on the operating table is less than two, it is recognized that there is no accumulation of household appliances to be disassembled. Personnel theft behavior: When it is detected that a worker has privately kept the disassembled products, theft behavior is recognized. Violent dismantling behavior: When a worker is detected using a hammer to dismantle items, the worker is identified as engaging in violent dismantling behavior. Smashing behavior: When a worker is detected using a smashing or throwing method to dismantle items, the worker is identified as engaging in smashing behavior. Dangerous behavior: When a situation that is about to endanger the worker's safety is detected, the worker is identified as about to engage in dangerous behavior. Personnel taking off / wearing a helmet behavior: When a worker is detected taking off a helmet, the worker is identified as taking off a helmet. When a worker is detected wearing a helmet, the worker is identified as wearing a helmet.
3. The method for training a personnel behavior recognition model during product disassembly process according to claim 1, characterized in that: The image description annotation method includes annotation based on the Gemini large model and manual annotation. The annotation based on the Gemini large model includes self-designed prompt word templates and calling the Gemini API to generate image descriptions; the manual annotation includes designing annotation management methods, annotation methods and annotation rules. The annotation management method includes, first, a data annotation team is formed by two people (A and B), 20% of the total data set is taken out, and A and B negotiate to jointly annotate this part of the data, aiming to reach a consensus on the annotation standard. Then, the remaining 80% is evenly divided between A and B, and A and B annotate respectively. Finally, A and B exchange the annotated data and check the annotated data with each other. For controversial annotation items, two people need to make a decision together. In the annotation method, first, the annotator needs to be familiar with the definitions of eight types of gestures and body movements, then the annotator watches the video and disassembles the operation instructions twice, and finally gives a text description of the video and saves the text in a csv file. The marking rules are as follows: use Chinese to organize sentences; if there are no workers in the picture, mark "There are no workers in the picture."; if the workers cannot be clearly observed through the video, mark "The picture is blurred."; if there are workers in the picture, use the workers as the subject and describe it in the third person; try to use the action nouns mentioned in the disassembly operation instruction to describe the workers' actions; use phrases such as "first", "then", "next", and "finally" to organize sentences; if there are multiple workers in the picture, describe the workers' actions in order from left to right.
4. The method for training a personnel behavior recognition model during product disassembly process according to claim 1, characterized in that: The question-answer pair annotation method includes a Gemini-based annotation method, and the Gemini-based annotation method includes setting a prompt word template and calling a Gemini API.
5. The method for training a personnel behavior recognition model during product disassembly process according to claim 1, characterized in that: The image-text alignment training method includes supervised training using an image description dataset.
6. The method for training a personnel behavior recognition model during product disassembly process according to claim 1, characterized in that: The instruction fine-tuning training method includes using a question-answering pair dataset for supervised training.
7. A method for identifying personnel behavior during product disassembly, the method comprising the following steps: Obtain camera videos on the factory disassembly station and prompt words corresponding to the identification of personnel behaviors, and input the video data and prompt words into the disassembly process personnel behavior recognition model in the product disassembly process personnel behavior recognition model training method as described in any one of claims 1 to 6 to obtain the recognition results of the eight types of personnel behaviors.
8. A personnel behavior recognition system during product disassembly process, including front-end and back-end designs, characterized in that: The front end uses Websocket+timer to receive video metadata, inference results, and target frames and video playback progress of synchronized inference results. The back end is built using Flask+Websocket, and frame synchronization is achieved by front-end millisecond-level frame rate control. When executed, the system implements the steps of the method as described in any one of claims 1 to 7.