Universal object detection method in industrial scene based on multi-modal large model
Through the application of multimodal large model in industrial scenarios, using video streams and template images to extract prompt words, combined with VIT networks for feature comparison, solving the problems of high cost and low efficiency in traditional methods, and achieving efficient and accurate objects Inspection and data set annotation have promoted the development of industrial automation and intelligence.
Patent Information
- Application Number
- CN202411120554.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2025-01-07
AI Technical Summary
In industrial scenarios, traditional object detection methods rely on a large number of training samples and high-cost data annotations, resulting in insufficient data volume and difficult to support the application of deep learning algorithms. In addition, the general model is accurate and robust when detecting various objects. Insufficient in nature, it takes a long time to adapt to different detection tasks.
Using a multimodal large model method, by streaming videos into pictures, manually selecting template pictures, using Blip-2 and Glip models to extract prompt words, combining VIT network for feature extraction and similarity comparison, achieving low cost and high efficiency object detection.
It significantly reduces the cost and time-consuming model training, improves the accuracy and robustness of object detection, solves the adaptation problems of different detection tasks, and realizes efficient data set annotation and lean manufacturing support in the field of intelligent manufacturing.
Smart Images
Figure CN119273944A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of object detection and relates to a general object detection method in an industrial scenario based on a multimodal large model. Background Art
[0002] In an era of unprecedented development of intelligent algorithms, some algorithms are often used to replace or supervise workers in existing intelligent industrial production lines. Action recognition is the core task of these algorithms. Existing research has explored various technical routes for using action recognition algorithms to identify objects or workers from different angles. Detection of object position and category in industrial scenes is one of the core technologies of intelligent manufacturing and is inseparable from product monitoring in the manufacturing process. However, the types and forms of objects in industrial scenes vary greatly. How to quickly and accurately detect the position and category of any object is crucial. With the development of industrial technology, the demand for object detection in the industrial field is increasing. Although there are encouraging results, the traditional industrial object detection steps include supervised learning using models such as yolo, which relies on the collection of training data, often requires more training sample input and considers issues such as positive and negative sample balance, and the implementation cost is high. The magnitude and quality of industrial data sets are highly correlated with the accuracy of the final industrial object detection model. Some industrial data sets are insufficient in data volume due to difficulties in collection, collection affecting workers' work, short collection time, and low collection frequency, making it difficult to support the application of advanced algorithms such as deep learning. Therefore, it is crucial to achieve low-cost, low-time, intelligent and even fully automatic dataset annotation. Summary of the invention
[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a general object detection method in industrial scenarios based on a multimodal large model.
[0004] The present invention provides a general object detection method in an industrial scenario based on a multimodal large model, comprising:
[0005] Step 1: Convert the video stream of the actual production line into pictures of the actual production line frame by frame;
[0006] Step 2: Manually select pictures containing the target objects to be detected from the actual production line pictures as template pictures;
[0007] Step 3: Input the template image into Blip-2 and output the text description of each template image;
[0008] Step 4: Use a word frequency statistics strategy to automatically extract prompt words from the text description based on the frequency and relevance of the words;
[0009] Step 5: Input the prompt words and all the actual production line pictures into the Glip model, and find the target box of the target object to be detected in the actual production line pictures according to the prompt words;
[0010] Step 6: Cut the actual production line image according to the target frame to obtain multiple target frame area images;
[0011] Step 7: Input the target frame area image and the template image into the VIT network and extract the corresponding feature maps respectively;
[0012] Step 8: Perform similarity comparison between the feature map extracted from the target frame area image and the feature map of the template image, and select the target frame area image with a similarity higher than the threshold.
[0013] The present invention provides a general object detection method in industrial scenarios based on a multimodal large model, which has the following beneficial effects:
[0014] In the field of intelligent manufacturing, the traditional box selection and labeling method not only consumes a lot of manpower and time, but is also costly, and often requires thousands of labeled data to train the model. The method of the present invention can successfully reduce this cost to a minimum, and only a few box selection data are needed to achieve efficient model training. In addition, in view of the problem that general models are difficult to accurately detect various objects in industrial scenarios, the present invention optimizes the recognition effect by introducing templates and additional information, and significantly improves the accuracy and robustness of the model. More importantly, the method of the present invention solves the adaptation problem of different detection tasks in intelligent manufacturing scenarios, greatly shortens the time consumption, makes production line inspections, problem discovery and elimination of potential customer complaints more efficient, and provides strong technical support for the realization of lean manufacturing. These innovations not only promote the development of industrial automation and intelligence, but also provide new ideas and methods for the transformation and upgrading of the manufacturing industry. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 It is a flow chart of a general object detection method in an industrial scenario based on a multimodal large model of the present invention. DETAILED DESCRIPTION
[0016] like Figure 1 As shown, a general object detection method in an industrial scenario based on a multimodal large model of the present invention includes:
[0017] Step 1: Convert the video stream of the actual production line into pictures of the actual production line frame by frame.
[0018] Step 2: Manually select pictures containing the target objects to be detected from the actual production line pictures as template pictures.
[0019] In the specific implementation, 8-12 template images are manually selected to ensure that the target object to be detected in the template image is clear and complete. For example, if the target object to be detected is a box, the box in the image must be clear and complete when manually selecting the template image. Ensure that the text description output by Blip-2 contains the field of the target object to be detected, so as to achieve the purpose of automatically extracting high-quality prompt words.
[0020] Step 3: Input the template image into Blip-2 and output the text description of each template image.
[0021] Step 4: Use a word frequency statistics strategy to automatically extract prompt words from the text description based on the frequency and relevance of the words, specifically:
[0022] Step 4.1: Use the regularization method to clean the text description, remove punctuation marks in the text description and convert the uppercase letters in the text description to lowercase letters.
[0023] Step 4.2: Use the string split method to split the text descriptions of all cleaned template images into word lists.
[0024] Step 4.3: Design a stop word list based on industrial scenarios and Blip-2 common words, and use the stop word list to filter the word list.
[0025] Step 4.4: Use the Counter library function to count the frequency of the filtered word list.
[0026] Step 4.5: Select the most frequent and relevant nouns as the prompt words for the task.
[0027] The logic of prompt word extraction may be different in different scenarios or specific scenarios. Prompt words are usually provided by domain experts. For some specific fields, domain experts provide a list of prompt words to ensure the accuracy and relevance of prompt words. We use natural language processing techniques such as word frequency statistics, TF-IDF, topic models and other methods to extract prompt words. These methods can automatically extract prompt words from text and sort and filter them according to factors such as frequency of occurrence and context. The basis for selecting prompt words can be a variety of factors, including frequency of occurrence, relevance, importance, etc. In most scenarios, frequency of occurrence is an important factor because words with high frequency of occurrence are usually more representative and relevant, and the target of detection is an object, that is, a noun. Through these constraints, we can filter out high-quality prompt words.
[0028] Step 5: Input the prompt words and all the actual production line pictures into the Glip model, and find the target box of the target object to be detected in the actual production line pictures according to the prompt words, specifically:
[0029] Step 5.1: The encoding layer of the Glip model includes a text encoder, an image encoder, and a cross-modal deep fusion module. The prompt word is input into the text encoder. The actual production line picture is first detected by the RPN network to obtain the candidate target frame, and then passed through the image encoder. The cross-modal deep fusion module is used to fuse the information of the two modalities of text and picture in the encoding stage, and the text features and the image features of the candidate target frame are output respectively.
[0030] Step 5.2: Calculate the vector similarity between the text features and the candidate target box image features, and select the candidate target box with the greatest similarity to the text words.
[0031] Step 5.3: Map the candidate target frame with the greatest similarity to the original actual production line image to generate the target frame of the target object to be detected.
[0032] Step 6: Cut the actual production line image according to the target frame to obtain multiple target frame area images, specifically:
[0033] Step 6.1: Load the image to be cut.
[0034] Step 6.2: Determine the area to be cut, that is, the parameters of the target box obtained in step 5, which consists of the coordinates of the upper left corner of the rectangle, the width and height of the rectangle.
[0035] Step 6.3: Use the image cutting method and the coordinates and dimensions determined in step 6.2 to cut the image.
[0036] Step 6.4: Save the cut target frame area images to the file system.
[0037] Step 7: Input the target frame area image and the template image into the VIT network and extract the corresponding feature maps respectively, as follows:
[0038] Step 7.1: Use bilinear interpolation to adjust the target box area image and the template image to a uniform size of 3*1024*1024, where 3 represents the number of color channels (RGB).
[0039] Step 7.2: Normalize the resized image and scale it to between 0 and 1 so that its pixel value range is suitable for subsequent processing or model input requirements.
[0040] Step 7.3: Use the attention mechanism layer of the VIT network to process the input image to capture global and local contextual information, and pass the output of the attention mechanism layer to the multi-layer perceptron layer to complete a more complex feature representation.
[0041] Step 7.4: Finally, output the feature map of the target box area image and the feature map of the template image with a size of 256*64*64.
[0042] Step 8: Perform similarity comparison between the feature map extracted from the target frame area image and the feature map of the template image, and select the target frame area image with a similarity higher than the threshold, specifically:
[0043] Step 8.1: Calculate the cosine similarity between the feature map of each target box area image and the feature map of each template image.
[0044] Step 8.2: Calculate the average cosine similarity between the feature map of the target box area image and the feature map of each template image.
[0045] Step 8.3: Select the target box area images whose average cosine similarity value is higher than the threshold.
[0046] Step 8.4: Save the target frame area image that meets the requirements, and correspond it back to the original actual production line image with the target frame to complete the task of intelligent industrial data set labeling.
[0047] In the field of intelligent manufacturing, the traditional scene manual box selection and labeling method is not only costly but also inefficient. Moreover, traditional supervised algorithms require a lot of training time and resources for each industrial scene. By introducing a large visual model, the present invention not only solves the problem of the accuracy of detection of various objects in industrial scenes by general models, but also significantly improves the recognition accuracy by integrating multiple modal information. In addition, these algorithms also effectively solve the adaptation problem of different detection tasks in intelligent manufacturing scenarios, shorten the time consumption, and thus provide strong technical support for industrial automation and intelligence.
[0048] The above description is only a preferred embodiment of the present invention and is not intended to limit the concept of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A general object detection method in industrial scenarios based on multimodal large models, characterized in that: include: Step 1: Convert the video stream of the actual production line into pictures of the actual production line frame by frame; Step 2: Manually select pictures containing the target objects to be detected from the actual production line pictures as template pictures; Step 3: Input the template image into Blip-2 and output the text description of each template image; Step 4: Use a word frequency statistics strategy to automatically extract prompt words from the text description based on the frequency and relevance of the words; Step 5: Input the prompt words and all the actual production line pictures into the Glip model, and find the target box of the target object to be detected in the actual production line pictures according to the prompt words; Step 6: Cut the actual production line image according to the target frame to obtain multiple target frame area images; Step 7: Input the target frame area image and the template image into the VIT network and extract the corresponding feature maps respectively; Step 8: Perform similarity comparison between the feature map extracted from the target frame area image and the feature map of the template image, and select the target frame area image with a similarity higher than the threshold.
2. The general object detection method in industrial scenarios based on multimodal large models according to claim 1, characterized in that: In step 2, 8-12 template images are manually selected to ensure that the target objects to be detected in the template images are clear and complete.
3. The general object detection method in industrial scenarios based on multimodal large models as claimed in claim 1, characterized in that: The step 4 is specifically as follows: Step 4.1: Use the regularization method to clean the text description, remove punctuation marks in the text description and convert the uppercase letters in the text description to lowercase letters; Step 4.2: Use the string split method to split the text descriptions of all cleaned template images into word lists; Step 4.3: Design a stop word list based on industrial scenarios and Blip-2 common words, and use the stop word list to filter the word list; Step 4.4: Count the frequency of the filtered word list; Step 4.5: Select the most frequent and relevant nouns as the prompt words for the task.
4. The general object detection method in industrial scenarios based on multimodal large models as claimed in claim 1, characterized in that: The step 5 is specifically as follows: Step 5.1: The encoding layer of the Glip model includes a text encoder, an image encoder, and a cross-modal deep fusion module. The prompt word is input into the text encoder. The actual production line picture is first detected by the RPN network to obtain the candidate target frame. Then, it passes through the image encoder and the cross-modal deep fusion module fuses the information of the two modalities of text and image in the encoding stage, and outputs the text features and the image features of the candidate target frame respectively. Step 5.2: Calculate the vector similarity between the text features and the candidate target frame image features, and select the candidate target frame with the greatest similarity to the text words; Step 5.3: Map the candidate target frame with the greatest similarity to the original actual production line image to generate the target frame of the target object to be detected.
5. The general object detection method in industrial scenarios based on multimodal large models according to claim 1, characterized in that: The step 6 is specifically as follows: Step 6.1: Load the picture to be cut; Step 6.2: Determine the area to be cut, that is, the parameters of the target box obtained in step 5, which consists of the coordinates of the upper left corner of the rectangle, the width and height of the rectangle; Step 6.3: Use the image cutting method and the coordinates and dimensions determined in step 6.2 to cut the image; Step 6.4: Save the cut target frame area images to the file system.
6. The general object detection method in industrial scenes based on multimodal large models according to claim 1, characterized in that: The step 7 is specifically as follows: Step 7.1: Use bilinear interpolation to adjust the target frame area image and the template image to a uniform size of 3*1024*1024, where 3 represents the number of color channels (RGB); Step 7.2: Normalize the resized image and scale it to between 0 and 1 so that its pixel value range is suitable for subsequent processing or model input requirements; Step 7.3: Use the attention mechanism layer of the VIT network to process the input image to capture global and local context information, and pass the output of the attention mechanism layer to the multi-layer perceptron layer to complete a more complex feature representation; Step 7.4: Finally, output the feature map of the target box area image and the feature map of the template image with a size of 256*64*64.
7. The general object detection method in industrial scenarios based on multimodal large models as claimed in claim 1, characterized in that: The step 8 is specifically as follows: Step 8.1: Calculate the cosine similarity between the feature map of each target box area image and the feature map of each template image; Step 8.2: Calculate the average cosine similarity between the feature map of the target frame area image and the feature map of each template image; Step 8.3: Select the target box area images whose average cosine similarity value is higher than the threshold; Step 8.4: Save the target frame area image that meets the requirements, and correspond it back to the original actual production line image with the target frame to complete the task of intelligent industrial data set labeling.