Multi-modal fusion industrial filling scene multi-scale target detection method and device and storage medium

Abstract

The application relates to the technical field of industrial visual inspection, in particular to a multi-modal fusion industrial filling scene multi-scale target detection method and device and a storage medium, the method is based on an OV-DINO model, a multi-modal fusion industrial filling scene multi-scale target detection model MQF-OV-DINO is proposed, a hierarchical feature enhancement module HFE, a language condition query initialization module LCQI and a query and feature self-adaptive fusion module AQFF are introduced, and the problems of small target missed detection, background interference mis-detection and insufficient text semantic guidance of an existing target detection method in filling barrel surface defect detection in an industrial filling scene are solved.

Description

Technical Field

[0001] This application relates to the field of industrial vision inspection technology, and in particular to a multimodal fusion method, device and storage medium for multi-scale target detection in industrial filling scenarios. Background Technology

[0002] With the development of industrial automation and intelligent manufacturing technologies, machine vision-based target detection technology has been widely applied in product surface quality inspection tasks. Compared with general scenarios, the targets to be detected in industrial filling scenarios, such as drips, QR codes, paint peeling, scratches, dirt, and splashes, are usually characterized by small size, large differences, irregular shape, and weak texture features. At the same time, the production environment also has problems such as changes in lighting, metal reflection, and background interference, which can easily lead to insufficient representation of target features, resulting in missed detections, false detections, and inaccurate positioning.

[0003] Current industrial target detection methods primarily employ traditional machine vision approaches or deep learning-based detection schemes based on single visual information. Traditional machine vision methods typically require manual design of features and thresholds for different targets, resulting in poor adaptability. When faced with variations in lighting, metallic reflections, background texture interference, or changes in target shape within the production environment, they are prone to missed detections and false positives. While deep learning-based target detection methods can automatically extract image features, they still suffer from insufficient feature representation and inaccurate localization when dealing with small-scale targets, weakly textured targets, and targets in edge regions within industrial scenarios. In industrial filling scenarios, targets exhibit large variations in scale and irregular shapes. Existing algorithms suffer from insufficient interaction between visual features and textual semantics, leading to missed detections of small-scale and edge targets. In recent years, multimodal object detection algorithms based on vision-language fusion have made some progress in general object detection tasks. However, when directly applied to the scenario of detecting defects on the surface of filling barrels, the following problems still exist: In industrial scenarios, the scale of targets varies greatly. Targets such as drips, scratches, and dirt are usually small in area, have unclear edges, and are extremely similar, making them prone to false detection and missed detection; The surface of the barrel has metallic reflections, shadows, and complex background textures. Targets with weak textures are easily confused with background interference areas, leading to false detection or inaccurate localization; Existing multimodal detection models do not make sufficient use of textual semantic information, and the fusion method between query representation, visual features, and textual semantics is not flexible enough, making it difficult to adaptively adjust the detection process according to different image content. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a multi-modal fusion method, apparatus, and storage medium for multi-scale target detection in industrial filling scenarios. This addresses the technical problems of existing target detection methods in industrial filling scenarios, where targets are small in scale, highly variable, irregular in shape, and have weak texture features. These problems arise from issues such as varying lighting conditions, metal reflections, and background interference in the production environment, leading to insufficient target feature representation and consequently, missed detections, false detections, and inaccurate localization. The specific solution is as follows:

[0005] In a first aspect, the present invention provides a multi-modal fusion method for multi-scale target detection in an industrial filling scenario, the method comprising:

[0006] S10, Construct a multi-scale object detection model, which includes an image encoder, a text encoder, a hierarchical feature enhancement module, an encoder, a language-aware query selection module, a query fusion module, a language-aware decoder, a matching module, a classification head, and a bounding box regression head;

[0007] S20, Obtain an image of the packaging container;

[0008] S30, construct target category text information, and input the target category name as a text prompt into the multi-scale target detection model to provide semantic information for target detection;

[0009] S40, the acquired packaging container image is input into the multi-scale target detection model for target detection. The image encoder extracts features from the input packaging container image to obtain multi-scale visual features F. The hierarchical feature enhancement module performs global context modeling and local spatial feature enhancement on the multi-scale visual features F, followed by a flattening operation to obtain the enhanced visual features. and enhanced visual features The input is fed into the encoder to obtain the encoded visual features. Simultaneously, the text encoder encodes the text information describing the target category to obtain text embeddings E representing the names of each category. t Then embed E into the text t Perform global pooling to obtain the global text embedding E, which represents all categories in the current detection task. g Then the encoded visual features E i With global text embedding E g The language-aware query selection module receives the input together from the encoded visual features E. i In the selection of candidate target features E that are related to the semantics of the text sel The query fusion module will combine candidate target features E sel Compared with the initial learnable query representation Perform fusion to generate a fused query representation. ; Query representation after merging The language-aware decoder updates the query representation layer by layer, resulting in an enhanced query representation. Finally, the matching module completes the correspondence between the predicted results and the real targets, and the classification head and the bounding box regression head output the detection results such as the target category and location information.

[0010] Further, in step S40, the hierarchical feature enhancement module first divides the multi-scale visual feature F into g groups along the channel dimension, and then processes each sub-feature... Perform two-branch modeling; in the local branch, first extract sub-features through 3×3 convolution. The local spatial information is used to enhance the local structural information of the target, resulting in local branch features. Next, local branch features Perform a softmax operation to obtain the spatial attention weight representation. Meanwhile, local branch features Perform average pooling to obtain the spatial description vector. In the global branch, sub-features are extracted separately through average pooling operations along the height and width directions. By incorporating global and positional information from different directions, the model's ability to model spatial context relationships is enhanced, resulting in average pooling results in the height direction. and the average pooling result in the width direction Subsequently, feature fusion is performed through concatenation and convolution operations to obtain a global feature representation. Next, the global feature representation will be... Split along both height and width to obtain the height-direction feature. Features in the width direction Then, the height-oriented attention weights are generated using the Sigmoid activation function. Attention weights in the width direction , and the height-oriented attention weight Attention weights in the width direction AND characteristics Element-wise multiplication is performed to achieve adaptive recalibration of the input features in different spatial directions, resulting in global branch enhancement features. Enhance features for global branches Perform softmax and average pooling operations to obtain the spatial attention weight representation. and spatial description vector The interaction relationship between local and global branches is established through matrix multiplication, resulting in spatial interaction weights that guide local branches based on global branch features. And the spatial interaction weights obtained by guiding the global branch from local branch features. The spatial interaction weights obtained by guiding local branches with global branch features will be used. Spatial interaction weights obtained by guiding global branches from local branch features The components are then fused, and the final spatial attention weights are generated using the Sigmoid activation function. Then, the final spatial attention weights are used. Pair features Element-wise recalibration is performed to obtain the enhanced feature representation. Finally, the enhanced feature representations of each group are... By splicing the images together, the enhanced visual features are obtained. .

[0011] Furthermore, in step S10, the constructed multi-scale object detection model further includes a language-conditional query initialization module, which is located before the query fusion module; in step S40, the global text is embedded into E... g The query initialization module, which inputs the language conditions, initializes the initial learnable query representation. Modulate the language to generate a language-guided query representation. This allows the query representation to have semantic priors related to the current detection category before entering the language-aware decoder; then, the query fusion module combines the candidate target features E sel Language-guided query representation Perform fusion to generate a fused query representation. .

[0012] Furthermore, the language condition query initialization module first embeds the global text. As input, it is mapped through a multilayer perceptron to obtain the language query offset. Then set the language query offset. and initial learnable query representation To perform fusion, while retaining the initial learnable query table Introducing semantic compensation based on representation To obtain a language-guided query representation .

[0013] Furthermore, in step S10, the constructed multi-scale target detection model includes a query and feature adaptive fusion module in the language-aware decoder. This module comprises a static gated attention branch and a dynamic residual compensation branch, achieving multi-source information fusion through aggregation. In step S40, the query and feature adaptive fusion module uses the fused query representation as input to the language-aware decoder. Candidate target features The query representation is jointly modeled with its global context vector, and the fused query representation is modeled using residuals. The query representation is enhanced to obtain the enhanced query representation. .

[0014] Furthermore, in the static gated attention branch of the query and feature adaptive fusion module, the fused query representation is... and candidate target features The input is fed into a multi-head attention mechanism to establish the association between the query representation and the target region. To control the fusion strength, a globally learnable gating parameter is introduced. The attention output is modulated to obtain static fusion features. In the dynamic residual compensation branch of the query and feature adaptive fusion module, the candidate target features are first processed. Perform global average pooling to obtain a context vector that characterizes the overall semantic information of the image. Then, the context vector Query representation after fusion Concatenate the data to form a context-guided query representation. Based on this, compensation guided by context Generate dynamic compensation items Simultaneously, a scaling factor β is introduced to adjust the dynamic compensation term. The degree of influence is used to obtain the dynamic residual compensation characteristics. During the fusion phase, the fused query representation will be... Static fusion features and dynamic residual compensation characteristics Aggregation is performed to achieve residual fusion of multi-source features, resulting in the final enhanced query representation. .

[0015] Secondly, the present invention also provides a multi-modal fusion multi-scale target detection device for industrial filling scenarios, the device comprising:

[0016] The image acquisition module is used to acquire images of packaging containers;

[0017] The image and text input module is used to preprocess the acquired packaging container images to meet the input requirements for object detection; at the same time, it constructs target category text information to provide semantic information for object detection.

[0018] The target detection module is used to perform target detection on the acquired packaging container images using a multi-scale target detection model. This module includes a visual feature enhancement module, a text semantic guidance module, a multi-modal candidate feature selection module, and a decoding detection module. The visual feature enhancement module extracts multi-scale visual features from the preprocessed packaging container images using an image encoder. Then, a hierarchical feature enhancement module performs global context modeling and local spatial feature enhancement on the multi-scale visual features. After flattening, the enhanced visual features are obtained, and the encoder encodes these enhanced visual features to obtain encoded visual features. The text semantic guidance module encodes the text information describing the target category using a text encoder to obtain the encoded visual features. The table contains text embeddings of the category names and generates global text embeddings representing all categories in the current detection task through global pooling. The multimodal candidate feature filtering module filters candidate target features related to the target category from visual features through the language-aware query selection module to obtain candidate target features and provide accurate target candidate information. The decoding detection module fuses the candidate target features with the initial learnable query representation through the query fusion module to generate a fused query representation. Then, the language-aware decoder updates the fused query representation layer by layer to obtain an enhanced query representation. Finally, the matching module completes the correspondence between the prediction result and the real target, and the classification head and the bounding box regression head output the detection results such as target category and location information.

[0019] Furthermore, the text semantic guidance module also includes a language condition query initialization module, which is located before the query fusion module. The language condition query initialization module is used to modulate the initial learnable query representation through global text embedding to generate a language-guided query representation, so that the query representation has semantic prior related to the current detection category before entering the decoder.

[0020] Furthermore, the decoding and detection module also includes a query and feature adaptive fusion module; the query and feature adaptive fusion module is used to jointly model the fused query representation, candidate target features and their global context vectors as input to the language-aware decoder, and enhance the fused query representation through residual method to obtain an enhanced query representation.

[0021] Thirdly, the present invention also provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the above-described multi-modal fusion multi-scale target detection method for industrial filling scenarios.

[0022] One or more technical solutions proposed in this application have at least the following advantages compared with the prior art:

[0023] This invention provides a multimodal fusion method, device, and storage medium for multi-scale target detection in industrial filling scenarios. Addressing the issues of large target scale variations and insufficient feature representation for small and weakly textured targets in industrial filling scenarios, it introduces a hierarchical feature enhancement module. This module enhances the feature representation capability of detected targets in complex backgrounds through channel grouping, global context modeling, and local spatial feature extraction, improving the model's perception of small, weakly textured, and irregularly shaped targets such as drips, scratches, and dirt on packaging containers. Simultaneously, through channel grouping, multi-branch feature extraction, and spatial relationship modeling, it further strengthens visual feature representation, improving the model's perception capability for small targets, weakly textured targets, and complex background areas while controlling computational overhead, providing more effective feature support for subsequent processing.

[0024] This invention provides a multimodal fusion method, device, and storage medium for multi-scale target detection in industrial filling scenarios. Addressing the problems of traditional target detection models, such as the lack of category semantic guidance in the initial query representation and the strong blindness in the target search process, this invention proposes a language-conditional query initialization module. This module generates query offsets using global semantic representations of target category texts such as dripping liquid, QR codes, paint chips, scratches, dirt, and splashes. This modulates the initial learnable query representation, enabling it to acquire semantic priors relevant to the current detection task before entering the decoder, thus reducing the blindness in the target search process. The language-conditional query initialization module achieves a mapping from text semantics to query initialization, giving the language-guided query representation category awareness before entering the decoder. Compared to random initialization, this effectively guides the model to focus on semantically relevant target regions, realizing a shift from "visual-driven detection" to "visual and semantically guided detection," improving the target detection accuracy of the target detection model, and reducing interference from irrelevant areas such as background reflections, shadows, and barrel surface textures.

[0025] This invention provides a multimodal fusion method, device, and storage medium for multi-scale target detection in industrial filling scenarios. Addressing the problems of fixed cross-modal interaction methods and difficulty in adaptively adjusting query representations and the fusion strength of visual region features and category text semantic features in traditional target detection model decoders, this invention constructs a query and feature adaptive fusion module. It generates adaptive compensation terms based on visual context features, achieving dynamic fusion of query representations, visual features, and text semantics, thereby enhancing the target model's ability to detect multi-scale targets. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 The flowchart illustrates a multi-modal fusion multi-scale target detection method for an industrial filling scenario, according to an exemplary embodiment.

[0028] Figure 2 The diagram illustrates the structure of a multi-scale target detection model MQF-OV-DINO according to an exemplary embodiment.

[0029] Figure 3 This is a structural diagram of a hierarchical feature enhancement module (HFE) according to an exemplary embodiment.

[0030] Figure 4 This is a structural diagram of the Language Conditions Query Initialization Module (LCQI) according to an exemplary embodiment.

[0031] Figure 5 This is a structural diagram of the query and feature adaptive fusion module AQFF according to an exemplary embodiment.

[0032] Figure 6 This is a diagram illustrating the inference process of the multi-scale object detection model MQF-OV-DINO on an input image according to an exemplary embodiment.

[0033] Figure 7 The figure shows the comparative experimental results of the effect of the Hierarchical Feature Enhancement (HFE) module on different hierarchical features according to an exemplary embodiment.

[0034] Figure 8 This is an exemplary embodiment illustrating the overall process flow of a filling operation using a multi-modal fusion multi-scale target detection method in an industrial filling scenario.

[0035] Figure 9 This is a flowchart illustrating a multi-modal fusion multi-scale target detection method for industrial filling scenarios to detect defects on the surface of containers and perform subsequent filling operations, according to an exemplary embodiment.

[0036] Figure 10 The image is an example of a homemade filling barrel drip detection dataset according to an exemplary embodiment.

[0037] Figure 11 The results are shown in the comparative experiment of each model on the filling barrel drip detection dataset according to an exemplary embodiment.

[0038] Figure 12 The image shows the target detection results of each model on the filling barrel drip detection dataset in a comparative experiment according to an exemplary embodiment.

[0039] Figure 13 This is a framework diagram of a multi-modal fusion multi-scale target detection device for an industrial filling scenario, according to an exemplary embodiment.

[0040] In the above figure:

[0041] 100. Image acquisition module.

[0042] 200. Image and text input module.

[0043] 300. Target detection module, 310. Visual feature enhancement module, 320. Text semantic guidance module, 330. Multimodal candidate feature selection module, 340. Decoding detection module. Detailed Implementation

[0044] To better explain the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application. In some instances, methods, means, elements, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of this application. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0045] Example 1

[0046] See Figure 1 This embodiment provides a multi-modal fusion method for multi-scale target detection in industrial filling scenarios, the method comprising:

[0047] S10, construct the multi-scale target detection model MQF-OV-DINO.

[0048] See Figure 2 In step S10, the constructed multi-scale target detection model MQF-OV-DINO includes an image encoder, a text encoder, a hierarchical feature enhancement module (HFE), an encoder (Transformer Encoder), a language-aware query selection module (LAQS), a language-conditional query initialization module (LCQI), a query fusion module (Query Fusion), a language-aware decoder (Transformer Decoder), a matching module (Matching), a classification head (Cls), and a bounding box regression head (Box). The language-aware decoder (Transformer Decoder) includes a query and feature adaptive fusion module (AQFF).

[0049] In this embodiment, the Image Encoder, the Text Encoder, the Transformer Encoder, the Language Aware Query Selection Module (LAQS), the Query Fusion Module (Query Fusion), the Matching Module (Matching), the Classification Head (Cls), and the Bounding Box Regression Head (Box) all refer to the existing OV-DINO model, and their specific structures will not be described in detail here.

[0050] In this embodiment, the hierarchical feature enhancement module (HFE) is introduced after the image encoder and before the encoder (TransformerEncoder), and its structure is as follows: Figure 3 As shown, the Hierarchical Feature Enhancement (HFE) module groups the multi-scale features output by the Image Encoder into channels and extracts local detail information and global context information respectively. Then, the two types of features are fused to generate spatial attention weights to enhance key regions. Finally, the enhanced features are concatenated and input into the Transformer Encoder.

[0051] In this embodiment, a query initialization module LCQI based on language conditions is introduced before the query fusion module Query Fusion, and its specific structure is as follows: Figure 4As shown, the linguistic condition query initialization module LCQI first encodes the target category text using a text encoder to obtain category text embedding features; then, it generates a global text semantic representation through a global pooling operation to characterize the category semantic information in the current detection task. Next, the global text semantic representation is input into a multilayer perceptron to generate a language query offset, which is then fused with the initial learnable query representation to obtain a language-guided query representation.

[0052] In this embodiment, the Language Aware Decoder (Transformer Decoder) includes a Query and Feature Adaptive Fusion (AQFF) module, the specific structure of which is as follows: Figure 5 As shown. The query and feature adaptive fusion module AQFF includes a static gated attention branch and a dynamic residual compensation branch, which achieves multi-source information fusion through aggregation.

[0053] S20, Obtain the image of the packaging container.

[0054] S30, construct target category text information, and input the target category name as a text prompt into the multi-scale target detection model MQF-OV-DINO to provide semantic information for target detection.

[0055] For example, category names such as dripping liquid, QR code, paint chipping, scratches, dirt, and splashes can be used as text prompts input into the multi-scale target detection model MQF-OV-DINO to provide semantic information for subsequent multimodal detection.

[0056] S40, the acquired packaging container image is input into the multi-scale target detection model MQF-OV-DINO for target detection.

[0057] See Figure 6 Step S40 specifically includes the following steps:

[0058] S41, the image encoder extracts features from the input packaging container image to obtain multi-scale visual features F.

[0059] S42, the hierarchical feature enhancement module HFE performs global context modeling and local spatial feature enhancement on the multi-scale visual feature F, and then performs a flattening operation to obtain the enhanced visual feature F', and inputs the enhanced visual feature F' into the encoder Transformer Encoder to obtain the encoded visual feature Ei.

[0060] Specifically, the hierarchical feature enhancement module HFE first divides the multi-scale visual features F into g groups along the channel dimension, and then processes each sub-feature... Perform two-branch modeling.

[0061] In the local branch, sub-features are first extracted using 3×3 convolution. The local spatial information is used to enhance the local structural information of the target, resulting in local branch features. The calculation formula is: ,in, Represents a 3×3 convolution operation

[0062] Next, local branch features Perform a softmax operation to obtain the spatial attention weight representation. The calculation formula is: ,in, This represents the normalized activation function; simultaneously, it addresses local branch features. Perform average pooling to obtain the spatial description vector. The calculation formula is: ,in, This indicates the average pooling operation.

[0063] In the global branch, sub-features are extracted separately through average pooling operations along the height and width directions. By incorporating global and positional information from different directions, the model's ability to model spatial context relationships is enhanced, resulting in average pooling results in the height direction. and the average pooling result in the width direction The calculation formula is: ,in, and These represent the average pooling operations in the height and width dimensions, respectively.

[0064] Subsequently, feature fusion is performed through concatenation and convolution operations to obtain a global feature representation. The calculation formula is: ,in, This indicates a splicing operation. This represents a 1×1 convolution operation.

[0065] Next, the global feature representation will be... Split along both height and width to obtain the height-direction feature. Features in the width direction The calculation formula is: ,in, This is a splitting function.

[0066] Then, the height-oriented attention weights are generated using the Sigmoid activation function. Attention weights in the width direction The calculation formula is: ,in, This represents the Sigmoid activation function.

[0067] Weight the height orientation attention Attention weights in the width direction AND characteristics Element-wise multiplication is performed to achieve adaptive recalibration of the input features in different spatial directions, resulting in global branch enhancement features. The calculation formula is: ,in, This indicates element-wise multiplication.

[0068] Enhanced features for global branches Perform softmax and average pooling operations to obtain the spatial attention weight representation. and spatial description vector The calculation formula is: , .

[0069] Based on this, the interaction relationship between local and global branches is established through matrix multiplication, resulting in two sets of spatial interaction weights. , The calculation formula is: , ,in, This represents the spatial interaction weights obtained by guiding local branches with global branch features. This represents the spatial interaction weights obtained by guiding the global branch from local branch features. This indicates matrix multiplication calculation.

[0070] Spatial interaction weights obtained by guiding local branches with global branch features Spatial interaction weights obtained by guiding global branches from local branch features The components are then fused, and the final spatial attention weights are generated using the Sigmoid activation function. The calculation formula is: .

[0071] Then, the final spatial attention weights are used. Pair features Element-wise recalibration is performed to obtain the enhanced feature representation. The calculation formula is: .

[0072] Finally, the enhanced feature representations of each group are... By splicing the images together, the enhanced visual features are obtained. The calculation formula is: .

[0073] To verify the effectiveness of the Hierarchical Feature Enhancement (HFE) module on different feature levels, while maintaining consistency in model structure and training strategy, the HFE module was applied to different feature layers. Here, P1, P2, P3, and P4 represent the four scale features obtained after feature extraction, where P1 is the low-level detail feature with the highest spatial resolution, P2 and P3 represent intermediate-level features, and P4 is the high-level semantic feature with the lowest spatial resolution. Experimental results are as follows: Figure 7 As shown.

[0074] When the hierarchical feature enhancement module (HFE) is applied only to layer F4, the multi-scale object detection model MQF-OV-DINO achieves optimal results across all four evaluation metrics. Compared to applying HFE to layer F4+F3, when applied only to layer F4, the AP, APs, APm, and APl of the multi-scale object detection model MQF-OV-DINO are improved by 0.485%, 0.307%, 0.580%, and 0.337%, respectively. This demonstrates that targeted enhancement of high-level semantic features can simultaneously improve the scale-based object detection capability of the multi-scale object detection model MQF-OV-DINO.

[0075] When the Hierarchical Feature Enhancement (HFE) module's operating layers are increased to F4+F3+F2 and F4+F3+F2+F1, the performance of the multi-scale target detection model MQF-OV-DINO decreases. Compared to the setting where it only operates on layer F4, when it operates on layer F4+F3+F2, the AP, APs, APm, and APl of the multi-scale target detection model MQF-OV-DINO decrease by 1.305%, 2.117%, 2.107%, and 1.772%, respectively; when it operates on layer F4+F3+F2+F1, the AP, APs, APm, and APl of the multi-scale target detection model MQF-OV-DINO decrease by 1.199%, 1.674%, 1.063%, and 0.809%, respectively. The results show that excessive enhancement of low-level features introduces redundant details and background noise, interfering with the feature representation of key target regions by the multi-scale target detection model MQF-OV-DINO. Considering the overall detection performance and multi-scale target detection effect of the multi-scale target detection model MQF-OV-DINO, this embodiment adopts the method of introducing the hierarchical feature enhancement module HFE only on the F4 high-level semantic features.

[0076] S43, the text encoder encodes the text information describing the target category to obtain the text embedding E representing the name of each category. t Then embed E into the text tPerform global pooling to obtain the global text embedding E, which represents all categories in the current detection task. g .

[0077] S44, then encode the visual features E i With global text embedding E g The common input is the Language Aware Query Selection (LAQS) module, which selects the encoded visual features E. i In the selection of candidate target features E that are related to the semantics of the text sel .

[0078] S45, embed global text into E g The LCQI query initialization module, which inputs the language conditions, performs initial learnable query representation. Modulate the language to generate a language-guided query representation. This allows the query representation to have semantic priors related to the current detection category before entering the language-aware decoder, Transformer Decoder.

[0079] Specifically, the LCQI query initialization module for language conditions first embeds the global text. As input, the language query offset is obtained by mapping through a multilayer perceptron (MLP). The calculation formula is: ,in, This represents the mapping function of a multilayer perceptron.

[0080] Then set the language query offset. and initial learnable query representation The fusion is performed while preserving the initial learnable query representation. Introducing semantic compensation To obtain a language-guided query representation The calculation formula is: ,in, This is a learnable scaling factor.

[0081] Through the above operations, the LCQI (Language Condition Query Initialization) module realizes the mapping from text semantics to query initialization, enabling language-guided query representation. The model possesses category awareness capabilities before entering the language-aware decoder, TransformerDecoder. Compared to random initialization, this effectively guides the model to focus on semantically relevant target regions, achieving a shift from "visual-driven detection" to "visually and semantically guided detection," thereby improving the model's target detection accuracy.

[0082] S46, the query fusion module Query Fusion will convert the candidate target features Esel Language-guided query representation Perform fusion to generate a fused query representation. .

[0083] S47, Query representation after fusion After being updated layer by layer by the language-aware decoder Transformer Decoder, the query and feature adaptive fusion module AQFF will use the fused query representation, which is the input of the language-aware decoder Transformer Decoder, as the result. Candidate target features The query representation is jointly modeled with its global context vector, and the fused query representation is modeled using residuals. The query representation is enhanced to obtain the enhanced query representation. .

[0084] Specifically, the query and feature adaptive fusion module AQFF includes a static gated attention branch and a dynamic residual compensation branch, which achieves multi-source information fusion through aggregation.

[0085] In the static gated attention branch of the query and feature adaptive fusion module AQFF, the fused query representation is... and candidate target features The input is fed into a multi-head attention mechanism to establish the association between the query representation and the target region. To control the fusion strength, a globally learnable gating parameter is introduced. The attention output is modulated to obtain static fusion features. The calculation formula is: ,in, Used to constrain the gating value within a certain range, controlling the strength of the influence of the attention output on the query representation. This indicates a multi-headed attention operation.

[0086] In the dynamic residual compensation branch of the query and feature adaptive fusion module AQFF, the candidate target features are first processed. Perform global average pooling to obtain a context vector that characterizes the overall semantic information of the image. The calculation formula is: .

[0087] Then, the context vector Query representation after fusion Concatenate the data to form a context-guided query representation. Based on this, compensation guided by context Generate dynamic compensation items The calculation formula is: .

[0088] Simultaneously, a scaling factor β is introduced to adjust the dynamic compensation term. The degree of influence is used to obtain the dynamic residual compensation characteristics. The calculation formula is: .

[0089] During the fusion phase, the fused query representation will be... Static fusion features and dynamic residual compensation characteristics Aggregation is performed to achieve residual fusion of multi-source features, resulting in the final enhanced query representation. The calculation formula is: ,in, This indicates an element-wise addition operation.

[0090] S48 Finally, the Matching module completes the correspondence between the prediction results and the real targets, and the classification head Cls and the bounding box regression head Box output the detection results such as the target category and location information.

[0091] The following is a detailed description of the specific application of this method in the packaging container filling business, which is applicable to the detection of defects on the surface of filling barrels.

[0092] Taking a fully automatic filling machine as an example, the fully automatic filling machine has a host computer, a slave computer, an industrial camera for acquiring images of packaging containers, a filling business execution module, and machine vision defect detection software. The machine vision defect detection software has the aforementioned multi-scale target detection model MQF-OV-DINO embedded in it.

[0093] See Figure 8 , Figure 8 The overall process of filling operations using a multi-modal fusion multi-scale target detection method in industrial filling scenarios is shown.

[0094] First, the host computer, the slave computer, and the machine vision defect detection software communicate in real time. After the host computer issues the detection task, the industrial camera captures real-time images, which are then preprocessed and input into the multi-scale target detection model MQF-OV-DINO for barrel surface defect detection. The processing results are then returned to the slave computer, which controls the filling business execution module to complete the subsequent filling task.

[0095] See Figure 9 , Figure 9 This document illustrates a workflow diagram for detecting barrel surface defects and executing subsequent filling operations using a multi-modal fusion multi-scale target detection method in an industrial filling scenario. The specific process includes:

[0096] S1, Image Acquisition Step: Acquire images of packaging containers on the filling production line.

[0097] S2, Image and Text Input Step: The acquired packaging container image undergoes preprocessing operations such as size adjustment and normalization to meet the input requirements of the multi-scale target detection model; simultaneously, target category text information is constructed, and defect category names such as dripping, QR code, paint peeling, scratches, dirt, and splashes are used as text prompts input into the multi-scale target detection model MQF-OV-DINO to provide semantic information for subsequent multimodal defect detection.

[0098] S3, Visual Feature Enhancement Step: The preprocessed packaging container image is input into the ImageEncoder to extract multi-scale visual features; then, the Hierarchical Feature Enhancement (HFE) module is used to perform global context modeling and local spatial feature enhancement on the visual features, thereby improving the model's ability to express the features of defective targets such as small-scale droplets, weakly textured targets, and edge regions.

[0099] S4, Text Semantic Guidance Step: Input the defect category text information into the Text Encoder to obtain text embedding features representing the names of each defect category, and generate a global text embedding to represent all defect categories in the current detection task through global pooling; The Language Condition Query Initialization Module (LCQI) modulates the initial learnable query representation using the global text embedding to generate a language-guided query representation, so that the query representation has semantic priors related to the current defect detection category before entering the Language Aware Decoder (Transformer Decoder), reducing the blindness of the defect target search process.

[0100] S5, the multimodal candidate feature screening step, inputs the encoded visual features and global text embedding into the Language Aware Query Selection Module (LAQS) to screen candidate target features related to text semantics from the encoded visual features, reducing interference from irrelevant areas such as background reflection, shadows and barrel texture, and providing more accurate target candidate information for subsequent decoding and detection.

[0101] S6, Decoding and Detection Step: The candidate target features and the language-guided query representation are fused using the Query Fusion module to generate a fused query representation as input to the Language-Aware Decoder (Transformer Decoder). This fused representation is then updated layer by layer within the Transformer Decoder. The Query and Feature Adaptive Fusion module (AQFF) jointly models the fused query representation, candidate target features, and their global context vectors as input to the Transformer Decoder, and enhances the fused query representation using residuals to obtain an enhanced query representation. Finally, the Matching module matches the predicted results with the true target, and the classification head (Cls) and the bounding box regression head (Box) output the defect detection results.

[0102] S7 controls the execution steps, returns the defect detection results to the lower-level machine, and the lower-level machine controls the filling business execution module to complete the subsequent filling task.

[0103] To verify the defect detection performance of this method in industrial filling scenarios, experiments were conducted using a self-made Filling Barrel Droplet Detection Dataset (FBDD).

[0104] This dataset contains 2208 images (1440×1080 pixels) of cans captured using a Hikvision 1.6MP color camera on a filling production line. The images are used to create an industrial filling scene target detection dataset. The dataset primarily consists of top-down views of the can surfaces, including different lighting conditions and filling line backgrounds. The can surface colors are dark green and blue. The training and validation set ratio is 9:1. It includes six detection categories: water, QR codes, paint drops, scratches, dirt, and splashes. The primary detection target is water droplets on the can surface. However, since QR codes, paint drops, scratches, and dirt on the can surface can easily visually interfere with water droplets in a filling scene, these areas are also labeled as target categories to reduce the model's false detection rate. Some image examples are shown below. Figure 10 As shown.

[0105] In the image, the droplets appear as small dots or irregular areas, with a small scale of about 15×15 pixels, accounting for 0.014% of the total image area. They are scattered and their features are not obvious under the interference of barrel surface reflection, shadow occlusion and background texture. The scratches, dirt and paint chips are irregular in shape and are easily confused with droplets, barrel surface reflection, shadow or background texture, which further increases the difficulty of defect target detection.

[0106] To verify the effectiveness of this method, this embodiment uses average precision (AP), average precision of targets at different scales (APs, APm, and APl), as well as the number of parameters (Params), floating-point operations (FLOPs), and the number of images processed per second (Frames Per Second, FPS) as evaluation metrics.

[0107] AP: Measures the overall detection performance under different recall conditions. The higher the value, the better the performance in classification and localization. The calculation formula is as follows: ,in, This represents the number of IoU thresholds; in the COCO standard, N=10. This represents the average accuracy calculated at the IoU threshold.

[0108] Aps is used to determine the detection accuracy for targets smaller than 322 pixels, APm is used to determine the model's detection accuracy for targets between 322 and 962 pixels, and Apl is used to determine the model's detection accuracy for targets larger than 962 pixels.

[0109] Experimental results on the filling barrel drip detection dataset are as follows: Figure 11 As shown in the figure. Compared with the traditional OV-DINO model, the AP, APs, APm, and APl of the multi-scale target detection model in this method are improved by 0.09%, 0.851%, 1.634%, and 0.092%, respectively, indicating that the proposed method can enhance the model's feature representation ability for targets of different scales and improve the detection accuracy of small-scale targets. The target detection results on the filling barrel drip detection dataset are shown in the figure. Figure 12 As shown, the left column of the image is the complete image, and the last two columns are magnified views of the edge area of ​​the bucket. The red boxes indicate manually marked missed detection locations, used to compare the droplet detection effects of different methods. Figure 12None of the methods showed false detections, but there were significant differences in the detection of small droplets on the edge of the bucket. In the upper left edge region, there were two adjacent droplets in the original image. All four comparison models could only detect one droplet, while the multi-scale target detection model MQF-OV-DINO in our method could detect both droplets simultaneously. In the right edge region, there was one droplet. Both the traditional OV-DINO model and the multi-scale target detection model MQF-OV-DINO in our method could detect the droplet, but the latter was more accurate in detecting the droplet location. In summary, the multi-scale target detection model MQF-OV-DINO in our method can detect more small droplets on the edge of the bucket, showing better detection results.

[0110] Example 2

[0111] See Figure 13 Based on the same technical concept, this embodiment provides a multi-modal fusion multi-scale target detection device for industrial filling scenarios, the device comprising:

[0112] Image acquisition module 100 is used to acquire images of packaging containers.

[0113] The image and text input module 200 is used to preprocess the acquired packaging container images to meet the input requirements for object detection; at the same time, it constructs target category text information to provide semantic information for object detection.

[0114] The target detection module 300 is used to perform target detection on the acquired packaging container images using the multi-scale target detection model MQF-OV-DINO.

[0115] The multi-scale object detection model MQF-OV-DINO includes an image encoder, a text encoder, a hierarchical feature enhancement module (HFE), an encoder (Transformer Encoder), a language-aware query selection module (LAQS), a language-conditional query initialization module (LCQI), a query fusion module (Query Fusion), a language-aware decoder (Transformer Decoder), a matching module (Matching), a classification head (Cls), and a bounding box regression head (Box). The language-aware decoder (Transformer Decoder) includes a query and feature adaptive fusion module (AQFF).

[0116] Since the multi-scale target detection model MQF-OV-DINO is the same as the multi-scale target detection model MQF-OV-DINO in the above method embodiment, the specific structure of the multi-scale target detection model MQF-OV-DINO will not be described in detail here.

[0117] The target detection module 300 includes a visual feature enhancement module 310, a text semantic guidance module 320, a multimodal candidate feature screening module 330, and a decoding detection module 340.

[0118] The visual feature enhancement module 310 is used to extract multi-scale visual features from the preprocessed packaging container image using the Image Encoder. Then, the hierarchical feature enhancement module HFE is used to perform global context modeling and local spatial feature enhancement on the multi-scale visual features. After flattening, the enhanced visual features are obtained, and the enhanced visual features are encoded by the Transformer Encoder to obtain the encoded visual features.

[0119] The text semantic guidance module 320 is used to encode the text information describing the target category of the identification using the text encoder to obtain text embeddings representing the names of each category, and to generate global text embeddings representing all categories in the current detection task through global pooling. The text semantic guidance module 320 is also used to modulate the initial learnable query representation using the global text embeddings through the language conditional query initialization module LCQI to generate a language-guided query representation, so that the query representation has semantic priors related to the current detection category before entering the decoder.

[0120] The multimodal candidate feature filtering module 330 is used to filter candidate target features related to the target category from visual features through the language-aware query selection module LAQS, thereby obtaining candidate target features and providing accurate target candidate information.

[0121] The decoding and detection module 340 is used to fuse candidate target features with an initial learnable query representation through the query fusion module Query Fusion to generate a fused query representation. The query and feature adaptive fusion module AQFF jointly models the fused query representation, candidate target features, and their global context vectors (inputs to the language-aware decoder Transformer Decoder), and enhances the fused query representation using residuals to obtain an enhanced query representation. The language-aware decoder Transformer Decoder then updates the fused query representation layer by layer. Finally, the matching module Matching completes the mapping between the predicted results and the true target, and the classification head Cls and the bounding box regression head Box output the detection results, including target category and location information.

[0122] Since this device performs target detection on the acquired packaging container images using the multi-scale target detection model MQF-OV-DINO, the principle for solving the problem is the same as the method in the above method embodiment. Therefore, the implementation of this device can refer to the implementation of the method, and the repeated parts will not be described again.

[0123] Example 3

[0124] Based on the same technical concept, this embodiment also provides a non-volatile computer-readable storage medium storing a computer program or instructions thereon. When the computer program or instructions are executed, the multi-modal fusion multi-scale target detection method for industrial filling scenarios described in this application embodiment is implemented.

[0125] Furthermore, in the various embodiments of this application, each functional module can be integrated into one processing module, or each module can exist independently, or two or more modules can be integrated into one module.

[0126] The aforementioned storage media include, but are not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), Cache, Hard Disk Drive (HDD), or Memory Card. The memory can be used to store computer program instructions.

[0127] Since the principle of the computer-readable storage medium used to solve the problem is similar to the multi-scale target detection method in the multi-modal fusion industrial filling scenario, the implementation of the computer-readable storage medium can be found in the implementation of the method, and the repetitive parts will not be repeated.

[0128] The foregoing has provided a detailed description of a multimodal fusion method, apparatus, and storage medium for multi-scale target detection in an industrial filling scenario, as provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0129] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation can be achieved by means of software plus necessary general-purpose hardware platforms, and of course, it can also be achieved by hardware. Furthermore, the computer program in the memory can be implemented as a software functional unit and, when sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the above-mentioned technical solution, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

Claims

1. A multi-modal fusion method for multi-scale target detection in industrial filling scenarios, characterized in that, The method includes: S10, Construct a multi-scale object detection model, which includes an image encoder, a text encoder, a hierarchical feature enhancement module, an encoder, a language-aware query selection module, a query fusion module, a language-aware decoder, a matching module, a classification head, and a bounding box regression head; S20, Obtain an image of the packaging container; S30, construct target category text information, and input the target category name as a text prompt into the multi-scale target detection model to provide semantic information for target detection; S40, the acquired packaging container image is input into the multi-scale target detection model for target detection. The image encoder extracts features from the input packaging container image to obtain multi-scale visual features F. The hierarchical feature enhancement module performs global context modeling and local spatial feature enhancement on the multi-scale visual features F, followed by a flattening operation to obtain the enhanced visual features. and enhanced visual features The input is fed into the encoder to obtain the encoded visual features. Simultaneously, the text encoder encodes the text information describing the target category to obtain text embeddings representing the names of each category. E t Then embed the text E t Perform global pooling to obtain global text embeddings that represent all categories in the current detection task. E g Then encode the visual features E i With global text embedding E g The language-aware query selection module receives input from the encoded visual features. E i Filter candidate target features that are semantically relevant to the text. E sel The query fusion module will combine candidate target features E sel Compared with the initial learnable query representation Perform fusion to generate a fused query representation. ; Query representation after merging The language-aware decoder updates the query representation layer by layer, resulting in an enhanced query representation. Finally, the matching module completes the correspondence between the predicted results and the real targets, and the classification head and the bounding box regression head output the detection results such as the target category and location information.

2. The method according to claim 1, characterized in that, In step S40, the hierarchical feature enhancement module first divides the multi-scale visual feature F into g groups along the channel dimension, and then processes each sub-feature... Perform two-branch modeling; in the local branch, first extract sub-features through 3×3 convolution. The local spatial information is used to enhance the local structural information of the target, resulting in local branch features. Next, local branch features Perform a softmax operation to obtain the spatial attention weight representation. Meanwhile, local branch features Perform average pooling to obtain the spatial description vector. In the global branch, sub-features are extracted separately through average pooling operations along the height and width directions. By incorporating global and positional information from different directions, the model's ability to model spatial context relationships is enhanced, resulting in average pooling results in the height direction. and the average pooling result in the width direction Subsequently, feature fusion is performed through concatenation and convolution operations to obtain a global feature representation. Next, the global feature representation will be... Split along both height and width to obtain the height-direction feature. Features in the width direction Then, the height-oriented attention weights are generated using the Sigmoid activation function. Attention weights in the width direction , and the height-oriented attention weight Attention weights in the width direction AND characteristics Element-wise multiplication is performed to achieve adaptive recalibration of the input features in different spatial directions, resulting in global branch enhancement features. Enhance features for global branches Perform softmax and average pooling operations to obtain the spatial attention weight representation. and spatial description vector The interaction relationship between local and global branches is established through matrix multiplication, resulting in spatial interaction weights that guide local branches based on global branch features. And the spatial interaction weights obtained by guiding the global branch from local branch features. The spatial interaction weights obtained by guiding local branches with global branch features will be used. Spatial interaction weights obtained by guiding global branches from local branch features The components are then fused, and the final spatial attention weights are generated using the Sigmoid activation function. Then, the final spatial attention weights are used. Pair features Element-wise recalibration is performed to obtain the enhanced feature representation. Finally, the enhanced feature representations of each group are... By splicing the images together, the enhanced visual features are obtained. .

3. The method according to claim 2, characterized in that, In step S10, the constructed multi-scale object detection model further includes a language-conditional query initialization module, which is located before the query fusion module; in step S40, the global text is embedded into E. g The query initialization module, which inputs the language conditions, initializes the initial learnable query representation. Modulate the language to generate a language-guided query representation. This allows the query representation to have semantic priors related to the current detection category before entering the language-aware decoder; then, the query fusion module combines the candidate target features E sel Language-guided query representation Perform fusion to generate a fused query representation. .

4. The method according to claim 3, characterized in that, The query initialization module for the language conditions first embeds the global text. As input, it is mapped through a multilayer perceptron to obtain the language query offset. Then set the language query offset. and initial learnable query representation To perform fusion, while retaining the initial learnable query table Introducing semantic compensation based on representation To obtain a language-guided query representation .

5. The method according to claim 4, characterized in that, In step S10, the constructed multi-scale target detection model includes a query and feature adaptive fusion module in the language-aware decoder. This module comprises a static gated attention branch and a dynamic residual compensation branch, achieving multi-source information fusion through aggregation. In step S40, the query and feature adaptive fusion module uses the fused query representation as input to the language-aware decoder. Candidate target features The query representation is jointly modeled with its global context vector, and the fused query representation is modeled using residuals. The query representation is enhanced to obtain the enhanced query representation. .

6. The method according to claim 5, characterized in that, In the static gated attention branch of the query and feature adaptive fusion module, the fused query representation is... and candidate target features The input is fed into a multi-head attention mechanism to establish the association between the query representation and the target region. To control the fusion strength, a globally learnable gating parameter is introduced. The attention output is modulated to obtain static fusion features. In the dynamic residual compensation branch of the query and feature adaptive fusion module, the candidate target features are first processed. Perform global average pooling to obtain a context vector that characterizes the overall semantic information of the image. Then, the context vector Query representation after fusion Concatenate the data to form a context-guided query representation. Based on this, compensation guided by context Generate dynamic compensation items Simultaneously, a scaling factor β is introduced to adjust the dynamic compensation term. The degree of influence is used to obtain the dynamic residual compensation characteristics. During the fusion phase, the fused query representation will be... Static fusion features and dynamic residual compensation characteristics Aggregation is performed to achieve residual fusion of multi-source features, resulting in the final enhanced query representation. .

7. A multi-modal fusion multi-scale target detection device for industrial filling scenarios, characterized in that, The device includes: The image acquisition module is used to acquire images of packaging containers; The image and text input module is used to preprocess the acquired packaging container images to meet the input requirements for object detection; at the same time, it constructs target category text information to provide semantic information for object detection. The target detection module is used to perform target detection on the acquired packaging container images using a multi-scale target detection model. This module includes a visual feature enhancement module, a text semantic guidance module, a multi-modal candidate feature selection module, and a decoding detection module. The visual feature enhancement module extracts multi-scale visual features from the preprocessed packaging container images using an image encoder. Then, a hierarchical feature enhancement module performs global context modeling and local spatial feature enhancement on the multi-scale visual features. After flattening, the enhanced visual features are obtained, and the encoder encodes these enhanced visual features to obtain encoded visual features. The text semantic guidance module encodes the text information describing the target category using a text encoder to obtain the encoded visual features. The table contains text embeddings of the category names and generates global text embeddings representing all categories in the current detection task through global pooling. The multimodal candidate feature filtering module filters candidate target features related to the target category from visual features through the language-aware query selection module to obtain candidate target features and provide accurate target candidate information. The decoding detection module fuses the candidate target features with the initial learnable query representation through the query fusion module to generate a fused query representation. Then, the language-aware decoder updates the fused query representation layer by layer to obtain an enhanced query representation. Finally, the matching module completes the correspondence between the prediction result and the real target, and the classification head and the bounding box regression head output the detection results such as target category and location information.

8. The apparatus according to claim 7, characterized in that, The text semantic guidance module also includes a language condition query initialization module, which is located before the query fusion module. The language condition query initialization module is used to modulate the initial learnable query representation through global text embedding to generate a language-guided query representation, so that the query representation has semantic prior related to the current detection category before entering the decoder.

9. The apparatus according to claim 8, characterized in that, The decoding and detection module further includes a query and feature adaptive fusion module; the query and feature adaptive fusion module is used to jointly model the fused query representation, candidate target features and their global context vectors as input to the language-aware decoder, and enhance the fused query representation through residual method to obtain an enhanced query representation.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the multimodal fusion multi-scale target detection method for industrial filling scenarios as described in any one of claims 1 to 6.

Images