Multi-frame liquid suspension detection method, system, device, and medium

Abstract

The application provides a multi-frame liquid suspension detection method, system, device and medium. The application initializes historical memory states containing content memory states and normalized memory states, and extracts image space features frame by frame; query features are generated based on the space features, read weights are obtained through mapping operation, and matched historical supplementary features are read according to the read weights; the space features and the historical supplementary features are fused to form enhanced features, and the position, category and confidence of the suspension foreign matter are output; meanwhile, keys, values and gate features are generated according to the space features, effective historical information is screened by relying on the gate features, and the memory increment is iteratively updated according to the key-value features to update the historical memory states, so as to continuously provide the time sequence context for subsequent frame detection, thereby improving the multi-frame liquid suspension detection precision.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, system, device and medium for detecting liquid suspended matter in multiple frames. Background Technology

[0002] In the field of pharmaceutical production quality inspection, single-frame deep learning visual inspection methods are widely used for detecting foreign object defects in liquids, ampoules, vials, and various pharmaceutical containers due to their advantages of simple operation and adaptability to common imaging scenarios. This method uses a single-frame image as input to complete feature extraction and target recognition, and it has good detection results for routine inspection tasks with clear targets, stable backgrounds, and large target sizes, meeting the needs of routine industrial quality inspection.

[0003] However, single-frame detection technology has significant limitations in detecting minute foreign objects in pharmaceuticals. Common anomalies in pharmaceutical detection include tiny suspended matter and particulate impurities. These targets are small in size, have low contrast, and are easily affected by factors such as bottle wall reflection, changes in the pharmaceutical liquid background, imaging noise, and motion blur, making identification extremely difficult. Furthermore, single-frame detection models have limited information utilization, only able to access information from the current frame and unable to utilize the spatial location and temporal characteristics of foreign objects in consecutive frames. Ultimately, this makes it difficult to adapt to the high-precision detection scenarios of complex and subtle foreign objects in pharmaceutical tablets. Therefore, an effective multi-frame liquid suspension detection method is urgently needed to address these issues. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to overcome or at least partially solve the above problems.

[0005] To achieve the above and other related objectives, the present invention provides a multi-frame liquid suspension detection method, the method comprising: The historical memory state is initialized, and the current frame image is obtained frame by frame from the continuously acquired image sequence. The current frame image is input into the target detection network to extract the spatial features of the current frame image. The historical memory state includes the content memory state and the normalized memory state. Query features are generated based on the spatial features of the current frame image, and the query features are used to perform mapping operations to obtain reading weights. Based on the reading weights, historical supplementary features that match the current frame image are read from the historical memory state. The historical supplementary features are fused with the spatial features to obtain enhanced features, which are then input into the decoder and the detection head to output the suspended foreign object detection results of the current frame image; wherein, the suspended foreign object detection results include the location of the foreign object, the type of the foreign object, and the detection confidence level; Based on the spatial features, key features, value features, and gating features are generated respectively; the gating features are used to filter the historical memory state to obtain filtered historical memory information; the key features and value features are used to construct a memory increment; and the historical memory state is updated by combining the historical memory information and the memory increment for the detection of suspended objects in the next frame image.

[0006] Optionally, the step of inputting the current frame image into the target detection network to extract the spatial features of the current frame image includes: The ViT backbone network in the target detection network is used to divide the current frame image into image blocks, and image block tokens are generated through embedding mapping. Then, the spatial position dependency relationship within the current frame image is established through a self-attention mechanism to obtain visual features. The visual features are dimensionally adjusted using the projection module in the target detection network to obtain spatial features suitable for the decoder.

[0007] Optionally, the historical memory state includes a content memory state and a normalized memory state. The content memory state is used to store the compressed key-value information of the preceding frame, and the normalized memory state is used to stabilize the historical reading process. The step of generating query features based on the spatial features of the current frame, performing mapping operations using the query features to obtain reading weights, and reading historical supplementary features matching the current frame image from the historical memory state based on the reading weights includes: Perform a mapping operation on the query features to obtain the read weights; The reading weights are applied to the content memory state and the normalized memory state respectively to calculate the historical supplementary features that match the current frame image.

[0008] Optionally, fusing the historical supplementary features with the spatial features to obtain enhanced features includes: The dimensions of the historical supplementary features are adjusted to obtain aligned historical features that are consistent with the dimensions of the spatial features. The enhanced features are obtained by residual fusion of the alignment history features and the spatial features.

[0009] Optionally, the step of inputting the enhanced features into the decoder and the detection head, and outputting the suspended foreign object detection result of the current frame image, includes: Based on the target query vector preset by the decoder, the enhanced features are subjected to cross-attention decoding operation by the decoder to obtain the target hidden layer representation. The target hidden layer representation data is processed using the classification branch and bounding box regression branch in the detection head to output the foreign object location, foreign object category, and detection confidence of the current frame image.

[0010] Optionally, generating key features, value features, and gating features based on the spatial features includes: The spatial features of the current frame image are mapped and transformed to obtain key features and value features for writing into the historical memory state; The spatial features of the current frame image are mapped and transformed, and a Sigmoid activation operation is performed to obtain gating features used to regulate the retention ratio of historical memory.

[0011] Optionally, the step of using the gating feature to filter the historical memory state to obtain filtered historical memory information, constructing a memory increment from the key feature and the value feature, and updating the historical memory state by combining the historical memory information and the memory increment includes: The historical memory state is filtered using the gating feature to obtain filtered historical memory information; Based on the key features and the value features, construct the memory increment corresponding to the current frame image; By combining the historical memory information with the memory increment update content memory state, and updating the normalized memory state based on the key features, the updated historical memory state is obtained.

[0012] Secondly, the present invention also provides a multi-frame liquid suspension detection system, the system comprising: The extraction module is used to initialize the historical memory state, obtain the current frame image frame by frame from the continuously acquired image sequence, input the current frame image into the target detection network, and extract the spatial features of the current frame image; the historical memory state includes content memory state and normalized memory state; The reading module is used to generate query features based on the spatial features of the current frame image, perform mapping operations using the query features to obtain reading weights, and read historical supplementary features that match the current frame image from the historical memory state based on the reading weights. The output module is used to fuse the historical supplementary features with the spatial features to obtain enhanced features, and input the enhanced features into the decoder and the detection head to output the suspended object detection results of the current frame image; wherein, the suspended object detection results include the location of the object, the type of object, and the detection confidence level; The update module is used to generate key features, value features and gating features based on the spatial features, obtain historical memory information by filtering through the gating features, and update the historical memory state by combining the key features and the value features for the detection of suspended objects in the next frame image.

[0013] Thirdly, the present invention provides an electronic device comprising: a memory and a processor; the memory for storing a computer program; and the processor for executing the computer program stored in the memory to enable the electronic device to perform the multi-frame liquid suspension detection method as described above.

[0014] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-frame liquid suspension detection method as described above.

[0015] The above-described one or more technical solutions provided by this invention can have the following advantages or at least achieve the following technical effects: This invention initializes a historical memory state containing content memory state and normalized memory state, extracting spatial features frame by frame. Based on these spatial features, query features are generated, and reading weights are obtained through mapping operations. Matching historical supplementary features are then read according to these reading weights. Spatial features and historical supplementary features are fused to form enhanced features, outputting the location, category, and confidence level of suspended foreign objects. Simultaneously, key-value and gating features are generated based on the spatial features. Valid historical information is filtered using gating features, and the historical memory state is iteratively updated using memory increments constructed from key-value features, continuously providing temporal context for subsequent frame detection. This solution is applied to light inspection scenarios for transparent liquid containers such as ampoules and vials. It strengthens the features of weak suspended objects by relying on multi-frame temporal information, filters background noise through reading weights and gating mechanisms, and uses compressed historical memory states to reduce computational and storage requirements, effectively improving the detection accuracy of suspended foreign objects in liquids. Attached Figure Description

[0016] Figure 1 The diagram shown is a flowchart of a multi-frame liquid suspension detection method according to an embodiment of the present invention.

[0017] Figure 2 The diagram shows the overall process of a time-memory-enhanced multi-frame liquid suspension detection network in one embodiment of the present invention.

[0018] Figure 3 The image shown is a schematic diagram of a single-frame liquid suspension target imaging in an embodiment of the present invention.

[0019] Figure 4 The image shown is a schematic diagram of multi-frame motion trajectory imaging of a liquid suspension in one embodiment of the present invention.

[0020] Figure 5 The diagram shows a functional module schematic of a multi-frame liquid suspended matter detection system in one embodiment of the present invention.

[0021] Figure 6 The diagram shown is a schematic representation of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0022] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0023] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0024] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0025] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0026] Unless otherwise stated, the term "multiple" means two or more.

[0027] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0028] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0029] The technical solutions of the present invention will now be described in detail with reference to the accompanying drawings.

[0030] Please see Figure 1An embodiment of the present invention provides a multi-frame liquid suspension detection method, the method may include the following steps S10~S40: Step S10: Initialize the historical memory state, obtain the current frame image frame by frame from the continuously acquired image sequence, input the current frame image into the target detection network, and extract the spatial features of the current frame image; the historical memory state includes the content memory state and the normalized memory state.

[0031] Among them, the historical memory state is used to store the compressed feature information of the preceding frame. This state consists of the content memory state and the normalized memory state; at the initial moment, the content memory state ( ) and normalized memory state ( All are initialized to zero, i.e. , .

[0032] Content memory state ( This carries the key-value features of the preceding frame after dimensionality reduction or projection compression, and is used to store the key-value information obtained by compression of the preceding frame.

[0033] Normalized memory state ( This is used to constrain numerical stability during time-series reading, avoid fluctuations in memory retrieval, and stabilize the historical supplementary feature reading process.

[0034] Image sequence This refers to a sequentially arranged set of images obtained by continuously imaging ampoules, vials, or other transparent medicine containers containing liquid medication. The detection network sequentially uses each frame in this image sequence as the current frame for suspended object detection, thereby identifying small suspended objects within the image. Let represent the image at frame t, and T represent the length of the input sequence.

[0035] The current frame image is the image that is being processed, acquired frame by frame in chronological order from a continuously acquired image sequence.

[0036] The object detection network is a multi-frame object detection framework with temporal modeling capabilities, formed by embedding a history memory read / write module on top of the standard RF-DETR (Roboflow Detection Transformer) basic network architecture. This network retains the standard RF-DETR's ViT (Vision Transformer) backbone network and Transformer-based decoder structure, and adds a history memory storage unit and a temporal feature fusion unit.

[0037] The historical memory storage unit is used to save the compressed feature information of the preceding frame, and the temporal feature fusion unit is used to fuse the read historical supplementary features with the spatial features of the current frame, thereby expanding the single-frame detection network into a multi-frame detection system that can utilize the temporal correlation information of multiple frames to improve the detection performance of weak suspended targets.

[0038] Please see Figure 2 The figure shows the overall process of the temporal memory-enhanced liquid suspension detection network. The image sequence is input into the feature extraction module of the target detection network in time sequence to extract the spatial features of the current frame image. The spatial features are divided into two paths: one path constructs query features to read the historical memory state to obtain historical supplementary features. After fusing the historical supplementary features with the spatial features, the detection result is output through the decoder and the detection head; the other path generates key features, value features, and gating features, and iteratively updates the historical memory state for subsequent frame processing.

[0039] Spatial features are visual features obtained by the target detection network after extracting features from the current frame image. They carry the spatial context information of the image and are used for subsequent historical memory retrieval and detection processes.

[0040] In the specific implementation, during the initial stage of operation, the historical memory state is first initialized. This historical memory state includes a content memory state and a normalized memory state. The content memory state is used to store the compressed key-value information of the previous frame, and the normalized memory state is used to stabilize the historical reading process and avoid the problem of numerical instability as the number of frames increases. After initialization, the network processes the image sequence in chronological order, acquiring the current frame image frame by frame from the continuously acquired image sequence, and inputting the current frame image into the target detection network. The target detection network performs feature extraction on the current frame image to obtain the spatial features of the current frame. These spatial features serve as intermediate features for subsequent historical memory reading and suspended object target detection processes.

[0041] Step S20: Generate query features based on the spatial features of the current frame image, and use the query features to perform mapping operations to obtain reading weights. Based on the reading weights, read the historical supplementary features that match the current frame image from the historical memory state.

[0042] Among them, the query feature refers to the feature vector generated by inputting the spatial features of the current frame image into the query mapping module and processing it, which is used as the retrieval condition for matching historical time sequence information at each spatial location of the current frame; the query feature is subjected to a mapping operation to obtain the reading weight, and historical supplementary features are read from the historical memory state according to the reading weight.

[0043] This query characteristic can be represented as:

[0044] In the formula, This indicates the query mapping module; This indicates the query features used to read the historical memory state of the current frame image; This represents the spatial features of the current frame image. The query features are mapped to generate read weights. Based on these read weights, the historical memory state corresponding to the (t-1)th frame image is processed to retrieve and extract supplementary historical features; where... This indicates the historical memory state formed by the recursion of the preceding frames.

[0045] Read weights are obtained from query features through mapping operations and are used to quantify the correlation between the current query features and each historical stored feature.

[0046] Historical supplementary features refer to temporal feature information obtained by weighted aggregation of feature information within historical memory states according to reading weights; the historical memory states store compressed feature information of previous frames, including content memory states and normalized memory states; enhanced features are generated by fusing historical supplementary features with spatial features of the current frame.

[0047] In the specific implementation, the extracted spatial features of the current frame image are input into the query mapping module. After mapping processing, query features are generated, which serve as retrieval conditions for historical information at each spatial location of the current frame image. Then, a mapping operation is performed on the query features to obtain reading weights. The historical memory state, containing both content memory state and normalized memory state, is used as the historical feature information to be read. Based on the reading weights, the feature information in the historical memory state is weighted and aggregated, thereby obtaining historical supplementary features matching the current frame image from the historical memory state. Thus, through the attention interaction between the query features and the historical memory state, adaptive fusion of the spatial features of the current frame image and the historical features of the preceding frame is achieved, enabling the read historical supplementary features to effectively supplement the temporal context information of the current frame image.

[0048] Step S30: The historical supplementary features are fused with the spatial features to obtain enhanced features, and the enhanced features are input into the decoder and the detection head to output the suspended foreign object detection result of the current frame image; wherein, the suspended foreign object detection result includes the location of the foreign object, the type of the foreign object, and the detection confidence level.

[0049] Among them, enhanced features refer to the feature representation obtained by fusing historical supplementary features with the spatial features of the current frame image after dimensional alignment. This feature representation carries both the spatial visual information of the current frame and the temporal information of the previous frame.

[0050] The suspended foreign object detection result is the target information output by the decoder and the detection head after processing, including the location of the foreign object, the type of foreign object, and the detection confidence level.

[0051] Please see Figure 3 The figure is a schematic diagram of single-frame liquid suspended object imaging. The suspended object features are weak in a single frame image, but the target has motion response, dark lines or suspicious textures in the previous image frame. The aforementioned temporal memory fusion mechanism can retrieve such historical feature information to participate in the feature enhancement of the current frame, thereby improving the weak target recognition capability.

[0052] Please see Figure 4 The figure is a schematic diagram of multi-frame motion trajectory imaging of liquid suspended objects. The temporal memory feature fusion process accumulates weak target responses in multiple consecutive frames in the feature dimension, and uses multi-frame motion trajectory information to fill in the missing target features in the current frame, thereby strengthening the characterization of suspended object targets.

[0053] In the specific implementation, the read historical supplementary features are fused with the spatial features of the current frame image to obtain enhanced features. These enhanced features simultaneously contain the visual spatial information of the current frame and the historical temporal context information of the previous frames. Then, the enhanced features are input into a decoder for decoding, and the decoded features are input into a detection head, which outputs the suspended foreign object detection results for the current frame image. These results include the foreign object location, foreign object category, and detection confidence level, respectively characterizing the spatial location, category attribute, and detection confidence level of the suspended foreign object in the current frame image.

[0054] Step S40: Generate key features, value features, and gating features based on the spatial features; use the gating features to filter the historical memory state to obtain filtered historical memory information; construct a memory increment based on the key features and the value features; and update the historical memory state by combining the historical memory information and the memory increment.

[0055] Among them, the key feature refers to the feature vector obtained by mapping and transforming the spatial features of the current frame. It is used to perform operations with the normalized memory state to determine the position and weight allocation of the current frame feature information written into the historical memory state, and serves as the index information for writing into the historical memory state.

[0056] Value features refer to the feature vectors obtained by mapping and transforming the spatial features of the current frame. They are used to provide the feature content of the current frame to be written into the historical memory state and carry the target content information to be stored in the historical memory.

[0057] Gated features refer to feature vectors generated by mapping transformation and activation operations of the spatial features of the current frame. They are used to measure the correlation between the spatial features of the current frame and each memory unit in the historical memory state, regulate the retention ratio of old historical memory states, and suppress the long-term accumulation of imaging noise and false responses in historical memory.

[0058] Historical memory information is the feature information of the preceding frame stored in the historical memory state. When updating the memory state, the effective historical memory is retained by filtering through the gating features of the current frame, and the memory is updated by combining the key features and value features of the current frame, thus providing a temporal context for suspended object detection.

[0059] The memory increment refers to the amount of features to be updated, which is jointly constructed from the key features and value features of the current frame. It is the core update data used to iteratively update the historical memory state.

[0060] In the specific implementation, based on the spatial features of the current frame image, key features, value features, and gated features are generated by performing key mapping transformation, value mapping transformation, and gated mapping transformation respectively. The gated features are used to filter the historical memory state, removing outdated and irrelevant redundant memory information and retaining valid historical memories related to the temporal features of suspended objects, resulting in the filtered historical memory information. Subsequently, a memory increment is constructed using the key features and value features. The key features are used to define the position where the current frame features are written into the historical memory state and assign writing weights, while the value features are used to carry the spatial feature content of the current frame to be written. The filtered historical memory information and the constructed memory increment are combined to complete the iterative update of the historical memory state, synchronously updating the content memory state and the normalized memory state. The updated historical memory state can continuously store the compressed and processed valid feature information of the previous frame, providing stable and continuous temporal context support for the detection of suspended objects in the next frame image.

[0061] In this embodiment, by initializing a historical memory state containing content memory state and normalized memory state, spatial features of the image are extracted frame by frame. Query features are generated based on spatial features, and read weights are obtained through mapping operations. Matching historical supplementary features are read according to the read weights. Spatial features and historical supplementary features are fused to form enhanced features, outputting the location, category, and confidence level of suspended foreign objects. Simultaneously, key-value and gating features are generated based on spatial features. Valid historical information is filtered using gating features, and the historical memory state is iteratively updated using memory increments constructed from key-value features, continuously providing temporal context for subsequent frame detection. This solution is applied to light inspection scenarios for transparent liquid containers such as ampoules and vials. It strengthens the features of weak suspended objects by relying on multi-frame temporal information, filters background noise through read weights and gating mechanisms, and uses compressed historical memory states to reduce computing power and storage requirements, effectively improving the detection accuracy of suspended foreign objects in liquids.

[0062] Based on the foregoing embodiments, a second embodiment of the multi-frame liquid suspension detection method of the present invention is proposed. In this embodiment, step S10 may include the following sub-steps S101~S102: Sub-step S101 involves using the ViT backbone network in the target detection network to generate image block tokens from the current frame image through image block embedding mapping, and then establishing the spatial position dependency relationship within the current frame image through a self-attention mechanism to obtain visual features.

[0063] The image block token is formed by dividing the image into several fixed-size image blocks. Each image block is converted into a one-dimensional feature vector with a fixed dimension through a linear embedding mapping. This feature vector is called the image block token.

[0064] Visual features refer to the features extracted by the ViT backbone network of the object detection network after dividing the current frame image into image blocks (Tokens) and modeling spatial dependencies using a self-attention mechanism.

[0065] The internal self-attention of a ViT can be represented as:

[0066] In the formula, Q, K, and V represent the query, key, and value in ViT self-attention, respectively; d represents the feature dimension; this self-attention is used to model the spatial context relationship within the current frame image.

[0067] The spatial features of the current frame are obtained after passing through the ViT Backbone and Projector (projection module):

[0068] In the formula, This refers to the ViT Backbone (ViT backbone network, used to extract basic visual features of images). This refers to the Projector (projection module, used to perform dimensional projection transformation on the main output features). Represents the image of frame t; This represents the spatial features of the current frame.

[0069] In the specific implementation, the current frame image is input into the ViT backbone network of the object detection network. First, the current frame image is divided into a sequence of image blocks of fixed size. Linear embedding mapping is performed on the image block sequence and position encoding is superimposed to generate image block tokens, thus obtaining a serialized feature representation. Then, the above serialized feature representation is input into a multi-layer Transformer encoding structure. The global association weights between each image block are calculated through a self-attention mechanism to establish long-distance dependencies between any spatial locations within the current frame image. Global contextual information is extracted and fused layer by layer to finally obtain visual features.

[0070] Sub-step S102 involves using the projection module in the target detection network to adjust the dimensions of the visual features, thereby obtaining the spatial features of the adaptive decoder.

[0071] In the specific implementation, the projection module in the object detection network is used to process the visual features extracted by the ViT backbone network. Through linear transformation, the visual features are mapped to feature representations adapted to the input dimension of the decoder, thus obtaining the spatial features of the current frame image. These spatial features serve as the basis for subsequent processing, used to generate query features, key features, value features, and gating features, respectively. They are then input into the decoder as objects to be fused with historical supplementary features, ensuring the smooth execution of suspended object detection.

[0072] In this embodiment, the current frame image is input into the target detection network. The ViT backbone network in the target detection network is used to divide the current frame image into image blocks. Image block tokens are generated through embedding mapping. The spatial position dependencies within the current frame image are established through a self-attention mechanism to obtain visual features. Then, the projection module in the target detection network is used to adjust the dimensions of the visual features to obtain spatial features for the adaptive decoder. This provides the spatial visual information foundation of the current frame for subsequent generation of query features, key features, value features, and gating features based on the spatial features, as well as for performing historical memory retrieval and enhanced feature fusion.

[0073] Based on the foregoing embodiments, a third embodiment of the multi-frame liquid suspension detection method of the present invention is proposed. In this embodiment, step S20 may include the following sub-steps S201~S202: Sub-step S201: Perform a mapping operation on the query features to obtain the read weights.

[0074] In the specific implementation, the query features are used as retrieval vectors, and the reading weights are obtained through mapping operations. These reading weights quantify the correlation between the current query features and historically stored features, enabling selective retrieval of temporal memories. This provides a filtering basis for fusing temporal context information from previous frames, supporting subsequent enhanced feature generation and suspended object / foreign object detection.

[0075] In sub-step S202, the reading weights are applied to the content memory state and the normalized memory state respectively to calculate the historical supplementary features that match the current frame image.

[0076] Among them, the historical supplementary features can be represented as:

[0077] In the formula, Represents a nonnegative mapping function; This represents a constant to prevent the denominator from being zero; Represents the query features of the current frame image; This represents the supplementary historical features retrieved from the historical memory state for the current frame image.

[0078] This expression represents the query features of the current frame. After nonnegative mapping, read weights are generated and applied to the content memory states of the preceding frames. and normalized memory state We perform weighted normalization operations to obtain the historical supplementary features related to the current frame from the previous frame.

[0079] In the specific implementation, the reading weight is applied to the content memory state and the normalized memory state in the historical memory state for weighted operation, thereby retrieving and extracting historical supplementary features. A matrix multiplication operation is performed between the reading weight and the content memory state to obtain the content reading value; then, a matrix multiplication operation is performed between the reading weight and the normalized memory state to obtain the normalized reading value; finally, the content reading value and the normalized reading value are normalized to obtain the historical supplementary features. During the reading process, the reading results of the content memory state are normalized using the normalized memory state to suppress the numerical offset caused by the accumulation of temporal information, ensuring that the historical supplementary features reliably represent the effective visual information of the preceding frame, and providing stable supplementary information for the fusion of the spatial features and temporal historical features of the current frame image.

[0080] In this embodiment, a reading weight is obtained by performing a mapping operation on the query features. The reading weight is then applied to the content memory state and the normalized memory state to obtain historical supplementary features that match the current frame image. This supports the fusion of spatial features of the current frame image and historical temporal features, thereby achieving temporal enhancement of image features.

[0081] Based on the foregoing embodiments, a fourth embodiment of the multi-frame liquid suspension detection method of the present invention is proposed. In this embodiment, in order to achieve effective fusion of historical temporal information and current frame spatial information, and to enhance the temporal correlation and feature representation capability of liquid suspension detection, step S30 can complete feature enhancement through feature reading, dimension alignment and spatiotemporal fusion, which may specifically include the following sub-steps S301~S302: Sub-step S301: Adjust the dimensions of the historical supplementary features to obtain aligned historical features that are consistent with the dimensions of the spatial features.

[0082] Among them, the historical features are aligned, and the historical supplementary features are adjusted in dimension to obtain the temporal features. Their dimensions are matched with the current frame spatial features, so as to realize the fusion of historical temporal information and current frame spatial visual information in the same feature space.

[0083] In the specific implementation, historical supplementary features are input into the dimension adjustment module. A learnable linear transformation is then performed by the linear projection layer within the module to map the historical supplementary features to a dimension specification consistent with the spatial features, resulting in aligned historical features. The linear projection layer achieves dimensional alignment between the historical read results and the spatial features of the current frame image by performing weighted recombination and channel mapping on the historical supplementary features.

[0084] Sub-step S302 involves performing residual fusion of the alignment history features and the spatial features to obtain the enhanced features.

[0085] The enhanced features can be described as follows:

[0086] In the formula, This indicates the readout feature mapping module, used to add historical supplementary features. Transformation to spatial features Consistent feature dimensions; This refers to supplementary historical features retrieved from the historical memory state for the current frame image; This represents the fusion coefficient, which can be set as a learnable parameter or a preset fixed weight. This residual fusion method can preserve the spatial detail features of the current frame image itself, while incorporating effective temporal supplementary information from previous frames.

[0087] In the specific implementation, the alignment historical features and spatial features are added element-wise. Residual connection mechanisms are used to fuse historical temporal information with current frame visual information to obtain enhanced features. Residual fusion establishes a shortcut mapping between alignment historical features and spatial features. While ensuring the stability of the main path representation of the current frame's spatial features, effective historical memory information from previous frames is superimposed, achieving complementary enhancement of temporal and spatial features. The enhanced features simultaneously carry visual details of the current frame and temporal context information from historical frames, providing a data foundation for the decoder to detect the location, category, and confidence level of foreign objects based on the fused features.

[0088] Furthermore, in one embodiment, based on the completion of fusion of historical features and spatial feature residuals and the generation of enhanced features (i.e., in the specific implementation of sub-steps S301 to S302), in order to fully utilize the spatiotemporal information carried by the enhanced features to identify liquid suspended matter, cross-attention feature decoding can also be carried out based on the decoder, specifically including the following sub-steps S303 to S304: Sub-step S303: Based on the target query vector preset by the decoder, the decoder performs cross-attention decoding operation on the enhanced features to obtain the target hidden layer representation.

[0089] The target query vector (Q) refers to the learnable feature vector preset by the decoder, which serves as the query input for the cross-attention operation. It corresponds to the potential foreign object target to be detected in the image and is used to retrieve and aggregate target-related spatiotemporal information from the enhanced features.

[0090] Target hidden layer representation ( (This refers to the intermediate features output by cross-attention decoding, which aggregate spatiotemporal information from the enhanced features, carry abstract features related to the location and category of the foreign object, and supply the detection head to complete the foreign object detection prediction.)

[0091] In the specific implementation, the target query vector preset by the decoder is used as the query input, and the enhanced features are used as the key-value input. Cross-attention decoding operation is performed within the decoder. Attention weights are calculated and features are aggregated through the target query vector and enhanced features, and the spatial and temporal information contained in the enhanced features is gathered into each target query to obtain the target hidden layer representation. This target query vector provides prior information about candidate targets, and the cross-attention mechanism constructs a global dependency relationship between the target query and the enhanced features, so that the target hidden layer representation carries the abstract information of the location and category of the foreign object to be detected, providing a decoding output basis for the detection head to predict the location, category and confidence of the foreign object.

[0092] Sub-step S304: The target hidden layer representation data is processed using the classification branch and bounding box regression branch in the detection head, and the foreign object location, foreign object category and detection confidence of the current frame image are output.

[0093] In the specific implementation, the target hidden layer representation is input in parallel to the classification branch and bounding box regression branch of the detection head. The classification branch performs semantic mapping and probability estimation, outputting the foreign object category and detection confidence score corresponding to the suspended object. The bounding box regression branch performs spatial coordinate regression on the target hidden layer representation, outputting the location bounding box information (i.e., foreign object location) of the suspended object or suspected abnormal region. The classification branch and bounding box regression branch work together to transform the target hidden layer representation into a foreign object location, foreign object category (feathers, glass fibers, etc.), and detection confidence score with semantic information, outputting the liquid suspended object detection result corresponding to the current frame image.

[0094] Based on the target query vector preset by the decoder, the enhanced features are subjected to cross-attention decoding operation to obtain the target hidden layer representation. The classification branch and bounding box regression branch in the detection head are used to process the target hidden layer representation, thereby accurately outputting the location, category and detection confidence of foreign objects in the current frame image, so as to realize the detection and recognition of foreign objects.

[0095] In this embodiment, aligned historical features are obtained by adjusting the dimensions of historical supplementary features, and then the aligned historical features are fused with spatial features in a residual manner to generate enhanced features, providing spatiotemporal fusion feature support for subsequent decoding operations and suspended object detection.

[0096] Based on the foregoing embodiments, a fifth embodiment of the multi-frame liquid suspension detection method of the present invention is proposed. In this embodiment, in order to achieve adaptive iterative update of temporal memory information by relying on key features, value features and gating features, step S40 may include the following sub-steps S401~S402: Sub-step S401: Map and transform the spatial features of the current frame image to obtain key features and value features for writing the historical memory state.

[0097] Among them, the spatial features of the current frame image Generate key features, value features, and gated features respectively:

[0098]

[0099]

[0100] In the formula, , , Represents the feature mapping module; Represents the Sigmoid function; This represents the index key feature used when the current frame image is written to memory; This represents the content value feature written to memory for the current frame image; Gating features represent the degree to which historical memory states are retained, and can adaptively control the retention ratio of historical foreign object feature information.

[0101]

[0102] The content to be written to memory for the current frame image can be represented as:

[0103] This operation compresses the key-value association information in the current frame's spatial dimension into a state increment of fixed dimension; where This is a predefined mapping function. Therefore, the scale of historical memory states is mainly determined by... and It is determined, and does not increase linearly with the number of frames T in the image sequence.

[0104] In the specific implementation, the spatial features of the current frame image are input into the projection module. A learnable linear transformation is performed through the linear projection layer within the module to generate key features and value features. The linear projection layer performs weighted recombination and channel dimension mapping on the spatial features, and outputs key features and value features adapted to the historical memory state, which are then written into the historical memory state and used to update the content memory state.

[0105] Sub-step S402 involves mapping and transforming the spatial features of the current frame image and performing a Sigmoid activation operation to obtain gating features used to regulate the retention ratio of historical memories.

[0106] The retention ratio is the retention weight of the gating feature for each dimension of the original historical memory information. The gating feature, after being activated by Sigmoid, takes a value between 0 and 1, which is used to adaptively control the degree of retention of historical memory information.

[0107] In the specific implementation, the spatial features of the current frame image are input into the projection module, where a learnable linear transformation is performed by the linear projection layer within the module, and a gated feature is generated by combining it with a Sigmoid activation operation. The linear projection layer performs weighted recombination and channel dimension mapping on the spatial features, and then the Sigmoid activation function constrains the mapping result to the 0~1 interval, resulting in a gated feature used to regulate the proportion of historical memory retention, thereby controlling the degree of information retention of historical memory states.

[0108] Furthermore, in the detection of suspended solids in liquids, value characteristics Both directly from the spatial features of the current frame image The generation can also incorporate auxiliary temporal evidence obtained from consecutive frames. They jointly participate in the construction of value features and gating features:

[0109]

[0110] In the formula, Spatial features of the current frame image; To assist in the development of time series features; Indicates the feature and Perform feature splicing; , These are trainable feature transformation networks; This is the gating activation function. It can be obtained through inter-frame difference, multi-frame overlay, or local motion response maps. For example... Figure 4 The multi-frame overlay results show that suspended objects can exhibit clearer motion trajectories and local responses in consecutive frames compared to single frames. This type of temporal response can be used as auxiliary temporal evidence to improve the feature quality of memory writing.

[0111] Furthermore, in one embodiment, based on the extraction of key features, value features, and gating features (i.e., in the specific implementation of sub-steps S401 to S402), in order to maintain the temporal memory system based on the above features iteratively, synchronously generate a normalized memory state that can be used for similarity retrieval, and achieve continuous suppression of interference information, step S40 may also include the following sub-steps S403 to S405: Sub-step S403: The historical memory state is filtered using the gating feature to obtain filtered historical memory information; Sub-step S404: Based on the key features and the value features, construct the memory increment corresponding to the current frame image; Sub-step S405: Combine the historical memory information with the memory increment update content memory state, and update the normalized memory state based on the key features, thereby obtaining the updated historical memory state.

[0112] The content memory state update method is as follows:

[0113] in, This indicates the memory status of the updated content; This indicates the memory status of the content from the previous frame; Gating features that indicate the degree to which the control history memory state of the current frame image is preserved; This represents element-wise multiplication; This indicates the historical memory information retained after gating and filtering. Key features representing the current frame image; This represents the value features of the current frame image; Indicates the preset mapping function; This indicates the memory increment to be written to the current frame image.

[0114] This update method indicates that the updated content memory state consists of two parts: one part is the historical memory information filtered and retained by the gating mechanism, and the other part is the memory increment introduced by the current frame image. Gating features It can adaptively adjust the retention ratio of historical memory information based on the visual content of the current frame image, and suppress the long-term accumulation of bottle wall reflection, background texture, imaging noise and false detection response in the historical memory state.

[0115] During normalized memory state update, the key features of the current frame are non-negatively mapped, and the mapped key features are then subjected to matrix operations with a one-dimensional vector to obtain the normalized memory increment corresponding to the current frame. This normalized memory increment is then added to the normalized memory state of the previous frame to obtain the updated normalized memory state. The normalized memory state is used to record the cumulative normalized weights of historical key features and serves as a normalization term during historical memory retrieval, working in conjunction with the content memory state to complete the retrieval of supplementary historical features.

[0116] The normalized memory state update method is as follows:

[0117] In the formula, This indicates the updated normalized memory state. This represents the normalized memory state of the previous frame. Represents the key features of the current frame image, where 1 indicates a vector of all ones corresponding to the number of spatial locations in the current frame; This represents the normalized memory increment corresponding to the current frame image; This indicates the preset mapping function.

[0118] In the specific implementation, the gating features are multiplied element-wise with the content memory state of the previous frame to obtain the filtered historical memory information; the key features and value features are multiplied together to construct the memory increment corresponding to the current frame image; then, the filtered historical memory information and the memory increment are added element-wise to update the new content memory state; the key features of the current frame are mapped to obtain the normalized memory increment, and the normalized memory state is updated in combination with the normalized memory increment; the updated historical memory state is composed of the updated content memory state and the updated normalized memory state. Specifically, the gating filtering process uses gating features to weight the retention of content memory states, thereby controlling the proportion of historical information retained; the outer product operation generates the memory increment by multiplying key features and value features; the overlay process iteratively updates the content memory state by adding the filtered historical memory information and the memory increment; and the normalized memory increment obtained based on the key feature mapping is used to iteratively update the normalized memory state. By using a gating mechanism to filter historical memory information and integrating the current frame memory increment generated based on key features and value features to complete the update of historical memory state, it can achieve adaptive forgetting of temporal information and continuous writing of new features, effectively suppressing the long-term accumulation of noise and false detection information in historical states.

[0119] In this embodiment, key features, value features, and gating features are obtained by mapping and transforming the spatial features of the current frame, respectively, providing feature support for incremental writing of temporal memory and adaptive filtering of historical information, and realizing dynamic adjustment of the retention degree of historical memory information based on image content.

[0120] Based on the same inventive concept, the sixth embodiment of this invention also provides a multi-frame liquid suspended matter detection system corresponding to the multi-frame liquid suspended matter detection method of the foregoing embodiments. Since the principle of the system in the fifth embodiment of this invention is similar to the multi-frame liquid suspended matter detection method of the foregoing embodiments, the implementation of the system can refer to the implementation of the method; repeated details will not be elaborated further. Please refer to... Figure 5 The present invention provides a multi-frame liquid suspension detection system, the system comprising: Extraction module 10 is used to initialize the historical memory state, obtain the current frame image frame by frame from the continuously acquired image sequence, input the current frame image into the target detection network, and extract the spatial features of the current frame image; the historical memory state includes content memory state and normalized memory state; The reading module 20 is used to generate query features based on the spatial features of the current frame image, and to perform mapping operations using the query features to obtain reading weights, and to read historical supplementary features that match the current frame image from the historical memory state based on the reading weights. Output module 30 is used to fuse the historical supplementary features with the spatial features to obtain enhanced features, and input the enhanced features into the decoder and the detection head to output the suspended object detection result of the current frame image; wherein, the suspended object detection result includes the object location, object type and detection confidence level; The update module 40 is used to generate key features, value features and gating features based on the spatial features; use the gating features to filter the historical memory state to obtain filtered historical memory information; construct a memory increment from the key features and the value features; and update the historical memory state by combining the historical memory information and the memory increment for the detection of suspended objects in the next frame image.

[0121] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described multi-frame liquid suspension detection method.

[0122] Figure 6 This is a schematic block diagram of the electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the electronic device are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 6 The general will label all buses as bus systems.

[0123] The user interface 405 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0124] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0125] In this embodiment of the invention, the memory 402 is used to store various types of data to support the operation of the electronic device 400. Examples of this data include: any executable program for operation on the electronic device 400, such as the operating system 4021 and application program 4022; the operating system 4021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 4022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The implementation of the multi-frame liquid suspended matter detection method provided in this embodiment of the invention can be included in the application program 4022.

[0126] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 401 may be a microprocessor or any conventional processor, etc. The steps of the multi-frame liquid suspended matter detection method provided in the embodiments of the present invention can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0127] In an exemplary embodiment, the electronic device 400 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0128] In summary, this invention initializes a historical memory state containing content memory state and normalized memory state, extracting spatial features frame by frame; generates query features based on spatial features, obtains read weights through mapping operations, and reads matching historical supplementary features according to the read weights; fuses spatial features and historical supplementary features to form enhanced features, outputting the location, category, and confidence level of suspended foreign objects; simultaneously generates key, value, and gating features based on spatial features, filters effective historical information using gating features, and iteratively updates the historical memory state using memory increments constructed with key-value features, continuously providing temporal context for subsequent frame detection. This solution is applied to light inspection scenarios for transparent liquid containers such as ampoules and vials, strengthening weak suspended object features with multi-frame temporal information, filtering background noise through read weights and gating mechanisms; and employs compressed historical memory states to reduce computational and storage requirements, effectively improving the detection accuracy of suspended foreign objects in liquids.

[0129] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A multi-frame liquid suspension detection method, characterized in that, The method includes: The historical memory state is initialized, and the current frame image is obtained frame by frame from the continuously acquired image sequence. The current frame image is input into the target detection network to extract the spatial features of the current frame image. The historical memory state includes the content memory state and the normalized memory state. Query features are generated based on the spatial features of the current frame image, and the query features are used to perform mapping operations to obtain reading weights. Based on the reading weights, historical supplementary features that match the current frame image are read from the historical memory state. The historical supplementary features are fused with the spatial features to obtain enhanced features, which are then input into the decoder and the detection head to output the suspended foreign object detection results of the current frame image; wherein, the suspended foreign object detection results include the location of the foreign object, the type of the foreign object, and the detection confidence level; Based on the spatial features, key features, value features, and gating features are generated respectively; the gating features are used to filter the historical memory state to obtain filtered historical memory information; the key features and value features are used to construct a memory increment; and the historical memory state is updated by combining the historical memory information and the memory increment for the detection of suspended objects in the next frame image.

2. The method according to claim 1, characterized in that, The step of inputting the current frame image into the target detection network and extracting the spatial features of the current frame image includes: The ViT backbone network in the target detection network is used to divide the current frame image into image blocks, and image block tokens are generated through embedding mapping. Then, the spatial position dependency relationship within the current frame image is established through a self-attention mechanism to obtain visual features. The visual features are dimensionally adjusted using the projection module in the target detection network to obtain spatial features suitable for the decoder.

3. The method according to claim 1, characterized in that, The step of using the query features to perform mapping operations to obtain the reading weight, and reading historical supplementary features matching the current frame image from the historical memory state based on the reading weight, includes: Perform a mapping operation on the query features to obtain the read weights; The reading weights are applied to the content memory state and the normalized memory state respectively to calculate the historical supplementary features that match the current frame image.

4. The method according to claim 1, characterized in that, The process of fusing the historical supplementary features with the spatial features to obtain enhanced features includes: The dimensions of the historical supplementary features are adjusted to obtain aligned historical features that are consistent with the dimensions of the spatial features. The enhanced features are obtained by residual fusion of the alignment history features and the spatial features.

5. The method according to claim 1, characterized in that, The step of inputting the enhanced features into the decoder and the detection head, and outputting the suspended foreign object detection result of the current frame image, includes: Based on the target query vector preset by the decoder, the enhanced features are subjected to cross-attention decoding operation by the decoder to obtain the target hidden layer representation. The target hidden layer representation data is processed using the classification branch and bounding box regression branch in the detection head to output the foreign object location, foreign object category, and detection confidence of the current frame image.

6. The method according to claim 1, characterized in that, The generation of key features, value features, and gated features based on the spatial features includes: The spatial features of the current frame image are mapped and transformed to obtain key features and value features for writing into the historical memory state; The spatial features of the current frame image are mapped and transformed, and a Sigmoid activation operation is performed to obtain gating features used to regulate the retention ratio of historical memory.

7. The method according to claim 1, characterized in that, The process of filtering historical memory states using the gating features to obtain filtered historical memory information, constructing a memory increment using the key features and the value features, and updating the historical memory state by combining the historical memory information and the memory increment includes: The historical memory state is filtered using the gating feature to obtain filtered historical memory information; Based on the key features and the value features, construct the memory increment corresponding to the current frame image; By combining the historical memory information with the memory increment update content memory state, and updating the normalized memory state based on the key features, the updated historical memory state is obtained.

8. A multi-frame liquid suspended matter detection system, characterized in that, The system includes: The extraction module is used to initialize the historical memory state, obtain the current frame image frame by frame from the continuously acquired image sequence, input the current frame image into the target detection network, and extract the spatial features of the current frame image; the historical memory state includes content memory state and normalized memory state; The reading module is used to generate query features based on the spatial features of the current frame image, perform mapping operations using the query features to obtain reading weights, and read historical supplementary features that match the current frame image from the historical memory state based on the reading weights. The output module is used to fuse the historical supplementary features with the spatial features to obtain enhanced features, and input the enhanced features into the decoder and the detection head to output the suspended object detection results of the current frame image; wherein, the suspended object detection results include the location of the object, the type of object, and the detection confidence level; The update module is used to generate key features, value features, and gating features based on the spatial features; to filter the historical memory state using the gating features to obtain filtered historical memory information; to construct a memory increment using the key features and the value features; and to update the historical memory state by combining the historical memory information and the memory increment for the detection of suspended objects in the next frame image.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the processor to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images