Prompt interaction-based security check contraband identification method and device, storage medium and computer device

Abstract

The security check forbidden article identification method and device based on prompt interaction provided in the application, the storage medium and the computer equipment, through responding to the selection operation of the user to the target region in the security check image, the first positioning information and the segmentation mask of the target region are obtained as the interactive prompt, the model is guided to focus on identifying the target region specified by the user, the security check officer can directly perform the interactive operation such as clicking or frame selecting on the security check image in the review link, without manually searching for the positioning in the global detection result, the human-computer interaction efficiency and the operation convenience in the review stage are greatly improved; on this basis, the application extracts the target object feature and the boundary feature, and fuses the background information feature of the target region, so that the model can comprehensively utilize the self information and the morphological boundary information of the target object for comprehensive judgment, and the identification precision is significantly improved; finally, the decoder outputs the separation image and the fine-grained classification result of the target object, and the actual security check demand is met.

Description

Technical Field

[0001] This application relates to the field of security inspection technology, and in particular to a method, device, storage medium and computer equipment for identifying prohibited items in security inspections based on prompting interaction. Background Technology

[0002] With the continuous development of security inspection technology, in typical security inspection scenarios, security personnel use security scanners to detect prohibited items in bags. Currently, mainstream security inspection solutions generally employ deep learning-based full-image object detection methods, which perform end-to-end detection on the entire security image in a single operation, outputting the category labels and bounding box coordinates of each prohibited item in the image. However, in actual security inspection processes, the review process by security personnel is equally crucial. When a security personnel have doubts about a certain area or item in a package, a secondary, detailed examination is required. In this review scenario, existing technologies have the following significant shortcomings:

[0003] 1. Lack of focused recognition capability for specific targets. Existing methods can only perform global detection on the entire security inspection image and cannot perform secondary focused recognition on specific areas based on the security personnel's interactive intentions (such as clicking or selecting).

[0004] 2. The classification of prohibited items is coarse-grained. Existing technologies can usually only output labels for broad categories of prohibited items, which is difficult to meet the needs of fine-grained classification in the review process.

[0005] 3. Insufficient ability to separate targets in occluded scenarios. Stacking and mutual occlusion of items inside packages are common phenomena. Existing methods are difficult to effectively separate the target from the occluded background, resulting in decreased detection accuracy or even missed detection.

[0006] In summary, existing technologies lack effective interactive focusing recognition methods during the security check and verification stage, making it difficult to meet the actual needs for fine-grained identification and occlusion separation of specific targets. Summary of the Invention

[0007] The purpose of this application is to at least address one of the aforementioned technical deficiencies, particularly the lack of effective interactive focusing recognition methods in the existing technology during the security check and verification stage, which makes it difficult to meet the actual needs of fine-grained recognition and occlusion separation of specific targets.

[0008] This application provides a method for identifying prohibited items during security checks based on interactive prompts, the method comprising:

[0009] In response to a user's selection of a target region in a security inspection image, the first positioning information and the first segmentation mask of the target region are obtained;

[0010] Based on the first positioning information and the first segmentation mask, extract the target object features and boundary features of the target object in the target region;

[0011] The target object features, the boundary features, and the background information features of the target region are fused to obtain the first fused feature;

[0012] Based on the occlusion status of the target object, at least the first fusion feature is input into the decoder to obtain the separated image of the target object and the fine-grained classification result output by the decoder.

[0013] Optionally, the step of obtaining first positioning information and a first segmentation mask of the target region in response to a user's selection operation of a target region in a security inspection image includes:

[0014] In response to a user's click operation on a target object in a security inspection image, the detection coordinates of the target object are obtained as the first positioning information, and the area where the target object is located is segmented from the security inspection image based on the detection coordinates to obtain the first segmentation mask of the target object;

[0015] Alternatively, in response to a user's filtering instruction for similar items, the detection coordinates of all similar items in the security inspection image are obtained as the first positioning information, and the area where each similar item is located is segmented from the security inspection image according to each detection coordinate, to obtain the first segmentation mask for each similar item.

[0016] Alternatively, in response to a user's selection of any region in the security inspection image, the coordinates of the arbitrary region are obtained as the first positioning information, and the arbitrary region is segmented from the security inspection image based on the coordinates of the arbitrary region to obtain the first segmentation mask of the arbitrary region.

[0017] Optionally, the step of extracting target object features and boundary features of the target object in the target region based on the first positioning information and the first segmentation mask includes:

[0018] The first encoder extracts features from the region corresponding to the first segmentation mask to obtain the target object features in the target region.

[0019] The boundary features of the target object in the target area are obtained by extracting features from the boundary region corresponding to the first positioning information using the second encoder.

[0020] Optionally, fusing the target object features, the boundary features, and the background information features of the target region to obtain the first fused feature includes:

[0021] Obtain the background information features of the target area;

[0022] The target object features, the boundary features, and the background information features are input into a feature fusion layer and fused to obtain the first fused feature.

[0023] Optionally, the step of inputting at least the first fused feature into the decoder based on the occlusion of the target object includes:

[0024] When the target object has no obstructions, only the first fused feature is input into the decoder;

[0025] When the target object is obstructed, the obstruction feature of the obstruction and the background layout feature of the target area are obtained. The obstruction feature, the background layout feature and the first fusion feature are fused to obtain the second fusion feature, and the second fusion feature is input into the decoder.

[0026] Optionally, obtaining the occlusion features of the occluder and the background layout features of the target area includes:

[0027] Obtain the second positioning information and the second segmentation mask of the obstruction;

[0028] The second segmentation mask of the occluder and the background information features of the target area are input into the third encoder, and the occlusion features of the occluder are extracted by the third encoder;

[0029] The second positioning information of the obstruction, the shape information corresponding to the second segmentation mask, and the overall image shape information of the security inspection image are stitched together and input into the layout feature extraction layer. The background layout features of the target area are extracted through the layout feature extraction layer.

[0030] Optionally, obtaining the separated image and fine-grained classification result of the target object output by the decoder includes:

[0031] The first fusion feature input to the decoder is input in parallel into the target object separation branch and the target object classification branch;

[0032] A clear image of the target object is identified and separated from the first fusion feature through the target object separation branch, resulting in a separated image;

[0033] The separated target objects are further classified into fine-grained categories by the target object classification branch to obtain the fine-grained classification results of the target objects.

[0034] This application also provides a security inspection prohibited item identification device based on prompting interaction, including:

[0035] The data acquisition module is used to acquire the first positioning information and the first segmentation mask of the target area in response to the user's selection operation of the target area in the security inspection image;

[0036] The feature extraction module is used to extract the target object features and boundary features of the target object in the target region based on the first positioning information and the first segmentation mask;

[0037] The feature fusion module is used to fuse the target object features, the boundary features, and the background information features of the target region to obtain a first fused feature;

[0038] An image recognition module is used to input at least the first fused feature into a decoder based on the occlusion of the target object, so as to obtain a separated image of the target object and a fine-grained classification result output by the decoder.

[0039] This application also provides a computer-readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the security inspection contraband identification method based on prompting interaction as described in any of the above embodiments.

[0040] This application also provides a computer device, including: one or more processors, and memory;

[0041] The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the security inspection contraband identification method based on prompting interaction as described in any of the above embodiments.

[0042] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0043] The method, apparatus, storage medium, and computer equipment for identifying prohibited items in security checks based on interactive prompts provided in this application, respond to the user's selection of a target region in a security check image, obtain the first location information and segmentation mask of the target region as interactive prompts, and guide the model to focus on and identify the target region specified by the user. During the review process, security personnel can directly interact with the security check image by clicking or selecting boxes, without having to manually search for the location in the global detection results, which greatly improves the efficiency and convenience of human-computer interaction in the review stage. On this basis, this application extracts the features and boundary features of the target object and fuses them with the background information features of the target region, enabling the model to comprehensively utilize the target object's own information and morphological boundary information for comprehensive judgment. When the target object is occluded, the background features of the occluding object can be further introduced into the fusion, effectively enhancing the model's ability to separate the target object from the occluded background and significantly improving the recognition accuracy in occluded scenarios. Finally, the decoder outputs the separated image of the target object and fine-grained classification results. The finer classification granularity can provide security personnel with more sufficient and accurate decision-making basis for the final judgment. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A flowchart illustrating a security inspection method for identifying prohibited items based on interactive prompts, provided as an embodiment of this application;

[0046] Figure 2 A schematic diagram of the prohibited item identification process of a security inspection machine with a promptable interactive paradigm provided in this application embodiment;

[0047] Figure 3 This is a schematic diagram illustrating the process of identifying contraband when there are obstructions, as provided in an embodiment of this application.

[0048] Figure 4 A schematic diagram of a security inspection prohibited item identification device based on prompting interaction is provided for an embodiment of this application;

[0049] Figure 5 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] In one embodiment, such as Figure 1 As shown, Figure 1 This application provides a flowchart illustrating a method for identifying prohibited items in security checks based on interactive prompts. The method may include:

[0052] S110: In response to the user's selection operation of the target area in the security inspection image, obtain the first positioning information and the first segmentation mask of the target area.

[0053] In this step, when using a security scanner to detect prohibited items in bags, if the security personnel have doubts about a certain area or item in the security image and need to conduct a second review, they can directly select the target area by clicking, selecting a box, or filtering similar items. The system will directly obtain the first location information of the corresponding target area based on the user's operation, and segment the first segmentation mask of the target area. This information is used as an interactive prompt to input into the subsequent recognition model, enabling the model to focus on the user-specified area for recognition.

[0054] Specifically, this application supports multiple interactive selection methods to adapt to different review scenarios: when a security inspector has doubts about only a single suspicious target, they can directly click on the target, and the system uses the target's detection coordinates as the first positioning information to segment the target's area from the security image to obtain the first segmentation mask; when a security inspector needs to review multiple similar suspicious items in batches, they can send a filtering command for similar items, and the system obtains the detection coordinates of all similar items as the first positioning information, and segments each similar item to obtain the first segmentation mask; when a security inspector has doubts about any irregular suspicious area in the security image, they can directly select the area, and the system uses the area's coordinates as the first positioning information to segment the area to obtain the first segmentation mask.

[0055] In this application, the target area refers to the area that security personnel deem suspicious and requires secondary verification. It can be a single, complete target item or any suspicious area within the security image, adaptable flexibly to the security personnel's actual interaction intent. The first positioning information refers to the coordinate position of the target area within the entire security image, used to determine the suspicious area indicated by the user's interaction intent, providing a positioning basis for subsequent focused identification. The first segmentation mask is a target area mask image obtained through binarized segmentation, used to distinguish the target area from other non-target areas in the security image, facilitating accurate extraction of feature information within the target area by the model.

[0056] S120: Based on the first positioning information and the first segmentation mask, extract the target object features and boundary features of the target object in the target region.

[0057] In this step, after obtaining the first positioning information and the first segmentation mask of the target area through S110, this application can extract different types of features for the target area itself and the boundary of the target area, such as the target object features and boundary features.

[0058] Among them, the target object features refer to the basic semantic features such as texture, grayscale, and shape contained within the target area, which are used to reflect the inherent attributes of the target object itself and help the model determine the category of the target object; the target object boundary features refer to the contour, grayscale gradient, and other features at the boundary between the edge of the target area and the surrounding background / other objects, which are used to reflect the boundary information between the target object and the surrounding objects, help the model accurately distinguish the target object from adjacent occluders, and improve the accuracy of subsequent separation and recognition.

[0059] In practical implementation, this application can extract two types of features separately using two independent encoders. For example, by extracting features from the region corresponding to the first segmentation mask using the first encoder, the target object features in the target region can be obtained; by extracting features from the boundary region corresponding to the first positioning information using the second encoder, the boundary features of the target object in the target region can be obtained. This split extraction method can avoid mutual interference between the two types of features and improve the effectiveness and accuracy of feature extraction.

[0060] Of course, this application can also extract target features and boundary features separately using an encoder. It only needs to divide different output branches at the encoder output end, which can also achieve effective extraction of the two types of features. This application does not impose too many restrictions on this.

[0061] It should be noted that the target objects in the target area of ​​this application refer to the items to be identified contained in the suspicious area selected by the user interaction operation. These may be a single complete contraband or suspected contraband, or a suspicious item that is partially exposed after being obscured by other items, or a collection of a certain type of items. These are the core objects of this focused identification and separation classification.

[0062] S130: The target object features, boundary features, and background information features of the target area are fused to obtain the first fused feature.

[0063] In this step, after extracting the target object features and boundary features of the target object in the target area through S120, this application further combines the background information features of the target area for fusion, so that the model can not only focus on the target area specified by the user, but also take into account the global context information of the entire security inspection image, avoiding judgment bias caused by the lack of local features.

[0064] In practical implementation, this application can first obtain the global background features of the entire security inspection image, and then crop the background information features corresponding to the target area from the global background features based on the first positioning information, ensuring that the background information features can accurately correspond to the position of the target area. Subsequently, the extracted target object features, boundary features, and background information features are sent together to a pre-set feature fusion layer. The feature fusion layer performs weighted splicing or adaptive fusion on the multi-source features, and after integrating the effective information of the three types of features, the first fused feature is output. The first fused feature can simultaneously carry the target object's own attributes, boundary differentiation information, and contextual background information, providing a more comprehensive feature foundation for subsequent recognition and separation.

[0065] In this application, when obtaining the global background features of the entire security inspection image, the features of the entire security inspection image can be extracted in advance through a backbone convolutional neural network, and the output multi-scale feature map can be used as the global background features. This application utilizes the powerful general feature extraction capability of the backbone network to fully explore the basic semantic information of each region in the security inspection image. Without the need to design a complex extraction structure, high-quality global background features can be obtained quickly, which is suitable for the needs of rapid identification and verification in actual security inspection scenarios.

[0066] It is understood that the global background features in this application refer to the feature set obtained by multi-scale feature extraction of the entire security inspection image by the backbone convolutional neural network, which contains semantic information of different levels of the entire security inspection image. It can cover everything from low-level edge and texture information to high-level semantic and structural information, providing complete global context support for feature fusion of the target area and avoiding the problem of information loss caused by using only local features.

[0067] In this application, the background information feature of the target area refers to the background feature corresponding to the location of the target area, which is obtained by cropping the global background feature based on the first positioning information. It includes not only the background information inside the target area except for the target object, but also the background context information of the adjacent areas around the target object. This can help the model better distinguish between the target object and the background and determine the actual shape of the target object.

[0068] S140: Based on the occlusion of the target object, at least the first fusion feature is input into the decoder to obtain the target object separation image and fine-grained classification result output by the decoder.

[0069] In this step, after obtaining the first fusion feature containing multi-dimensional information of the target area through S130, this application can further adopt different feature input strategies based on whether the target object is currently occluded by other objects, adapting to the recognition needs of different scenarios: when the target object is not occluded and its outline and features are fully exposed, only the first fusion feature needs to be input into the decoder; when the target object is occluded and some features are covered by the occluder, the background features of the occluder are first obtained, and the background features are fused with the first fusion feature to obtain the second fusion feature, and then the second fusion feature is input into the decoder, so that the model can fully grasp the position and shape information of the occluder and more accurately separate the target object from the occluded background.

[0070] Specifically, after obtaining the fusion features of the input decoder, this application can complete the target separation and fine-grained classification tasks through two parallel branches of the decoder: the target separation branch identifies and separates a clear image of the target from the fusion features, removes interference from obstructions and irrelevant backgrounds, and obtains a separated image that can be directly used by security inspectors for judgment; the target classification branch performs fine-grained category division on the separated target and directly outputs the specific fine-grained classification results of the target, without requiring security inspectors to further manually determine the category, providing a direct reference for security inspection judgment.

[0071] Understandably, during the model training phase, this application pre-assigns corresponding labels to the training data. These labels act as "navigation markers," clarifying the learning direction for the model. Simultaneously, combined with key elements such as the designed training loss function, the model can continuously adjust its parameters based on the label information, optimizing separation and classification performance, and ultimately achieving effective separation and accurate classification of the target object.

[0072] For example, during the training phase, the decoder of this application can simultaneously set the separation task loss and the classification task loss. The two task losses are weighted and summed to obtain the total training loss. The total training loss is backpropagated to update the parameters of the entire model network, so that the model can simultaneously optimize the target object separation ability and fine-grained classification ability, and finally obtain a deployed and usable trained model.

[0073] In the above embodiments, in response to the user's selection of a target area in the security inspection image, the first positioning information and segmentation mask of the target area are obtained as interactive prompts to guide the model to focus on and identify the target area specified by the user. During the review process, security personnel can directly interact with the security inspection image by clicking or selecting boxes, without having to manually search for the location in the global detection results, which greatly improves the efficiency and convenience of human-computer interaction in the review stage. On this basis, this application extracts the features and boundary features of the target object and fuses them with the background information features of the target area, enabling the model to comprehensively utilize the target object's own information and morphological boundary information for comprehensive judgment. When the target object is occluded, the background features of the occluding object can be further introduced to participate in the fusion, which effectively enhances the model's ability to separate the target object from the occluded background and significantly improves the recognition accuracy in occluded scenarios. Finally, the decoder outputs the separated image of the target object and fine-grained classification results. The classification granularity is finer, which can provide more sufficient and accurate decision-making basis for the security personnel's final judgment.

[0074] In one embodiment, S110, in response to a user's selection operation of a target region in a security inspection image, obtaining first positioning information and a first segmentation mask of the target region, may include:

[0075] In response to a user's click operation on a target object in a security inspection image, the detection coordinates of the target object are obtained as the first positioning information, and the area where the target object is located is segmented from the security inspection image based on the detection coordinates to obtain the first segmentation mask of the target object.

[0076] Alternatively, in response to a user's filtering instruction for similar items, the detection coordinates of all similar items in the security inspection image are obtained as the first positioning information, and the areas where each similar item is located are segmented from the security inspection image according to each detection coordinate, to obtain the first segmentation mask for each similar item.

[0077] Alternatively, in response to a user's selection of any region in the security inspection image, the coordinates of the arbitrary region are obtained as the first positioning information, and the arbitrary region is segmented from the security inspection image based on the coordinates of the arbitrary region to obtain the first segmentation mask of the arbitrary region.

[0078] In this embodiment, during security checks, the system can support multiple interactive selection methods to adapt to the verification needs in different scenarios. Whether the security inspector is only confirming a single suspicious target, needs to check a batch of similar suspicious items, or has doubts about an irregular suspicious area, they can quickly locate the target area through the corresponding interactive operation. There is no need to adjust the operating habits or generate unnecessary operation steps. It can adapt to the operating habits of different security inspectors in different security check scenarios, and further improve the overall efficiency of the verification process.

[0079] Indicatively, such as Figure 2 As shown, Figure 2 A schematic diagram of the prohibited item identification process of a security inspection machine with a promptable interactive paradigm provided in this application embodiment; Figure 2 In this system, when a security officer discovers a single, blurry, suspicious object in the baggage inspection image, they can simply click on the suspicious location. The system quickly obtains the corresponding detection coordinates as the first positioning information and automatically segments the object into a first segmentation mask. This object is then used as the input prompt for focusing and recognition, resulting in a separated image and fine-grained classification of the object. This eliminates the need for manual adjustment of the selection area, making the operation convenient and efficient. If multiple suspicious items resembling lighters are detected in the baggage inspection image, the officer can directly send a filtering command for similar items. The system can then obtain the detection coordinates of all similar items at once and generate a first segmentation mask for each item. This allows officers to perform batch verification without manually selecting items one by one, significantly reducing the time required for batch verification. If a rectangular area in the security image shows abnormal grayscale or is suspicious, the officer can manually select the rectangular area. The system will directly use the selected coordinates as the first positioning information to segment the corresponding area into a first segmentation mask, adapting to the verification needs of suspicious areas of any shape and ensuring no potential contraband risks are overlooked.

[0080] It is understood that the above examples are merely provided to facilitate understanding of the interaction concept of this solution. In actual applications, more interaction options can be developed based on specific security check scenarios and needs, and this application does not impose any specific limitations on this.

[0081] In one embodiment, extracting the target object features and boundary features of the target object in the target region based on the first positioning information and the first segmentation mask in S120 may include:

[0082] S121: The first encoder extracts features from the region corresponding to the first segmentation mask to obtain the target object features in the target region.

[0083] S122: The second encoder extracts features from the boundary region corresponding to the first positioning information to obtain the boundary features of the target object in the target region.

[0084] In this embodiment, when extracting the target object features and boundary features of the target object in the target area, this application can use two independent encoders to extract two types of features respectively. The first encoder focuses on learning the semantic features inside the target area, and the second encoder focuses on learning the boundary features of the boundary area. The two encoding paths are independent of each other and will not interfere with each other, which can ensure the purity of the two types of feature extraction, avoid recognition deviation caused by feature confusion, and effectively improve the accuracy of subsequent fusion recognition.

[0085] Among them, the boundary region for boundary feature extraction is a ring-shaped region obtained by extending outward from the target region's positioning coordinates by a preset pixel width. It can completely cover the boundary between the target object and surrounding objects, fully collect boundary information such as contour and grayscale gradient, and provide sufficient boundary basis for subsequent separation of occluded and overlapping target objects.

[0086] Furthermore, during the model training phase, this application allows setting corresponding loss function branches for the two encoders, enabling the first encoder to continuously optimize its ability to extract semantic features within the target region during training, while the second encoder focuses on optimizing its ability to extract boundary features. The two encoders each perform their respective tasks, updating parameters in their corresponding optimization directions, thereby further ensuring the effectiveness and independence of the two types of feature extraction.

[0087] The above embodiments use separate encoding to achieve optimal extraction of internal and boundary features of the target, which avoids recognition interference caused by feature confusion and provides a more accurate feature basis for subsequent models to separate occluded targets and determine target categories, thereby further improving the accuracy of contraband identification in complex overlapping scenarios.

[0088] In one embodiment, fusing the target object features, the boundary features, and the background information features of the target region in step S130 to obtain a first fused feature may include:

[0089] S131: Obtain the background information features of the target area.

[0090] S132: The target object features, the boundary features, and the overall image features are input into the feature fusion layer and fused to obtain the first fused feature.

[0091] In this embodiment, when fusing the target object features, boundary features and background information features of the target area, the entire security inspection image is first extracted using a backbone convolutional neural network. The output multi-scale feature map is used as the global background feature. Then, the background information features corresponding to the target area are cropped from the global background features based on the first positioning information. This ensures that the background information features can accurately correspond to the location of the target area and that they can be directly called during subsequent feature fusion without adding extra time loss in the inference stage.

[0092] Subsequently, this application inputs the extracted target features, boundary features, and pre-extracted background information features of the target region into the feature fusion layer. The spatial alignment of different features is completed through a self-attention mechanism, so that the target features, boundary features, and background information features can be accurately matched in the semantic space. Then, the multi-source features are integrated through convolution operation, and finally the first fusion feature is obtained, which simultaneously contains the target's own attributes, boundary information, and contextual background information. This ensures that the model can focus on the suspicious target specified by the user, while not deviating from the background information of the entire security inspection image, thus avoiding misjudgment due to insufficient local information.

[0093] In this application, when performing feature fusion, a multi-head self-attention mechanism can be used to model the relationship between different features, fully explore the semantic relationship between target features, boundary features, and background information features, further improve the sufficiency of feature fusion, and make the fused features contain richer decision information.

[0094] This layered extraction and fusion approach can fully utilize information from different dimensions to assist decision-making and effectively improve the accuracy of the identification results.

[0095] In one embodiment, inputting the first fused feature into the decoder at least according to the occlusion status of the target object in step S140 may include:

[0096] S141: When the target object does not have any obstructions, only the first fused feature is input into the decoder.

[0097] S142: When the target object is obstructed, the obstruction feature of the obstruction and the background layout feature of the target area are obtained, the obstruction feature and the background layout feature are fused with the first fusion feature to obtain the second fusion feature, and the second fusion feature is input into the decoder.

[0098] In this embodiment, when inputting the fused features to the decoder, this application can adapt different input strategies according to whether the target object is occluded. When the target object is fully exposed without occlusion, only the first fused feature needs to be input to meet the recognition requirements, without introducing extra computation, which can speed up the model inference speed and adapt to the needs of rapid verification scenarios. When the target object is occluded by other objects, the occlusion features of the occluder and the background layout features of the target area are additionally introduced to participate in the fusion, allowing the model to grasp the position, shape and texture information of the occluder in advance, which can more accurately distinguish the boundary between the target object and the occluder, improve the completeness and accuracy of the separation results, avoid misjudging the features of the occluder as the features of the target object itself, and effectively optimize the recognition effect in occluded scenarios.

[0099] In this application, the occlusion features refer to the semantic features of the occlusion itself, extracted from the security inspection image by the encoder based on the segmented region corresponding to the occlusion. These features fully reflect the texture, shape, and other attribute information of the occlusion. The background layout features of the target region, on the other hand, are the overall spatial distribution information of the entire target region, including the target and the occlusion. This helps the model clarify the front-back positional relationship between the target and the occlusion, further assisting the model in separation. By supplementing the occlusion-related feature information, the model can more clearly reconstruct the complete shape of the occluded target during inference, avoiding feature interference caused by the occlusion and effectively improving the recognition accuracy in occluded scenarios.

[0100] This scenario-specific adaptation strategy balances model inference efficiency with recognition accuracy in complex scenarios. It can flexibly adjust the allocation of computing resources according to the actual scenario, thereby maximizing the speed of security check verification while ensuring recognition accuracy.

[0101] In one embodiment, obtaining the occlusion features of the occlusion and the background layout features of the target area in S142 may include:

[0102] S1421: Obtain the second positioning information and the second segmentation mask of the obstruction.

[0103] S1422: Input the second segmentation mask of the occluder and the background information features of the target area into the third encoder, and extract the occluder features of the occluder through the third encoder.

[0104] S1423: The second positioning information of the obstruction, the shape information corresponding to the second segmentation mask, and the overall image shape information of the security inspection image are spliced ​​together and input into the layout feature extraction layer. The background layout features of the target area are extracted through the layout feature extraction layer.

[0105] In this embodiment, when acquiring the background features of the occluded object, this application first acquires the second positioning information and the second segmentation mask of the occluded object, and then extracts background information of different dimensions through two branches. One branch extracts the semantic features of the occluded object itself through the third encoder, so that the model can know in advance what the occluded object is and what texture features it has. The other branch acquires the spatial layout information such as the relative position and size ratio of the occluded object and the target object through the layout feature extraction layer, which helps the model to understand the spatial relationship between different objects in the target area.

[0106] Understandably, the two branches in this application supplement occlusion-related information from both semantic and spatial dimensions, enabling the model to more clearly distinguish between the target object and the occluder, and preventing the features of the occluder from being mixed into the classification results of the target object. This effectively reduces the misclassification rate in occlusion scenarios and further improves the accuracy of contraband identification in overlapping occlusion scenarios.

[0107] After extraction, this application can combine the occlusion features, background layout features and the previously obtained first fusion features to obtain the second fusion feature, which is then input into the decoder for subsequent separation and classification processing, providing the decoder with a complete and comprehensive decision-making basis.

[0108] In this application, the second location information of the occluder refers to the coordinate information of the occluder area specified synchronously when the user selects the target area interactively. The second segmentation mask is the segmentation result of the corresponding occluder area, which can also be obtained through interactive methods such as clicking and selecting boxes. There is no need for the model to automatically detect and locate. The range of the occluder can be quickly locked by the manual judgment of the security inspector. This can not only simplify the inference steps of the model, but also improve the accuracy of the occluder location by using manual prediction, and further accelerate the efficiency of subsequent feature extraction and fusion.

[0109] The third encoder in this application can also have an independent loss function branch set during the model training phase to optimize the third encoder's ability to extract semantic features of occluded objects, ensuring the accuracy of occluded object feature extraction and preventing confusion with target object features. At the same time, the layout feature extraction layer can also optimize the modeling ability of spatial information through the corresponding loss function branch, further improving the effectiveness of layout feature extraction.

[0110] The layout feature extraction layer in this application can adopt a structure combining convolution and fully connected layers. First, the spliced ​​spatial shape information is downsampled and compressed, and then mapped to a fixed-dimensional background layout feature output through the fully connected layer. This can efficiently integrate spatial location information without introducing too much computation, ensuring that the model's inference speed is not significantly affected.

[0111] In a specific implementation, such as Figure 3 As shown, Figure 3This is a schematic diagram illustrating the process of identifying contraband when there are obstructions, as provided in an embodiment of this application. Figure 3 The specific process is detailed below:

[0112] S1: When a user needs to verify a detected item, clicking on the item will retrieve its corresponding target area. After segmentation, the first segmentation mask and the first positioning information of the item are further obtained. The first encoder is used to extract the features of the target object, while the second encoder is responsible for extracting features such as the target object's boundary information.

[0113] S2: If the object occludes other objects, the method in S1 is used to obtain the second segmentation mask and second positioning information of the occluding objects. Then, the second segmentation mask and the background information features of the target area are input into the third encoder to extract the background-related information of the target object. If multiple objects occlude the target object, this step needs to be repeated; if the target object is not occluded, this step is not required. Therefore, in the above framework diagram, this item is marked as optional.

[0114] S3: The shape information of the second positioning information obtained in S2, the shape information of the corresponding second segmentation mask, and the overall image shape information of the whole image are concatenated and sent to the layout feature extraction layer. The function of the layout feature extraction layer is to extract features from the background layer information of the target region.

[0115] The shape information includes height, width, and number of channels; similarly, the overall image shape information includes the overall image height, width, and number of channels. The layout feature extraction layer consists of a convolutional neural network (CNN, containing convolutional layers), fully connected layers (FC), and activation functions. This layer's function is to transmit signals to the model indicating the presence of the target object and background information in the current region. Through feature extraction, it enhances the model's ability to separate the target object from the background; the extracted features are key to achieving this separation.

[0116] S3: Perform feature fusion on the features extracted from S1 and S2. Feature fusion operations include, but are not limited to, feature concatenation, feature addition, and fusion through CNN layers or FC layers.

[0117] S4: Input the fusion features obtained in S3 into the decoder to obtain the separated image of the target object and the fine-grained classification result of the target object.

[0118] In the above embodiments, by using branch feature extraction and scene-specific input adaptation design, the model can accurately separate and identify contraband regardless of whether the target object is occluded by other objects. This not only avoids adding extra computational burden in unoccluded scenes, but also effectively solves the problems of high difficulty in target differentiation and low recognition accuracy in overlapping occlusion scenes, greatly improving the recognition accuracy and processing efficiency of the security inspection contraband verification process.

[0119] In one embodiment, obtaining the separated image and fine-grained classification result of the target object output by the decoder in S140 may include:

[0120] S401: Input the first fusion feature into the decoder in parallel into the target separation branch and the target classification branch.

[0121] S402: Identify and separate a clear image of the target object from the first fusion feature through the target object separation branch to obtain a separated image.

[0122] S403: The separated target objects are classified into fine-grained categories through the target object classification branch to obtain the fine-grained classification result of the target objects.

[0123] In this embodiment, after the first fused feature is input into the decoder, the decoder sets up two parallel output branches, which are responsible for the two tasks of target object separation and fine-grained classification, respectively. The two branches share the feature information extracted in the previous encoding and fusion stages, and can learn exclusive decision parameters for their respective tasks. The target object separation branch focuses on learning to restore the complete shape of the target object from complex backgrounds and occlusions, and outputs a clearer separated image; the target object classification branch focuses on learning to complete fine-grained classification based on the fused features and the separated target image, and outputs a more accurate category result.

[0124] The parallel branch design allows two tasks to infer and output simultaneously without being executed sequentially, reducing the overall inference time and enabling faster output of recognition results. This meets the needs of rapid clearance in security check scenarios. At the same time, sharing features reduces resource consumption caused by redundant feature calculations. While ensuring the accuracy of both tasks, it further improves the model's inference efficiency and meets the real-time verification needs of security check sites.

[0125] Furthermore, when occlusions are present, the input to the decoder is a second fused feature that incorporates background features of the occlusion. The two parallel branches also simultaneously perform target separation and fine-grained classification based on this second fused feature. The additional occlusion background features help the separation branch more accurately cut the boundary between the target and the occlusion, and help the classification branch eliminate interference from occlusion features, further improving the output accuracy of both tasks. This parallel output design can efficiently output complete recognition results regardless of whether the target is occluded, balancing inference speed and task accuracy, and better adapting to the actual application needs of security check verification.

[0126] The following describes the security inspection prohibited item identification device based on prompt interaction provided in the embodiments of this application. The security inspection prohibited item identification device based on prompt interaction described below can be referred to in correspondence with the security inspection prohibited item identification method based on prompt interaction described above.

[0127] In one embodiment, such as Figure 4 As shown, Figure 4 This application provides a schematic diagram of a security inspection prohibited item identification device based on prompting interaction, which may include a data acquisition module 210, a feature extraction module 220, a feature fusion module 230, and an image recognition module 240, as detailed below:

[0128] The data acquisition module 210 is used to acquire the first positioning information and the first segmentation mask of the target area in response to the user's selection operation of the target area in the security inspection image.

[0129] The feature extraction module 220 is used to extract the target object features and boundary features of the target object in the target region based on the first positioning information and the first segmentation mask.

[0130] The feature fusion module 230 is used to fuse the target object features, the boundary features and the background information features of the target area to obtain a first fused feature.

[0131] The image recognition module 240 is used to input at least the first fused feature into the decoder according to the occlusion of the target object, so as to obtain the separated image of the target object and the fine-grained classification result output by the decoder.

[0132] In the above embodiments, in response to the user's selection of a target area in the security inspection image, the first positioning information and segmentation mask of the target area are obtained as interactive prompts to guide the model to focus on and identify the target area specified by the user. During the review process, security personnel can directly interact with the security inspection image by clicking or selecting boxes, without having to manually search for the location in the global detection results, which greatly improves the efficiency and convenience of human-computer interaction in the review stage. On this basis, this application extracts the features and boundary features of the target object and fuses them with the background information features of the target area, enabling the model to comprehensively utilize the target object's own information and morphological boundary information for comprehensive judgment. When the target object is occluded, the background features of the occluding object can be further introduced to participate in the fusion, which effectively enhances the model's ability to separate the target object from the occluded background and significantly improves the recognition accuracy in occluded scenarios. Finally, the decoder outputs the separated image of the target object and fine-grained classification results. The classification granularity is finer, which can provide more sufficient and accurate decision-making basis for the security personnel's final judgment.

[0133] In one embodiment, this application also provides a computer-readable storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the security inspection contraband identification method based on prompting interaction as described in any of the above embodiments.

[0134] In one embodiment, this application also provides a computer device, including: one or more processors, and memory.

[0135] The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the security inspection contraband identification method based on prompting interaction as described in any of the above embodiments.

[0136] Indicatively, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the internal structure of a computer device 300 provided in an embodiment of this application. The computer device 300 can be provided as a server. (Refer to...) Figure 5 The computer device 300 includes a processing component 302, which further includes one or more processors, and memory resources represented by memory 301 for storing instructions executable by the processing component 302, such as application programs. The application programs stored in memory 301 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 302 is configured to execute instructions to perform the security screening contraband identification method based on prompting interaction of any of the above embodiments.

[0137] The computer device 300 may also include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate on an operating system stored in memory 301, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0138] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0139] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0140] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0141] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying prohibited items during security checks based on interactive prompts, characterized in that, The method includes: In response to a user's selection of a target region in a security inspection image, the first positioning information and the first segmentation mask of the target region are obtained; Based on the first positioning information and the first segmentation mask, extract the target object features and boundary features of the target object in the target region; The target object features, the boundary features, and the background information features of the target region are fused to obtain the first fused feature; Based on the occlusion status of the target object, at least the first fusion feature is input into the decoder to obtain the separated image of the target object and the fine-grained classification result output by the decoder.

2. The method according to claim 1, characterized in that, The step of responding to a user's selection of a target region in a security inspection image and obtaining the first location information and first segmentation mask of the target region includes: In response to a user's click operation on a target object in a security inspection image, the detection coordinates of the target object are obtained as the first positioning information, and the area where the target object is located is segmented from the security inspection image based on the detection coordinates to obtain the first segmentation mask of the target object; Alternatively, in response to a user's filtering instruction for similar items, the detection coordinates of all similar items in the security inspection image are obtained as the first positioning information, and the area where each similar item is located is segmented from the security inspection image according to each detection coordinate, to obtain the first segmentation mask for each similar item. Alternatively, in response to a user's selection of any region in the security inspection image, the coordinates of the arbitrary region are obtained as the first positioning information, and the arbitrary region is segmented from the security inspection image based on the coordinates of the arbitrary region to obtain the first segmentation mask of the arbitrary region.

3. The method according to claim 1, characterized in that, The step of extracting target object features and boundary features of target objects in the target region based on the first positioning information and the first segmentation mask includes: The first encoder extracts features from the region corresponding to the first segmentation mask to obtain the target object features in the target region. The boundary features of the target object in the target area are obtained by extracting features from the boundary region corresponding to the first positioning information using the second encoder.

4. The method according to claim 1, characterized in that, The step of fusing the target object features, the boundary features, and the background information features of the target region to obtain a first fused feature includes: Obtain the background information features of the target area; The target object features, the boundary features, and the background information features are input into a feature fusion layer and fused to obtain the first fused feature.

5. The method according to claim 1, characterized in that, The step of inputting at least the first fused feature into the decoder based on the occlusion status of the target object includes: When the target object has no obstructions, only the first fused feature is input into the decoder; When the target object is obstructed, the obstruction feature of the obstruction and the background layout feature of the target area are obtained. The obstruction feature, the background layout feature and the first fusion feature are fused to obtain the second fusion feature, and the second fusion feature is input into the decoder.

6. The method according to claim 5, characterized in that, The acquisition of the occlusion features of the occlusion and the background layout features of the target area includes: Obtain the second positioning information and the second segmentation mask of the obstruction; The second segmentation mask of the occluder and the background information features of the target area are input into the third encoder, and the occlusion features of the occluder are extracted by the third encoder; The second positioning information of the obstruction, the shape information corresponding to the second segmentation mask, and the overall image shape information of the security inspection image are stitched together and input into the layout feature extraction layer. The background layout features of the target area are extracted through the layout feature extraction layer.

7. The method according to claim 1, characterized in that, The process of obtaining the separated image and fine-grained classification result of the target object output by the decoder includes: The first fusion feature input to the decoder is input in parallel into the target object separation branch and the target object classification branch; A clear image of the target object is identified and separated from the first fusion feature through the target object separation branch, resulting in a separated image; The separated target objects are further classified into fine-grained categories by the target object classification branch to obtain the fine-grained classification results of the target objects.

8. A security inspection device for identifying prohibited items based on interactive prompts, characterized in that, include: The data acquisition module is used to acquire the first positioning information and the first segmentation mask of the target area in response to the user's selection operation of the target area in the security inspection image; The feature extraction module is used to extract the target object features and boundary features of the target object in the target region based on the first positioning information and the first segmentation mask; The feature fusion module is used to fuse the target object features, the boundary features, and the background information features of the target region to obtain a first fused feature; An image recognition module is used to input at least the first fused feature into a decoder based on the occlusion of the target object, so as to obtain a separated image of the target object and a fine-grained classification result output by the decoder.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the security inspection contraband identification method based on prompting interaction as described in any one of claims 1 to 7.

10. A computer device, characterized in that, include: One or more processors, and memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the security inspection contraband identification method based on prompting interaction as described in any one of claims 1 to 7.

Images