A biological attack detection method, apparatus and device

Abstract

The embodiment of the specification discloses a biological attack detection method, device and equipment, the method comprises: acquiring multiple different modal images, respectively extracting the features of each modal image, obtaining the image features corresponding to each modal image, and extracting the content contained in each modal image, obtaining the image content text corresponding to each modal image, respectively fusing the image features of each modal and the image content text of each modal, obtaining the first fusion features, then, based on the first fusion features, the first fusion features corresponding to the multiple different modal images are processed by the preset cross attention algorithm, the corresponding processing result is obtained, and the processing result is fused to determine the corresponding second fusion features, finally, based on the second fusion features, the biological attack detection result corresponding to the multiple different modal images is determined.

Description

Technical Field

[0001] This document relates to the field of computer technology, and in particular to a method, apparatus and equipment for detecting biological attacks. Background Technology

[0002] With the continuous development of facial recognition systems in recent years and people's increasing attention to their privacy data, biometric attack detection has become an indispensable part of facial recognition systems. Biometric attack detection can effectively intercept non-biological attack samples.

[0003] With the continuous increase in facial recognition traffic, the number of users and the scenarios in which they use facial recognition are also increasing, leading to a greater variety of user attributes and combinations. As facial recognition traffic grows, feature extraction algorithms need to cover more and more scenarios and domains. Therefore, how to fully utilize multimodal data to achieve multi-scenario and multi-domain adaptation has become a key research direction with significant practical deployment implications. To this end, a technical solution is needed to improve the accuracy of biometric attack detection mechanisms and their adaptability across multiple scenarios. Summary of the Invention

[0004] The purpose of the embodiments in this specification is to provide a technical solution to improve the accuracy of biological attack detection mechanisms and their adaptability in multiple scenarios.

[0005] To achieve the above technical solution, the embodiments in this specification are implemented as follows:

[0006] This specification provides a method for detecting biological attacks, comprising: acquiring multiple images of different modalities for biological attack detection; extracting features from each modal image to obtain image features corresponding to each modal image, extracting the content contained in each modal image to obtain image content text corresponding to each modal image; fusing the image features and image content text corresponding to each modal image to obtain a first fused feature corresponding to each modal image; performing cross-attention processing on the first fused features corresponding to each modal image using a preset cross-attention algorithm to obtain corresponding processing results, and performing fusion processing on the processing results to determine a second fused feature corresponding to the multiple modal images; and determining the biological attack detection result corresponding to the multiple modal images based on the second fused features corresponding to the multiple modal images.

[0007] This specification provides an embodiment of a bio-attack detection method, which includes: acquiring multiple images of different modalities for bio-attack detection; extracting features from the images of each modality based on a feature extraction sub-model in a bio-attack detection model to obtain image features corresponding to each modality; extracting the content contained in the images of each modality based on an image description generation sub-model in the bio-attack detection model to obtain image content text corresponding to each modality; fusing the image features and image content text corresponding to each modality to obtain a first fused feature corresponding to each modality; performing attention processing on the first fused features corresponding to the images of each modality using a preset attention module in the bio-attack detection model to obtain corresponding processing results; and performing fusion processing based on the processing results to determine a second fused feature corresponding to the images of each modality; and inputting the second fused features corresponding to the images of each modality into a classification prediction sub-model in the bio-attack detection model to obtain bio-attack detection results corresponding to the images of each modality.

[0008] This specification provides a biological attack detection device, comprising: an image acquisition module for acquiring multiple images of different modalities for biological attack detection; a first fusion module for extracting features from each modal image to obtain image features corresponding to each modal image, extracting content from each modal image to obtain image content text corresponding to each modal image, and fusing the image features and image content text corresponding to each modal image to obtain a first fusion feature corresponding to each modal image; a second fusion module for performing cross-attention processing on the first fusion features corresponding to each modal image using a preset cross-attention algorithm to obtain corresponding processing results, and performing fusion processing based on the processing results to determine a second fusion feature corresponding to the multiple modal images; and a detection module for determining the biological attack detection result corresponding to the multiple modal images based on the second fusion features corresponding to the multiple modal images.

[0009] This specification provides an embodiment of a bio-attack detection device, comprising: an image acquisition module for acquiring multiple images of different modalities for bio-attack detection; a first model processing module for extracting features from the images of each modality based on a feature extraction sub-model in a bio-attack detection model, obtaining image features corresponding to each modality, and extracting the content contained in the images of each modality based on an image description generation sub-model in the bio-attack detection model, obtaining image content text corresponding to each modality; and fusing the image features and image content text corresponding to each modality to obtain a first fused feature corresponding to each modality; a second model processing module for performing attention processing on the first fused features corresponding to the images of each modality using a preset attention module in the bio-attack detection model, obtaining corresponding processing results, and performing fusion processing based on the processing results to determine a second fused feature corresponding to the images of each modality; and a model prediction module for inputting the second fused features corresponding to the images of each modality into a classification prediction sub-model in the bio-attack detection model to obtain bio-attack detection results corresponding to the images of each modality.

[0010] This specification provides an embodiment of a bio-attack detection device, comprising: a processor; and a memory arranged to store computer-executable instructions, wherein the executable instructions, when executed, cause the processor to: acquire multiple images of different modalities for bio-attack detection; extract features from the images of each modality to obtain image features corresponding to each modality, extract the content contained in the images of each modality to obtain image content text corresponding to each modality, fuse the image features and image content text corresponding to each modality to obtain a first fused feature corresponding to each modality; based on the first fused feature corresponding to each modality, perform cross-attention processing on the first fused features corresponding to the multiple images of different modalities using a preset cross-attention algorithm to obtain corresponding processing results, and perform fusion processing based on the processing results to determine a second fused feature corresponding to the multiple images of different modalities; and determine the bio-attack detection result corresponding to the multiple images of different modalities based on the second fused feature corresponding to the multiple images of different modalities.

[0011] This specification provides an embodiment of a bio-attack detection device, comprising: a processor; and a memory arranged to store computer-executable instructions, wherein the executable instructions, when executed, cause the processor to: acquire images of multiple different modalities for bio-attack detection; extract features from the images of each modality based on a feature extraction sub-model in a bio-attack detection model to obtain image features corresponding to each modality; extract the content contained in the images of each modality based on an image description generation sub-model in the bio-attack detection model to obtain image content text corresponding to each modality; fuse the image features and image content text corresponding to each modality to obtain a first fused feature corresponding to each modality; perform attention processing on the first fused features corresponding to each modality through a preset attention module in the bio-attack detection model to obtain corresponding processing results; and perform fusion processing based on the processing results to determine a second fused feature corresponding to each modality; input the second fused features corresponding to each modality into a classification prediction sub-model in the bio-attack detection model to obtain bio-attack detection results corresponding to each modality.

[0012] This specification also provides a storage medium for storing computer-executable instructions. When executed by a processor, these instructions implement the following process: acquiring multiple images of different modalities for bio-attack detection; extracting features from each modal image to obtain image features corresponding to each modal image; extracting the content contained in each modal image to obtain image content text corresponding to each modal image; fusing the image features and image content text corresponding to each modal image to obtain a first fused feature corresponding to each modal image; performing cross-attention processing on the first fused features corresponding to each modal image using a preset cross-attention algorithm to obtain corresponding processing results; and performing fusion processing based on the processing results to determine a second fused feature corresponding to the multiple modal images; and determining the bio-attack detection result corresponding to the multiple modal images based on the second fused features corresponding to the multiple modal images.

[0013] This specification also provides a storage medium for storing computer-executable instructions. When executed by a processor, these instructions implement the following process: acquiring multiple images of different modalities for biological attack detection; extracting features from each modal image based on a feature extraction sub-model within the biological attack detection model to obtain image features corresponding to each modal image; extracting content from each modal image based on an image description generation sub-model within the biological attack detection model to obtain image content text corresponding to each modal image; fusing the image features and image content text corresponding to each modal image to obtain a first fused feature corresponding to each modal image; performing attention processing on the first fused features corresponding to each modal image using a preset attention module within the biological attack detection model to obtain corresponding processing results; and performing fusion processing based on these processing results to determine a second fused feature corresponding to the multiple modal images; and inputting the second fused features corresponding to the multiple modal images into a classification prediction sub-model within the biological attack detection model to obtain biological attack detection results corresponding to the multiple modal images. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1A This is an embodiment of a biological attack detection method described in this specification;

[0016] Figure 1B This is a schematic diagram illustrating the generation process of a first fusion feature as described in this specification;

[0017] Figure 2 This is a schematic diagram of the structure of a feature extraction model in this specification;

[0018] Figure 3 This is another embodiment of a biological attack detection method described in this specification;

[0019] Figure 4 This is yet another embodiment of a biological attack detection method described in this specification;

[0020] Figure 5 This is a schematic diagram of a biological attack detection process described in this specification;

[0021] Figure 6 This is yet another embodiment of a biological attack detection method described in this specification;

[0022] Figure 7 This is a schematic diagram of another biological attack detection process described in this manual;

[0023] Figure 8 This is an embodiment of a biological attack detection device described in this specification;

[0024] Figure 9 This is another embodiment of a biological attack detection device described in this specification;

[0025] Figure 10 This is an embodiment of a biological attack detection device described in this specification. Detailed Implementation

[0026] This specification provides a method, apparatus, and device for detecting biological attacks.

[0027] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0028] This specification provides a biometric attack detection mechanism based on the Transformer architecture. With the continuous development of facial recognition systems in recent years and the increasing emphasis on privacy, biometric attack detection has become an indispensable part of facial recognition systems. Biometric attack detection can effectively intercept non-biological attack samples, including printed paper images, digital images displayed on electronic product screens, silicone masks, and 3D head models. As the volume of facial recognition services continues to increase, the number of users and the scenarios they occupy are also increasing, leading to more variations and combinations of user attributes. With the increasing volume of facial recognition services, feature extraction algorithms need to cover more and more scenarios and domains. Therefore, how to fully utilize multimodal data to achieve multi-scenario and multi-domain adaptation has become a key research direction with significant practical deployment implications.

[0029] In one embodiment, an end-to-end binary classification approach for biometric attack detection can be used. This approach employs an end-to-end binary classification network structure to directly classify input images as either biometric or attack-related. In this method, the model can learn global image features, but the region of interest is relatively fixed for different input images, resulting in relatively weak generalization performance. In another embodiment, a biometric attack detection approach based on early-stage or late-stage feature fusion can be used. This involves either fusion directly at the beginning of the multimodal image input or fusion after independent feature extraction for each modality at the end of the network. However, both fusion methods struggle to fully maximize the benefits of feature fusion and have high model complexity, thus affecting the performance and deployment difficulty of practical biometric attack detection models. To address this, this specification proposes a bio-attack detection method based on multimodal hybrid feature fusion using a Transformer architecture. This method acquires raw data of RGB, IR, and depth images using a multimodal image acquisition device. For each modality, corresponding textual description information is first extracted using an image description mechanism. Then, the image and text information are fused. Finally, a feature fusion processing is performed using a feature extraction mechanism based on cross-attention and hybrid feature fusion. This mechanism increases the weight of the attention regions of each feature, making the fused features more separable and fully utilizing the characteristics of images from different modalities. Specific processing details can be found in the following embodiments.

[0030] like Figure 1A As shown in the embodiments of this specification, a method for detecting biological attacks is provided. The execution subject of this method can be a terminal device or a server, etc. The terminal device can be a mobile terminal device such as a mobile phone or tablet computer, a computer device such as a laptop or desktop computer, or an IoT device (specifically, a smartwatch, an in-vehicle device, etc.). The server can be a single server or a server cluster composed of multiple servers. The server can be a backend server for financial or online shopping services, or a backend server for an application. This embodiment uses a server as the execution subject for detailed description. For the case where the execution subject is a terminal device, please refer to the following server-side processing, which will not be repeated here. The method may specifically include the following steps:

[0031] In step S102, images of various modalities are acquired for biological attack detection.

[0032] Biometric attack detection can be a detection mechanism used to determine the true physiological characteristics of an object in certain identity verification scenarios (such as identity verification scenarios in payment transactions, money transfer transactions, instant messaging transactions, shopping transactions, personal privacy data acquisition transactions, or other sensitive data acquisition transactions). In facial recognition applications, biometric attack detection can verify whether a user is operating the device themselves by using facial key point localization and facial tracking through combined actions such as blinking, opening the mouth, shaking the head, and nodding. It can effectively resist common attack methods such as photos, videos, face swaps, masks, occlusions, 3D animations, and screen re-photographing, thereby helping users identify fraudulent behavior and protecting their interests. Images of various modalities can be used to describe images acquired by different types of image acquisition components. For example, they can include RGB modal images, IR modal images, and depth images. RGB modal images contain the texture, color, and appearance information of the object being photographed. IR modal images can be infrared images output by infrared image sensors or infrared image acquisition components. Depth images can be matrices composed of depth data of all points within the field of view of image acquisition components such as depth cameras, containing the geometric information of the object being photographed.

[0033] In implementation, various methods can be used to acquire images of different modalities for biometric attack detection. For example, an algorithm for biometric attack detection can be deployed in a specified service. When a user needs to perform the specified service, various image acquisition components can be activated to acquire corresponding images, thereby obtaining images of various modalities for biometric attack detection during the user's execution of the specified service. Alternatively, when a user needs to perform the specified service through a device (such as a terminal device (such as a mobile phone or tablet computer) or machine (such as a vending machine), the device can activate various image acquisition components to acquire corresponding images, thereby obtaining images of various modalities for biometric attack detection during the user's execution of the specified service. These images can then be sent to a server, which can then acquire them. The specific implementation can be tailored to the actual situation, and this specification does not limit the specific implementation.

[0034] In step S104, feature extraction is performed on the image of each modality to obtain the image features corresponding to each modality. The content contained in the image of each modality is extracted to obtain the image content text corresponding to each modality. The image features and image content text corresponding to each modality are fused to obtain the first fused feature corresponding to each modality.

[0035] Image features can include various types, such as color features, texture features, shape features, and spatial relationship features. Color features can be a global feature that describes the surface properties of the scene corresponding to the image or image region. Texture features are also a global feature that describes the surface properties of the scene corresponding to the image or image region. Shape features have two types of representation: contour features and region features. Contour features of an image mainly target the outer boundary of an object, while region features of an image relate to the entire shape region. Spatial relationship features can be the spatial position or relative direction relationship between multiple targets segmented in the image. The above relationships can also be divided into connection / adjacency relationships, overlap / intersection relationships, and inclusion / containment relationships. Image content text can be text that transforms visual information in an image into semantic information. For example, it can extract text with corresponding semantic information through the visual features of the image (such as color, texture, and shape, and their combinations), or it can extract text with corresponding semantic information by identifying and reasoning to determine the specific target objects in the image and the relationships between them. It can also extract text with corresponding semantic information by performing high-level management of the objects, scenes, and goals contained in the image (which may include text with scene semantic information, text with behavioral semantic information, and text with emotional semantic information, etc.).

[0036] In practice, image feature extraction algorithms for different modalities can be pre-defined according to the actual situation. Alternatively, corresponding feature extraction models can be constructed using image feature extraction algorithms. Then, feature extraction can be performed on images of a specified modality based on the corresponding image feature extraction algorithm or feature extraction model, thereby extracting the image features contained in each modality and obtaining the image features corresponding to each modality.

[0037] For example, a neural network model (such as a convolutional neural network model or a deep neural network model) can be trained to enable the neural network model to extract features from images of different modalities. In this case, the images of each modality can be input into the corresponding trained neural network model, and the trained neural network model can extract features from the images of each modality to obtain the image features corresponding to each modality.

[0038] Image description extraction algorithms for different modalities can be pre-defined according to actual conditions. Alternatively, corresponding image description extraction models can be constructed using image description extraction algorithms. Then, the content contained in the image of a specified modality can be extracted based on the corresponding image description extraction algorithm or image description extraction model, thereby extracting the image description content contained in the image of each modality and obtaining the image content text corresponding to the image of each modality.

[0039] For example, a template-based approach can be used to extract the content of images for each modality, resulting in image content text for each modality. In this method, the generated text sentences have a fixed template. By detecting relevant elements such as objects, scenes, and actions in each modality's images, relevant words can be filled into the template to form sentences. Alternatively, a retrieval-based approach can be used, specifically image matching. First, a large number of images and image descriptions are stored in a database. Then, images for each modality are compared with images in the database to find similar images. The image descriptions corresponding to these similar images are used as candidate descriptions. These candidate descriptions are then processed to generate image descriptions for each modality, thus obtaining the image content text for each modality. It is also possible to train a specified deep learning model (such as a convolutional neural network model, a deep neural network model, a long short-term memory network LSTM, or a combination thereof) to enable the deep learning model to extract image descriptions for images of different modalities. In this case, the images of each modality can be input into the corresponding trained deep learning model, and the trained deep learning model can extract the content contained in the images of each modality to obtain the image content text corresponding to each modality.

[0040] After obtaining the image features and image content text corresponding to each modality through the above method, a fusion algorithm can be pre-set according to the actual situation, for example, such as... Figure 1B As shown, the image content text corresponding to each modality can be converted into a vector (using methods such as...). Figure 1B The encoder (i.e., the text encoder) in the code converts the text into vectors T1, T2, T3…T N (where N represents the number of vectors), and the image features corresponding to each modality can also be converted into vectors (through methods such as...). Figure 1B The encoder (i.e., the image encoder) in the image is converted into vectors I1, I2, I3…I M (where M represents the number of vectors), and then, the vectors of the image content text corresponding to images of the same modality can be fused with the vectors of the corresponding image features (e.g., Figure 1B In the I1T1, I1T2, I1T3…IM T N The image features and text content of each modality are fused separately to obtain the first fused feature of each modality.

[0041] In step S106, based on the first fusion feature corresponding to each modality of the image, a preset cross-attention algorithm is used to perform cross-attention processing on the first fusion features corresponding to multiple different modalities of the image to obtain the corresponding processing results. Based on the above processing results, fusion processing is performed to determine the second fusion feature corresponding to multiple different modalities of the image.

[0042] In implementation, to increase the weight of the attention regions for two different features, a cross-attention algorithm can be pre-defined according to the actual situation. This cross-attention algorithm can generate a query vector for each position in the decoder, used to calculate the attention weight for each position. Then, a set of key vectors and value vectors can be generated for each position in the encoder. The query vector and key vector can be used to perform a dot product operation, and the attention weights can be obtained through the softmax function. The attention weights are multiplied by the value vectors, and the results are summed to obtain the output data adjusted by the encoder. Based on this cross-attention algorithm, cross-attention processing can be performed on the first fusion features corresponding to images of different modalities, thereby realizing cross-attention processing on the first fusion features corresponding to images of multiple different modalities and obtaining the corresponding processing results.

[0043] A fusion algorithm can be pre-set according to the actual situation to fuse the processing results of images of different modalities. Specifically, it can be executed through the processing process of the cross-attention algorithm mentioned above to perform fusion processing based on the above processing results. Finally, the second fusion features corresponding to images of multiple different modalities can be obtained.

[0044] In step S108, based on the second fusion features corresponding to images of multiple different modalities, the biological attack detection results corresponding to images of multiple different modalities are determined.

[0045] In implementation, the biometric attack detection results corresponding to images of multiple different modalities can be determined in various ways. For example, according to a pre-set weight allocation rule, corresponding weights can be assigned to the second fusion features corresponding to the images of multiple different modalities. Then, a weighted summation calculation can be performed on the second fusion features corresponding to the images of multiple different modalities. The calculation result can be presented numerically. Afterward, the calculation result can be compared with a preset threshold. If the calculation result is less than the preset threshold, the biometric attack detection result corresponding to the images of multiple different modalities can be determined as an attack. If the calculation result is greater than the preset threshold, the biometric attack detection result corresponding to the video data can be determined as a non-attack. Alternatively, other algorithms or models can be used to determine the biometric attack detection results corresponding to the images of multiple different modalities using the second fusion features corresponding to the images of multiple different modalities. The specific settings can be determined according to the actual situation, and this specification does not limit this aspect.

[0046] This specification provides a method for detecting biological attacks. It involves acquiring multiple images of different modalities for biological attack detection, extracting features from each modal image to obtain image features corresponding to each modal, extracting the content contained in each modal image to obtain image content text corresponding to each modal image, fusing the image features and image content text corresponding to each modal image to obtain a first fused feature corresponding to each modal image, and then applying a preset cross-attention algorithm to the first fused features corresponding to the multiple modal images based on the first fused features corresponding to each modal image to obtain corresponding processing results. Finally, based on these processing results, a fusion process is performed to determine the image content corresponding to the multiple modal images. The second fusion feature, based on the second fusion features corresponding to images of multiple different modalities, determines the biological attack detection results corresponding to images of multiple different modalities. In this way, the raw data of images of multiple different modalities are collected by the multimodal image acquisition component. For each modal image, the corresponding image content text is extracted first through the image content, and then the image features and image content text are fused together. Then, a cross-attention algorithm is used to perform cross-attention processing on the first fusion features corresponding to images of multiple different modalities, thereby increasing the weight of the attention regions of each feature, making the fusion features more separable, and making full use of the characteristics between different modal images to a greater extent to adaptively leverage the separability of each modality in the feature space, thereby improving the accuracy of the biological attack detection mechanism and its adaptability in multiple scenarios.

[0047] In the embodiments of this specification, images of various modalities may include one or more of RGB modal images, infrared (IR) modal images, and depth images.

[0048] The relevant content regarding RGB mode images, IR mode images, and depth images can be found in the foregoing descriptions and will not be repeated here. In practical applications, images of various modes may include not only RGB mode images, IR mode images, and depth images, but also images of other modes, such as point cloud mode images and laser mode images. The specific mode can be set according to the actual situation, and this specification does not limit this.

[0049] Furthermore, there are various ways to process the image features of each modality in step S104 above. Here is one optional processing method, which may include the following: inputting the image of each modality into the feature extraction model corresponding to the modality to obtain the image features of each modality. The feature extraction model is a model built based on one or more ResBlock modules.

[0050] In implementation, taking multi-modal images, including images with three modalities, as an example, such as... Figure 2 As shown, the feature extraction model can include three branches, each of which can be used to perform image feature extraction processing on an image of one modality. Each branch can include one or more ResBlock modules (where...). Figure 2Each branch consists of three sequentially connected and distinct ResBlock modules (res1, res2, and res3). Each ResBlock module can be configured with appropriate data dimensions, such as 64 or 128, depending on the specific requirements. If the images include RGB, IR, and depth modalities, a branch for image feature extraction can be created for the RGB modal images. This branch, consisting of the three ResBlock modules (res1, res2, and res3), allows for feature extraction of the RGB modal images, yielding the corresponding image features. In addition, a branch for image feature extraction can be set up for IR mode images, consisting of three ResBlock modules: res4, res5, and res6 (in practical applications, res4 can have the same network structure as res1, res5 as res2, and res6 as res3, etc.). Image features of the IR mode image can be extracted using the three ResBlock modules in this branch, yielding the corresponding image features. Similarly, a branch for image feature extraction can be set up for depth images, consisting of three ResBlock modules: res7, res8, and res9 (in practical applications, res7 can have the same network structure as res1, res8 as res2, and res9 as res3, etc.). Image features of the depth image can be extracted using the three ResBlock modules in this branch, yielding the corresponding image features.

[0051] Based on the structure of the above feature extraction model, training samples for training the feature extraction model can be obtained, and a corresponding loss function can be set. The training samples can include images of one or more different modalities. The feature extraction model can be trained using the training samples, that is, the branch structures corresponding to multiple different modalities in the feature extraction model can be jointly trained until the loss function converges, and the trained feature extraction model can be obtained. The above image feature extraction processing can be performed based on the trained feature extraction model.

[0052] In the embodiments described in this specification, such as Figure 3As shown, the first fusion feature corresponding to each modality of the image in step S106 above can be processed by a preset cross-attention algorithm to obtain the corresponding processing result. The specific processing method can be varied. The following provides an optional processing method, which can specifically include the processing of steps S1062 and S1064.

[0053] In step S1062, the first fusion feature corresponding to the first modality is obtained from the first fusion features corresponding to the images of multiple different modalities. The first modality is the modality whose modality separability is higher than a preset threshold among multiple different modalities.

[0054] The modal separability degree can be defined as the degree to which a specific modality can be accurately distinguished from multiple different modalities. For example, among RGB modal images, IR modal images, and depth images, RGB modal images and IR modal images can usually be easily distinguished, thus the modal separability degree corresponding to RGB and IR modes is relatively high. The preset threshold can be set according to the actual situation, and the embodiments in this specification do not limit it.

[0055] In implementation, a preset threshold can be set according to the actual situation. By setting the preset threshold, an image of the first modality that is higher than the preset threshold can be determined from multiple images of different modalities, thereby obtaining the first fusion feature corresponding to the first modality image. For example, if the multiple images of different modalities include RGB modal images, IR modal images, and depth images, the preset threshold can be used to select the IR modal image as the first modal image, thereby obtaining the first fusion feature corresponding to the IR modal image.

[0056] In step S1064, the image of the first modality is used as the image of the base modality. Through a preset cross-attention algorithm, the first fusion feature corresponding to the image of the first modality is cross-attention processed with the first fusion feature corresponding to the images of other modalities besides the first modality, respectively, to obtain the corresponding processing results.

[0057] In implementation, based on the above example, the image of the IR modality can be used as the base modality image. Then, the first fusion feature corresponding to the IR modality image and the first fusion feature corresponding to the RGB modality image can be cross-attention processed using a cross-attention algorithm to obtain the processing result of the above two (which can be referred to as the first processing result for the sake of convenience in the following description). At the same time, the first fusion feature corresponding to the IR modality image and the first fusion feature corresponding to the depth image can be cross-attention processed using a cross-attention algorithm to obtain the processing result of the above two (which can be referred to as the second processing result for the sake of convenience in the following description). Finally, the first processing result and the second processing result can be obtained. The above is only an optional example. In practical applications, depending on the different images of the determined first modality and the number of images of different modalities, a preset cross-attention algorithm can be used to cross-attention process the first fusion feature corresponding to the image of the first modality with the first fusion features corresponding to images of other modalities besides the first modality in multiple different modalities to obtain the corresponding processing results.

[0058] In practical applications, the first modality image may include an RGB modality image.

[0059] In implementation, based on the above, the RGB modal image can be used as the base modal image. Then, the first fusion feature corresponding to the RGB modal image and the first fusion feature corresponding to the IR modal image can be cross-attention processed using a cross-attention algorithm to obtain the processing results of the above two (which can be referred to as the third processing result for the convenience of subsequent description). At the same time, the first fusion feature corresponding to the RGB modal image and the first fusion feature corresponding to the depth image can be cross-attention processed using a cross-attention algorithm to obtain the processing results of the above two (which can be referred to as the fourth processing result for the convenience of subsequent description). Finally, the third processing result and the fourth processing result can be obtained.

[0060] In the embodiments described in this specification, such as Figure 4 As shown, the specific processing method for determining the second fusion feature corresponding to multiple different modalities of the image by performing fusion processing based on the above processing results in step S106 can be varied. The following provides an optional processing method, which may specifically include the processing steps S1066 and S1068.

[0061] In step S1066, based on the obtained processing result and the first fusion feature corresponding to the first modality image, the correlation information between the processing result and the first fusion feature corresponding to the first modality image is determined by a correlation algorithm.

[0062] In implementation, the corresponding processing results can be fused with the first fusion feature corresponding to the first modality image by calculating the correlation. Specifically, a correlation algorithm can be pre-defined, and the correlation between the corresponding processing results and the first fusion feature corresponding to the first modality image can be calculated based on this algorithm to obtain the corresponding correlation information. For example, taking an RGB modality image as the base modality image, the first fusion feature corresponding to the RGB modality image and the first fusion feature corresponding to the IR modality image are cross-attention processed using a cross-attention algorithm. The results of these two processing methods are then used to calculate the correlation between the first fusion feature corresponding to the RGB modality image and the first fusion feature corresponding to the depth image. This results in the correlation between the first fusion feature corresponding to the RGB modality image and the first fusion feature corresponding to the depth image, and the results of these two processing methods are then used to calculate the correlation between the first fusion feature corresponding to the RGB modality image and the first fusion feature corresponding to the depth image. This results in the correlation between the first fusion feature corresponding to the RGB modality image and the first fusion feature corresponding to the depth image. This results in the correlation between the first fusion feature corresponding to the RGB modality image and the first fusion feature corresponding to the depth image. This results in the correlation between the first fusion feature corresponding to the RGB modality image and the first fusion feature corresponding to the depth image. This results in the correlation between the first fusion feature corresponding to the depth ...

[0063] It should be noted that the above is only an optional example. In practical applications, the correlation information between different processing results and the first fusion feature corresponding to the first modality image can be determined by using a correlation algorithm, based on the different images of the determined first modality, the number of images of different modalities, and the different processing results obtained.

[0064] In step S1068, the above-mentioned correlation information and the first fusion feature corresponding to the first modality image are fused to obtain the second fusion feature corresponding to multiple different modalities.

[0065] In implementation, a fusion algorithm can be pre-defined. This algorithm can be used to fuse the aforementioned correlation information and the first fusion feature corresponding to the first modality image to obtain second fusion features corresponding to multiple different modalities. For example, based on the above example, this fusion algorithm can be used to fuse the first correlation information, the second correlation information, and the first fusion feature corresponding to the first modality image to obtain second fusion features corresponding to multiple different modalities.

[0066] There are various ways to process step S108 above. The following is an optional processing method, which may include the following: inputting the second fusion features corresponding to images of multiple different modalities into a pre-trained biological attack detection model to obtain biological attack detection results corresponding to images of multiple different modalities. The biological attack detection model is a model built based on a preset classification algorithm.

[0067] The classification algorithm can include various types, such as binary classification algorithm, decision tree algorithm, neural network algorithm, etc. The specific algorithm can be set according to the actual situation, and the embodiments in this specification do not limit it.

[0068] In implementation, a suitable classification algorithm can be selected based on the actual situation, and a model architecture for a biohazard detection model can be constructed based on this algorithm. Then, training samples (i.e., training samples constructed using the second fusion features determined in the above manner) can be obtained for training the biohazard detection model. A corresponding loss function can be set, and the biohazard detection model can be trained using these training samples until the loss function converges, resulting in a trained biohazard detection model. The input data of the trained biohazard detection model can be the second fusion features determined in the above manner, and the output data can be the corresponding biohazard detection results. The second fusion features corresponding to images of various modalities can be input into the trained biohazard detection model to obtain biohazard detection results corresponding to images of various modalities.

[0069] In the embodiments described in this specification, such as Figure 5 As shown, the specific processing methods for extracting the content contained in the image of each modality in step S104 above to obtain the image content text corresponding to each modality can be varied. The following provides an optional processing method, which may include the following: extracting the content contained in the image of each modality through an image description generation model to obtain the image content text corresponding to each modality. The image description generation model is a pre-trained model containing a Transformer module; or, based on the image description mapping relationship corresponding to different modalities, extracting the content contained in the image of each modality to obtain the image content text corresponding to each modality.

[0070] In implementation, the image captioning generation model can be based on a Transformer module and an LSTM network structure. Then, the model can be trained using appropriate training samples until the set loss function converges, resulting in a trained image captioning generation model. Images of each modality can be input into the trained model to extract the content from each modality's image, thus obtaining the corresponding image content text for each modality.

[0071] In addition to determining the image content text corresponding to each modality of the image through the methods described above, there are many other ways to determine the image content text corresponding to each modality of the image. For example, the mapping relationship of text description can be customized according to the characteristics of images of different modalities (i.e., the image description mapping relationship corresponding to different modalities). Through the above image description mapping relationship, better text descriptions with different modal differences can be extracted from images of each modality, so as to extract the content contained in images of each modality and obtain the image content text corresponding to each modality of the image.

[0072] like Figure 6 As shown in the embodiments of this specification, a method for detecting biological attacks is provided. The execution subject of this method can be a terminal device or a server, etc. The terminal device can be a mobile terminal device such as a mobile phone or tablet computer, a computer device such as a laptop or desktop computer, or an IoT device (specifically, a smartwatch, an in-vehicle device, etc.). The server can be a single server or a server cluster composed of multiple servers. The server can be a backend server for financial or online shopping services, or a backend server for an application. This embodiment uses a server as the execution subject for detailed description. For the case where the execution subject is a terminal device, please refer to the following server-side processing, which will not be repeated here. The method may specifically include the following steps:

[0073] In step S602, images of various modalities are acquired for biological attack detection.

[0074] In step S604, feature extraction is performed on the image of each modality based on the feature extraction sub-model in the biological attack detection model to obtain the image features corresponding to each modality. The content contained in the image of each modality is extracted based on the image description generation sub-model in the biological attack detection model to obtain the image content text corresponding to each modality. The image features and image content text corresponding to each modality are fused to obtain the first fused feature corresponding to each modality.

[0075] The biological attack detection model may include several different parts, such as in this embodiment, for example... Figure 7As shown, the biological attack detection model can sequentially include a feature extraction sub-model, an image description generation sub-model, an attention module, and a classification prediction sub-model. The feature extraction sub-model can be constructed using various algorithms or deep networks. Specifically, it can be constructed using convolutional neural networks or deep neural networks, or using a specified feature extraction algorithm. The specific implementation can be determined according to the actual situation, and this specification does not limit this. The image description generation sub-model can be based on a pre-trained sub-model containing a Transformer module or a sub-model constructed based on the image description mapping relationship corresponding to different modalities. The specific implementation can be determined according to the actual situation.

[0076] It should be noted that the biological attack detection method provided in the above embodiments is different from the biological attack detection method in this embodiment. In the biological attack detection method provided in the above embodiments, steps S102 to S108 and their related steps use independent algorithms or independent models (there is no subordinate relationship between different models) for corresponding processing, such as... Figure 7 As shown, in the biological attack detection method of this embodiment, each step from S602 to S608 and its related steps include a complete model, namely the biological attack detection model. Moreover, the biological attack detection model includes multiple different parts (i.e., sub-models), including feature extraction sub-model, image description generation sub-model, attention module and classification prediction sub-model, etc. That is, the feature extraction sub-model, image description generation sub-model, attention module and classification prediction sub-model, etc., belong to the biological attack detection model.

[0077] In step S606, based on the first fusion feature corresponding to each modality of the image, attention processing is performed on the first fusion feature corresponding to multiple different modalities of the image through the preset attention module in the biological attack detection model to obtain the corresponding processing result, and fusion processing is performed based on the above processing result to determine the second fusion feature corresponding to multiple different modalities of the image.

[0078] The attention module can include algorithms such as cross-attention.

[0079] In step S608, the second fusion features corresponding to images of multiple different modalities are input into the classification prediction sub-model in the biological attack detection model to obtain biological attack detection results corresponding to images of multiple different modalities.

[0080] The classification prediction sub-model can be constructed using various different algorithms, such as binary classification algorithms or neural network algorithms. The specific model can be set according to the actual situation, and the embodiments in this specification do not limit it.

[0081] The specific processing procedures for steps S602 to S608 above can be found in the relevant content above, and will not be repeated here.

[0082] This specification provides a method for detecting biological attacks. It involves acquiring multiple images of different modalities for biological attack detection, extracting features from each modal image using a feature extraction sub-model within the biological attack detection model to obtain image features corresponding to each modal, and extracting the content contained in each modal image using an image description generation sub-model within the biological attack detection model to obtain image content text corresponding to each modal image. The image features and image content text corresponding to each modal image are then fused to obtain a first fused feature corresponding to each modal image. Based on this first fused feature, an attention module pre-defined within the biological attack detection model performs attention processing on the first fused features corresponding to the multiple modal images to obtain corresponding processing results, and finally, fusion processing is performed based on these results. A second fusion feature corresponding to multiple different modalities is determined, and this second fusion feature is input into the classification prediction sub-model in the bioattack detection model to obtain bioattack detection results for multiple different modalities. In this way, the raw data of multiple different modalities of images are collected by a multimodal image acquisition component. For each modality, the corresponding image content text is extracted first, and then the image features and image content text are fused. Then, an attention-based module is used to perform attention processing on the first fusion feature corresponding to multiple different modalities, thereby increasing the weight of the attention regions of each feature, making the fusion feature more separable, and fully utilizing the characteristics between different modalities of images to a greater extent to adaptively leverage the separability of each modality in the feature space, thus improving the accuracy of the bioattack detection mechanism and its adaptability in multiple scenarios.

[0083] In practical applications, the attention module described above can be built based on cross-attention algorithms, channel attention algorithms, or Transformer-based self-attention algorithms. During the cross-attention processing of the first fusion features corresponding to multiple different modalities, the attention module can execute the process according to the corresponding attention algorithm used, which will not be elaborated further here.

[0084] In the embodiments of this specification, images of various modalities include one or more of RGB modal images, infrared (IR) modal images, and depth images.

[0085] In the embodiments described in this specification, the feature extraction sub-model is a model built based on one or more ResBlock modules, such as... Figure 7As shown, the feature extraction sub-model includes three branches, each corresponding to a modality, and each branch is built based on three ResBlock modules.

[0086] In the embodiments of this specification, the first fusion feature corresponding to each modality of the image in step S606 above can be processed by the attention module preset in the biological attack detection model to obtain the corresponding processing result. The specific processing method can be varied. Here is another optional processing method, which can specifically include the processing of steps A2 and A4.

[0087] In step A2, the first fusion feature corresponding to the first modality is obtained from the first fusion features corresponding to images of multiple different modalities. The first modality is the modality whose modality separability is higher than a preset threshold among multiple different modalities.

[0088] In step A4, the image of the first modality is used as the image of the base modality. Through the attention module preset in the biological attack detection model, the first fusion feature corresponding to the image of the first modality is subjected to attention processing with the first fusion features corresponding to the images of other modalities besides the first modality in a variety of different modalities, and the corresponding processing results are obtained.

[0089] The specific processing procedures for steps A2 and A4 above can be found in the aforementioned related content, and will not be repeated here.

[0090] In practical applications, the first modality image may include an RGB modality image.

[0091] In the embodiments of this specification, the specific processing method for determining the second fusion feature corresponding to multiple different modalities of images by performing fusion processing based on the above processing results in step S606 can be varied. Here, we provide another optional processing method, which may specifically include the processing of steps B2 and B4.

[0092] In step B2, based on the obtained processing result and the first fusion feature corresponding to the first modality image, the correlation information between the above processing result and the first fusion feature corresponding to the first modality image is determined by the correlation algorithm in the biological attack detection model.

[0093] In step B4, in the biological attack detection model, the above-mentioned correlation information and the first fusion feature corresponding to the first modality image are fused to obtain the second fusion feature corresponding to multiple different modalities.

[0094] The specific processing procedures for steps B2 and B4 above can be found in the aforementioned related content, and will not be repeated here.

[0095] In practical applications, classification prediction sub-models can be sub-models built based on preset classification algorithms.

[0096] This specification provides a method for detecting biological attacks. On the one hand, it makes full use of the text features of image descriptions to enhance the spatial representation of image features. On the other hand, it performs full cross-attention fusion of mixed features of image content text and image features from different modalities, thereby making greater adaptive use of the separability of each modality in the feature space, thereby improving the accuracy of the biological attack detection mechanism and its adaptability in multiple scenarios.

[0097] The above are the biological attack detection methods provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide a biological attack detection device, such as... Figure 8 As shown.

[0098] The biological attack detection device includes: an image acquisition module 801, a first fusion module 802, a second fusion module 803, and a detection module 804, wherein:

[0099] Image acquisition module 801 acquires images of various modalities for biological attack detection;

[0100] The first fusion module 802 extracts features from the image of each modality to obtain the image features corresponding to each modality, extracts the content contained in the image of each modality to obtain the image content text corresponding to each modality, and performs fusion processing on the image features and image content text corresponding to each modality to obtain the first fusion feature corresponding to each modality.

[0101] The second fusion module 803, based on the first fusion feature corresponding to each modality of the image, performs cross-attention processing on the first fusion feature corresponding to multiple different modalities of the image through a preset cross-attention algorithm to obtain the corresponding processing result, and performs fusion processing based on the processing result to determine the second fusion feature corresponding to multiple different modalities of the image;

[0102] The detection module 804 determines the biological attack detection results corresponding to the images of the various different modalities based on the second fusion features corresponding to the images of the various different modalities.

[0103] In the embodiments of this specification, the various modal images include one or more of RGB modal images, infrared (IR) modal images, and depth images.

[0104] In the embodiments of this specification, the first fusion module 802 inputs the image of each modality into the feature extraction model corresponding to the corresponding modality to obtain the image features corresponding to each modality. The feature extraction model is a model built based on one or more ResBlock modules.

[0105] In this embodiment of the specification, the second fusion module 803 includes:

[0106] The basic mode determination unit obtains the first fusion feature corresponding to the first mode from the first fusion features corresponding to images of multiple different modes. The first mode is the mode whose modality separability is higher than a preset threshold among multiple different modes.

[0107] The attention unit uses the image of the first modality as the image of the base modality. Through a preset cross-attention algorithm, it performs cross-attention processing on the first fusion feature corresponding to the image of the first modality and the first fusion feature corresponding to the images of other modalities besides the first modality, respectively, to obtain the corresponding processing results.

[0108] In the embodiments described in this specification, the image of the first modality includes an image of the RGB modality.

[0109] In this embodiment of the specification, the second fusion module 803 includes:

[0110] The correlation determination unit determines the correlation information between the processing result and the first fusion feature corresponding to the first modality image based on the obtained processing result and the first fusion feature corresponding to the first modality image through a correlation algorithm.

[0111] The fusion unit fuses the correlation information and the first fusion feature corresponding to the first modality image to obtain the second fusion feature corresponding to multiple different modalities.

[0112] In this embodiment of the specification, the detection module 804 inputs the second fusion features corresponding to the multiple different modalities of the images into a pre-trained biological attack detection model to obtain biological attack detection results corresponding to the multiple different modalities of the images. The biological attack detection model is a model constructed based on a preset classification algorithm.

[0113] In the embodiments of this specification, the first fusion module 802 extracts the content contained in the image of each modality through the image description generation model to obtain the image content text corresponding to the image of each modality. The image description generation model is based on a pre-trained model containing a Transformer module; or, based on the image description mapping relationship corresponding to different modalities, it extracts the content contained in the image of each modality to obtain the image content text corresponding to the image of each modality.

[0114] This specification provides a biological attack detection device. It acquires multiple images of different modalities for biological attack detection, extracts features from each modal image to obtain image features corresponding to each modal, and extracts the content contained in each modal image to obtain image content text corresponding to each modal image. It then fuses the image features and image content text corresponding to each modal image to obtain a first fused feature corresponding to each modal image. Based on the first fused feature corresponding to each modal image, it performs cross-attention processing on the first fused features corresponding to the multiple modal images using a preset cross-attention algorithm to obtain corresponding processing results. Finally, it performs fusion processing based on these processing results to determine the image content corresponding to the multiple modal images. The second fusion feature, based on the second fusion features corresponding to images of multiple different modalities, determines the biological attack detection results corresponding to images of multiple different modalities. In this way, the raw data of images of multiple different modalities are collected by the multimodal image acquisition component. For each modal image, the corresponding image content text is extracted first through the image content, and then the image features and image content text are fused together. Then, a cross-attention algorithm is used to perform cross-attention processing on the first fusion features corresponding to images of multiple different modalities, thereby increasing the weight of the attention regions of each feature, making the fusion features more separable, and making full use of the characteristics between different modal images to a greater extent to adaptively leverage the separability of each modality in the feature space, thereby improving the accuracy of the biological attack detection mechanism and its adaptability in multiple scenarios.

[0115] Following the same line of thought, this specification also provides a biological attack detection device, such as... Figure 9 As shown.

[0116] The biological attack detection device includes: an image acquisition module 901, a first model processing module 902, a second model processing module 903, and a model prediction module 904, wherein:

[0117] Image acquisition module 901 acquires images of various modalities for biological attack detection;

[0118] The first model processing module 902 performs feature extraction on the image of each modality based on the feature extraction sub-model in the biological attack detection model to obtain the image features corresponding to each modality. It also extracts the content contained in the image of each modality based on the image description generation sub-model in the biological attack detection model to obtain the image content text corresponding to each modality. Finally, it performs fusion processing on the image features and image content text corresponding to each modality to obtain the first fused feature corresponding to each modality.

[0119] The second model processing module 903, based on the first fusion feature corresponding to each modality of the image, performs attention processing on the first fusion feature corresponding to multiple different modalities of the image through the preset attention module in the biological attack detection model, obtains the corresponding processing result, and performs fusion processing based on the processing result to determine the second fusion feature corresponding to multiple different modalities of the image;

[0120] The model prediction module 904 inputs the second fusion features corresponding to the images of the various different modalities into the classification prediction sub-model in the biological attack detection model to obtain the biological attack detection results corresponding to the images of the various different modalities.

[0121] In the embodiments of this specification, the attention module is a module built based on the cross-attention algorithm, the channel attention algorithm, or the self-attention algorithm based on Transformer, and the feature extraction sub-model is a sub-model built based on one or more ResBlock modules.

[0122] In the embodiments of this specification, the various modal images include one or more of RGB modal images, infrared (IR) modal images, and depth images.

[0123] In the embodiments described in this specification, the feature extraction sub-model is a sub-model built based on one or more ResBlock modules.

[0124] In this embodiment of the specification, the second model processing module 903 includes:

[0125] The basic mode determination unit obtains the first fusion feature corresponding to the first mode from the first fusion features corresponding to images of multiple different modes. The first mode is the mode whose modality separability is higher than a preset threshold among multiple different modes.

[0126] The attention unit uses the image of the first modality as the image of the base modality. Through the attention module preset in the biological attack detection model, it performs attention processing on the first fusion feature corresponding to the image of the first modality and the first fusion feature corresponding to the images of other modalities besides the first modality, respectively, to obtain the corresponding processing results.

[0127] In the embodiments described in this specification, the image of the first modality includes an image of the RGB modality.

[0128] In this embodiment of the specification, the second model processing module 903 includes:

[0129] The correlation determination unit determines the correlation information between the processing result and the first fusion feature corresponding to the first modality image based on the obtained processing result and the first fusion feature corresponding to the first modality image through a preset correlation algorithm in the biological attack detection model.

[0130] In the biological attack detection model, the fusion unit fuses the correlation information and the first fusion feature corresponding to the first modality image to obtain the second fusion feature corresponding to multiple different modalities.

[0131] In the embodiments described in this specification, the classification prediction sub-model is a sub-model constructed based on a preset classification algorithm.

[0132] In the embodiments described in this specification, the image description generation sub-model is a pre-trained sub-model containing a Transformer module or a sub-model constructed based on the image description mapping relationship corresponding to different modalities.

[0133] This specification provides a biometric attack detection device. It acquires multiple images of different modalities for biometric attack detection. Based on a feature extraction sub-model within the biometric attack detection model, it extracts features from each modal image to obtain image features corresponding to each modality. Then, based on an image description generation sub-model within the biometric attack detection model, it extracts the content contained in each modal image to obtain image content text corresponding to each modality. The image features and image content text corresponding to each modality are then fused to obtain a first fused feature corresponding to each modality. Based on this first fused feature, an attention module preset within the biometric attack detection model performs attention processing on the first fused features corresponding to the multiple modal images to obtain corresponding processing results. Finally, it performs fusion processing based on these processing results. A second fusion feature corresponding to multiple different modalities is determined, and this second fusion feature is input into the classification prediction sub-model in the bioattack detection model to obtain bioattack detection results for multiple different modalities. In this way, the raw data of multiple different modalities of images are collected by a multimodal image acquisition component. For each modality, the corresponding image content text is extracted first, and then the image features and image content text are fused. Then, an attention-based module is used to perform attention processing on the first fusion feature corresponding to multiple different modalities, thereby increasing the weight of the attention regions of each feature, making the fusion feature more separable, and fully utilizing the characteristics between different modalities of images to a greater extent to adaptively leverage the separability of each modality in the feature space, thus improving the accuracy of the bioattack detection mechanism and its adaptability in multiple scenarios.

[0134] The above are examples of biological attack detection devices provided in the embodiments of this specification. Based on the same concept, embodiments of this specification also provide a biological attack detection device, such as... Figure 10 As shown.

[0135] The biological attack detection device can provide terminal devices or servers, etc., for the above embodiments.

[0136] Biometric attack detection devices can vary significantly in configuration and performance. They may include one or more processors 1001 and memory 1002, with memory 1002 storing one or more application programs or data. Memory 1002 can be temporary or persistent storage. The application programs stored in memory 1002 may include one or more modules (not shown), each module including a series of computer-executable instructions for the biometric attack detection device. Furthermore, processor 1001 may be configured to communicate with memory 1002, executing the series of computer-executable instructions stored in memory 1002 on the biometric attack detection device. The biometric attack detection device may also include one or more power supplies 1003, one or more wired or wireless network interfaces 1004, one or more input / output interfaces 1005, and one or more keyboards 1006.

[0137] Specifically, in this embodiment, the bio-attack detection device includes a memory and one or more programs, wherein one or more programs are stored in the memory, and one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the bio-attack detection device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0138] Acquire images in multiple different modalities for biological attack detection;

[0139] Feature extraction is performed on the image of each modality to obtain the image features corresponding to each modality. The content contained in the image of each modality is extracted to obtain the image content text corresponding to each modality. The image features and image content text corresponding to each modality are fused to obtain the first fused feature corresponding to each modality.

[0140] Based on the first fusion feature corresponding to each modality of the image, a preset cross-attention algorithm is used to perform cross-attention processing on the first fusion features corresponding to multiple different modalities of the image to obtain the corresponding processing results. Based on the processing results, fusion processing is performed to determine the second fusion feature corresponding to multiple different modalities of the image.

[0141] Based on the second fusion features corresponding to the images of the various different modalities, the biological attack detection results corresponding to the images of the various different modalities are determined.

[0142] Furthermore, specifically in this embodiment, the bio-attack detection device includes a memory and one or more programs, wherein one or more programs are stored in the memory, and one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the bio-attack detection device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0143] Acquire images in multiple different modalities for biological attack detection;

[0144] Based on the feature extraction sub-model in the biological attack detection model, features are extracted from the images of each modality to obtain the image features corresponding to each modality. Based on the image description generation sub-model in the biological attack detection model, the content contained in the images of each modality is extracted to obtain the image content text corresponding to each modality. The image features and image content text corresponding to each modality are fused to obtain the first fused feature corresponding to each modality.

[0145] Based on the first fusion feature corresponding to each modality of the image, attention processing is performed on the first fusion feature corresponding to multiple different modalities of the image through the preset attention module in the biological attack detection model to obtain the corresponding processing result, and fusion processing is performed based on the processing result to determine the second fusion feature corresponding to multiple different modalities of the image.

[0146] The second fusion features corresponding to the images of the various different modalities are input into the classification prediction sub-model in the biological attack detection model to obtain the biological attack detection results corresponding to the images of the various different modalities.

[0147] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for biological attack detection devices are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0148] This specification provides a biological attack detection device. It acquires images of various modalities for biological attack detection, extracts features from each modal image to obtain image features corresponding to each modal, and extracts the content contained in each modal image to obtain image content text corresponding to each modal image. The image features and image content text corresponding to each modal image are then fused to obtain a first fused feature corresponding to each modal image. Based on the first fused feature corresponding to each modal image, a preset cross-attention algorithm is used to perform cross-attention processing on the first fused features corresponding to the multiple modal images to obtain corresponding processing results. Based on these processing results, fusion processing is performed to determine the corresponding features of the multiple modal images. The second fusion feature, based on the second fusion features corresponding to images of multiple different modalities, determines the biological attack detection results corresponding to images of multiple different modalities. In this way, the raw data of images of multiple different modalities are collected by the multimodal image acquisition component. For each modal image, the corresponding image content text is extracted first through the image content, and then the image features and image content text are fused together. Then, a cross-attention algorithm is used to perform cross-attention processing on the first fusion features corresponding to images of multiple different modalities, thereby increasing the weight of the attention regions of each feature, making the fusion features more separable, and making full use of the characteristics between different modal images to a greater extent to adaptively leverage the separability of each modality in the feature space, thereby improving the accuracy of the biological attack detection mechanism and its adaptability in multiple scenarios.

[0149] Furthermore, based on the above Figures 1A to 7 The method shown in this specification, along with one or more embodiments, also provides a storage medium for storing computer-executable instruction information. In one specific embodiment, the storage medium can be a USB flash drive, optical disc, hard disk, etc. When the computer-executable instruction information stored in the storage medium is executed by a processor, it can achieve the following process:

[0150] Acquire images in multiple different modalities for biological attack detection;

[0151] Feature extraction is performed on the image of each modality to obtain the image features corresponding to each modality. The content contained in the image of each modality is extracted to obtain the image content text corresponding to each modality. The image features and image content text corresponding to each modality are fused to obtain the first fused feature corresponding to each modality.

[0152] Based on the first fusion feature corresponding to each modality of the image, a preset cross-attention algorithm is used to perform cross-attention processing on the first fusion features corresponding to multiple different modalities of the image to obtain the corresponding processing results. Based on the processing results, fusion processing is performed to determine the second fusion feature corresponding to multiple different modalities of the image.

[0153] Based on the second fusion features corresponding to the images of the various different modalities, the biological attack detection results corresponding to the images of the various different modalities are determined.

[0154] In another specific embodiment, the storage medium can be a USB flash drive, optical disc, hard disk, etc., and the computer-executable instruction information stored in the storage medium can achieve the following process when executed by the processor:

[0155] Acquire images in multiple different modalities for biological attack detection;

[0156] Based on the feature extraction sub-model in the biological attack detection model, features are extracted from the images of each modality to obtain the image features corresponding to each modality. Based on the image description generation sub-model in the biological attack detection model, the content contained in the images of each modality is extracted to obtain the image content text corresponding to each modality. The image features and image content text corresponding to each modality are fused to obtain the first fused feature corresponding to each modality.

[0157] Based on the first fusion feature corresponding to each modality of the image, attention processing is performed on the first fusion feature corresponding to multiple different modalities of the image through the preset attention module in the biological attack detection model to obtain the corresponding processing result, and fusion processing is performed based on the processing result to determine the second fusion feature corresponding to multiple different modalities of the image.

[0158] The second fusion features corresponding to the images of the various different modalities are input into the classification prediction sub-model in the biological attack detection model to obtain the biological attack detection results corresponding to the images of the various different modalities.

[0159] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the above-described storage medium embodiment is basically similar to the method embodiment, so the description is relatively simple; relevant parts can be referred to the description of the method embodiment.

[0160] This specification provides a storage medium that acquires multiple images of different modalities for biological attack detection. Features are extracted from each modal image to obtain image features corresponding to each modal. The content contained in each modal image is also extracted to obtain image content text corresponding to each modal. The image features and image content text corresponding to each modal image are then fused to obtain a first fused feature corresponding to each modal image. Based on the first fused feature corresponding to each modal image, a preset cross-attention algorithm is used to perform cross-attention processing on the first fused features corresponding to the multiple different modal images to obtain corresponding processing results. Based on these processing results, fusion processing is performed to determine the second fused feature corresponding to the multiple different modal images. The fusion feature, based on the second fusion feature corresponding to multiple different modalities of images, determines the biological attack detection results corresponding to multiple different modalities of images. In this way, the raw data of multiple different modalities of images are collected by the multimodal image acquisition component. For each modality of image, the corresponding image content text is extracted first through the image content, and then the image features and image content text are fused together. Then, a cross-attention algorithm is used to perform cross-attention processing on the first fusion feature corresponding to multiple different modalities of images, thereby increasing the weight of the attention region of each feature, making the fusion feature more separable, and making full use of the characteristics between different modalities of images, so as to better adapt to the separability of each modality in the feature space, thereby improving the accuracy of the biological attack detection mechanism and its adaptability in multiple scenarios.

[0161] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0162] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using a hardware physical module. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0163] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0164] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0165] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0166] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] The embodiments described herein are illustrated with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable parallel device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable parallel device, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0168] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable fraud device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0169] These computer program instructions may also be loaded onto a computer or other programmable device in parallel or in series, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable device, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0170] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0171] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0172] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0173] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0174] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0176] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0177] The above description is merely an embodiment of this specification and is not intended to limit this document. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for detecting biological attacks, the method comprising: Acquire images in multiple different modalities for biological attack detection; Feature extraction is performed on the image of each modality to obtain the image features corresponding to each modality. The content contained in the image of each modality is extracted to obtain the image content text corresponding to each modality. The image features and image content text corresponding to each modality are fused to obtain the first fused feature corresponding to each modality. Based on the first fusion feature corresponding to each modality of the image, a preset cross-attention algorithm is used to perform cross-attention processing on the first fusion features corresponding to multiple different modalities of the image to obtain the corresponding processing results; Based on the obtained processing result and the first fusion feature corresponding to the image of the first modality, the correlation information between the processing result and the first fusion feature corresponding to the image of the first modality is determined by a correlation algorithm. The first modality is a modality whose modality separability is higher than a preset threshold among multiple different modalities. The correlation information and the first fusion feature corresponding to the first modality image are fused together to obtain the second fusion feature corresponding to multiple different modalities; Based on the second fusion features corresponding to the images of the various different modalities, the biological attack detection results corresponding to the images of the various different modalities are determined.

2. The method according to claim 1, wherein the multiple different modal images include one or more of RGB modal images, infrared (IR) modal images, and depth images.

3. The method according to claim 2, wherein extracting features from the image of each modality to obtain the image features corresponding to each modality includes: The image of each modality is input into the feature extraction model corresponding to the modality to obtain the image features corresponding to each modality. The feature extraction model is a model built based on one or more ResBlock modules.

4. The method according to claim 3, wherein the first fusion feature corresponding to each modality image is subjected to cross-attention processing using a preset cross-attention algorithm to obtain the corresponding processing result, including: The first fusion feature corresponding to the first modality is obtained from the first fusion features corresponding to images of multiple different modalities; Using the image of the first modality as the image of the base modality, a preset cross-attention algorithm is used to perform cross-attention processing on the first fusion feature corresponding to the image of the first modality and the first fusion feature corresponding to the images of other modalities besides the first modality, respectively, to obtain the corresponding processing results.

5. The method according to claim 4, wherein the image of the first modality includes an image of the RGB modality.

6. The method according to claim 1, wherein determining the biological attack detection result corresponding to the multiple different modalities of the images based on the second fusion features corresponding to the multiple different modalities includes: The second fusion features corresponding to the images of the various different modalities are input into a pre-trained biological attack detection model to obtain biological attack detection results corresponding to the images of the various different modalities. The biological attack detection model is a model constructed based on a preset classification algorithm.

7. The method according to claim 1, wherein extracting the content contained in the image of each modality to obtain the image content text corresponding to each modality includes: The image description generation model extracts the content of images for each modality to obtain the corresponding image content text for each modality. This image description generation model is based on a pre-trained model that includes a Transformer module; or... Based on the image description mapping relationship corresponding to different modalities, the content contained in the image of each modality is extracted to obtain the image content text corresponding to each modality.

8. A method for detecting biological attacks, the method comprising: Acquire images in multiple different modalities for biological attack detection; Based on the feature extraction sub-model in the biological attack detection model, features are extracted from the images of each modality to obtain the image features corresponding to each modality. Based on the image description generation sub-model in the biological attack detection model, the content contained in the images of each modality is extracted to obtain the image content text corresponding to each modality. The image features and image content text corresponding to each modality are fused to obtain the first fused feature corresponding to each modality. Based on the first fusion feature corresponding to each modality of the image, attention processing is performed on the first fusion feature corresponding to multiple different modalities of the image through the preset attention module in the biological attack detection model to obtain the corresponding processing results; Based on the obtained processing result and the first fusion feature corresponding to the image of the first modality, the correlation information between the processing result and the first fusion feature corresponding to the image of the first modality is determined by a correlation algorithm. The first modality is a modality whose modality separability is higher than a preset threshold among multiple different modalities. The correlation information and the first fusion feature corresponding to the first modality image are fused together to obtain the second fusion feature corresponding to multiple different modalities; The second fusion features corresponding to the images of the various different modalities are input into the classification prediction sub-model in the biological attack detection model to obtain the biological attack detection results corresponding to the images of the various different modalities.

9. The method according to claim 8, wherein the attention module is a module constructed based on a cross-attention algorithm, a channel attention algorithm, or a Transformer-based self-attention algorithm, and the feature extraction sub-model is a sub-model constructed based on one or more ResBlock modules.

10. A biological attack detection device, the device comprising: The image acquisition module acquires images of various modalities for biological attack detection. The first fusion module extracts features from the image of each modality to obtain the image features corresponding to each modality, extracts the content contained in the image of each modality to obtain the image content text corresponding to each modality, and fuses the image features and image content text corresponding to each modality to obtain the first fusion feature corresponding to each modality. The second fusion module, based on the first fusion feature corresponding to each modality of the image, performs cross-attention processing on the first fusion feature corresponding to multiple different modalities of the image through a preset cross-attention algorithm to obtain the corresponding processing results; Based on the obtained processing result and the first fusion feature corresponding to the image of the first modality, the correlation information between the processing result and the first fusion feature corresponding to the image of the first modality is determined by a correlation algorithm. The first modality is a modality whose modality separability is higher than a preset threshold among multiple different modalities. The correlation information and the first fusion feature corresponding to the first modality image are fused together to obtain the second fusion feature corresponding to multiple different modalities; The detection module determines the biological attack detection results corresponding to the images of the various different modalities based on the second fusion features.

11. A biological attack detection device, the device comprising: The image acquisition module acquires images of various modalities for biological attack detection. The first model processing module extracts features from the images of each modality based on the feature extraction sub-model in the biological attack detection model to obtain the image features corresponding to each modality. It also extracts the content contained in the images of each modality based on the image description generation sub-model in the biological attack detection model to obtain the image content text corresponding to each modality. Finally, it fuses the image features and image content text corresponding to each modality to obtain the first fused feature corresponding to each modality. The second model processing module, based on the first fusion feature corresponding to each modality of the image, performs attention processing on the first fusion feature corresponding to multiple different modalities of the image through the preset attention module in the biological attack detection model, and obtains the corresponding processing results; Based on the obtained processing result and the first fusion feature corresponding to the image of the first modality, the correlation information between the processing result and the first fusion feature corresponding to the image of the first modality is determined by a correlation algorithm. The first modality is a modality whose modality separability is higher than a preset threshold among multiple different modalities. The correlation information and the first fusion feature corresponding to the first modality image are fused together to obtain the second fusion feature corresponding to multiple different modalities; The model prediction module inputs the second fusion features corresponding to the images of the various different modalities into the classification prediction sub-model in the biological attack detection model to obtain the biological attack detection results corresponding to the images of the various different modalities.

12. A biological attack detection device, the biological attack detection device comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to: Acquire images in multiple different modalities for biological attack detection; Feature extraction is performed on the image of each modality to obtain the image features corresponding to each modality. The content contained in the image of each modality is extracted to obtain the image content text corresponding to each modality. The image features and image content text corresponding to each modality are fused to obtain the first fused feature corresponding to each modality. Based on the first fusion feature corresponding to each modality of the image, a preset cross-attention algorithm is used to perform cross-attention processing on the first fusion features corresponding to multiple different modalities of the image to obtain the corresponding processing results; Based on the obtained processing result and the first fusion feature corresponding to the image of the first modality, the correlation information between the processing result and the first fusion feature corresponding to the image of the first modality is determined by a correlation algorithm. The first modality is a modality whose modality separability is higher than a preset threshold among multiple different modalities. The correlation information and the first fusion feature corresponding to the first modality image are fused together to obtain the second fusion feature corresponding to multiple different modalities; Based on the second fusion features corresponding to the images of the various different modalities, the biological attack detection results corresponding to the images of the various different modalities are determined.

13. A biological attack detection device, the biological attack detection device comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to: Acquire images in multiple different modalities for biological attack detection; Based on the feature extraction sub-model in the biological attack detection model, features are extracted from the images of each modality to obtain the image features corresponding to each modality. Based on the image description generation sub-model in the biological attack detection model, the content contained in the images of each modality is extracted to obtain the image content text corresponding to each modality. The image features and image content text corresponding to each modality are fused to obtain the first fused feature corresponding to each modality. Based on the first fusion feature corresponding to each modality of the image, attention processing is performed on the first fusion feature corresponding to multiple different modalities of the image through the preset attention module in the biological attack detection model to obtain the corresponding processing results; Based on the obtained processing result and the first fusion feature corresponding to the image of the first modality, the correlation information between the processing result and the first fusion feature corresponding to the image of the first modality is determined by a correlation algorithm. The first modality is a modality whose modality separability is higher than a preset threshold among multiple different modalities. The correlation information and the first fusion feature corresponding to the first modality image are fused together to obtain the second fusion feature corresponding to multiple different modalities; The second fusion features corresponding to the images of the various different modalities are input into the classification prediction sub-model in the biological attack detection model to obtain the biological attack detection results corresponding to the images of the various different modalities.

Images