Multi-modal adversarial processing method and device for low-resource secure devices, electronic device, readable medium

Abstract

The application discloses a multi-modal confrontation processing method and device for a low-resource security device, an electronic device and a readable medium. The method comprises the following steps: determining whether an abnormal attack exists on a signal source according to real-time multi-source data of the acquired multi-modal; generating abnormal attack detection information when it is determined that at least one signal source is subjected to an abnormal attack; when the abnormal attack detection information indicates that an abnormal attack exists, performing a control step on the unmanned forklift: determining a park location; controlling the unmanned forklift to stop moving based on the park location supporting staying within a target time length; packaging the evidence file to upload to a park intelligent platform to obtain attack processing response information; and controlling the unmanned forklift to move according to the attack processing response information. Through the two-level security anomaly detection method corresponding to the unmanned forklift, the application can effectively confront abnormal signal source data in the multi-modal mode, so that the unmanned forklift can avoid being subjected to an abnormal attack on the signal source during movement.

Description

Technical Field

[0001] The embodiments disclosed in this invention relate to the field of computer technology, specifically to a multimodal adversarial processing method and apparatus for low-resource security devices, electronic devices, and computer-readable media. Background Technology

[0002] Currently, with the continuous development of artificial intelligence, the deployment of intelligent equipment (such as unmanned forklifts) in smart parks (e.g., logistics parks) has become one of the main development directions. For handling anomalies in unmanned forklifts, the common approach is to utilize the anomaly handling module deployed on the unmanned forklift (i.e., a module equipped with an anomaly detection model based on multi-source data) to handle any abnormal issues that occur.

[0003] However, when using the aforementioned method, the following technical problems often arise:

[0004] For the anomaly detection and handling of unmanned forklifts, it is necessary to send data from multiple sources to the anomaly handling module in real time. This results in a large number of data communication operations, which can affect the handling of anomalies when they occur. In the event of anomaly attacks on sources during operation, timely processing is not possible. Summary of the Invention

[0005] The purpose of this invention is to solve one or more of the technical problems mentioned in the background section above, and to provide a multimodal countermeasure processing method and apparatus, electronic device, and computer-readable medium for low-resource security devices. The multimodal countermeasure processing method for low-resource security devices disclosed in this invention, through a two-level security anomaly detection method corresponding to the unmanned forklift, can effectively counteract abnormal information source data under multimodal conditions, thereby preventing the unmanned forklift from being attacked by abnormal information sources during its movement.

[0006] In a first aspect, the present invention discloses a multimodal adversarial processing method for low-resource security devices, comprising:

[0007] Based on the real-time multimodal multi-source data acquired by multiple low-resource security devices under a target time in a non-extreme environment, it is determined whether there is an abnormal attack on the information source. Each low-resource security device is a low-resource information source processing sensor that supports security detection under the corresponding information source and is installed on the unmanned forklift.

[0008] When it is determined that at least one information source is under abnormal attack, the multi-source abnormal attack detection model corresponding to the unmanned forklift is used to generate abnormal attack detection information based on the at least one information source and the real-time multi-source data.

[0009] When an abnormal attack is detected based on the abnormal attack detection information, the following control steps are executed for the unmanned forklift: determine the location of the unmanned forklift in the current park environment; based on the park location supporting a stop within a target duration, control the unmanned forklift to stop moving; package the real-time multi-source data, abnormal attack detection information, and the corresponding stop information of the unmanned forklift into an evidence file and upload it to the park's intelligent platform to obtain attack handling response information; and control the unmanned forklift to move according to the attack handling response information.

[0010] Secondly, the present invention discloses a multimodal countermeasure processing device for low-resource security devices, comprising:

[0011] The determination unit is configured to determine whether an information source is under abnormal attack based on real-time multimodal multi-source data acquired by multiple low-resource security devices under a target time in a non-extreme environment. Each low-resource security device is a low-resource information source processing sensor that supports security detection under the corresponding information source and is installed on an unmanned forklift.

[0012] The generation unit is configured to, when it is determined that at least one information source is under abnormal attack, use the multi-source abnormal attack detection model corresponding to the unmanned forklift to generate abnormal attack detection information based on the at least one information source and the real-time multi-source data.

[0013] The execution unit is configured to, when an abnormal attack is detected based on the abnormal attack detection information, perform the following control steps for the unmanned forklift: determine the location of the unmanned forklift in the current park environment; control the unmanned forklift to stop moving based on the park location supporting a target duration of stay; package the real-time multi-source data, abnormal attack detection information, and the corresponding stay information of the unmanned forklift into an evidence file and upload it to the park's intelligent platform to obtain attack handling response information; and control the unmanned forklift to move according to the attack handling response information.

[0014] Thirdly, the present invention discloses an electronic device including one or more processors and a storage device, wherein one or more programs are stored on the storage device, and when one or more programs are executed by one or more processors, the one or more processors implement the multimodal adversarial processing method as described in any one aspect.

[0015] Fourthly, the present invention discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multimodal adversarial processing method as described in any one of the first aspects.

[0016] This invention discloses a multimodal countermeasure processing method for low-resource security devices. First, based on real-time multimodal multi-source data acquired by multiple low-resource security devices at a target time under non-extreme environments, it determines whether any information sources are under abnormal attack. Each low-resource security device is a low-resource information source processing sensor, mounted on an unmanned forklift, supporting security detection at its corresponding information source. By setting up multiple low-resource security devices on the unmanned forklift, first-level security anomaly detection can be achieved for each information source. This allows for timely attack countermeasures and anomaly handling when an abnormal attack occurs at a particular information source, eliminating the need to constantly send information source data from multiple sources to the corresponding anomaly handling module for real-time detection.

[0017] Then, when it is determined that at least one information source is under abnormal attack, the multi-source abnormal attack detection model corresponding to the unmanned forklift can accurately generate abnormal attack detection information based on the at least one information source and the real-time multi-source data. In summary, when various low-resource security devices detect abnormal attacks on their respective information sources, the multi-source abnormal attack detection model can integrate the current information source data from each source to achieve accurate detection of abnormal attacks. Therefore, by using low-resource security devices for source attack detection and the multi-source abnormal attack detection model for overall multi-source attack detection, two levels of security protection and timely anomaly handling can be achieved.

[0018] Finally, based on the anomaly detection information indicating the presence of an anomaly attack, the following control steps are executed for the unmanned forklift: determining the unmanned forklift's location within the current park environment; controlling the unmanned forklift to stop moving based on the park location's ability to remain stationary for a target duration; packaging the real-time multi-source data, anomaly detection information, and the unmanned forklift's corresponding stationary information into an evidence file and uploading it to the park's intelligent platform to obtain attack handling response information; and controlling the unmanned forklift to move according to the attack handling response information. Here, by executing these control steps, the park aims to prevent property damage caused by abnormal behavior of the unmanned forklift when attacked. Furthermore, by utilizing the park's intelligent platform and leveraging the intelligent decision-making capabilities of the target park, accurate generation of attack handling response information is achieved, enabling efficient handling of anomalies related to the unmanned forklift.

[0019] In summary, by implementing a two-tiered safety anomaly detection method for unmanned forklifts, effective countermeasures against anomalous data sources in multimodal environments can be achieved, thus preventing unmanned forklifts from being attacked by anomalous data sources during operation. Attached Figure Description

[0020] The described and other features, advantages, and aspects of the various embodiments disclosed herein will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0021] Figure 1 This is a flowchart of some embodiments of the multimodal adversarial processing method for low-resource security devices disclosed in this invention;

[0022] Figure 2 This is a schematic diagram of the structure of some embodiments of the multimodal countermeasure processing device for low-resource security equipment disclosed in this invention;

[0023] Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present invention. Detailed Implementation

[0024] The embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments disclosed herein are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0025] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this invention can be combined with each other.

[0026] It should be noted that the concepts of "first" and "second" mentioned in this invention are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0027] It should be noted that the terms "a" and "a plurality of" used in this invention disclosure are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0028] The names of messages or information exchanged between multiple devices in the embodiments of this invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0029] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0030] refer to Figure 1 The diagram illustrates a flow 100 of some embodiments of a multimodal adversarial processing method for low-resource security devices disclosed in this invention. This multimodal adversarial processing method for low-resource security devices includes the following steps:

[0031] Step S101: Based on the real-time multi-modal multi-source data acquired by multiple low-resource security devices under a non-extreme environment at a target time, determine whether there is an abnormal attack on the information source; wherein, each low-resource security device is a low-resource information source processing sensor that supports security detection under the corresponding information source and is installed on the unmanned forklift.

[0032] In some embodiments, after acquiring real-time multi-source data from multiple low-resource security devices under a non-extreme environment at a target time, the executing entity (e.g., an electronic device) of the multi-modal adversarial processing method for low-resource security devices can determine whether an information source is under abnormal attack based on the real-time multi-source data. Here, the non-extreme environment can be non-extreme weather. For example, non-extreme weather can be foggy weather, heavy rain weather, or other weather with low visibility. The low-resource security devices correspond to the information source processing sensors in unmanned forklifts with limited original computing resources, used for security control. Limited computing resources are due to the fact that, in the context of unmanned forklifts, the original information source processing sensors are terminal sensors deployed to acquire information source data for safe operation. For example, for unmanned forklifts operating within a park, the corresponding information source processing sensors may include: lidar sensors and vision sensors. LiDAR sensors have corresponding deployed abnormal attack detection models. Similarly, vision sensors have deployed abnormal attack detection models. Here, abnormal attack detection under the corresponding information source is achieved by deploying corresponding chips that do not require high computing power on each sensor. In practice, each chip can be a low-power and simple chip, and can be an edge microcontroller (MCU) equipped with a lightweight AI accelerator. Specifically, for vision sensors, chips with extremely small die size (3.4×3.4mm) and power consumption of 10mW at a working speed of up to 240 frames per second can be used. For LiDAR, STM32H7 series chips are recommended. In practice, the signal source for LiDAR sensors is the LiDAR signal source, and the corresponding signal source data is the acquired LiDAR data. The signal source for vision sensors is the visual signal source, and the corresponding signal source data is the acquired visual video. The target time can be the current movement time of the unmanned forklift. It should be noted here that the park where the unmanned forklift operates is one with low requirements for its functions. Specifically, the unmanned forklift is a forklift with automatic navigation and automatic cargo handling, which does not require highly complex intelligence. Multimodal operation can include: video mode and LiDAR mode. Real-time multi-source data can be a collection of data from various sources obtained by an unmanned forklift through various low-resource safety devices at the current time.

[0033] In practice, real-time multi-source data can include visual video and LiDAR data. Anomalies affecting these data sources can occur during data acquisition due to interference from interfering objects, leading to safety risks for unmanned forklifts. For example, for visual data sources, anomalies might involve attaching special patterns to the ground or shelves to interfere with visual navigation, or shining a focused laser at the camera, causing localized overexposure or permanent damage. For LiDAR data sources, anomalies might involve using a high-power laser emitter to emit synchronized fake pulse signals to the LiDAR, generating false obstacles in the point cloud, or modifying the real point cloud data with minor perturbations (e.g., adding a small amount of noise points), causing the model to identify pedestrians as thin air or shelves as aisles.

[0034] As an example, the executing entity can utilize multiple anomaly detection models in multiple low-resource security devices to generate anomaly detection results for corresponding source data, thereby determining whether any source has been attacked. Here, there is a one-to-one correspondence between the multiple anomaly detection models and the multiple sources. The anomaly detection models can be lightweight deep learning models (e.g., convolutional layers with a few concatenated layers) or machine learning models.

[0035] In some optional implementations of certain embodiments, the real-time multi-source data includes: multiple source data corresponding to multiple information sources. Each information source has unique corresponding source data. The source data may include: information received by the unmanned forklift on the corresponding information source and the information source sending object. That is, the information source sending object may be the information source object.

[0036] Optionally, the executing entity can determine whether an information source has been subjected to an abnormal attack based on the real-time multi-source data, including the following steps:

[0037] The first step involves using pre-defined alignment rules for information alignment between multiple information sources to determine if any sources are under abnormal attack. These alignment rules determine whether alignment is possible between different information sources. In practice, alignment rules can be generated based on historical experience. For example, an alignment rule could be "the detection orientation of visual inspection and LiDAR detection of a physical object is the same," or "the detection contour of visual inspection and LiDAR detection of a physical object is the same." It's important to note that these alignment rules are personalized for specific information source types. These alignment rules can be personalized rules set by technical personnel on the multi-source processing page. In practice, the multi-source processing page can be a page for configuring information related to information sources. In practice, the multi-source processing page can support various operations such as adding, deleting, modifying, and querying information sources, and it can also support rule configuration operations between at least one information source.

[0038] As an example, alignment rules are used to determine whether multiple sources in real-time multi-source data experience content conflicts. If a content conflict is confirmed, it indicates that one source has been subjected to an anomalous attack. If no content conflict is confirmed, it indicates that no source has been subjected to an anomalous attack.

[0039] The second step, when it is determined that at least one information source has been subjected to an abnormal attack, is to perform the following verification steps for each of the at least one information source:

[0040] Sub-step 1: Obtain the anomaly information corresponding to the source output by the alignment rule. The anomaly information can be the verification result after the alignment rule performs source anomaly verification. That is, based on the alignment rule, it outputs whether the source has an anomaly. In practice, the anomaly information may include: an anomaly identifier indicating whether an anomaly exists, the anomaly category, and the anomaly detection time. The anomaly category can be the anomaly type corresponding to the source. The anomaly detection time can be the generation time corresponding to the anomaly information.

[0041] Sub-step 2 involves determining the anomaly category sequence corresponding to the information source based on the information source data corresponding to the aforementioned information source and at least one information source data corresponding to at least one information source that has not been subjected to anomaly attacks, using at least one attack identification machine learning model corresponding to at least one candidate anomaly category. The at least one candidate anomaly category refers to various anomaly categories corresponding to the anomaly information.

[0042] Sub-step 3: If the overall confidence level corresponding to the abnormal category sequence is higher than the first confidence level, it is determined that the information source has been subjected to an abnormal attack.

[0043] Sub-step 4: If the overall confidence level corresponding to the abnormal category sequence is not higher than the first confidence level but higher than the second confidence level, generate abnormal attack confirmation information for the information source; the first confidence level is higher than the second confidence level.

[0044] The third step is to determine that the information source has not been subjected to an abnormal attack if the overall confidence level corresponding to the abnormal category sequence is not higher than the second confidence level.

[0045] Optionally, the steps also include:

[0046] Based on the information obtained regarding the various abnormal attacks to be confirmed, determine whether each corresponding information source has been subjected to an abnormal attack.

[0047] Step S102: When it is determined that at least one information source is under abnormal attack, the multi-source abnormal attack detection model corresponding to the unmanned forklift is used to generate abnormal attack detection information based on the at least one information source and the real-time multi-source data.

[0048] In some embodiments, when it is determined that at least one information source is under an abnormal attack, the executing entity can utilize the multi-source abnormal attack detection model corresponding to the unmanned forklift to generate abnormal attack detection information based on the at least one information source and the real-time multi-source data. The multi-source abnormal attack detection model can detect whether an abnormal attack exists under at least one labeled information source based on data from multiple information sources. In practice, the input to the multi-source abnormal attack detection model can be multi-source data (i.e., source data corresponding to multiple information sources), and the corresponding output is the attack detection result indicating whether at least one information source is under an abnormal attack. It should be noted that the multi-source abnormal attack detection model can support diverse information source data inputs. That is, the abnormal attack detection information here can include: the detection result indicating whether at least one information source is under an abnormal attack.

[0049] For example, a multi-source anomaly attack detection model may include: an input layer corresponding to the source data, an encoding layer that encodes the source data for each source, a fusion layer that fuses multimodal source information, and output layers for each source. The input layer can be a network layer that inputs the source data. The input layer also supports word embedding processing of the source data to convert it into corresponding vector information. Each source has a corresponding encoding layer used to encode the source data to extract key semantic content. In practice, the input layer may include a data input layer and multiple parallel encoding layers. Each parallel encoding layer has a one-to-one correspondence with each source. Each encoding layer is a network layer used to encode the source data. In practice, different sources have different encoding layers. For example, for a source corresponding to a LiDAR sensor, the corresponding encoding layer could be a point-level feature encoding layer. For example, a point-level feature encoding layer can use a multilayer perceptron (MLP) to map the original attributes (X, Y, Z, Intensity, etc.) of each point to a high-dimensional feature space, and then aggregate global information through global pooling (such as Max Pooling). For the information sources corresponding to a visual sensor, the corresponding encoding layer could be an 11-layer concatenated convolutional layer. The fusion layer can be a network layer that fuses the key semantic content of each information source. In practice, the fusion layer can be a network layer based on a multi-head attention mechanism (e.g., the multi-head attention mechanism in the Transformer architecture). Here, the fusion layer can extract attention based on the common data corresponding to each information source, performing comprehensive source semantic content attention extraction. The multi-head attention mechanism network layer can include: a linear projection layer, a parallel scaling dot product attention layer, a concatenation layer, and a final linear projection layer. The linear projection layer can be a network layer that linearly projects the input data into corresponding query, key, and value matrices. Specifically, the linear projection layer can be three independent fully connected layers. The parallel scaling dot product attention layer can include: similarity calculation, scaling and normalization processing, and weighted summation processing. The fusion layer outputs a vector that fuses the semantic content of multi-source data. There is a one-to-one correspondence between each output layer and each source. Each output layer outputs an attack detection result indicating whether an anomalous attack exists in its corresponding source. Each output layer can be a fully connected layer with different parameter forms.

[0050] As an example, the executing entity can input real-time multi-source data into the multi-source anomaly attack detection model to obtain attack detection results corresponding to each source. Then, it selects at least one attack detection result corresponding to at least one source from the various attack detection results as anomaly attack detection information.

[0051] In some optional implementations of certain embodiments, the executing entity may utilize the multi-source anomaly attack detection model corresponding to the unmanned forklift to generate anomaly attack detection information based on the at least one information source and the real-time multi-source data, including the following steps:

[0052] First, for each of the at least one information source, perform the following generation steps:

[0053] Sub-step 1 involves acquiring multiple intelligent devices that support data communication between the unmanned forklift and the data source corresponding to the information source. Data permissions can be permissions that support data communication. The multiple intelligent devices can be various intelligent devices that assist the unmanned forklift in determining the source data through anomaly attacks. For example, intelligent devices can be intelligent loading and unloading vehicles, intelligent lights, video surveillance equipment, four-way shuttle vehicles, unmanned cleaning vehicles, and unmanned security vehicles.

[0054] As an example, firstly, a list of devices supporting data communication with the data source corresponding to the unmanned forklift at the target time is obtained. Then, multiple intelligent devices currently in operation are extracted from the device list.

[0055] Sub-step 2 involves obtaining a set of target intelligent devices from among the multiple intelligent devices, based on the location of the unmanned forklift within the park, where there is a proximity relationship between them. The target intelligent devices can be intelligent devices within an area centered on the location of the unmanned forklift within the park, with a target distance as the radius. The target distance can be a pre-set distance, for example, 5 meters.

[0056] Sub-step 3 involves sending the source-specific abnormal attack confirmation information to the target intelligent device set to obtain at least one abnormal attack confirmation message. The source-specific abnormal attack confirmation message can be a confirmation request to confirm whether there is an abnormal attack on the source data corresponding to the source. The abnormal attack confirmation message can also be the confirmation result of the target intelligent device after confirming the attack in response to the confirmation request corresponding to the source-specific abnormal attack confirmation message. Each target intelligent device has corresponding abnormal attack confirmation information.

[0057] As an example, the executing entity can bring the source data corresponding to the source into the source abnormal attack confirmation template to obtain source abnormal attack confirmation information, and then send it to the target intelligent device set.

[0058] Sub-step 4: Based on the at least one abnormal attack confirmation information, generate preliminary determination information regarding whether the information source is subject to an abnormal attack. This preliminary determination information characterizes the preliminary determination result of whether the information source has been subjected to an abnormal attack.

[0059] As an example, the executing entity can combine at least one abnormal attack confirmation message to obtain preliminary confirmation information.

[0060] The second step involves generating anomaly detection information based on at least one preliminary determination information, the at least one information source, and the real-time multi-source data, using the multi-source anomaly attack detection model corresponding to the unmanned forklift.

[0061] As an example, firstly, for each piece of preliminary information, if all the abnormal attack confirmation information in the preliminary information indicates that there is no abnormality, then the corresponding information source is determined as the target information source. Then, each target information source is removed from at least one information source to obtain the remaining information sources. Next, using the remaining information sources as at least one information source, and based on at least one information source and real-time multi-source data, an abnormal attack detection information is generated using the multi-source abnormal attack detection model corresponding to the unmanned forklift.

[0062] In some optional implementations of certain embodiments, the executing entity may generate anomaly attack detection information based on at least one preliminary determination information, the at least one information source, and the real-time multi-source data, using the multi-source anomaly attack detection model corresponding to the unmanned forklift, including the following steps:

[0063] The first step, for each piece of initially determined information, is to perform the following processing steps:

[0064] Sub-step 1: Based on the preliminary information, determine the anomaly ratio of the corresponding information source. The corresponding information source can be the information source corresponding to the preliminary information. The anomaly ratio can be the proportion of confirmation information indicating the existence of an anomaly attack among the various anomaly attack confirmation information in the preliminary information.

[0065] As an example, firstly, the number of first messages indicating the presence of an anomalous attack and the number of second messages indicating the absence of an anomalous attack are determined in each anomalous attack confirmation message. Then, the ratio of the number of first messages to the number of second messages is determined as the anomalous ratio.

[0066] Sub-step 2 involves acquiring at least one source data corresponding to at least one target intelligent device associated with the information source. Here, the at least one target intelligent device refers to any intelligent device confirmed to be subject to an abnormal attack. There is a one-to-one correspondence between the target intelligent devices and the source data in the at least one source data set. The specific implementation details are omitted here.

[0067] Sub-step 3 involves setting at least one data weight corresponding to the at least one source data based on the anomaly ratio. Each source data has a corresponding data weight. The data weight characterizes the importance of the source data. The data weight can be a value between 0 and 1. A higher value indicates greater data importance.

[0068] As an example, firstly, for each source data, the probability information representing the existence of an anomalous attack corresponding to the source data is determined. Then, the probability information and the probability of the source having anomalies corresponding to the anomalous proportion (which can be the result of dividing the first information number by the sum of the first information number and the second information number) are multiplied to obtain a multiplication result. Then, at least one multiplication result is normalized to obtain at least one data weight.

[0069] Sub-step 4 involves fusing the at least one data source data and the corresponding source data from the real-time multi-source data according to the at least one data weight, to obtain fused data. The fused data may be the fused content of various source data from multiple intelligent devices (including unmanned forklifts) at corresponding source sources.

[0070] As an example, the executing entity can perform data fusion on the at least one source data and the corresponding source data in the real-time multi-source data according to at least one data weight and the weight 1 of the source data corresponding to the unmanned forklift, to obtain fused data.

[0071] The second step involves generating anomaly detection information based on at least one fused data, the at least one information source, and the real-time multi-source data, using the multi-source anomaly attack detection model corresponding to the unmanned forklift.

[0072] As an example, firstly, the executing entity can replace at least one source data corresponding to at least one source in the real-time multi-source data with at least one fused data to obtain the replaced multi-source data. Then, based on the replaced multi-source data and the at least one source, anomaly attack detection information is generated using the multi-source anomaly attack detection model corresponding to the unmanned forklift.

[0073] In some optional implementations of certain embodiments, the multi-source anomaly attack detection model includes: multiple lightweight source feature processing layers, multiple lightweight source feature cross-layers, a lightweight intermediate feature processing layer, and an anomaly attack detection information output layer. There is a one-to-one correspondence between the multiple lightweight source feature processing layers and the individual sources. Each source feature processing layer can be a network layer used to perform data feature processing on the input data corresponding to the source. Here, the network structure of the corresponding source feature processing layer differs depending on the source. The data feature processing can involve vectorizing and encoding the source data. Specifically, for example, for a source corresponding to a LiDAR sensor, the corresponding source feature processing layer can be a point-level feature encoding layer. For a source corresponding to a vision sensor, the corresponding source feature processing layer can be an 11-layer concatenated convolutional layer. Alternatively, the source feature processing layer can be the extracted network layers from a heavyweight source feature processing layer. That is, the heavyweight source feature processing layer can include multiple network layers, while the lightweight source feature processing layer can be the result of extracting key network layers from the heavyweight source feature processing layer. For example, the heavyweight source feature processing layer includes a complete point-level feature encoding layer, and the corresponding lightweight source feature processing layer can include at least one key perceptron layer from the complete point-level feature encoding layer. The source feature cross layer can be a network layer that performs feature cross processing on at least one source feature processing result corresponding to at least one source feature processing layer. In practice, the source feature processing result can be the output feature information corresponding to the source feature processing layer. Feature cross processing can be a processing operation that extracts attention feature information from at least one source feature processing result. In practice, the source feature cross layer can be the attention mechanism layer in the Transformer architecture. Different source feature cross layers have multiple corresponding sources, and the corresponding structure can be an attention mechanism layer, with different model training parameters. The lightweight source feature cross layer can be the result extracted from the heavyweight source feature cross layer after extracting key network layers. Further details will not be elaborated further. The key network here can be selected by relevant technicians during the model architecture construction process. The intermediate feature processing layer can be a network layer that processes the cross-results of features from various information sources. Here, the intermediate feature processing layer can be a feature processing layer that integrates the outputs of the cross-results of various information source features to generate the overall semantic content of the information source. For example, the intermediate feature processing layer can be a multi-layered convolutional layer or a multi-layered downsampling layer. The lightweight intermediate feature processing layer can be interpreted as the extraction result after extraction from the heavyweight intermediate feature processing layer. The anomaly attack detection information output layer can be a network layer that outputs anomaly attack detection information. For example, the anomaly attack detection information output layer can be a fully connected layer. Optionally, the multi-source anomaly attack detection model is trained through the following steps:

[0074] The first step is to obtain the first and second adversarial training datasets for multiple sources. Here, the first and second adversarial training datasets are training datasets with the same data format and essentially the same content. The first adversarial training dataset is used to train a heavyweight initial multi-source attack detection model. The second adversarial training dataset is used to train a lightweight initial multi-source attack detection model. Their purposes differ. For example, the first adversarial training data can include: multiple source data, anomalies corresponding to the multiple source data, anomaly categories corresponding to the multiple source data, and an overall anomaly status. In practice, multiple source data can be multiple data points corresponding to multiple sources. Multiple source data can be collected or generated for the same object in the same environment. Anomalies can be a first anomaly label indicating whether anomalies exist in the multiple source data. Each source data point has a corresponding first anomaly label. The first anomaly label can be "00" or "01". When it is "00", it can indicate that the source data has anomalies. When it is "01", it can indicate that the source data does not have anomalies. The overall anomaly status can be the multi-source anomaly attack detection results.

[0075] The second step involves training a heavyweight initial multi-source attack detection model deployed at the edge of the target campus, based on the first adversarial training dataset, to obtain a multi-source attack detection model. This model includes multiple heavyweight source feature processing layers, multiple heavyweight source feature cross-processing layers, a heavyweight intermediate feature processing layer, and an anomaly attack detection information output layer. Each lightweight source feature processing layer is a network layer after key layer extraction from each network layer in the heavyweight source feature processing layer; each lightweight source feature cross-processing layer is a network layer after key layer extraction from each network layer in the heavyweight source feature cross-processing layer; and each lightweight intermediate feature processing layer is a network layer after key layer extraction from each network layer in the heavyweight intermediate feature processing layer. The model training method can be a conventional method, such as backpropagation. The initial multi-source attack detection model may be a multi-source attack detection model that has not yet finished training. The edge of the target campus may be an edge terminal deployed within the target campus for performing relevant computational tasks.

[0076] The third step involves obtaining, based on the multi-source attack detection model, the first network parameters corresponding to the multiple lightweight source feature processing layers, the second network parameters corresponding to the multiple lightweight source feature cross layers, the third network parameters corresponding to the lightweight intermediate feature processing layer, and the fourth network parameters corresponding to the abnormal attack detection information output layer.

[0077] The fourth step is to substitute the first network parameter, the second network parameter, the third network parameter, and the fourth network parameter into the initial multi-source anomaly attack detection model to obtain the preliminary multi-source anomaly attack detection model.

[0078] Fifth, based on the second adversarial training dataset, train the preliminary multi-source anomaly attack detection model to obtain the full multi-source anomaly attack detection model. Specific training methods will not be detailed here; conventional model training methods can be used.

[0079] In some optional implementations of some embodiments, the second adversarial training data in the second adversarial training dataset includes: multiple source data corresponding to multiple sources and anomaly attack detection labels.

[0080] Optionally, training the preliminary multi-source anomaly attack detection model based on the second adversarial training dataset to obtain the multi-source anomaly attack detection model includes the following steps:

[0081] The first step involves randomly removing at least one source data point from each set of second adversarial training data multiple times to obtain removed training data. This removed training data can be training data lacking a specific source. For example, the multiple source data points in the second adversarial training data might include: source data 1 corresponding to source 1, source data 2 corresponding to source 2, and source data 3 corresponding to source 3. Based on the randomly removed source data being source data 1, the removed training data would include: source data 2 corresponding to source 2 and source data 3 corresponding to source 3.

[0082] The second step is to combine the obtained training dataset (after removing the training dataset) with the second adversarial training dataset to obtain the training dataset.

[0083] Here, the training dataset and the second adversarial training dataset are combined to improve the ability of the subsequent multi-source anomaly attack detection model to identify source data with missing sources, thus avoiding the inability to identify subsequent anomalies due to the absence of a certain source.

[0084] The third step involves performing the following training steps for the target training data obtained from the training dataset:

[0085] Sub-step 1: Determine the source data and source labels corresponding to the target training data.

[0086] Sub-step 2: Based on the source identifier, perform the following network layer filtering steps:

[0087] The first sub-step involves selecting the corresponding source feature processing layers from the multiple source feature processing layers in the preliminary multi-source anomaly attack detection model.

[0088] The second sub-step involves selecting the corresponding source feature cross-layers from the multiple source feature cross-layers in the preliminary multi-source anomaly attack detection model.

[0089] Sub-step 3 involves inputting the data from each information source into the corresponding information source feature processing layer to obtain the information source feature processing information.

[0090] Sub-step 4 involves inputting the feature processing information of each source into the corresponding source feature cross-layer to obtain the cross-information of each source feature.

[0091] Sub-step 5 involves superimposing the cross-information of the various information source features into a lightweight intermediate feature processing layer to obtain intermediate anomaly attack feature information.

[0092] Sub-step 6: Input the intermediate abnormal attack feature information into the abnormal attack detection information output layer to obtain the first initial attack detection information.

[0093] Sub-step 7: Based on the first initial attack detection information and the corresponding abnormal attack detection label, determine whether the preliminary multi-source abnormal attack detection model has finished training.

[0094] Sub-step 8: At the end of training, the preliminary multi-source anomaly attack detection model is determined as a multi-source anomaly attack detection model.

[0095] Fourth, before training is complete, the model parameters of the initial multi-source anomaly detection model are updated based on the first initial attack detection information and the corresponding anomaly attack detection labels to obtain the updated model. Here, before the model training is complete, the loss value can be determined based on the first initial attack detection information and the corresponding anomaly attack detection labels, and the model parameters of the initial multi-source anomaly attack detection model can be updated based on the loss value to improve the accuracy of the model.

[0096] The fifth step involves using the updated model as the initial multi-source anomaly detection model and the reselected training data as the target training data to continue the training process. The reselected training data can be selected from the existing training dataset for training the initial multi-source anomaly detection model in the next stage.

[0097] Optionally, determining whether the preliminary multi-source anomaly attack detection model has finished training based on the first initial attack detection information and the corresponding anomaly attack detection label includes the following steps:

[0098] The first step is to perform the second processing step for each source feature intersection:

[0099] Sub-step 1: Extract the intermediate feature information corresponding to the source feature cross information from the intermediate anomaly attack feature information.

[0100] Sub-step 2 involves inputting the intermediate feature information into the target local anomaly attack detection information output layer to obtain the second initial attack detection information. The target local anomaly attack detection information output layer is an attack detection information output layer corresponding to at least one source based on the source feature cross-information. Specifically, the attack detection information output layer can be an output layer indicating whether an abnormal attack situation exists in the source. In practice, the attack detection information output layer can be a multi-layered convolutional layer plus a fully connected layer. The multiple convolutional layers plus fully connected layers are constructed based on the source feature cross-information. The specific construction method can be set by relevant technical personnel.

[0101] Sub-step 3 involves generating an initial local source loss based on the second initial attack detection information and the anomaly attack detection label. The initial local source loss characterizes the anomalous semantic difference between the second initial attack detection information and the anomaly attack detection label. A higher initial local source loss value corresponds to a greater anomalous semantic difference. The initial local source loss can be a pre-generated local source loss.

[0102] As an example, the second initial attack detection information and the abnormal attack detection label can be substituted into the cross-loss function to obtain the initial local source loss.

[0103] Sub-step 4 involves obtaining the source contribution degree of at least one source corresponding to the source feature cross-information for the attack category corresponding to the anomaly attack detection label. The source contribution degree can include at least one contribution degree corresponding to at least one source. Each source has a corresponding contribution degree. The contribution degree can be a value between 0 and 1. The higher the value, the greater the contribution of the source data of the corresponding source to the information of the attack category corresponding to the anomaly attack detection label. That is, in the process of determining the attack category, the semantic contribution degree of the source data corresponding to the source to the attack category.

[0104] As an example, firstly, at least one source data corresponding to at least one source is identified. Then, for each source data in the at least one source data set, the source data is input into an attack category classification algorithm (e.g., the attack category classification algorithm could be an SVM algorithm) to obtain the probability of various attack types corresponding to various attack types. Then, the attack type probability corresponding to the attack category is selected from the various attack type probabilities and used as the source contribution.

[0105] Sub-step 5: Generate local source loss based on the source contribution and the initial local source loss.

[0106] As an example, firstly, the average contribution value corresponding to at least one of the source contribution values ​​is determined. Then, the average contribution value is weighted with the initial local source loss to obtain the local source loss.

[0107] The second step is to generate a complete source loss based on the first initial attack detection information and the abnormal attack detection label. The complete source loss characterizes the semantic content of the information between the first initial attack detection information and the abnormal attack detection label.

[0108] As an example, the initial attack detection information and the abnormal attack detection label can be input into the cross-loss function to obtain the complete source loss.

[0109] The third step is to determine whether the preliminary multi-source anomaly attack detection model has finished training, based on the complete source loss and the obtained local source losses.

[0110] As an example, firstly, the loss from the complete source and the losses from each local source are weighted to obtain a weighted loss value. Then, when the sequence of loss values ​​corresponding to the weighted loss values ​​converges, the initial multi-source anomaly detection model training is considered complete. When the sequence of loss values ​​corresponding to the weighted loss values ​​fails to converge, the initial multi-source anomaly detection model training is considered incomplete.

[0111] Step S103: When the abnormal attack detection information indicates the presence of an abnormal attack, for the unmanned forklift, execute the following control steps:

[0112] Step S1031: Determine the location of the unmanned forklift in the current park environment.

[0113] In some embodiments, the executing entity can determine the location of the unmanned forklift within the current park environment. The park location can be the real-time location of the unmanned forklift within the target park. The current park environment can be the park environment within the target park where the unmanned forklift is located.

[0114] As an example, the location of the unmanned forklift in the park can be determined using a positioning system.

[0115] Step S1032: Based on the park location supporting the target duration of stay, control the unmanned forklift to stop moving.

[0116] In some embodiments, based on the location within the park, the execution entity can control the unmanned forklift to stop moving within a target duration. The target duration for stopping can be the target time for the unmanned forklift to remain stationary. In practice, the target duration can be the processing time for the park's intelligent platform to handle anomalies related to the unmanned forklift. That is, within the park, because unmanned forklifts are relatively dangerous equipment, to avoid economic losses and damage to the forklifts, when an abnormal attack is detected, the forklift can remain stationary while waiting for the park's intelligent platform to provide relevant anomaly handling methods.

[0117] Step S1033: Package the real-time multi-source data, abnormal attack detection information, and the corresponding dwell information of the unmanned forklift into an evidence file and upload it to the park's intelligent platform to obtain attack handling response information.

[0118] In some embodiments, the executing entity can package the real-time multi-source data, abnormal attack detection information, and the dwell information corresponding to the unmanned forklift into an evidence file and upload it to the park intelligent platform to obtain attack handling response information. The park intelligent platform can be a platform for intelligent processing of various events within the park. These events can include abnormal monitoring events of various intelligent machines, specifically, intelligent abnormal handling of unmanned forklifts. The park intelligent platform can deploy a multimodal large model for intelligent processing. The multimodal large model can adopt different model architectures and deployment methods based on the nature of the park. Here, the multimodal large model is not the focus of this solution and will not be elaborated further. Dwell information can include the dwell location of the unmanned forklift and its estimated dwell time. The attack handling response information can be the processing result of the park intelligent platform in response to the source attack on the unmanned forklift. In practice, the output can be based on the multimodal large model deployed on the park intelligent platform. The attack handling response information can be control instructions for the unmanned forklift, i.e., instructions on how to control the unmanned forklift to move in this scenario.

[0119] Step S1034: Based on the attack processing response information, control the unmanned forklift to move.

[0120] In some embodiments, the executing entity may control the unmanned forklift to move based on the attack processing response information.

[0121] As an example, the executing entity can control the unmanned forklift to move according to the control instructions corresponding to the attack handling response information.

[0122] The various embodiments disclosed in this invention have the following beneficial effects:

[0123] The multimodal countermeasure processing method for low-resource security devices disclosed in some embodiments of this invention, through the two-level security anomaly detection method corresponding to the unmanned forklift, can effectively counteract abnormal source data under multimodal conditions, so as to avoid the unmanned forklift being attacked by abnormal sources during its movement.

[0124] This invention discloses a multimodal countermeasure processing method for low-resource security devices. First, based on real-time multimodal multi-source data acquired by multiple low-resource security devices at a target time under non-extreme environments, it determines whether any information sources are under abnormal attack. Each low-resource security device is a low-resource information source processing sensor supporting security detection under its corresponding information source. Here, by setting multiple low-resource security devices for the unmanned forklift, security anomaly detection at the first level for each information source can be achieved. This allows for timely attack countermeasures and anomaly handling when an abnormal attack occurs at a particular information source, eliminating the need to constantly send information source data from multiple sources to the corresponding anomaly handling module for real-time detection. Then, when it is determined that at least one information source is under abnormal attack, the multi-source abnormal attack detection model corresponding to the unmanned forklift is used to accurately generate abnormal attack detection information based on the at least one information source and the real-time multi-source data. In summary, when each low-resource security device detects an abnormal attack at its corresponding information source, the multi-source abnormal attack detection model can integrate the current information source data from each source to achieve accurate detection of the abnormal attack. Therefore, by using low-resource security equipment for source attack detection and a multi-source anomaly attack detection model for overall multi-source attack detection, two-tiered security protection and timely anomaly handling can be achieved. Finally, based on the anomaly attack detection information indicating the presence of an anomaly attack, the following control steps are executed for the unmanned forklift: determining the unmanned forklift's location within the current park environment; controlling the unmanned forklift to stop moving based on the park location's ability to remain stationary for a target duration; packaging the real-time multi-source data, anomaly attack detection information, and the unmanned forklift's corresponding stationary information into an evidence file and uploading it to the park's intelligent platform to obtain attack handling response information; and controlling the unmanned forklift to move according to the attack handling response information. Here, by executing the relevant control steps, the park's property losses caused by abnormal behavior of the unmanned forklift are prevented when it is attacked. Furthermore, by utilizing the park's intelligent platform and leveraging the intelligent decision-making capabilities of the target park, accurate generation of attack handling response information is achieved, enabling efficient handling of anomalies related to the unmanned forklift. In summary, by implementing a two-tiered safety anomaly detection method for unmanned forklifts, effective countermeasures against anomalous data sources in multimodal environments can be achieved, thus preventing unmanned forklifts from being attacked by anomalous data sources during operation.

[0125] Further reference Figure 2As an implementation of the methods shown in the figures, this invention discloses some embodiments of a multimodal adversarial processing device for low-resource security devices. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, this multimodal adversarial processing device for low-resource security devices can be specifically applied to various electronic devices.

[0126] like Figure 2 As shown, a multimodal adversarial processing device 200 for low-resource security devices includes: a determination unit 201, a generation unit 202, and an execution unit 203. The determination unit 201 is configured to determine whether an information source is under abnormal attack based on real-time multi-source data acquired by multiple low-resource security devices under a target time in a non-extreme environment. Each low-resource security device is a low-resource information source processing sensor that supports security detection under the corresponding information source and is installed on the unmanned forklift. The generation unit 202 is configured to generate abnormal attack detection information based on the at least one information source and the real-time multi-source data when it is determined that at least one information source is under abnormal attack, using the multi-source abnormal attack detection model corresponding to the unmanned forklift. The execution unit 203 is configured to perform the following control steps for the unmanned forklift when the abnormal attack detection information indicates the existence of an abnormal attack: determine the location of the unmanned forklift in the current park environment; control the unmanned forklift to stop moving based on the park location supporting a stop within a target time period; package the real-time multi-source data, abnormal attack detection information, and the stop information corresponding to the unmanned forklift into an evidence file and upload it to the park intelligent platform to obtain attack handling response information; and control the unmanned forklift to move according to the attack handling response information.

[0127] It is understandable that the units described in the multimodal countermeasures processing apparatus 200 for low-resource security devices are similar to those in the reference... Figure 1 The steps in the described method correspond accordingly. Therefore, the operations, features, and beneficial effects described above for the method are also applicable to the multimodal adversarial processing apparatus 200 for low-resource security devices and the units contained therein, and will not be repeated here.

[0128] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device 300 suitable for implementing some embodiments of the present invention. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed in this invention.

[0129] like Figure 3As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0130] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0131] In particular, according to some embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 309, or installed from a storage device 308, or installed from a ROM 302. When the computer program is executed by the processing device 301, it performs the functions defined in the methods of some embodiments disclosed in this invention.

[0132] It should be noted that the computer-readable medium described in some embodiments of this invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0133] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0134] The computer-readable medium may be included in the electronic device or may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device causes the following actions: Based on real-time multi-source data acquired by multiple low-resource security devices under a non-extreme environment at a target time, determine whether any information source is under abnormal attack, wherein each low-resource security device is a low-resource information source processing sensor supporting security detection under the corresponding information source; when at least one information source is determined to be under abnormal attack, generate abnormal attack detection information based on the at least one information source and the real-time multi-source data using a multi-source abnormal attack detection model corresponding to the unmanned forklift; based on the abnormal attack detection information indicating the existence of an abnormal attack, for the unmanned forklift, execute the following control steps: determine the location of the unmanned forklift in the current park environment; based on the park location supporting a stop within a target time period, control the unmanned forklift to stop moving; package the real-time multi-source data, the abnormal attack detection information, and the stop information corresponding to the unmanned forklift into an evidence file and upload it to the park intelligent platform to obtain attack handling response information; and control the unmanned forklift to move according to the attack handling response information.

[0135] Computer program code for performing operations of some embodiments of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0137] The units described in some embodiments of this invention can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a determining unit, a generating unit, and an executing unit. The names of these units do not necessarily limit the specific unit itself. For instance, the generating unit may be described as "a unit that, when it is determined that at least one information source is under an abnormal attack, generates abnormal attack detection information based on the at least one information source and the real-time multi-source data using the multi-source abnormal attack detection model corresponding to the unmanned forklift."

[0138] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0139] The above description is merely a selection of preferred embodiments of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention as described in the embodiments is not limited to specific combinations of the described technical features, but should also cover other technical solutions formed by arbitrary combinations of the described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the described features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present invention.

Claims

1. A multimodal countermeasures method for low-resource security devices, characterized in that, include: Based on the real-time multimodal multi-source data acquired by multiple low-resource security devices under a target time in a non-extreme environment, it is determined whether there is an abnormal attack on the information source; wherein, each low-resource security device is a low-resource information source processing sensor that supports security detection under the corresponding information source, and is installed on the unmanned forklift. When it is determined that at least one information source is under abnormal attack, the multi-source abnormal attack detection model corresponding to the unmanned forklift is used to generate abnormal attack detection information based on the at least one information source and the real-time multi-source data. When an abnormal attack is detected based on the abnormal attack detection information, the following control steps are executed for the unmanned forklift: determine the location of the unmanned forklift in the current park environment; based on the park location supporting a stop within a target duration, control the unmanned forklift to stop moving; package the real-time multi-source data, abnormal attack detection information, and the corresponding stop information of the unmanned forklift into an evidence file and upload it to the park's intelligent platform to obtain attack handling response information; and control the unmanned forklift to move according to the attack handling response information.

2. The multimodal countermeasure processing method for low-resource security devices according to claim 1, characterized in that, The method of generating anomaly detection information based on the multi-source anomaly attack detection model corresponding to the unmanned forklift, according to the at least one information source and the real-time multi-source data, includes: For each of the at least one information source, perform the following generation steps: The unmanned forklift can acquire multiple intelligent devices that support data communication with the corresponding data permissions of the information source; Based on the location of the unmanned forklift in the park, target intelligent devices that have a proximity relationship are obtained from the multiple intelligent devices to obtain a set of target intelligent devices; Send the abnormal attack confirmation information corresponding to the information source to the target intelligent device set to obtain at least one abnormal attack confirmation information; Based on the at least one abnormal attack confirmation information, preliminary determination information is generated to determine whether the information source is subject to an abnormal attack. Based on at least one preliminary determination information, the at least one information source, and the real-time multi-source data, anomaly attack detection information is generated using the multi-source anomaly attack detection model corresponding to the unmanned forklift.

3. The multimodal countermeasures method for low-resource security devices according to claim 2, characterized in that, The step of generating anomaly attack detection information based on at least one preliminary determination information, the at least one information source, and the real-time multi-source data, using the multi-source anomaly attack detection model corresponding to the unmanned forklift, includes: For each piece of preliminary information, perform the following processing steps: Based on the preliminary information, determine the anomaly ratio of the corresponding information source; Acquire at least one source data corresponding to at least one target intelligent device corresponding to the information source, wherein the at least one target intelligent device is each intelligent device confirmed to have an abnormal attack; Based on the anomaly ratio, set at least one data weight corresponding to the at least one source data; Based on the at least one data weight, data fusion is performed on the at least one source data and the corresponding source data in the real-time multi-source data to obtain fused data; Based on at least one fused data, the at least one information source, and the real-time multi-source data, anomaly attack detection information is generated using the multi-source anomaly attack detection model corresponding to the unmanned forklift.

4. The multimodal countermeasure processing method for low-resource security devices according to claim 1, characterized in that, The real-time multi-source data includes multiple source data corresponding to multiple information sources; Based on the real-time multi-source data, determine whether any information source has been subjected to an abnormal attack, including: By using pre-set alignment rules for information alignment between multiple information sources, it can be determined whether any information source has been subjected to abnormal attacks. When it is determined that at least one information source has been subjected to an abnormal attack, the following verification steps are performed for each of the at least one information source: Obtain the abnormal information corresponding to the source output by the alignment rule; Based on the source data corresponding to the source and at least one source data corresponding to at least one source that has not been subjected to abnormal attacks, at least one attack identification machine learning model corresponding to at least one candidate anomaly category is used to determine the anomaly category sequence corresponding to the source, wherein the at least one candidate anomaly category is various anomaly categories corresponding to the anomaly information; When the overall confidence level corresponding to the anomaly category sequence is higher than the first confidence level, it is determined that the information source has been subjected to an abnormal attack. When the overall confidence level corresponding to the anomaly category sequence is not higher than the first confidence level but higher than the second confidence level, anomaly attack confirmation information for the information source is generated; the first confidence level is higher than the second confidence level. When the overall confidence level corresponding to the anomaly category sequence is not higher than the second confidence level, it is determined that the information source has not been subjected to an abnormal attack. The method further includes: Based on the information obtained regarding the various abnormal attacks to be confirmed, determine whether each corresponding information source has been subjected to an abnormal attack.

5. The multimodal countermeasures method for low-resource security devices according to claim 1, characterized in that, The multi-source anomaly attack detection model includes multiple lightweight source feature processing layers, multiple lightweight source feature cross-layers, a lightweight intermediate feature processing layer, and an anomaly attack detection information output layer. The multi-source anomaly attack detection model is trained through the following steps: Obtain the first and second adversarial training datasets for multiple information sources; Based on the first adversarial training dataset, a heavyweight initial multi-source attack detection model deployed at the edge of the target park is trained to obtain a multi-source attack detection model. The multi-source attack detection model includes multiple heavyweight source feature processing layers, multiple heavyweight source feature cross layers, a heavyweight intermediate feature processing layer, and an abnormal attack detection information output layer. Each lightweight source feature processing layer is a network layer after key layer extraction from each network layer in the heavyweight source feature processing layer. Each lightweight source feature cross layer is a network layer after key layer extraction from each network layer in the heavyweight source feature cross layer. Each lightweight intermediate feature processing layer is a network layer after key layer extraction from each network layer in the heavyweight intermediate feature processing layer. According to the multi-source attack detection model, the first network parameters corresponding to the multiple lightweight source feature processing layers, the second network parameters corresponding to the multiple lightweight source feature cross layers, the third network parameters corresponding to the lightweight intermediate feature processing layer, and the fourth network parameters corresponding to the abnormal attack detection information output layer are obtained. Substituting the first network parameter, the second network parameter, the third network parameter, and the fourth network parameter into the initial multi-source anomaly attack detection model, a preliminary multi-source anomaly attack detection model is obtained; The preliminary multi-source anomaly attack detection model is trained based on the second adversarial training dataset to obtain the multi-source anomaly attack detection model.

6. The multimodal countermeasure processing method for low-resource security devices according to claim 5, characterized in that, The second adversarial training dataset includes multiple source data corresponding to multiple sources and anomaly attack detection labels; The step of training the preliminary multi-source anomaly attack detection model based on the second adversarial training dataset to obtain the multi-source anomaly attack detection model includes: For each second adversarial training data, at least one source data is randomly removed from the second adversarial training data multiple times to obtain the removed training data. The training dataset is obtained by combining the training dataset (after removing the training dataset) and the second adversarial training dataset. Obtain the target training data from the training dataset and perform the following training steps: Determine the source data and source labels corresponding to the target training data; Based on the source identifier, perform the following network layer filtering steps: Each source feature processing layer corresponding to a source is selected from the multiple source feature processing layers in the preliminary multi-source anomaly attack detection model; From the multiple source feature cross layers in the preliminary multi-source anomaly attack detection model, each source feature cross layer corresponding to a source is selected; Each source data is input into its corresponding source feature processing layer to obtain source feature processing information. Each source feature processing information is input into each source feature cross layer to obtain the source feature cross information. The cross-information of the features from each source is superimposed and input into a lightweight intermediate feature processing layer to obtain intermediate anomaly attack feature information; The intermediate abnormal attack feature information is input into the abnormal attack detection information output layer to obtain the first initial attack detection information; Based on the first initial attack detection information and the corresponding abnormal attack detection label, determine whether the preliminary multi-source abnormal attack detection model has finished training; At the end of training, the preliminary multi-source anomaly attack detection model was determined to be a multi-source anomaly attack detection model; Before training is finished, the model parameters of the preliminary multi-source abnormal attack detection model are updated based on the first initial attack detection information and the corresponding abnormal attack detection label to obtain the updated model. The updated model is used as the initial multi-source anomaly attack detection model, and the newly selected training data is used as the target training data to continue the training process.

7. The multimodal countermeasure processing method for low-resource security devices according to claim 6, characterized in that, The step of determining whether the preliminary multi-source anomaly attack detection model has finished training based on the first initial attack detection information and the corresponding anomaly attack detection label includes: For each source feature intersection, perform the second processing step: Extract the intermediate feature information corresponding to the source feature cross information from the intermediate anomaly attack feature information; The intermediate feature information is input to the target local anomaly attack detection information output layer to obtain the second initial attack detection information. The target local anomaly attack detection information output layer is the attack detection information output layer corresponding to at least one source of the source feature cross information. Based on the second initial attack detection information and the abnormal attack detection label, an initial local source loss is generated; Obtain the source contribution of at least one source to the attack category corresponding to the abnormal attack detection label, based on the source feature cross information; Based on the source contribution and the initial local source loss, a local source loss is generated; Based on the first initial attack detection information and the corresponding abnormal attack detection label, a complete source loss is generated; Based on the complete source loss and the obtained local source losses, determine whether the preliminary multi-source anomaly attack detection model has finished training.

8. A multimodal countermeasure processing device for low-resource security equipment, characterized in that, include: The determination unit is configured to determine whether an information source is under abnormal attack based on real-time multi-modal multi-source data acquired by multiple low-resource security devices at a target time under non-extreme environments. Each low-resource security device is a low-resource information source processing sensor that supports security detection under the corresponding information source and is installed on an unmanned forklift. The generation unit is configured to, when it is determined that at least one information source is under abnormal attack, use the multi-source abnormal attack detection model corresponding to the unmanned forklift to generate abnormal attack detection information based on the at least one information source and the real-time multi-source data. The execution unit is configured to, when an abnormal attack is detected based on the abnormal attack detection information, perform the following control steps for the unmanned forklift: determine the location of the unmanned forklift in the current park environment; control the unmanned forklift to stop moving based on the park location supporting a target duration of stay; package the real-time multi-source data, abnormal attack detection information, and the corresponding stay information of the unmanned forklift into an evidence file and upload it to the park's intelligent platform to obtain attack handling response information; and control the unmanned forklift to move according to the attack handling response information.

9. An electronic device, characterized in that, include: One or more processors or storage devices; The storage device stores one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the multimodal adversarial processing method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, wherein the computer program, when executed by a processor, implements the multimodal adversarial processing method as described in any one of claims 1-7.

Images