Abnormality detection method, device, electronic device and computer-readable storage medium

Abstract

Embodiments of the present invention disclose an anomaly detection method, apparatus, electronic device, and computer-readable storage medium. The method comprises: obtaining N data, each corresponding to N sensors, where N is a positive integer greater than 1; and determining an abnormal sensor among the N sensors based on the N data and an adjacency matrix between the N sensors. Embodiments of the present invention combine sensor data with an adjacency matrix between sensors for anomaly detection. The adjacency matrix between sensors includes rich associations between sensors, thereby improving detection accuracy.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an anomaly detection method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the development of the Internet of Things (IoT) technology, IoT technology has been widely used in automation, medical health, smart energy, smart manufacturing and other fields.

[0003] Due to factors internal or external to the IoT system, sensor data collected by the system may contain anomalies (such as noise, adversarial examples, etc.). This anomaly can cause the system to make incorrect decisions. For example, in autonomous driving scenarios, anomalous data may cause the system to turn on the windshield wipers even when there is no rain. To address this issue, sensor anomaly detection in IoT systems is extremely important.

[0004] A common sensor anomaly detection method analyzes the data stream of a single sensor and uses statistical, classification, and clustering methods (such as support vector machines) to determine whether there is data that does not conform to the distribution pattern (i.e., anomaly data) in the sensor data stream. If anomaly data is present, the sensor is considered anomaly. This anomaly detection method cannot effectively identify anomaly data that conforms to the distribution pattern (such as adversarial examples), resulting in low accuracy. Summary of the Invention

[0005] The embodiments of the present invention disclose an anomaly detection method, an apparatus, an electronic device, and a computer-readable storage medium, which are used to improve detection accuracy.

[0006] A first aspect discloses an anomaly detection method, which can be applied to electronic devices, modules (e.g., chips) within electronic devices, or logic modules or software that implement all or part of the functions of electronic devices. The following description uses the application of electronic devices as an example. The method may include:

[0007] Obtain N data, where the N data correspond one-to-one to N sensors, and N is a positive integer greater than 1;

[0008] An abnormal sensor among the N sensors is determined according to the N data and an adjacency matrix between the N sensors.

[0009] In an embodiment of the present invention, when determining an abnormal sensor among N sensors, the electronic device uses an adjacency matrix between the N sensors. The adjacency matrix includes rich association relationships between the N sensors. Therefore, the electronic device can associate the N data through the adjacency matrix, and can more accurately determine the abnormal sensor among the N sensors, thereby improving detection accuracy.

[0010] As a possible implementation, determining an abnormal sensor among the N sensors according to the N data and an adjacency matrix between the N sensors includes:

[0011] Obtaining scene semantics according to the N data and an adjacency matrix between the N sensors, where the scene semantics includes semantics of the N sensors;

[0012] An abnormal sensor among the N sensors is determined according to the scene semantics.

[0013] In an embodiment of the present invention, the electronic device can first obtain scene semantics based on the N data and the adjacency matrix. The scene semantics is the scene semantics comprehensively described by the N sensors, including the semantics of the N sensors. Therefore, the electronic device can more accurately determine the abnormal sensor among the N sensors through the scene semantics.

[0014] As a possible implementation, obtaining scene semantics according to the N data and the adjacency matrix between the N sensors includes:

[0015] Obtaining N semantic vectors based on the N data, wherein the N semantic vectors correspond one-to-one to the N sensors;

[0016] The scene semantics are obtained based on the N semantic vectors and the adjacency matrix between the N sensors.

[0017] In this embodiment of the present invention, an electronic device can first obtain N semantic vectors from N data points, and then fuse these N semantic vectors based on the adjacency matrix to obtain scene semantics. Because scene semantics fuses the semantic vectors of various sensors, they can include richer and more accurate information, thereby facilitating anomaly detection in the electronic device.

[0018] As a possible implementation, obtaining N semantic vectors according to the N data includes:

[0019] Obtain N semantics based on the N data, where the N semantics correspond one-to-one to the N sensors;

[0020] The N semantic vectors are obtained according to the N semantics and the semantic library.

[0021] In an embodiment of the present invention, since the dimensions, data formats, and data contents of different sensor data may be different, the electronic device may first obtain N semantics based on the N data, and then obtain N semantic vectors based on the N semantics, so that the electronic device can fuse the semantics of the N sensors.

[0022] As a possible implementation, obtaining scene semantics based on the N semantic vectors and the adjacency matrix between the N sensors includes:

[0023] The N semantic vectors are concatenated to obtain the sensor semantic features;

[0024] Multiplying the adjacency matrix with the sensor semantic features to obtain the original scene semantics;

[0025] The original scene semantics is input into the encoder to obtain the scene semantics, and the dimension of the scene semantics is smaller than the dimension of the original scene semantics.

[0026] In an embodiment of the present invention, the electronic device may concatenate N semantic vectors to obtain sensor semantic features, and then multiply the sensor semantic features with the adjacency matrix to obtain the original scene semantics. The electronic device may then input the original scene semantics into an encoder to obtain embedded scene semantics (i.e., the aforementioned scene semantics). The dimension of the scene semantics may be smaller than that of the original scene semantics, thereby improving the efficiency of anomaly detection by the electronic device.

[0027] As a possible implementation, the method further includes:

[0028] Obtain N true value data and the scene label corresponding to the true value data. The N true value data correspond to the N sensors one by one.

[0029] According to the N true value data, the original scene semantics is obtained;

[0030] The original scene semantics is input into the initial encoder to obtain the scene semantics;

[0031] Input the scene semantics into the classifier to obtain the predicted scene label;

[0032] Determine a loss based on the predicted scene label and the scene label;

[0033] The parameters of the initial encoder are optimized according to the loss to obtain the encoder.

[0034] In an embodiment of the present invention, the electronic device may first train the initial encoder using true value data so as to obtain more accurate scene semantics and the scene semantic centroid of each scene, thereby improving the accuracy of anomaly detection.

[0035] As a possible implementation, the method further includes:

[0036] Calculate the similarity between the scene semantics and M scene semantic centroids to obtain M similarities, where the M scene semantic centroids correspond one-to-one to the M scenes, and the scene semantic centroid is the standard scene semantic corresponding to the scene, where M is a positive integer greater than or equal to 1;

[0037] The determining of an abnormal sensor among the N sensors according to the scene semantics includes:

[0038] When the M similarities are all smaller than the similarity threshold, an abnormal sensor among the N sensors is determined according to the scene semantics.

[0039] In an embodiment of the present invention, the electronic device can calculate the similarity between the scene semantics and M scene semantic centroids to obtain M similarities. Thereafter, the electronic device can determine the size between the M similarities and the similarity threshold. If there is a similarity greater than or equal to the similarity threshold among the M similarities, the electronic device can determine that there is no abnormal sensor among the N sensors and does not need to respond; if the M similarities are all less than the similarity threshold, the electronic device can determine that there is an abnormal sensor among the N sensors. Thereafter, the electronic device can determine the abnormal sensor among the N sensors based on the scene semantics. It can be seen that the electronic device can continue to determine the abnormal sensor only when it is determined that the M similarities are all less than the similarity threshold, which can improve the efficiency of abnormality detection.

[0040] As a possible implementation, the original scene semantics includes N scene semantic vectors, and the N scene semantic vectors correspond one-to-one to the N sensors; and determining an abnormal sensor among the N sensors based on the scene semantics includes:

[0041] Determine the scene semantic centroid with the highest semantic similarity to the scene to obtain the first scene semantic centroid;

[0042] Obtaining N standard semantic vectors corresponding to the semantic centroid of the first scene, where the N standard semantic vectors correspond one-to-one to the N sensors;

[0043] Calculating semantic distances between the N scene semantic vectors and the N standard semantic vectors to obtain N semantic distances, where the N semantic distances correspond one-to-one to the N sensors;

[0044] The largest k semantic distances among the N semantic distances are determined, and sensors corresponding to the k semantic distances are determined as abnormal sensors, where k is a positive integer greater than or equal to 1.

[0045] In an embodiment of the present invention, when an electronic device determines that there is an abnormal sensor among N sensors, the electronic device can first determine the scene semantic centroid with the highest semantic similarity to the scene to obtain a first scene semantic centroid. Afterwards, the electronic device can determine N semantic distances based on the N scene semantic vectors and the N standard semantic vectors corresponding to the first scene semantic centroid, and can determine the sensors corresponding to the k farthest semantic distances as abnormal sensors. Since the standard semantic vector can accurately measure the scene semantic vector that the sensor should represent under normal circumstances, the electronic device can find the sensor that is most likely to be abnormal by calculating the semantic distance between the scene semantic vector and the standard semantic vector.

[0046] As a possible implementation, the N sensors include a first sensor and a second sensor;

[0047] When the sensing ranges of the first sensor and the second sensor overlap, the values corresponding to the first sensor and the second sensor in the adjacency matrix are 1;

[0048] In the case that the sensing ranges of the first sensor and the second sensor do not overlap, the values corresponding to the first sensor and the second sensor in the adjacency matrix are 0.

[0049] A second aspect discloses an anomaly detection device, which may be an electronic device or a module (e.g., a chip) in an electronic device. The device may include:

[0050] As a possible implementation, the acquisition unit is configured to acquire N data, where the N data correspond one-to-one to N sensors, and N is a positive integer greater than 1;

[0051] A determining unit is configured to determine an abnormal sensor among the N sensors based on the N data and an adjacency matrix between the N sensors.

[0052] As a possible implementation manner, the determining unit determines an abnormal sensor among the N sensors according to the N data and an adjacency matrix between the N sensors, including:

[0053] Obtaining scene semantics according to the N data and an adjacency matrix between the N sensors, where the scene semantics includes semantics of the N sensors;

[0054] An abnormal sensor among the N sensors is determined according to the scene semantics.

[0055] As a possible implementation manner, the determining unit obtains scene semantics according to the N data and the adjacency matrix between the N sensors, including:

[0056] Obtaining N semantic vectors based on the N data, wherein the N semantic vectors correspond one-to-one to the N sensors;

[0057] The scene semantics are obtained based on the N semantic vectors and the adjacency matrix between the N sensors.

[0058] As a possible implementation manner, the determining unit obtains N semantic vectors according to the N data, including:

[0059] Obtain N semantics based on the N data, where the N semantics correspond one-to-one to the N sensors;

[0060] The N semantic vectors are obtained according to the N semantics and the semantic library.

[0061] As a possible implementation, the determining unit obtains scene semantics based on the N semantic vectors and the adjacency matrix between the N sensors, including:

[0062] The N semantic vectors are concatenated to obtain the sensor semantic features;

[0063] Multiplying the adjacency matrix with the sensor semantic features to obtain the original scene semantics;

[0064] The original scene semantics is input into the encoder to obtain the scene semantics, and the dimension of the scene semantics is smaller than the dimension of the original scene semantics.

[0065] As a possible implementation manner, the device further includes:

[0066] a processing unit, configured to calculate similarities between the scene semantics and M scene semantic centroids, to obtain M similarities, wherein the M scene semantic centroids correspond one-to-one to the M scenes, the scene semantic centroids being the standard scene semantics corresponding to the scene, and M being a positive integer greater than or equal to 1;

[0067] The determining unit determines, according to the scene semantics, an abnormal sensor among the N sensors, including:

[0068] When the M similarities are all smaller than the similarity threshold, an abnormal sensor among the N sensors is determined according to the scene semantics.

[0069] As a possible implementation, the original scene semantics includes N scene semantic vectors, and the N scene semantic vectors correspond one-to-one to the N sensors; and the determining unit determines an abnormal sensor among the N sensors according to the scene semantics, including:

[0070] Determine the scene semantic centroid with the highest semantic similarity to the scene to obtain the first scene semantic centroid;

[0071] Obtaining N standard semantic vectors corresponding to the semantic centroid of the first scene, where the N standard semantic vectors correspond one-to-one to the N sensors;

[0072] Calculating semantic distances between the N scene semantic vectors and the N standard semantic vectors to obtain N semantic distances, where the N semantic distances correspond one-to-one to the N sensors;

[0073] The largest k semantic distances among the N semantic distances are determined, and sensors corresponding to the k semantic distances are determined as abnormal sensors, where k is a positive integer greater than or equal to 1.

[0074] A third aspect discloses an electronic device comprising a processor and a memory. The memory is configured to store a computer program, and the processor is configured to invoke the computer program. When the processor executes the computer program stored in the memory, the processor performs the anomaly detection method disclosed in the first aspect or any embodiment of the first aspect.

[0075] A fourth aspect discloses a computer-readable storage medium having a computer program or computer instructions stored thereon. When the computer program or computer instructions are executed, the anomaly detection method disclosed in the above aspects is implemented.

[0076] A fifth aspect discloses a chip, comprising a processor for executing a program stored in a memory. When the program is executed, the chip executes the above method.

[0077] As a possible implementation, the memory is located outside the chip.

[0078] A sixth aspect discloses a computer program product, which includes a computer program code. When the computer program code is executed, the above-mentioned anomaly detection method is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 This is a flowchart of an anomaly detection method disclosed in an embodiment of the present invention;

[0080] Figure 2 This is a schematic diagram of a topological structure of a sensor network disclosed in an embodiment of the present invention;

[0081] Figure 3 is a schematic diagram of a scene semantic cluster disclosed in an embodiment of the present invention;

[0082] Figure 4 1 is a schematic diagram of a network architecture of a scene semantic centroid extractor disclosed in an embodiment of the present invention;

[0083] Figure 5is a schematic diagram of a semantic distance disclosed in an embodiment of the present invention;

[0084] Figure 6 is a schematic diagram of a scene semantic cluster disclosed in an embodiment of the present invention;

[0085] Figure 7 Schematic diagram comparing SSC-IDS and SSF-IDS disclosed in an embodiment of the present invention;

[0086] Figure 8 Schematic diagram comparing another SSC-IDS and SSF-IDS disclosed in an embodiment of the present invention;

[0087] Figure 9 This is a schematic diagram comparing different k values disclosed in an embodiment of the present invention;

[0088] Figure 10 It is a structural diagram of an abnormality detection device disclosed in an embodiment of the present invention;

[0089] Figure 11 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0090] The embodiments of the present invention disclose an anomaly detection method, device, electronic device, and computer-readable storage medium for improving detection accuracy. The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings.

[0091] Obviously, the described embodiments are only some of the embodiments of this application, and not all of them. Reference to "embodiments" herein means that the specific features, structures, or characteristics described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it represent independent or alternative embodiments that are mutually exclusive with other embodiments. It is understood, both explicitly and implicitly, that the embodiments described herein can be combined with other embodiments. All other embodiments derived by one of ordinary skill in the art without inventive effort based on the embodiments in this application are within the scope of protection of this application. In the specification, claims, and accompanying drawings of this application, the terms "first," "second," "third," and so on are used to distinguish between different objects, not to describe a specific order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a list of steps or elements may be included, or alternatively, steps or elements not listed may be included, or alternatively, other steps or elements inherent to such process, method, product, or device may be included.

[0092] Only part relevant to the present application is shown in the accompanying drawings, not all of it. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processing or methods depicted as flow charts. Although flow charts describe various operations (or steps) as sequential processing, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of various operations can be rearranged. When its operation is completed, the processing can be terminated, but can also have additional steps not included in the accompanying drawings. The processing can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0093] As used in this specification, the terms "component," "module," "system," "unit," and the like are used to refer to computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a unit can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or distributed between two or more computers. In addition, these units can be executed from various computer-readable media having various data structures stored thereon. Units can communicate, for example, through local and / or remote processes based on signals having one or more data packets (e.g., data from a second unit interacting with another unit in a local system, a distributed system, and / or a network. For example, the Internet interacts with other systems via signals).

[0094] In order to better understand the embodiments of the present invention, some terms and related technologies of the embodiments of the present invention are described below.

[0095] Artificial intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0096] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0097] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to imbue computers with intelligence. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, weakly supervised learning, and strongly supervised learning.

[0098] With the widespread adoption of IoT technology, its applications in automation, healthcare, energy, manufacturing, and other fields are increasing. By analyzing data collected by a large number of sensors in IoT systems, the entire IoT system can be assisted in making effective decisions. However, large-scale IoT devices (i.e., sensors) are vulnerable to attacks and damage, resulting in the collected data containing abnormal data such as noise (such as abnormal data and incomplete information) and adversarial examples. This abnormal data can cause the IoT system to make incorrect decisions. For example, in medical IoT systems, adversarial example attacks can cause electronic computed tomography (CT) image classification errors, increasing the risk of misdiagnosis. For another example, machine learning models used in autonomous driving are also highly susceptible to being deceived by adversarial examples, leading to incorrect decisions and increasing the risk of traffic accidents.

[0099] As can be seen, because IoT devices are vulnerable to attacks and damage, sensor data can contain abnormal data, which can pose a security threat to applications within the IoT system. For some high-risk applications (such as autonomous driving and telemedicine), abnormal data can lead to serious consequences. Therefore, IoT systems need to be highly robust and able to effectively identify abnormal data so that they can make reliable decisions based on accurate data and improve their security.

[0100] Among them, anomaly detection (intrusion detection) technology is a way to solve the above problems. Anomaly detection technology can improve the robustness of the Internet of Things system.

[0101] At present, anomaly detection in IoT systems mainly relies on analyzing the data of a single sensor and detecting sensor intrusions based on outlier detection methods. Specifically, for the sensor data stream, statistical, classification, clustering and other methods can be used to detect anomalies. Based on the sensor's historical data and predictions of future data, it can be determined whether the sensor's data stream includes outliers (i.e., abnormal data) that do not conform to the data stream distribution pattern. If there is data in the data stream that obviously does not conform to the distribution pattern, it can be considered that the sensor has been invaded or that the sensor is an abnormal sensor. However, for adversarial sample attacks, there are no anomalies in the data stream distribution (i.e., there are no outliers that do not conform to the data stream distribution pattern). Therefore, the above methods cannot effectively detect, resulting in low accuracy.

[0102] In order to better understand the embodiments of the present invention, the following first introduces the applicable scenarios of the embodiments of the present invention by way of example.

[0103] For example, the anomaly detection method provided by the embodiments of the present invention can be used to improve the accuracy of anomaly detection and can be used to perform anomaly detection on a sensor network including a large number of sensors. Furthermore, it is also applicable to a multi-source heterogeneous sensor network.

[0104] For example, with the development of intelligent vehicles, the number of sensors installed on smart cars is increasing, and becoming increasingly diverse, to facilitate the implementation of functions such as autonomous driving and assisted driving. A smart car may be equipped with one or more lidars, one or more cameras, one or more radars (such as millimeter-wave radars), and so on. These sensors can collectively form a sensor network. For autonomous driving scenarios, the sensors on a smart car can be in the same spatiotemporal environment. The data collected by these sensors can be used to implement functions such as automatic parking and emergency braking through machine learning and deep learning technologies. However, if the sensor data obtained by the autonomous driving system is erroneous, it may cause the autonomous driving system to make some incorrect decisions, thereby increasing driving safety risks. The anomaly detection method provided by the embodiments of the present invention can detect anomalies in the sensors on the smart car based on the acquired sensor data, identify abnormal sensors, and thus assist the autonomous driving system in making correct decisions.

[0105] The anomaly detection system implemented by the anomaly detection method provided by the embodiment of the present invention may be referred to as a scene semantic centroid intrusion detection system (SSC-IDS).

[0106] When the anomaly detection method provided by the embodiment of the present invention performs anomaly detection on sensor data, anomaly detection can be performed based on the scene semantics of the sensor network (system). For example, anomaly detection can be performed on the scene semantics of the sensor network on a smart car. In the case where the data of all sensors are normal data, the scene semantics corresponding to these data should also be normal scene semantics; in the case where abnormal data is included in the data of all sensors, the scene semantics corresponding to these data should be abnormal scene semantics. Therefore, if the scene semantics determined by the sensor data is abnormal scene semantics, it indicates that abnormal sensors may be included in the sensor network. For abnormal scene semantics, the semantic distance between the semantics of each sensor and the standard sensor semantics can be calculated, and k sensors with larger semantic distances can be determined as abnormal sensors, where k is an integer greater than 1.

[0107] It should be understood that the anomaly detection method provided by the embodiments of the present invention is applicable to all sensor systems in the Internet of Things. By using scene semantics-based anomaly detection, the efficiency and accuracy of anomaly detection can be greatly improved, thereby enhancing the quality of anomaly alerts in sensor systems. Furthermore, when an anomaly is detected in a sensor system, the method can also identify the sensor most likely to have experienced the anomaly.

[0108] The anomaly detection method provided in the embodiment of the present invention can be executed by an electronic device, and the electronic device includes but is not limited to a terminal device or a server. The above-mentioned server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms. The above-mentioned terminal device can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, car terminal, smart TV, etc., but is not limited to this. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this. At the same time, the electronic device can also be a chip, chip system, or processor that supports the electronic device to implement the anomaly method, or a logic module or software that can implement all or part of the functions of the electronic device.

[0109] See also Figure 1 , Figure 1 This is a flow chart of an anomaly detection method disclosed in an embodiment of the present invention. Figure 1 As shown, the anomaly detection method may include the following steps.

[0110] 101. Acquire N data, where the N data correspond one-to-one to the N sensors.

[0111] To detect whether a sensor network (such as the sensor network of a smart car) includes abnormal sensors, or to detect whether the sensor network has been invaded (such as by a cyber attack), an electronic device can acquire N data (i.e., data collected by N sensors) in real time, where the N data correspond one-to-one to the N sensors. The N sensors can belong to the same sensor network. The N sensors can be located in the same spatiotemporal environment, and the N data can also be data collected in the same spatiotemporal environment, where N is a positive integer greater than 1.

[0112] It should be understood that the N sensors may include different types of sensors (i.e., multi-source heterogeneous sensors), such as temperature sensors, lidars, cameras, millimeter-wave radars, etc., which are not limited in the embodiments of the present invention. Therefore, the data formats and data dimensions of the data collected by the N sensors may be different, as well as the information richness may be different. For example, a temperature sensor can collect real-time temperature information (such as 10 degrees) in the current time and space environment; a millimeter-wave radar can collect information about the surrounding physical environment and, based on analysis of this information, determine whether there are obstacles in the surrounding area; a camera can collect images or video information about the surrounding area. Compared to millimeter-wave radars, the information collected by cameras may include color information, such as the color of the traffic light in front of the camera and countdown information.

[0113] 102. Determine an abnormal sensor among the N sensors based on the N data and an adjacency matrix between the N sensors.

[0114] Since the N sensors may belong to the same sensor network and may be located in the same spatiotemporal environment, in order to more accurately determine the abnormal sensor among the N sensors, the electronic device may first determine the adjacency matrix between the N sensors. Thereafter, the electronic device may determine the abnormal sensor among the N sensors based on the N data and the adjacency matrix between the N sensors.

[0115] The adjacency matrix between N sensors can describe the spatial topology of the sensor network and include rich association information between the sensors. The size of the adjacency matrix is N*N. The sensing ranges of some or all of the N sensors may overlap. If the sensing ranges of two sensors overlap, the corresponding values in the adjacency matrix may be 1; if the sensing ranges of the two sensors do not overlap, the corresponding values in the adjacency matrix may be 0. Specifically, the N sensors may include a first sensor and a second sensor. If the sensing ranges of the first sensor and the second sensor overlap, the corresponding values in the adjacency matrix may be 1; if the sensing ranges of the first sensor and the second sensor do not overlap, the corresponding values in the adjacency matrix may be 0.

[0116] It should be noted that the sensor's perception range can be understood as the spatial range in which the sensor collects data. For example, for the camera and millimeter-wave radar on a smart car, if the camera can capture images of the first area and the millimeter-wave radar can capture physical environment information of the second area, if there are identical parts between the first and second areas, it can be considered that the perception ranges of the camera and millimeter-wave radar overlap. Assuming that the identical part of the first and second areas is the third area (that is, the intersection of the camera and millimeter-wave radar perception ranges is the third area), and there is a traffic light in the third area, the information collected by the millimeter-wave radar can determine that there is a traffic light in the third area, and the information collected by the camera can also determine that there is a traffic light in the third area and the status of the traffic light.

[0117] Electronic devices can model the sensor network as a graph structure, expressed as G = (V, E). V is a set of sensor nodes, and E is a set of edges. At the same time, the adjacency matrix of the graph is used to describe the connection relationship information between the sensor nodes. For example, suppose that the sensor system of a smart car includes 6 cameras (cameras), 5 radars and 1 lidar installed above the roof. The cameras are the front camera, the left front camera, the right front camera, the rear camera, the left rear camera and the right rear camera; the 5 radars are the front radar, the left front radar, the right front radar, the left rear radar and the right rear radar. Among them, an edge can be established between two sensors with overlapping perception ranges, and their topological structure can be as follows: Figure 2 As shown. Figure 2 It can be seen that there are connections (i.e., edges) between the lidar and all cameras and radars, indicating that the perception range of the lidar intersects with the perception range of any camera or radar. Figure 2 In the figure, the left rear radar can have edges with the rear camera, left rear camera, and lidar, which means that the perception range of the left rear radar can intersect with the perception range of the rear camera, left rear camera, and lidar respectively. Figure 2 The other sensors in the diagram can be understood similarly and will not be described in detail here.

[0118] An electronic device determines an abnormal sensor among the N sensors based on N data and an adjacency matrix between the N sensors. Specifically, the electronic device may include the following steps: the electronic device obtains scene semantics based on the N data and the adjacency matrix between the N sensors, and then the electronic device may determine the abnormal sensor among the N sensors based on the scene semantics. The scene semantics includes the semantics of the N sensors.

[0119] An electronic device can derive scene semantics from N data and an adjacency matrix between N sensors. Specifically, the electronic device can derive N semantic vectors from the N data, and then derive scene semantics based on the N semantic vectors and the adjacency matrix between the N sensors. The N semantic vectors correspond one-to-one to the N sensors.

[0120] To obtain N semantic vectors, the electronic device can first derive N semantics from N data. Then, the N semantic vectors can be obtained based on the N semantics and a semantic library. The N semantics correspond one-to-one to the N sensors. The electronic device can convert sensor data into semantics. For example, for images captured by a camera, the electronic device can convert the image into semantics using algorithms such as machine learning and deep learning.

[0121] The semantics of a sensor can be expressed as Sem i is the semantics of the i-th sensor in the sensor network (i.e., the N sensors mentioned above), is the kth semantics among all the semantics contained in the sensor data.

[0122] After obtaining the sensor's semantics, the electronic device can encode the sensor's semantics using the sensor's corresponding semantic library. This encoding method can be one-hot encoding. It should be noted that different sensor types require different semantic libraries for encoding. Sensors of the same type can also share the same semantic library.

[0123] The electronic device can encode N semantics through one-hot encoding to obtain N semantic vectors. Among them, the semantic vector (encoding) of the i-th sensor can be expressed as code i . code i The dimension of code is the same as the size of the semantic base used for encoding. i Each position in can represent a semantic, if the sensor semantic Sem i The semantics corresponding to a certain position is included in the value of the position, which can be 1. If the sensor semantics Sem i The semantics corresponding to a certain position is not included, and the value of this position can be 0.

[0124] For example, the semantic library for a camera can be {sunny, rainy, ..., snowy, cloudy}, the size of the semantic library can be 1000, and the semantics of a camera is {sunny}. Therefore, one-hot encoding the semantics of the camera through the semantic library can obtain a 1000-dimensional vector [1,0, ...,0,0], where only the position corresponding to sunny day has a value of 1 (that is, the value of the first position is 1), and the values of other positions are all 0.

[0125] After the electronic device obtains the semantic vectors of N sensors (i.e., the N semantic vectors mentioned above), it can obtain scene semantics based on the N semantic vectors and the adjacency matrix between the N sensors. Specifically, the electronic device can splice the N semantic vectors to obtain sensor semantic features; then, the adjacency matrix between the N sensors can be multiplied by the sensor semantic features to obtain the original scene semantics; then, the electronic device can input the original scene semantics into the encoder to obtain scene semantics. The dimension of the scene semantics is smaller than the dimension of the original scene semantics. The following is a detailed description of the steps for the electronic device to obtain scene semantics.

[0126] The electronic device can first concatenate N semantic vectors to obtain the sensor semantic feature, which can be expressed as a multidimensional vector S, S = (code1, code2, ... code N ) T Where N is the number of sensors in the sensor network, and S is the sensor semantic feature composed of the semantic vectors of all sensors, which can also be called the attribute matrix of the sensor network. In the sensor network, the adjacency matrix of the topology graph G can be expressed as A. The adjacency matrix A is specifically as follows:

[0127]

[0128] Among them, a ij The value of 1 indicates that the sensing ranges of sensor node i and sensor node j overlap (i.e., there is an intersection in the sensing ranges). ij The value of 0 means that the sensing ranges of sensor node i and sensor node j do not overlap, and i and j are less than N.

[0129] The original scene semantics can be obtained by the adjacency matrix A and the sensor semantic features S, which can be expressed as v, where v = A × S. It can be seen that the original scene semantics can be a graph vector, and the dimension of v can be N*M, where M can be the size of the largest semantic library among the semantic libraries corresponding to N sensors. The i-th row of v can be expressed as v i The original scene semantics may include N scene semantic vectors, and the N scene semantic vectors correspond one-to-one to the N sensors. Specifically, a row of the original scene semantics may be a scene semantic vector, whose dimension is 1*M.

[0130] Because the sensor semantic features S are obtained through one-hot encoding, S is high-dimensional and sparse. Therefore, the original scene semantics v are also high-dimensional and sparse. To reduce the dimensionality of the original scene semantics v and improve processing efficiency, the original scene semantics v can be encoded using an encoder. This encoder can be a graph neural network (GNN), and this encoder can be called ScSem2Vec.

[0131] The electronic device can compress the original scene semantics v from a high-dimensional sparse space to a low-dimensional dense space through ScSem2Vec. The number of layers of the encoder can be 3, 5, or other numbers, which is not limited in the embodiment of the present invention. In the encoder, each layer can aggregate the information of the neighboring nodes of each sensor node based on the correlation between the sensor nodes. In the k-th layer network of the encoder, the correlation coefficient between node i and node j can be calculated using the following formula (1):

[0132]

[0133] in, is a trainable parameter. Sigmoid is a Sigmoid function. σ can be an activation function ReLU or other activation functions, which are not limited in this embodiment of the present invention. When k is 1, and It can be the data of the i-th and j-th rows in the original scene semantics.

[0134] Electronic devices can be used to calculate the correlation coefficient (i.e. ) is standardized, and the standardization formula can be shown as the following formula (2).

[0135]

[0136] in, It can be the normalization coefficient between node i and node j. is the set of neighbor nodes of node (i.e. sensor node) i. The nodes in the kth layer can be represented as The details are shown in the following formula (3).

[0137]

[0138] The output obtained by the electronic device through the last layer of the encoder network is the embedded representation of the original scene semantics, which is the above-mentioned scene semantics.

[0139] It should be understood that in order to group together the original scene semantics under the same scene and the embedded scene semantics obtained after compression by the encoder, the encoder can be trained first. During the encoder training process, a large amount of real-valued data can be used to train the encoder, and cross-entropy can be used as the loss function. The output of the last layer of the encoder can be passed through a classifier to classify the scene semantics.

[0140] Specifically, the electronic device can obtain N true value data and the scene label corresponding to the true value data; based on the N true value data, the electronic device can obtain the original scene semantics; then, the electronic device can input the original scene semantics into the initial encoder to obtain the scene semantics; then, the electronic device can input the scene semantics into the classifier to obtain the predicted scene label; then, the electronic device can determine the loss based on the predicted scene label and the scene label; finally, the electronic device can optimize the parameters of the initial encoder based on the loss to obtain the encoder. N true value data and N sensors correspond one to one.

[0141] For example, autonomous driving scenarios can include eight scene semantic categories: daytime straight driving, daytime parking, daytime turning, daytime overtaking, nighttime straight driving, nighttime parking, nighttime turning, and nighttime overtaking. For each category, a large amount of ground truth data must be collected, and each ground truth data can be associated with a scene label (e.g., data collected in a daytime straight driving scenario can be labeled as daytime straight driving). The encoder is trained using this ground truth data and the corresponding scene labels.

[0142] It is understandable that during the iterative training of the model (i.e., the above-mentioned initial encoder), when the total loss value is less than the preset model error, the electronic device can stop training to obtain a trained encoder. Optionally, the electronic device can set a preset number of iterations for the initial encoder and record the number of training iterations of the initial encoder. When the number of training iterations is equal to the preset number of iterations, the training of the initial encoder can be stopped, and the model with the number of training iterations equal to the preset number of iterations can be determined as the final trained encoder. At the same time, during the model training process, the total loss values of multiple groups of different data can be obtained first, and then the total loss values of the multiple groups of data can be averaged to obtain an average total loss value. Afterwards, the model parameters can be optimized based on the average total loss value.

[0143] It should be noted that a large amount of real-value data in different scenarios, through the trained encoder, the scene semantics in each scenario will form scene semantic clusters. The definition of scene semantic clusters can be: given a set of clusters C = {C1,...,C k}, for a set of N samples of scene semantics V = v1, v2, ..., v N, k corresponding scene semantic clusters will be formed. That is,

[0144] Scene semantic clusters can have the properties shown in the following formula (4):

[0145]

[0146] Among them, dist(v i ,v j ) is the scene semantic sample v i and v j The semantic distance between them can be measured by cosine similarity. The calculation of cosine similarity is shown in the following formula (5). The scene semantic sample can be the scene semantics determined by the true value data of the sensor.

[0147]

[0148] Among them, θ i,j is the scene semantic sample v i and v j The angle between them. i || is the scene semantic sample v i The modulus length, ||v j || is the scene semantic sample v j The modulus length. i ·v j is the semantic sample v i and v j The dot product between .

[0149] To facilitate understanding of scene semantic anomaly detection, the concept of scene semantic centroid is introduced below. The scene semantic centroid of a scene can be the center vector of the scene semantic cluster of the scene. The scene semantic centroid can be calculated as shown in the following formula (6).

[0150]

[0151] Among them, class i Represents a scene category, centroid i is the scene semantic centroid corresponding to the scene, and Y is the number of scene semantic samples used to calculate the scene semantic centroid. j is the jth scene semantic sample.

[0152] It should be understood that the larger Y is (i.e., the more true value data corresponding to a scene), the more accurate the calculated scene semantic centroid can be. At the same time, when the value of Y is greater than a fixed value, the scene semantic centroid will converge. The scene semantic centroid is the expected value of all scene semantic samples within the scene semantic cluster to which it belongs, and can be the standard scene semantics corresponding to the scene. At the same time, it should also be understood that the various components of the scene semantic centroid can correspond to the semantic expectations of each single sensor in each true value data sample in the scene, as shown in the following formula (7).

[0153]

[0154] in, Represents class i In the scene, the semantic expectation of each single sensor corresponding to the j-th scene semantic sample. T represents the transpose of the matrix. The calculation of can be shown as the following formula (8).

[0155]

[0156] Where N is the number of sensor nodes. is the semantics corresponding to the kth sensor in the jth scene semantic sample.

[0157] It should be noted that the distribution plane of the scene semantic cluster can be fixed. When the acquired sensor data does not include abnormal data, the scene semantics obtained from the sensor data should be within the distribution plane of the scene semantic cluster. When the acquired sensor data includes abnormal data, the scene semantics obtained from the sensor data should be outside the distribution plane of the scene semantic cluster. Therefore, the distribution plane of the scene semantic cluster can be used as a standard for anomaly detection, and sensor anomaly detection can be performed through out of distribution detection (OOD). Please refer to Figure 3 , Figure 3 Schematic diagram of a scene semantic cluster disclosed in an embodiment of the present invention. Figure 3 As shown in the figure, it is assumed that there are 4 scenes, corresponding to 4 different clusters. In the figure, except for the abnormal data, the rest are normal data (i.e., true value data). From the three-dimensional visualization image, it can be seen that the normal data is more concentrated in the three-dimensional space, and a cluster can be formed around the scene semantic centroid of each scene (i.e., scene semantic cluster), while the abnormal data is obviously independent and far away from the semantic centroid of each scene. Compress the three-dimensional data to a two-dimensional plane for visualization (i.e., Figure 3 From the two-dimensional visualization image on the right of the figure, we can see that the clusters are basically distributed in a circle around the scene semantic centroid, while the abnormal data are far away from the scene semantic centroid.

[0158] The electronic device can determine an abnormal sensor among N sensors based on scene semantics. Specifically, the electronic device can calculate the similarity between the scene semantics and M scene semantic centroids to obtain M similarities; then, the electronic device can first determine the size of the M similarities and the similarity threshold. If it is determined that one of the M similarities is greater than or equal to the similarity threshold, it can be indicated that the N acquired data are all normal data, and it can be considered that there is no abnormal sensor among the N sensors; if it is determined that all the M similarities are less than the similarity threshold, it can be indicated that there is abnormal data among the N acquired data, and it can be considered that there is an abnormal sensor among the N sensors. Then, the electronic device can determine the abnormal sensor among the N sensors based on the scene semantics. Among them, the M scene semantic centroids correspond one-to-one to the M scenes, the scene semantic centroid is the standard scene semantics corresponding to the scene, and M is a positive integer greater than or equal to 1.

[0159] The electronic device can calculate the cosine similarity between the scene semantics and the M scene semantic centroids, and use the cosine similarity as a way to measure the similarity. The scene semantics v and the scene semantic centroid i The cosine similarity cosθ between i The calculation of can be shown as the following formula (9).

[0160]

[0161] Among them, θ i is the scene semantics v and the scene semantics centroid i The angle between them. ‖centroid i ‖ is the scene semantic centroid i The modulus length of , ‖v‖ is the modulus length of the scene semantics v.

[0162] It should be understood that the cosine similarity threshold may be 0.5, 0.7, or other values, and may be selected according to actual conditions, which is not limited in the embodiment of the present invention.

[0163] After the electronic device determines that the scene semantics are abnormal, it can determine the abnormal sensor among the N sensors based on the scene semantics. Specifically, the electronic device can determine the scene semantic centroid with the highest similarity to the scene semantics to obtain a first scene semantic centroid; then, the electronic device can obtain N standard semantic vectors corresponding to the first scene semantic centroid; then, the electronic device can calculate the semantic distances between the N scene semantic vectors and the N standard semantic vectors to obtain N semantic distances; then, the electronic device can determine the largest k semantic distances among the N semantic distances, and can determine the sensors corresponding to the k semantic distances as abnormal sensors. Wherein, k is a positive integer greater than or equal to 1, the N standard semantic vectors correspond one-to-one to the N sensors, the N semantic distances correspond one-to-one to the N sensors, the N scene semantic vectors correspond one-to-one to the N sensors, and the N scene semantic vectors correspond one-to-one to the N sensors, and the N scene semantic vectors are scene semantic vectors corresponding to the original scene semantics. The first scene semantic centroid can be the scene semantic centroid corresponding to the first scene (such as scenes such as driving straight during the day and overtaking during the day). The steps of the electronic device determining the abnormal sensor are described in detail below.

[0164] Specifically, the higher the similarity between the scene semantics and a certain scene semantics centroid, the more likely the scene semantics is to belong to the abnormal scene semantics of the scene. Therefore, after the electronic device determines that the scene semantics is abnormal scene semantics, it can determine the scene semantics centroid with the highest similarity to the scene semantics (such as the highest cosine similarity) to obtain the first scene semantics centroid. In particular, since the scene semantics can be a vector obtained by compressing the original scene semantics through an encoder; accordingly, the scene semantics centroid can also be regarded as a vector obtained by compressing the standard original scene semantics through an encoder. In particular, the standard original scene semantics can include N standard semantic vectors, and the N standard semantic vectors correspond one-to-one to the N sensors.

[0165] After the electronic device determines the scene semantic centroid with the highest similarity, it can obtain the standard semantic vector corresponding to the scene semantic centroid. Specifically, the electronic device can obtain the standard semantic vector corresponding to the scene semantic centroid through a decoder. Since the electronic device compresses the original scene semantics through an encoder to obtain embedded scene semantics. Therefore, the electronic device can also restore the embedded scene semantics to its original structure (i.e., the original scene semantics) through a decoder corresponding to the encoder. The decoder can be called Vec2ScSem.

[0166] It should be understood that in order for the decoder to restore the embedded scene semantics to the original scene semantics, the decoder can be trained with true value data. During the training of the decoder, the electronic device can freeze the parameters of the encoder. The decoder needs to reconstruct the original input of the scene semantic centroid (i.e., the original scene semantics), so it can go through a reverse encoding process corresponding to the encoder. In the k-th layer network of the decoder, the correlation coefficient between node i and node j can be calculated using the following formula (10):

[0167]

[0168] in, is the trainable parameter of the k-th layer network of the decoder. Sigmoid is the Sigmoid function. σ can be the activation function ReLU or other activation functions, which are not limited in this embodiment of the present invention. When k is 1, and It can be the data of the i-th and j-th rows in the scene semantics.

[0169] Electronic devices can be used to calculate the correlation coefficient (i.e. ) is standardized, and the standardization formula can be shown as the following formula (11).

[0170]

[0171] in, Can be the normalization coefficient between node i and node j, is the set of neighbor nodes of node i. After the k-1th layer of the decoder is reconstructed, the node i can be expressed as The details are shown in the following formula (12).

[0172]

[0173] Among them, σ can be the ReLU activation function.

[0174] The output obtained by the electronic device through the last layer of the decoder network is the original scene semantics corresponding to the scene semantics. The original scene semantics may include N standard semantic components (ie, N standard semantic vectors).

[0175] It should be understood that in order to obtain the correct original scene semantics after the scene semantics are decoded by the decoder, minimizing the reconstruction error can be used as an optimization goal during the decoder training process, as shown in the following formula (13).

[0176]

[0177] Among them, v ican be the original scene semantics obtained by the decoder, It can be the original scene semantics input to the encoder (that is, the original scene semantics obtained through sensor true value data). Can be The square root of the modulus length.

[0178] After training the decoder with a large amount of ground truth data, a trained decoder is obtained. The electronic device can use this decoder to decode the scene semantic centroids and obtain N standard semantic vectors corresponding to each scene semantic centroid. It should be noted that the number of network layers of the encoder and decoder can be the same.

[0179] The electronic device can compare the scene semantics with the first scene semantic centroid at the sensor semantic level. Specifically, assuming that the N standard semantic components obtained after the first scene semantic centroid is decoded by the decoder are respectively recorded as Scene Semanticsv a The N scene semantic components corresponding to the original scene semantics (i.e., the scene semantic components of each sensor) are recorded as The electronic device can calculate the semantic distance of each sensor based on the N scene semantic components of the original scene semantics and the N standard semantic components. The calculation of the semantic distance can be shown in the following formula (14).

[0180]

[0181] in, is the scene semantic vector of the i-th sensor The standard semantic vector corresponding to the sensor in the first scenario The distance between them. l is the dimension of semantic encoding of a single sensor. It can be the value of the mth dimension of the i-th sensor. Can be The value of the mth dimension.

[0182] The electronic device calculates the semantic distances between the N scene semantic components of the original scene semantics and the N standard semantic components to obtain N semantic distances. Afterwards, the electronic device can compare the sizes of the N semantic distances and determine the farthest k semantic distances (i.e., the k largest semantic distances). Thereafter, the electronic device can determine the k sensors corresponding to the farthest k semantic distances, and can determine the k sensors as abnormal sensors or invaded sensors.

[0183] See also Figure 5 , Figure 5 This is a schematic diagram of a semantic distance disclosed in an embodiment of the present invention. Figure 5The semantic distance in is the normalized semantic distance, Figure 5 In the example, L represents a laser radar, C1, C2, C3, C4, C5, and C6 represent front camera, left front camera, right front camera, rear camera, left rear camera, and right rear camera respectively; R1, R2, R3, R4, and R5 represent front radar, left front radar, right front radar, left rear radar, and right rear radar respectively. Figure 5 As shown, L and C6 may be abnormal sensors or invaded sensors, and the semantic distances corresponding to them are greater than the difference between normal sensors and the standard semantic vectors (ie, the semantic distance is greater).

[0184] It should be understood that the scene semantic centroid and the corresponding semantic components can be obtained by the collaboration of ScSem2Vec (encoder) and Vec2ScSem (decoder). These two model structures can be collectively referred to as the scene semantic centroid extractor. The specific network architecture can be as follows: Figure 4 As shown. Figure 4 As shown, the entire network architecture can include four parts. In the first part, the electronic device can input the original scene semantics into the encoder to obtain embedded scene semantics (i.e., the above-mentioned scene semantics). Through a large amount of true value data, the scene semantics under different scenes will form scene semantic clusters. During the encoder training process, the embedded scene semantics can be input into a classifier to obtain a predicted scene label. Afterwards, the loss can be calculated based on the predicted scene label and the real scene label. The encoder is optimized by the loss to obtain a trained encoder. The original scene semantics is obtained by the electronic device based on sensor data. In the second part, the scene semantics obtained by the trained encoder through a large amount of true value data will form scene semantic clusters. Based on the scene semantic clusters, the scene semantic centroid corresponding to each scene can be calculated. In the third part, the decoder can be trained. During the training process, the parameters of the encoder can be frozen, and the embedded scene semantics can be obtained through true value data. The embedded scene semantics are then input into the decoder to obtain predicted original scene semantics. Afterwards, the electronic device can obtain a reconstruction error (i.e., reconstruction loss) based on the original scene semantics and the predicted original scene semantics. The decoder can be optimized by this error to obtain a trained decoder. In the fourth part, through the trained decoder, the electronic device can input the scene semantic centroid into the decoder to obtain the standard original scene semantics, which includes N standard semantic vectors.

[0185] It should also be understood that in the embodiments of the present invention, the scene semantic centroid and standard semantic components can be obtained in advance and can be used directly. Therefore, the anomaly detection method disclosed in the embodiments of the present invention can greatly improve the efficiency of anomaly detection. Specifically, the electronic device can obtain a trained encoder and decoder in advance through pre-training, and can obtain the distribution plane of normal scene semantic samples and the scene semantic centroid.

[0186] To verify the effectiveness of the anomaly detection method proposed in this invention, the present invention's embodiments were validated using the Nuscenes dataset, a public dataset for autonomous driving. The Nuscenes dataset includes data collected by various types of sensors during autonomous driving, with timestamps aligned. The Nuscenes dataset also covers a variety of scenarios.

[0187] During the verification process, we selected data from eight different scenarios to train the encoder and decoder. The eight scenarios can be daytime straight driving, daytime parking, daytime turning, daytime overtaking, nighttime straight driving, nighttime parking, nighttime turning, and nighttime overtaking. Figure 6 , Figure 6 It is a schematic diagram of a scene semantic cluster disclosed in an embodiment of the present invention. Figure 6 This is the visualization of the scene semantic clusters corresponding to the above eight scenes on a two-dimensional plane, such as Figure 6 As shown in FIG, through the trained encoder, the above eight scenes will form eight scene semantic clusters, and the scene semantic centroid of each scene semantic cluster can be obtained.

[0188] In order to test the effectiveness of the above anomaly detection method, data with different timestamps can be mixed and crossed to create abnormal samples. For example, for samples in the daytime straight-ahead scenario, we can replace one or more sensor data with data in the nighttime straight-ahead scenario, so that some sensor data in the constructed abnormal samples are data from the daytime straight-ahead scenario, and the other sensor data are data from the nighttime straight-ahead scenario. In addition, data from different sensors can be cross-mixed to construct abnormal samples. For example, data from five sensors, namely the front camera (i.e., video camera), right front camera, front radar, left front radar, and lidar, can be cross-mixed. Please refer to Figure 6 , Figure 6 The abnormal scene semantics in is the scene semantics obtained through abnormal samples. Figure 6 It can be found that although the constructed abnormal samples do not appear abnormal on the surface (i.e., they conform to the normal distribution of the data stream), the scene semantics corresponding to the abnormal samples are independent of each scene semantic cluster.

[0189] In this embodiment of the present invention, the experimental effect is evaluated by the detection rate (DR) and false positive rate (FPR). The detection rate represents the percentage of detected true abnormal data out of the total number of abnormal data, and the false positive rate represents the percentage of normal data misclassified as abnormal data. The calculation formulas for the detection rate and false positive rate are shown in the following formulas (15) and (16).

[0190]

[0191]

[0192] Among them, TP (true positive) represents true positive examples, that is, positive examples (i.e. positive samples) that are correctly classified (detected); TN (true negative) represents true negative examples, that is, negative examples that are correctly classified; FP (false positive) represents false positive examples, that is, negative examples that are misclassified as positive examples; FN (false negative) represents false negative examples, that is, positive examples that are misclassified as negative examples.

[0193] In the embodiments of the present invention, the above-mentioned anomaly detection method is compared with traditional anomaly detection methods. For sensor networks, traditional anomaly detection methods analyze the data streams of a single sensor. If any sensor detects an outlier, the intrusion detection system will report an anomaly. In the anomaly detection method disclosed in the embodiments of the present invention, if the intrusion detection system determines that an abnormal scene semantics exists, it will report an anomaly. An intrusion detection system based on traditional anomaly detection methods can be referred to as a single sensor flow intrusion detection system (SSF-IDS).

[0194] See also Figure 7 , Figure 7 This is a schematic diagram comparing an SSC-IDS and an SSF-IDS disclosed in an embodiment of the present invention. Figure 7 The DR in can be the detection rate of sensor network anomalies (i.e., detecting whether there are abnormal sensors in the sensor network) detected by SSC-IDS and SSF-IDS, without having to detect which specific sensor has an abnormality. Figure 7 As shown in the figure, the detection rate of SSF-IDS is approximately 81.1%, while the detection rate of SSC-IDS, the scene semantic centroid-based intrusion detection system proposed in this embodiment of the present invention, is approximately 97.6%. This shows that SSC-IDS has a better detection effect than SSF-IDS, can avoid false alarms in the sensor system, and improve the quality of the intrusion detection system.

[0195] Furthermore, the ability of SSC-IDS and SSF-IDS to trace abnormal sensors can be compared. SSF-IDS uses the data of a single sensor to determine whether it is an abnormal sensor, while SSC-IDS uses the data of all sensors to detect abnormal sensors among all sensors.

[0196] See also Figure 8 , Figure 8 It is a schematic diagram comparing another SSC-IDS and SSF-IDS disclosed in an embodiment of the present invention. Figure 8 The DR in the figure represents the detection rate of abnormal sensors (i.e., specific sensors in the sensor network) detected by SSC-IDS and SSF-IDS. For single data streams from the front camera, right front camera, front radar, left front radar, and lidar, SSF-IDS can detect sensor anomalies with an average detection rate of approximately 57.5%. The detection rate of SSC-IDS is approximately 95.7%.

[0197] It can be seen that compared with SSF-IDS, SSC-IDS has better detection effect. SSC-IDS can determine abnormal sensors based on the standard semantic components of the scene semantic centroid, can effectively distinguish adversarial samples from normal data, and can improve the accuracy of anomaly detection and the quality of intrusion detection systems.

[0198] Assume that the detection rate of abnormal scenarios (i.e., the detection rate of only detecting whether the sensor network is abnormal) is expressed as DR scene , the detection rate of perceived intrusion (i.e., the detection rate of sensors that detect specific abnormalities in the sensor network) is expressed as DR sensing In the embodiment of the present invention, the overall detection rate DR total To evaluate the comprehensive performance of SSC-IDS and SSF-IDS, the overall detection rate is calculated as shown in the following formula (17).

[0199] DR total =DR scene ×DR sensing (17)

[0200] Through a large number of test samples, we can obtain an overall detection rate of about 46.3% for SSF-IDS and about 93.4% for SSC-IDS. This shows that SSC-IDS can effectively detect anomalies in sensor networks and identify the specific sensors with anomalies.

[0201] It should be noted that the hyperparameter k set when determining abnormal sensors through SSC-IDS may be 1. Therefore, the sensor corresponding to the largest semantic distance may be determined as the abnormal sensor.

[0202] The detection results may be different if the value of k is different. Figure 9 , Figure 9 This is a comparative diagram of different k values disclosed in an embodiment of the present invention. Figure 9 As shown in the figure, as the k value increases (from 1 to 5), the detection rate of SCC-IDS improves slightly, and more potentially compromised sensors can be detected. However, at the same time, more normal sensors may be misidentified as abnormal, resulting in a significant increase in the false positive rate, which in turn reduces detection efficiency.

[0203] It should be understood that, in actual practice, it is rare for multiple abnormal sensors to exist simultaneously (e.g., multiple sensors are invaded simultaneously). Therefore, a smaller k value of 1 or 2 may be preferred. The specific value of k can be determined based on actual circumstances and is not limited in this embodiment of the present invention.

[0204] In an embodiment of the present invention, the electronic device can first construct a sensor network topology map and obtain an adjacency matrix including the connection relationship between each sensor; then, the electronic device can fuse the semantics of each sensor to obtain scene semantics; then, the electronic device can perform anomaly detection on the scene semantics based on the scene semantic centroid; in the case where the scene semantics is detected to be abnormal scene semantics, the electronic device can trace the abnormal sensor based on the abnormal scene semantics, that is, by calculating the semantic distance between the semantic component of the abnormal scene semantics and the standard semantic component, the sensor corresponding to the largest k semantic distances is determined as the abnormal sensor. It can be seen that the electronic device can perform consistency verification based on the scene semantics described by the sensor network (i.e., perform similarity comparison with the centroid of each scene semantics), effectively detect whether there are abnormal sensors in the sensor network or whether the sensor system has been invaded, and can trace the abnormal sensor and issue an alarm. In addition, the detection method based on scene semantics can detect attacks by adversarial samples.

[0205] See also Figure 10 , Figure 10 1 is a schematic diagram of the structure of an abnormality detection device disclosed in an embodiment of the present invention. The abnormality detection device can be an electronic device or a module in an electronic device. Figure 10 As shown, the device may include:

[0206] An acquisition unit 1001 is configured to acquire N data, where the N data correspond one-to-one to N sensors, and N is a positive integer greater than 1;

[0207] The determining unit 1002 is configured to determine an abnormal sensor among the N sensors according to the N data and an adjacency matrix between the N sensors.

[0208] In one embodiment, the determining unit 1002 determines an abnormal sensor among the N sensors according to the N data and an adjacency matrix between the N sensors, including:

[0209] Obtaining scene semantics according to the N data and an adjacency matrix between the N sensors, where the scene semantics includes semantics of the N sensors;

[0210] An abnormal sensor among the N sensors is determined according to the scene semantics.

[0211] In one embodiment, the determining unit 1002 obtains scene semantics according to the N data and the adjacency matrix between the N sensors, including:

[0212] Obtaining N semantic vectors based on the N data, wherein the N semantic vectors correspond one-to-one to the N sensors;

[0213] The scene semantics are obtained based on the N semantic vectors and the adjacency matrix between the N sensors.

[0214] In one embodiment, the determining unit 1002 obtains N semantic vectors according to the N data, including:

[0215] Obtain N semantics based on the N data, where the N semantics correspond one-to-one to the N sensors;

[0216] The N semantic vectors are obtained according to the N semantics and the semantic library.

[0217] In one embodiment, the determining unit 1002 obtains scene semantics based on the N semantic vectors and the adjacency matrix between the N sensors, including:

[0218] The N semantic vectors are concatenated to obtain the sensor semantic features;

[0219] Multiplying the adjacency matrix with the sensor semantic features to obtain the original scene semantics;

[0220] The original scene semantics is input into the encoder to obtain the scene semantics, and the dimension of the scene semantics is smaller than the dimension of the original scene semantics.

[0221] In one embodiment, the apparatus may further include:

[0222] Processing unit 1003 is configured to calculate similarities between the scene semantics and M scene semantic centroids to obtain M similarities, where the M scene semantic centroids correspond one-to-one to the M scenes, the scene semantic centroids being the standard scene semantics corresponding to the scene, and M being a positive integer greater than or equal to 1;

[0223] The determining unit 1002 determines, according to the scene semantics, that an abnormal sensor among the N sensors includes:

[0224] When the M similarities are all smaller than the similarity threshold, an abnormal sensor among the N sensors is determined according to the scene semantics.

[0225] In one embodiment, the original scene semantics includes N scene semantic vectors, and the N scene semantic vectors correspond one-to-one to the N sensors; the determining unit 1002 determines the abnormal sensor among the N sensors according to the scene semantics, including:

[0226] Determine the scene semantic centroid with the highest semantic similarity to the scene to obtain the first scene semantic centroid;

[0227] Obtaining N standard semantic vectors corresponding to the semantic centroid of the first scene, where the N standard semantic vectors correspond one-to-one to the N sensors;

[0228] Calculating semantic distances between the N scene semantic vectors and the N standard semantic vectors to obtain N semantic distances, where the N semantic distances correspond one-to-one to the N sensors;

[0229] The largest k semantic distances among the N semantic distances are determined, and sensors corresponding to the k semantic distances are determined as abnormal sensors, where k is a positive integer greater than or equal to 1.

[0230] For a more detailed description of the acquisition unit 1001, the determination unit 1002 and the processing unit 1003, please refer to the above Figure 1 The relevant description in the illustrated method embodiment is directly available and will not be repeated here.

[0231] See Figure 11 , Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 11As shown, the electronic device 1100 may include: at least one processor 1101, such as a CPU, at least one memory 1105, and at least one communication bus 1102. Optionally, the electronic device 1100 may also include at least one network interface 1104 and a user interface 1103. The communication bus 1102 is used to achieve connection and communication between these components. The user interface 1103 may include a display screen (display) and a keyboard (keyboard), and the network interface 1104 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1105 may be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 1105 may optionally also be at least one storage device located away from the aforementioned processor 1101. As Figure 11 As shown, the memory 1105 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application.

[0232] exist Figure 11 In the electronic device 1100 shown, the network interface 1104 can provide a network communication function; the user interface 1103 is mainly used to provide an input interface for the user.

[0233] In one embodiment, the processor 1101 may be configured to call a device control application stored in the memory 1105 to implement:

[0234] Obtain N data, where the N data correspond one-to-one to N sensors, and N is a positive integer greater than 1;

[0235] An abnormal sensor among the N sensors is determined according to the N data and an adjacency matrix between the N sensors.

[0236] It should be understood that the electronic device 1100 described in the embodiment of the present application can be used to perform the above Figure 1 For the method executed by the electronic device in the method embodiment, reference can be made to the relevant description and will not be repeated here.

[0237] An embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed, executes the method in the above method embodiment.

[0238] An embodiment of the present invention further discloses a computer program product comprising instructions, which, when executed, performs the method in the above method embodiment.

[0239] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this application. It should be understood that the above description is only the specific implementation methods of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of this application should be included in the scope of protection of this application.

Claims

1. A method for detecting anomalies, characterized in that: include: Acquire N data, where the N data correspond one-to-one to N sensors, and the N data are data collected by the N sensors respectively, where N is a positive integer greater than 1; wherein the data collected by the sensors include images; Obtaining scene semantics based on the N data and an adjacency matrix between the N sensors, the scene semantics including the semantics of the N sensors; wherein, when the sensing ranges of two sensors overlap, the corresponding values of the two sensors in the adjacency matrix are 1; Calculate the similarity between the scene semantics and M scene semantic centroids to obtain M similarities, where the M scene semantic centroids correspond one-to-one to the M scenes, the scene semantic centroids are standard scene semantics corresponding to the scenes, and M is a positive integer greater than or equal to 1; When the M similarities are all smaller than a similarity threshold, an abnormal sensor among the N sensors is determined according to the scene semantics.

2. The method according to claim 1, characterized in that Obtaining scene semantics according to the N data and the adjacency matrix between the N sensors includes: Obtaining N semantic vectors according to the N data, wherein the N semantic vectors correspond one-to-one to the N sensors; Scene semantics are obtained based on the N semantic vectors and the adjacency matrix between the N sensors.

3. The method according to claim 2, characterized in that Obtaining N semantic vectors according to the N data includes: Obtaining N semantics according to the N data, wherein the N semantics correspond one-to-one to the N sensors; The N semantic vectors are obtained according to the N semantics and the semantic library.

4. The method according to claim 2, characterized in that The obtaining scene semantics based on the N semantic vectors and the adjacency matrix between the N sensors includes: Splicing the N semantic vectors to obtain sensor semantic features; Multiplying the adjacency matrix with the sensor semantic features to obtain original scene semantics; The original scene semantics are input into an encoder to obtain the scene semantics, where the dimension of the scene semantics is smaller than the dimension of the original scene semantics.

5. The method according to claim 4, characterized in that The original scene semantics includes N scene semantic vectors, and the N scene semantic vectors correspond one-to-one to the N sensors; and determining an abnormal sensor among the N sensors according to the scene semantics includes: Determine the scene semantic centroid with the highest semantic similarity to the scene to obtain a first scene semantic centroid; Obtaining N standard semantic vectors corresponding to the semantic centroid of the first scene, where the N standard semantic vectors correspond one-to-one to the N sensors; Calculating semantic distances between the N scene semantic vectors and the N standard semantic vectors to obtain N semantic distances, where the N semantic distances correspond one-to-one to the N sensors; Determine the largest k semantic distances among the N semantic distances, and determine sensors corresponding to the k semantic distances as abnormal sensors, where k is a positive integer greater than or equal to 1.

6. An abnormality detection device, characterized in that: include: an acquisition unit, configured to acquire N data, wherein the N data correspond one-to-one to the N sensors, the N data being data collected by the N sensors respectively, and N being a positive integer greater than 1; wherein the data collected by the sensors include images; a determining unit, configured to obtain scene semantics based on the N data and an adjacency matrix between the N sensors, the scene semantics including the semantics of the N sensors; wherein, when the sensing ranges of two sensors overlap, the corresponding values of the two sensors in the adjacency matrix are 1; a processing unit, configured to calculate similarities between the scene semantics and M scene semantic centroids to obtain M similarities, wherein the M scene semantic centroids correspond one-to-one to the M scenes, the scene semantic centroids being standard scene semantics corresponding to the scenes, and M being a positive integer greater than or equal to 1; The determining unit is further configured to determine an abnormal sensor among the N sensors according to the scene semantics when the M similarities are all less than a similarity threshold.

7. An electronic device, characterized in that: include: a memory and a processor; wherein: The memory is used to store a computer program, wherein the computer program includes program instructions; The processor is configured to call the program instructions so that the electronic device executes the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program or computer instructions, and when the computer program or computer instructions are executed, the method according to any one of claims 1 to 5 is implemented.

Images