Safety monitoring method and device, electronic equipment and storage medium

Abstract

The invention provides a safety monitoring method and device, electronic equipment and a storage medium, and the method comprises the steps: responding to a parking state of a vehicle, and collecting multi-mode sensing data of a monitoring region corresponding to the vehicle; and performing feature matching on at least one piece of event information and the multi-modal sensing data, and determining a target event matched with the multi-modal sensing data from the at least one piece of event information. According to the method, the corresponding event information and the target event are determined through matching analysis of the multi-modal sensing data of the monitoring area around the vehicle, monitoring of the events around the vehicle is achieved, the conditions around the vehicle can be better reflected through the multi-modal data, the events around the vehicle can be sensed and determined more accurately, and the monitoring efficiency is improved. And the safety of the vehicle is improved, and the method can be applied to the intelligent cabin of the vehicle.

Description

Technical Field

[0001] The present application relates to the field of security monitoring technology, and in particular to a security monitoring method, device, electronic device and storage medium. Background Art

[0002] With the development of intelligent vehicles, further improving vehicle safety through intelligent means has become a new trend. Vehicle safety not only addresses personal safety issues during intelligent driving but also vehicle safety during parking situations. Related technologies for detecting safety around vehicles have low accuracy, making it difficult to accurately detect events occurring around the vehicle. Summary of the Invention

[0003] The present application aims to solve, at least to some extent, the problem of low accuracy of safety detection around vehicles in the related art.

[0004] To this end, the present application proposes a method, an apparatus, an electronic device, and a storage medium.

[0005] In one aspect, an embodiment of the present application provides a security monitoring method, including:

[0006] In response to the vehicle being in a parked state, collecting multimodal perception data of a monitoring area corresponding to the vehicle;

[0007] Feature matching is performed on at least one event information and the multimodal perception data, and a target event matching the multimodal perception data is determined from the at least one event information.

[0008] Optionally, performing feature matching on at least one event information and the multimodal perception data, and determining a target event matching the multimodal perception data from the at least one event information includes:

[0009] Performing feature extraction on the multimodal sensing data to obtain multimodal features;

[0010] The multimodal feature is matched with first features respectively corresponding to at least one event information, a target first feature matching the multimodal feature among multiple first features is determined, and the event information corresponding to the target first feature is determined as the target event.

[0011] Optionally, matching the multimodal feature with first features respectively corresponding to at least one event information to determine a target first feature matching the multimodal feature among multiple first features includes:

[0012] A first feature having the highest similarity with the multimodal feature among the first features corresponding to at least one event information is determined as a target first feature.

[0013] Optionally, the extracting features from the multimodal sensing data to obtain multimodal features includes:

[0014] Determining a target object in the multimodal perception data, and generating a second feature corresponding to the target object;

[0015] Acquire a behavior tag and extract a third feature corresponding to the behavior tag, wherein the behavior tag is used to characterize the behavior existing in the monitoring area;

[0016] The second feature and the third feature are fused to generate the multimodal feature.

[0017] Optionally, the fusing the second feature and the third feature to generate the multimodal feature includes:

[0018] Splicing the second feature and the third feature to generate a fourth feature;

[0019] Performing feature projection on the fourth feature to generate an attention matrix;

[0020] The fourth feature is weighted according to the attention matrix to generate the multimodal feature.

[0021] Optionally, the method further includes:

[0022] Matching the multimodal perception data with a matching condition in a preset rule corresponding to at least one event information;

[0023] Determine a target preset rule corresponding to a matching condition satisfied by the multimodal perception data, and determine the target event according to event information corresponding to the target preset rule.

[0024] Optionally, the method further includes:

[0025] The at least one event information is determined in response to a first input, where the first input is an operation of a user inputting at least one event information or an operation of selecting at least one event information.

[0026] Another embodiment of the present application provides a safety monitoring device, comprising:

[0027] a collection module, configured to collect multimodal perception data of a monitoring area corresponding to the vehicle in response to the vehicle being in a parked state;

[0028] A processing module is used to perform feature matching on at least one event information and the multimodal perception data, and determine a target event that matches the multimodal perception data from the at least one event information.

[0029] Another embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method described in the above aspect is implemented.

[0030] Another aspect of the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described in the aforementioned aspect is implemented.

[0031] Another embodiment of the present application provides a chip, which includes a processing circuit configured to execute the method described in the above aspect.

[0032] Another embodiment of the present application provides a computer program product, which implements the method described in the above aspect when the program is executed by a processor.

[0033] The safety monitoring method, device, electronic device, chip and storage medium proposed in this application perform matching analysis on the multimodal perception data of the monitoring area around the vehicle to determine the corresponding event information and target events, thereby monitoring the events occurring around the vehicle. The multimodal data better reflects the situation around the vehicle, can more accurately perceive and determine the events occurring around the vehicle, and improve the safety of the vehicle.

[0034] By matching multimodal perception data based on user-configured event information, security detection can effectively meet user security needs, increase the flexibility of security detection, and enhance user experience. This application can be applied to scenarios such as smart cockpits in vehicles, industrial production security inspections, and home security, with a wide range of application scenarios.

[0035] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0037] Figure 1 A schematic diagram of a safety monitoring method provided in an embodiment of the present application;

[0038] Figure 2 A schematic structural diagram of a safety monitoring device provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0040] Figure 4 This is a schematic diagram of the structure of a chip proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0042] In related technologies, parking safety monitoring solutions are mainly divided into three types: vibration sensors, ultrasonic / millimeter wave radars, and surround-view cameras:

[0043] 1. Vibration sensor solution: This solution relies on the vehicle's own vibration and is triggered when the vibration intensity and duration exceed a certain level. It has the simplest application scenario and is only triggered when the sensor touches the vehicle itself.

[0044] 2. Ultrasonic / millimeter wave solution: This method uses ultrasonic / millimeter waves to measure the distance to objects around the vehicle and triggers the vehicle when the distance falls below a certain threshold. This method is low-cost but has low ranging accuracy and the triggering object is unknown.

[0045] 3. Vision solution: This system uses a combination of on-board cameras and deep learning algorithms for target detection to accurately identify and locate pedestrians, vehicles, obstacles, etc. around the vehicle, and triggers when the established strategy is met.

[0046] A single security detection solution is difficult to meet the increasingly high accuracy and speed requirements for security detection, and users' personalized security detection needs are increasing. The security detection solutions in related technologies are difficult to meet user needs.

[0047] The following describes the security monitoring method, device, electronic device, chip and storage medium of the embodiments of the present application with reference to the accompanying drawings.

[0048] Figure 1 A schematic diagram of a safety monitoring process provided in an embodiment of the present application.

[0049] As an implementation method, the security monitoring method of the embodiment of the present application can be configured in a security monitoring device, and the security monitoring device can be applied to any electronic device so that the electronic device can perform a security monitoring function.

[0050] Among them, the electronic device can be any device with computing capabilities, such as a mobile terminal. The mobile terminal can be, for example, a mobile phone, tablet computer, personal digital assistant, wearable device, or other hardware device with various operating systems, touch screens and / or display screens.

[0051] As another implementation method, the security monitoring method of the embodiment of the present application can also be executed by a chip with processing capabilities, including an image signal processing chip (Image Signal Processor, ISP), a central processing unit (Central Processing Unit, CPU), an application-specific integrated circuit (Application-Specific Integrated Circuit, ASIC), a microprocessor (Digital Signal Processor, DSP), a field programmable gate array (Field-Programmable Gate Array, FPGA), a system on chip (System On AChip, SOC), a reduced instruction set computer RISC (Reduced Instruction Set Computer, reduced instruction set computer), etc., which are not listed one by one here.

[0052] like Figure 1 As shown, the method may include the following steps:

[0053] Step 101: In response to a vehicle being in a parked state, multimodal sensing data of a monitoring area corresponding to the vehicle is collected;

[0054] Step 102: Perform feature matching on at least one event information and the multimodal perception data, and determine a target event that matches the multimodal perception data from the at least one event information.

[0055] In this embodiment, when the vehicle is in the parking state, the parking state refers to the state in which the vehicle is stopped. In the parking state, the vehicle can be in a state of being turned off and powered off, or the engine can be kept on but powered on (such as when the owner temporarily leaves the vehicle). In order to ensure the safety of the vehicle when parked, the vehicle's surrounding environment is monitored for safety. The vehicle is equipped with a variety of sensors for collecting multimodal perception data of the monitoring area around the vehicle. Since the vehicle's doors are locked when parked, the safety of the vehicle's interior can be guaranteed. The security threats faced by the vehicle mainly come from outside the vehicle, so the monitoring area around the vehicle is the area outside the vehicle centered on the vehicle. Optionally, the monitoring area is a circular area with the vehicle at the center.

[0056] Optionally, the event information represents an event occurring in the monitoring area corresponding to the vehicle, and includes information used to describe the event (such as text information or voice information, etc.). The event information can be a pre-set event or an event subsequently added by the user according to individual needs.

[0057] In step 102, features can be extracted from the multimodal perception data, and features corresponding to each event information can be extracted. The features can then be matched to determine event information that matches the multimodal perception data corresponding to the monitoring area, and the target event can be determined based on the event information. The multimodal perception data contains data from multiple modalities, which can more comprehensively characterize the conditions in the monitoring area. Matching the features of the multimodal perception data with features corresponding to the event information can more comprehensively consider factors within the monitoring area. The matched event information is more consistent with the conditions in the monitoring area around the vehicle, avoiding misjudgments caused by considering a single factor and improving the accuracy of safety monitoring. Optionally, the multimodal perception information can include, but is not limited to, any one or a combination of the following: image data, sound data, light intensity data, temperature data, etc. Optionally, feature extraction from the multimodal perception data can employ any one or a combination of the following: statistical extraction, principal component analysis, linear discriminant analysis, discrete cosine transform, extraction based on preset rules, automatic feature extraction using deep learning, and feature extraction based on clustering. During the implementation of statistical extraction methods, statistical features are extracted from multimodal sensory data, such as the mean, median, and mode, which describe the data's central tendency, and the variance, standard deviation, and range, which reflect the data's degree of dispersion. Principal component analysis (PCA) is a dimensionality reduction technique that extracts new features by identifying the most significant directions (principal components) in the data and projecting the original data onto these directions. Linear discriminant analysis (LDA) not only reduces dimensionality but also considers categorical information, aiming to find feature vectors that maximize inter-class separation. For example, in face recognition, LDA can extract features that best distinguish different faces, thereby improving classifier performance. The discrete cosine transform (DCT) is commonly used in image and audio signal processing to convert signals from the time or spatial domain to the frequency domain, extracting frequency components as features.

[0058] Optionally, feature matching refers to finding similar features between two or more sets of data. These features typically correspond to some kind of similarity or correlation. Matching features of multimodal sensory data with features corresponding to event information can employ any one or a combination of the following methods: distance-based matching, feature vector-based matching, graph-based matching, and deep learning-based matching. Distance-based matching methods calculate a specific mathematical distance based on features and determine the degree of feature matching based on the magnitude of this distance. For example, Euclidean distance calculates the straight-line distance between two feature vectors, Manhattan distance calculates the sum of the absolute differences between two feature vectors in each dimension, and cosine similarity measures similarity by calculating the cosine of the angle between two vectors. Among feature vector-based matching methods, nearest neighbor matching finds the nearest feature vector for each feature vector as a matching result and is widely used in scenarios such as image recognition. The SIFT image feature extraction algorithm extracts image feature points and their descriptors. During matching, matching feature points are found by calculating the Euclidean distance between descriptors. Graph-based matching methods model feature points as graph nodes and similarities as edge weights, finding the optimal match using graph algorithms (such as the Hungarian algorithm and spectral graph matching). Deep learning-based matching methods use deep neural networks to learn the embedding space of features, placing similar features closer together and dissimilar features farther apart.

[0059] In one possible embodiment, the monitoring area is within 5 meters around the vehicle. Pedestrians or other vehicles within this distance may cause damage to the vehicle. People or other vehicles beyond 5 meters are far away from the vehicle and are unlikely to cause damage to the vehicle. Therefore, safety monitoring is not performed for the range beyond 5 meters, which effectively saves the processor's computing power, avoids waste of computing power, and improves the efficiency of safety monitoring.

[0060] In a possible embodiment, the pre-configured event information may include but is not limited to any one or a combination of any two of the following: a collision with the door opening in a narrow parking space, the vehicle being touched, the vehicle being scratched, pedestrians or vehicles passing by the vehicle, the windows, doors or sunroof being open, water possibly entering the vehicle (for example, the windows, sunroof or doors are not closed and it is raining), there may be foreign objects in the ventilation duct, the license plate is damaged, the license plate is abnormal, the rearview mirror is damaged, the rearview mirror is abnormal, etc. These are common scenarios with a wide range of applications.

[0061] In one possible embodiment, user-defined event information may include, but is not limited to, any one or a combination of the following: pedestrians knocking on the vehicle window and checking the chassis, pedestrians or vehicles lingering around the vehicle for a long time (excluding pedestrians passing by, such as pedestrians or vehicles lingering in the monitoring area for more than 30 seconds, which is considered a long time), and non-security personnel approaching the vehicle. Scenarios supplemented by users based on their own needs are more adaptable to users' vehicle safety requirements. For example, the event information of "non-security personnel approaching the vehicle" can ignore the scenario of security personnel patrolling and approaching the vehicle, avoiding the judgment of security personnel patrols as security threats, making it more humane and avoiding frequent false prompts.

[0062] In one possible embodiment, the sensors installed on the vehicle include any one or more of the following: image sensors, onboard cameras, ultrasonic radar, millimeter-wave radar, lidar, infrared radar, sound sensors, spirit levels, speed sensors, pressure sensors, position sensors, temperature sensors, linear acceleration sensors, angular acceleration sensors, air flow sensors, gas sensors, etc., which are used to collect multimodal sensor data, provide comprehensive data support for subsequent safety monitoring, and improve the accuracy of safety monitoring. For example, the image sensor is used to collect image monitoring data of the monitoring area. The image data may include ordinary image data and infrared sensor image data (used to reflect temperature distribution and is suitable for scenes with low light at night). The radar sensor is used to collect radar data of the monitoring area. The radar data is used to generate point cloud data, which is a set of vectors in a three-dimensional coordinate system. Scanning data is recorded in the form of points. Each point contains three-dimensional coordinates and can carry other information about the point's attributes, such as color, reflectivity, and intensity. The main characteristics of point cloud data are high precision, high resolution, and high-dimensional geometric information, which can intuitively represent information such as the shape, surface, and texture of objects in space. The sound sensor is used to collect sound data in the monitoring area. The sound data reflects the sound generated by objects in the monitoring environment. The sound can reflect the events that have occurred. For example, the presence of knocking sounds indicates that someone or an object may be knocking on the vehicle, or the voice of people around the vehicle can be collected and converted into text to understand the people's intentions.

[0063] Optionally, step 102 of performing feature matching on at least one event information and the multimodal perception data, and determining a target event matching the multimodal perception data from the at least one event information, includes:

[0064] Performing feature extraction on the multimodal sensing data to obtain multimodal features;

[0065] The multimodal feature is matched with first features respectively corresponding to at least one event information, a target first feature matching the multimodal feature among multiple first features is determined, and the event information corresponding to the target first feature is determined as the target event.

[0066] In this embodiment, features in multimodal perception data are extracted through a multimodal model, wherein the multimodal model may include multiple neural network structures for extracting features from data of different modalities through different processing methods, and fusing the features corresponding to the perception data of each modality to ultimately output multimodal features. The neural network structure in the multimodal model may include, but is not limited to, any one or a combination of any two or more of the following: convolutional neural network (CNN), recurrent neural network (RNN), transfer transformer network, multilayer perceptron (MLP), generative adversarial network (GAN), etc. Among them, convolutional neural network is suitable for processing data with spatial structure such as images and videos. Its convolution layer can automatically extract local features in the data, such as edges and textures, while the pooling layer can reduce the dimension and abstract the features. In multimodal perception, CNN can be used to process image modalities, for example, using classic CNN architectures such as ResNet and DenseNet to extract deep features of images. Recurrent neural networks are mainly used to process sequence data, such as text and speech. In multimodal perception, they can be used to process text modalities or speech modalities, for example, to encode text sequences or model speech signals. The Transformer network, based on the self-attention mechanism, performs well in processing long sequences of data and capturing global dependencies. It is able to simultaneously consider the relationships between all positions in the sequence. In multimodal perception tasks, such as image-text matching in vision-language tasks and multimodal text generation, Transformer and its variants are widely used to process text and image modalities. Generative adversarial networks consist of a generator and a discriminator, and are trained through a two-person game. The goal of the generator is to generate data that is as realistic as possible to deceive the discriminator, while the discriminator attempts to distinguish between real data and generated data. In the field of multimodal perception, GANs can be used for tasks such as image generation, data augmentation, and cross-modal conversion, such as generating high-resolution images and converting sketches into color images.

[0067] The neural network structure described above is trained using pre-prepared training data. Training methods may include, but are not limited to, any one or a combination of the following: supervised learning, unsupervised learning, self-supervised learning, and transfer learning. Supervised learning utilizes labeled multimodal data for training. The model learns feature representations and prediction functions based on the input multimodal data and corresponding labels. For example, in image-text matching tasks, the model is trained using data labeled with images and corresponding text descriptions. This allows the model to learn the association between images and text, accurately predicting the corresponding text description given an image, or finding the best matching image given a text. Unsupervised learning discovers underlying structures and patterns in unlabeled multimodal data by learning features. For example, autoencoders are used to extract and reconstruct features from multimodal data, or contrastive learning methods are used to teach the model that similar multimodal data have similar feature representations. Self-supervised learning assigns the model self-supervised tasks, allowing the model to generate its own supervisory signals to learn feature representations. For example, in image-text data, the image can be used as input and the model can predict certain masked words in the corresponding text description, or the text description can be used as input and the model can predict certain local features of the image. Self-supervised learning can effectively leverage multimodal data for learning without requiring extensive manual annotation. Transfer learning uses a model pre-trained on large-scale data as an initialization, which is then fine-tuned on a specific multimodal task. The pre-trained model has already learned general feature representations and knowledge, and fine-tuning can make it better adaptable to new tasks and data distributions. For example, in multimodal perception tasks, a pre-trained CNN network model can be used to extract image features, and a pre-trained language model can be used to extract text features. The entire model can then be fine-tuned for the specific task. A feature extraction model extracts features from event information to obtain a high-dimensional feature vector, or first feature. The feature extraction model includes a neural network that extracts features from the text to obtain the first feature. The text of each event information is fed into the feature extraction model for processing, extracting the first feature corresponding to each event information. Feature extraction models may include, but are not limited to, any one or a combination of the following: recurrent neural networks (RNNs), convolutional neural networks (CNNs), and transfer transformer networks. RNNs are a type of neural network designed to process sequential data. They have a memory function and can model historical information in a sequence. In text feature extraction, text can be viewed as a sequence of characters or words. For example, for the text "I love natural language processing," an RNN processes each character or word in turn. The output at each time step depends not only on the current input but also on the memory of previous time steps.The Transformer network is based on a self-attention mechanism, which allows the model to attend to all positions in the sequence when processing each position in the sequence, thereby capturing global dependencies. The MLP is a feedforward neural network composed of multiple layers, each containing multiple neurons, and fully connected between adjacent layers. In text feature extraction, text can be represented as a bag-of-words model or word embedding vector and then input into the MLP for feature extraction. For example, for a sentiment analysis task, after converting the text into a bag-of-words vector and inputting it into the MLP, the hidden layer of the MLP can learn the emotional characteristics represented by word combinations in the text, such as the possibility that certain negative word combinations correspond to anger.

[0068] The feature extraction model is trained using pre-prepared training data, and the training method may include but is not limited to any one or a combination of the following: supervised learning, unsupervised learning, self-supervised learning, transfer learning, etc. Among them, supervised learning uses labeled multimodal data for training, and the model learns feature representations and prediction functions based on the input multimodal data and the corresponding labels. For example, in the image-text matching task, the model is trained using data labeled with images and corresponding text descriptions, so that the model can learn the association between images and texts, so that it can accurately predict the corresponding text description when given an image, or find the most matching image when given text. Unsupervised learning discovers the potential structure and patterns in the data by learning features from unlabeled multimodal data.

[0069] The multimodal feature reflects the features in the monitoring area, and the first feature reflects the features in the event information. In order to determine the event occurring in the monitoring area, the multimodal feature is matched with the first feature respectively to determine the proximity of the features, and the event information corresponding to the first feature closest to the multimodal feature is determined as the target event. In such a matching process, the features corresponding to the multimodal data in the monitoring area are taken into account, and the first feature corresponding to the event information is matched, which can accurately determine the target event and improve the accuracy of security monitoring. Feature matching refers to finding similar features in two or more groups of data. These features usually correspond to some kind of similarity or correlation. In matching the features of multimodal perception data with the features corresponding to the event information, any one of the following or any combination of multiple methods can be used: distance-based matching method, feature vector-based matching method, graph-based matching method, and deep learning-based matching method. Distance-based matching methods calculate specific mathematical distances based on features and determine the degree of feature matching based on the size of the distance. For example, Euclidean distance is used to calculate the straight-line distance between two feature vectors, Manhattan distance is used to calculate the sum of the absolute differences between two feature vectors in each dimension, and cosine similarity measures the similarity between two vectors by calculating the cosine of the angle between them. Among feature vector-based matching methods, nearest neighbor matching finds the closest feature vector for each feature vector as the matching result and is widely used in scenarios such as image recognition. The SIFT image feature extraction algorithm is used to extract image feature points and their descriptors.

[0070] In one possible embodiment, feature extraction is performed on image data in multimodal perception data. In computer vision, images are typically represented in matrix form. This matrix consists of multiple vectors, and image features can be extracted through one or a combination of operations such as convolution, activation functions, and pooling. In this process, each vector represents certain information in the image, such as edges and textures. Finally, in the fully connected layer, all feature vectors are flattened into a single vector to represent the features of the entire image. This vector can be thought of as splitting the image into several blocks from left to right and from top to bottom and arranging them in sequence. In this way, complex image information can be converted into a vector representation.

[0071] In one possible embodiment, feature extraction is performed on the sound data in the multimodal perception data, and the sound data is input into a speech-to-text model to attempt to convert it into text. If the conversion into text cannot be performed, it means that no one is speaking in the monitoring area. The sound data is then input into a feature extraction model to directly extract features from the speech. These features reflect the characteristics of the sound in the monitoring area, and it can be analyzed based on what object the sound is emitted from and whether it is the sound caused by contact with a vehicle. If the sound data can be converted into text, it means that someone is speaking in the detection area. The text corresponding to the sound data is split into multiple words, the word vector corresponding to each word is determined, and the feature vector corresponding to the entire text is obtained. The feature vector reflects the characteristics of the text, and the intention contained in the text can be analyzed based on this.

[0072] In a possible embodiment, for the feature vectors extracted from the perception data of each modality, the size of the feature vectors is adjusted to unify the sizes of the feature vectors, and the feature vectors with unified sizes are spliced ​​together to fuse the feature vectors of each modality to obtain the final multimodal features.

[0073] Optionally, matching the multimodal feature with first features respectively corresponding to at least one event information to determine a target first feature matching the multimodal feature among multiple first features includes:

[0074] A first feature having the highest similarity with the multimodal feature among the first features corresponding to at least one event information is determined as a target first feature.

[0075] In this embodiment, in the process of matching the multimodal features corresponding to the multimodal sensing data with the first features corresponding to the event information, it is necessary to specify the matching criteria in order to determine the event that best matches the situation in the monitored area. The similarity between the multimodal features and the first features is used as the matching criterion. By determining the event information that matches the multimodal sensing data based on the similarity of the features, the information in the first feature and the multimodal features is fully considered, and the target event can be determined more accurately, thereby improving the accuracy of safety detection, enhancing the safety of the vehicle, and enhancing the user experience of the vehicle.

[0076] In a possible embodiment, a cosine similarity calculation method is used to determine the similarity between features, which is used to measure the cosine value of the angle between two feature vectors, thereby providing a way to evaluate the relationship between feature vectors. The calculation method is: a·b / (||a||*||b||), where a is a multimodal feature, b is the first feature, a and b are both vectors, a·b is the inner product of a and b, ||a|| is the modulus of a, and ||b|| is the modulus of b. The value range of cosine similarity is [0,1]. The higher the value of cosine similarity, the higher the similarity between the first feature and the multimodal feature, and the more the event information fits the monitored multimodal perception data. The first feature with the highest cosine similarity with the multimodal feature is selected as the target first feature, and the event information corresponding to the target first feature is determined as the target event. Optionally, the straight-line distance between the first feature and the multimodal feature can be calculated using the Euclidean distance, or the sum of the absolute differences between the first feature and the multimodal feature in each dimension can be calculated using the Manhattan distance as the similarity.

[0077] Optionally, the extracting features from the multimodal sensing data to obtain multimodal features includes:

[0078] Determining a target object in the multimodal perception data, and generating a second feature corresponding to the target object;

[0079] Acquire a behavior tag and extract a third feature corresponding to the behavior tag, wherein the behavior tag is used to characterize the behavior existing in the monitoring area;

[0080] The second feature and the third feature are fused to generate the multimodal feature.

[0081] In this embodiment, when feature extraction is performed on image data in multimodal perception data (such as images taken by ordinary cameras, images taken by infrared cameras, etc.), if feature extraction is performed on the entire image data, the required amount of calculation is large and the processing efficiency is low. In this embodiment, the multimodal perception data is first input into a target detection model to extract the target object therein. The target detection model includes a multi-layer neural network, which is trained by inputting image data and detection frame labels. The target object is an object that may pose a threat to vehicle safety, such as pedestrians, other vehicles, spheres, etc. By extracting the target object and extracting features from the multimodal perception data corresponding to the target object to obtain a second feature, the extracted second feature is more targeted, which avoids irrelevant factors in the image from affecting the accuracy of the feature and improves the accuracy of safety detection.

[0082] Behavior tags are used to characterize behaviors within the monitoring area. Since the vehicle's doors are locked when parked, the safety of the interior of the vehicle is guaranteed. Security threats to the vehicle primarily originate from outside the vehicle. Therefore, the monitoring area surrounding the vehicle is the area outside the vehicle centered on the vehicle. Optionally, the monitoring area is a circular area with the vehicle at its center. Behaviors within the monitoring area are those of external people, animals, vehicles, and so on. Examples include pedestrians approaching, moving away from, or knocking on the vehicle; animals approaching, moving away from, or barking at the vehicle; and other vehicles approaching, moving away from, or colliding with the vehicle. A third feature is extracted based on the text in the behavior tag; the third feature characterizes the behavior. The second feature is fused with the third feature to generate the multimodal feature. The second feature is enhanced by the third feature corresponding to the behavior tag, so that the multimodal feature contains more information related to the behavior. This increases the effective information within the multimodal feature and enables more accurate identification of target events occurring within the monitoring area during the matching process.

[0083] In a possible embodiment, a detection frame is generated by a target detection model, and the image in the detection frame is the image corresponding to the target object. The image data is cropped according to the detection frame to obtain the image of the target object, and features are extracted from the image of the target object to obtain a second feature.

[0084] In a possible embodiment, the behavior tags include standing still / walking / running / squatting / holding objects / throwing objects / knocking / climbing / wandering / falling down / left behind objects / abnormal movements, etc., and one or more extracted features are selected from the behavior tags to generate the third feature.

[0085] The second feature is fused with the third feature to generate the multimodal feature, and the second feature corresponding to the target object is strengthened by the third feature corresponding to the behavior label, so that the feature can more accurately characterize the events occurring in the monitoring area. Optionally, the fusion can adopt any one of the following methods or a combination of multiple methods: Feature concatenation: multiple feature vectors are connected end to end to form a long feature vector. This is the simplest and most direct method and is suitable for situations where the feature dimensions do not conflict. Feature weighted fusion: different weights are assigned to different features, and then a weighted sum is performed. The weight can be determined based on the importance or relevance of the features. Feature transformation and alignment: different features are transformed so that they are represented in the same feature space, and then fused. For example, features are aligned through linear transformation or nonlinear mapping.

[0086] Optionally, the fusing the second feature and the third feature to generate the multimodal feature includes:

[0087] Splicing the second feature and the third feature to generate a fourth feature;

[0088] Performing feature projection on the fourth feature to generate an attention matrix;

[0089] The fourth feature is weighted according to the attention matrix to generate the multimodal feature.

[0090] In this embodiment, the second feature and the third feature are fused into a fourth feature by means of feature splicing, and the fourth feature is further enhanced by the attention mechanism. The attention mechanism is a technology widely used in deep learning, mainly used to process sequence data (such as sentences and video data in natural language processing). Its core idea is to allow the model to dynamically focus on the part most relevant to the current task when processing a certain input, rather than treating all inputs equally. The generated attention matrix reflects the importance of each part of the fourth feature, and the importance is related to the correlation between the part and the event. Weighting the fourth feature by the attention matrix can enhance information related to the event and weaken information unrelated to the event. The multimodal features finally obtained can more accurately characterize the events occurring in the monitoring area.

[0091] Optionally, the method further includes:

[0092] Matching the multimodal perception data with a matching condition in a preset rule corresponding to at least one event information;

[0093] Determine a target preset rule corresponding to a matching condition satisfied by the multimodal perception data, and determine the target event according to event information corresponding to the target preset rule.

[0094] In this embodiment, users can configure rules corresponding to event information and perform quantitative matching based on the matching conditions in the rules. This matching method simply compares multimodal sensor data with the corresponding conditions. If the multimodal sensor data meets all the conditions in the preset rules, the corresponding event is determined to have occurred. The matching conditions in the rules can include a combination of one or more conditions, which are related to the multimodal data collected by sensors installed on the vehicle. The conditions are typically set by setting thresholds. For example, the condition is met if a certain data is above a certain threshold; if a certain data is below a certain threshold; or if a certain data is above a first threshold and below a second threshold.

[0095] In one possible embodiment, the conditions in the preset rule corresponding to the event information "Nighttime Pedestrian Approaching" include: light intensity less than 50 lux, the distance between the person and the vehicle less than 1.5 meters, and a duration greater than 30 seconds. After collecting multimodal perception data, the illumination data and radar data are matched with the conditions in the rule. The illumination is determined based on the illumination data, and the distance between the person and the vehicle is determined based on the radar data. If both the illumination and distance meet the conditions, the duration is further determined. If the duration is greater than 30 seconds, the match is determined to be successful, and the target event "Nighttime Pedestrian Approaching" has been confirmed to have occurred.

[0096] Optionally, the method further includes:

[0097] The at least one event information is determined in response to a first input, where the first input is an operation of a user inputting at least one event information or an operation of selecting at least one event information.

[0098] In this embodiment, event information is user-configured based on vehicle safety requirements. Users can enter event information into the system for configuration. Event information can include qualitative descriptions of the event, such as "multiple people scratching the vehicle," or quantitative descriptions, such as "more than two people surrounded the vehicle for more than 20 minutes." By first inputting customized event information, users can customize safety monitoring based on their vehicle safety needs. This makes the safety monitoring process more targeted, effectively meeting user needs and improving the user experience.

[0099] In one possible embodiment, the user inputs event information via voice, which is converted into corresponding text by a speech-to-text conversion model, and the text serves as the event information. Alternatively, the user configures event information by inputting text, and the text serves as the event information.

[0100] In a possible embodiment, if the user attaches great importance to the safety of the vehicle during nighttime parking and is worried about animals approaching the vehicle at night and causing damage to the vehicle, an event message such as "animals approaching the vehicle at night" can be set. After matching the multimodal perception data that matches the event information, the vehicle's control system will respond, such as turning on the lights or playing a sound to drive away the animals, and sending a reminder message to the owner's vehicle control application account.

[0101] In a possible embodiment, if the user believes that a person in uniform (such as a security guard or cleaning staff) approaching the vehicle will not pose a threat to the safety of the vehicle, the user can set event information such as "a person not in uniform has been near the vehicle for more than 10 minutes". In this way, the multimodal features obtained when a person in uniform approaches the vehicle will not match the event information, thus avoiding false alarms.

[0102] In one possible embodiment, a user sees a safety incident involving a parked vehicle on their phone or in-vehicle app. To avoid the incident and prioritize it during vehicle safety monitoring, the user can select text from the incident or a screenshot from the app. The in-vehicle app can then generate event information based on the text, or input the screenshot into a text recognition model to retrieve the text and generate event information based on the text.

[0103] In a possible embodiment, a user may configure a score corresponding to the event information. The score reflects the severity of the security event corresponding to the event information, and different scores correspond to different response decisions.

[0104] In a possible embodiment, the score ranges from 0.5 to 1, and the responses for different scores are as follows:

[0105] 0.7-0.8: Local storage event logs include recorded videos and triggered events.

[0106] 0.8-0.9: Based on the previous level, the vehicle body light module is controlled through the bus to flash as a warning.

[0107] 0.9 and above: Emergency notification to the car owner via app push, phone call, text message, etc.

[0108] In order to implement the above embodiment, the embodiment of the present application also proposes a safety monitoring device.

[0109] In a possible embodiment, in response to the vehicle being in a parked state, multimodal perception data of a monitoring area corresponding to the vehicle is collected through sensors configured on the vehicle, including but not limited to any one or a combination of the following: image data, sound data, light intensity data, temperature data, etc.;

[0110] Then, a target object (such as a vehicle, pedestrian, animal, etc.) in the multimodal perception data is detected using a pre-trained multimodal model, and a feature extraction neural network structure in the multimodal model performs feature extraction on the image of the target object to obtain a second feature corresponding to the target object;

[0111] Obtaining a pre-configured behavior label, and extracting a third feature from the behavior label text using a multimodal model, wherein the behavior label is used to characterize the behavior existing in the monitoring area;

[0112] In the multimodal model, the second and third features are concatenated, and the second feature corresponding to the target object is enhanced with the third feature corresponding to the behavior label to generate a fourth feature. The fourth feature is then feature-projected in the multimodal model to generate an attention matrix reflecting the importance of each component of the fourth feature. Finally, the fourth feature is weighted in the multimodal model based on the attention matrix to generate a multimodal feature.

[0113] In response to a first input from a user, at least one event information is determined, wherein the first input is an operation in which the user inputs at least one event information or selects at least one event information. The event information input by the user is integrated with the pre-configured event information. The first feature corresponding to each event information is extracted, each first feature is matched with the multimodal feature, and the similarity between the first feature and the multimodal feature is calculated. After the calculation is completed, the first feature corresponding to the highest similarity is determined as the target first feature, and then the event information corresponding to the target first feature is determined as the target event; wherein, determining at least one event information / extracting the first feature corresponding to each event information can be determined in advance or determined in real time by the user when the user starts the function corresponding to the security monitoring method (such as turning on the sentry mode), such as determining at least one event information / extracting the first feature corresponding to each event information once, and the security monitoring method can be performed multiple times within a certain period of time using the determined event information / first feature, or the at least one event information / extracting the first feature corresponding to each event information can be regularly updated or re-determined, and the present invention is not limited here.

[0114] Optionally, when determining the target event, the multimodal perception data can also be matched with one or more matching conditions in a preset rule corresponding to at least one event information; the target preset rule corresponding to the matching condition satisfied by the multimodal perception data is determined, and the target event is determined based on the event information corresponding to the target preset rule.

[0115] In a possible embodiment, the user sets the sentry mode to be turned on and pre-configures one or more event information through voice input or text input, and then starts monitoring the monitoring area around the vehicle after the vehicle is turned off.

[0116] 1. Multimodal Data Collection and Preprocessing: When the vehicle enters parking mode (ignition off and locked), one or more of the following sensors are activated: surround-view camera, millimeter-wave radar, ultrasonic sensor, and ambient light sensor. After the sensors collect data, the multimodal data is transmitted to the onboard computing unit via the CAN bus.

[0117] 2. Feature extraction:

[0118] (1) The on-board computing unit calls the pre-trained multimodal model to perform target detection on the image data captured by the camera, detects targets that may pose a threat to the vehicle (such as people, other vehicles, animals, etc.), obtains the target frame corresponding to the target, extracts the image in each target frame, and uses the visual encoder in the multimodal model to encode the image in the target frame to generate the second feature V.

[0119] (2) Multi-label behavior classification: A lightweight multi-label classifier is constructed for classification to generate multiple behavior labels (for example, the following behavior labels: {stationary / walking / running / squatting / holding objects / throwing objects / knocking / climbing / wandering / falling / leftover objects / abnormal actions}). The behavior labels are input into the text encoder in the multimodal model and converted into the third feature T.

[0120] 3. Dynamic feature pool construction:

[0121] (1) Multimodal feature fusion: The second feature V and the third feature T are fused, and the second feature corresponding to the target object is enhanced by the third feature corresponding to the behavior label to generate the fourth feature F.

[0122] (2) The fourth feature is projected through the attention network in the multimodal model to generate an attention matrix W that reflects the importance of each part of the fourth feature. The fourth feature is weighted according to the attention matrix to generate a multimodal feature F'=W⊙F.

[0123] 4. Dual-channel matching engine

[0124] (1) Semantic channel matching: One or more pre-configured event information is converted into the first feature through a text encoder, and the cosine similarity between each first feature and the feature F' after multimodal projection is calculated to obtain Score_S. The event information corresponding to the first feature corresponding to the maximum value of Score_S is determined as the target event.

[0125] (2) Rule-based channel matching: Use pre-designed rules to perform hard matching on multimodal data. The rule corresponding to an event information includes one or more matching conditions. The example rules are as follows:

[0126] Event information: The matching rules for night approach events are:

[0127] When the ambient light is less than 50 lux, the distance between the pedestrian and the vehicle is less than 1.5 m, and the duration of the above situation is greater than 3 seconds, it is determined that a night approach event has occurred, and a score pre-configured for the night approach event, Score_R=0.8, is obtained.

[0128] (3) Custom arbitration algorithm: Calculate the comprehensive score based on Score_S and Score_R = α·Score_R + β·Score_S, where α and β are weights that can be adjusted by the user according to preference.

[0129] 5. Tiered response decision: Different responses are implemented based on the comprehensive score, including three levels:

[0130] (1) 0.7-0.8: Local storage event logs including recorded video, triggered events

[0131] (2) 0.8-0.9: Based on the previous level, the vehicle body light module is controlled through the bus to flash warning

[0132] (3) 0.9 or above: Emergency notification to the owner via app push, phone call, SMS, etc.

[0133] Figure 2 A schematic structural diagram of a safety monitoring device provided in an embodiment of the present application.

[0134] like Figure 2 As shown, the device may include:

[0135] The acquisition module 210 is configured to acquire multimodal sensing data of a monitoring area corresponding to the vehicle in response to the vehicle being in a parked state;

[0136] The processing module 220 is configured to perform feature matching on at least one event information and the multimodal perception data, and determine a target event that matches the multimodal perception data from the at least one event information.

[0137] It should be noted that the above explanation of the method embodiment is also applicable to the device of this embodiment and will not be repeated here.

[0138] In order to implement the above embodiments, the present application also proposes a non-transitory computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in the above method embodiments is implemented.

[0139] In order to implement the above embodiments, the present application further proposes a computer program product on which a computer program is stored. When the computer program is executed by a processor, the method described in the above method embodiments is implemented.

[0140] In order to implement the above embodiments, the present application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method described in the above method embodiments is implemented.

[0141] Figure 3This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. For example, the electronic device 800 can be a mobile phone, a computer, a vehicle, a car computer, a smart cockpit, etc.

[0142] Reference Figure 3 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0143] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0144] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0145] The power component 806 provides power to the various components of the electronic device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0146] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0147] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0148] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0149] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of components, such as the display and keypad of the electronic device 800. The sensor assembly 814 can also detect changes in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and temperature changes of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0150] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0151] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0152] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the electronic device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0153] In order to implement the above embodiments, the present application further proposes a chip, including: the chip includes a processing circuit, and the processing circuit is configured to execute the method provided in the above embodiments.

[0154] Figure 4 This is a schematic diagram of the structure of a chip proposed in the embodiment of this application. Figure 4 The structure of the chip 1100 is shown, but is not limited thereto.

[0155] The chip 1100 includes a processing circuit 1101 , which is configured to execute any of the above methods.

[0156] In some embodiments, chip 1100 further includes one or more interface circuits 1102. Optionally, interface circuit 1102 is connected to memory 1103. Interface circuit 1102 can be used to receive signals from memory 1103 or other devices, and can be used to send signals to memory 1103 or other devices. For example, interface circuit 1102 can read instructions stored in memory 1103 and send the instructions to processing circuit 1101.

[0157] In some embodiments, the interface circuit 1102 performs at least one of the communication steps such as sending and / or receiving in the above method, and the processing circuit 1101 performs the other steps.

[0158] In some embodiments, terms such as interface circuit, interface, transceiver pin, and transceiver may be used interchangeably.

[0159] In some embodiments, the chip 1100 further includes one or more memories 1103 for storing instructions. Alternatively, all or part of the memories 1103 may be located outside the chip 1100 .

[0160] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0161] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0162] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0163] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0164] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0165] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0166] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0167] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A safety monitoring method, characterized in that: include: In response to the vehicle being in a parked state, collecting multimodal perception data of a monitoring area corresponding to the vehicle; Feature matching is performed on at least one event information and the multimodal perception data, and a target event matching the multimodal perception data is determined from the at least one event information.

2. The method according to claim 1, characterized in that The performing feature matching on at least one event information and the multimodal perception data, and determining a target event matching the multimodal perception data from the at least one event information, includes: Performing feature extraction on the multimodal sensing data to obtain multimodal features; The multimodal feature is matched with first features respectively corresponding to at least one event information, a target first feature matching the multimodal feature among multiple first features is determined, and the event information corresponding to the target first feature is determined as the target event.

3. The method according to claim 2, characterized in that The matching of the multimodal feature with the first features respectively corresponding to at least one event information to determine a target first feature matching the multimodal feature among the plurality of first features includes: A first feature having the highest similarity with the multimodal feature among the first features corresponding to at least one event information is determined as a target first feature.

4. The method according to claim 2, characterized in that The extracting features from the multimodal sensing data to obtain multimodal features includes: Determining a target object in the multimodal perception data, and generating a second feature corresponding to the target object; Acquire a behavior tag and extract a third feature corresponding to the behavior tag, wherein the behavior tag is used to characterize the behavior existing in the monitoring area; The second feature and the third feature are fused to generate the multimodal feature.

5. The method according to claim 4, characterized in that The fusing the second feature and the third feature to generate the multimodal feature includes: Splicing the second feature and the third feature to generate a fourth feature; Performing feature projection on the fourth feature to generate an attention matrix; The fourth feature is weighted according to the attention matrix to generate the multimodal feature.

6. The method according to claim 1, characterized in that The method further comprises: Matching the multimodal perception data with a matching condition in a preset rule corresponding to at least one event information; Determine a target preset rule corresponding to a matching condition satisfied by the multimodal perception data, and determine the target event according to event information corresponding to the target preset rule.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: The at least one event information is determined in response to a first input, where the first input is an operation of a user inputting at least one event information or an operation of selecting at least one event information.

8. A safety monitoring device, characterized in that: include: a collection module, configured to collect multimodal perception data of a monitoring area corresponding to the vehicle in response to the vehicle being in a parked state; A processing module is used to perform feature matching on at least one event information and the multimodal perception data, and determine a target event that matches the multimodal perception data from the at least one event information.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A chip, characterized in that: The chip comprises a processing circuit configured to execute the method according to any one of claims 1 to 7.

12. A vehicle, characterized in that: The vehicle is used to perform the method according to any one of claims 1 to 7.

13. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images