Mobile unmanned security method and device and electronic equipment

Abstract

The invention discloses a mobile unmanned security method and device and electronic equipment, and relates to the field of robots. The method comprises the following steps: moving in a park according to a preset patrol route, and acquiring various sensor data; determining data types corresponding to the sensor data, wherein the data types comprise visual sensor data, acoustic sensor data and environmental sensor data; matching a target feature extraction algorithm corresponding to the target sensor data according to the target data type; performing feature extraction on the target sensor data by adopting a target feature extraction algorithm to obtain target features; the target features are input into a preset anomaly recognition model, whether abnormal conditions exist or not is judged, and the abnormal conditions comprise one or more of the vehicle abnormal conditions, the personnel abnormal conditions and the environment abnormal conditions; and if it is determined that there is an abnormal condition, triggering an alarm. According to the technical scheme, visual sense, sound sense and environment data can be movably acquired in an omnibearing mode, after it is determined that an abnormal condition exists, an alarm is triggered in time, and the response speed of security and protection is increased.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a mobile unmanned security method, device and electronic equipment. Background Art

[0002] With the acceleration of globalization and industrialization, security monitoring of industrial parks has become one of the core issues of enterprise management. Industrial parks usually involve a large number of assets and complex production activities, covering many aspects from production line operation to employee safety. An effective security system can not only monitor theft and vandalism, but also ensure the smooth progress of the production process and the safety of employees. In addition, with the strictness of safety regulations and standards, it has become particularly important to carry out effective security monitoring of industrial parks, which requires the security system to have high reliability and real-time response capabilities.

[0003] At present, although there are many security technologies on the market, including fixed surveillance cameras, traditional patrol personnel, etc., these methods are still insufficient in the application of industrial parks. These traditional security methods are difficult to analyze and provide immediate feedback on the received data in real time, making the response speed in emergency situations unsatisfactory.

[0004] Therefore, there is an urgent need for a mobile unmanned security method, device and electronic equipment. Summary of the invention

[0005] The present application provides a mobile unmanned security method, device and electronic equipment, which can obtain visual, acoustic and environmental sensor data in an all-round manner, and trigger an alarm in time after determining that an abnormal situation exists, thereby improving the response speed of security.

[0006] In a first aspect of the present application, a mobile unmanned security method is provided, in which a preset patrol route is used to move within a park, and a variety of sensor data are obtained; the data source of each of the sensor data is traced to obtain a data type corresponding to each of the sensor data; the data type includes visual sensor data, acoustic sensor data, and environmental sensor data; according to the target data type, a target feature extraction algorithm corresponding to the target sensor data is matched, the target sensor data is any one of the multiple sensor data, and the target data type is the data type corresponding to the target sensor data; the target feature extraction algorithm is used to extract features from the target sensor data to obtain target features; the target features are input into a preset abnormality recognition model to determine whether there is an abnormal situation, the abnormal situation includes one or more of vehicle abnormality, personnel abnormality, and environmental abnormality; if it is determined that an abnormal situation exists, an alarm is triggered, and an alarm message is sent to the park security personnel, the alarm message includes the abnormal situation type, the abnormality location, the abnormality detection time, and the abnormal situation picture.

[0007] By adopting the above technical solutions, we can move around the park and obtain a variety of sensor data, realizing all-round and multi-angle perception of the park environment. By tracing the data source of the sensor data, we can accurately distinguish the visual sensor data, acoustic sensor data and environmental sensor data, laying the foundation for the subsequent adoption of matching feature extraction algorithms for different sensor data types. By using the matching target feature extraction algorithm, we can efficiently extract key information reflecting the target characteristics from the massive sensor data, and realize intelligent abnormal behavior analysis and risk assessment by inputting the preset abnormality recognition model. Once a vehicle abnormality, personnel abnormality or environmental abnormality is found, an alarm can be automatically triggered and an alarm message containing abnormal details can be sent to the security personnel, greatly improving the real-time response capability and disposal efficiency of the park security.

[0008] Optionally, the target feature extraction algorithm is used to perform feature extraction on the target sensor data to obtain target features, specifically including: if it is determined that the target data type is visual sensor data, a convolutional neural network is used to perform feature extraction on the visual sensor data to obtain vehicle appearance features and personnel appearance features; if it is determined that the target data type is acoustic sensor data, a Mel-frequency cepstral coefficient extraction algorithm is used to perform feature extraction on the acoustic sensor data to obtain vehicle sound features and personnel sound features; if it is determined that the target data type is environmental sensor data, a statistical feature algorithm is used to perform feature extraction on the environmental sensor data to obtain temperature features, humidity features and air quality features.

[0009] By adopting the above technical solutions, targeted feature extraction algorithms are used according to different data types to give full play to the data endowments of different sensors and maximize the mining of feature information related to abnormal behavior recognition. For visual sensor data, a convolutional neural network model is used. For acoustic sensor data, a Mel frequency cepstral coefficient extraction algorithm is used to adaptively extract vehicle sound features and personnel sound features from sound signals by simulating the frequency selection characteristics of the human ear. For environmental sensor data, a statistical feature algorithm is used to calculate temperature features, humidity features, and air quality features, laying a solid data foundation for intelligent security decision-making.

[0010] Optionally, the use of a convolutional neural network to perform feature extraction on the visual sensor data to obtain appearance features specifically includes: performing a preprocessing operation on the visual sensor data to obtain a preprocessed image, wherein the preprocessing operation includes image scaling and image enhancement; inputting the preprocessed image into a convolutional neural network, and extracting local features and global features of the preprocessed image through multi-layer convolution and pooling operations; at the end of the convolutional neural network, setting different fully connected layers to output vehicle appearance features and personnel appearance features according to the local features and the global features, respectively.

[0011] By adopting the above technical solution, introducing image preprocessing operations, scaling and enhancing the input visual sensor data, the image size and distribution can be unified, and the quality and efficiency of subsequent feature extraction can be improved. The preprocessed image is sent to the convolutional neural network. Through layer-by-layer convolution and pooling operations, the local and global features in the image data can be automatically learned, and the visual information contained in the image can be fully mined. At the end of the network, multiple fully connected layers are connected in parallel to output the appearance features of vehicles and personnel respectively, realizing the decoupling and parallel extraction of vehicle visual features and personnel visual features, which not only retains the personalized features of the two types of targets, but also takes into account the efficiency of calculation.

[0012] Optionally, the use of a Mel-frequency cepstral coefficient extraction algorithm to perform feature extraction on the acoustic sensor data to obtain vehicle sound features and personnel sound features specifically includes: pre-emphasizing and framing the acoustic sensor data to obtain multiple frame data; performing fast Fourier transform on each frame data to obtain a spectrum signal; inputting the spectrum signal into a Mel filter group to obtain a Mel spectrum; performing discrete cosine transform on the Mel spectrum to obtain Mel-frequency cepstral coefficients; and determining corresponding vehicle sound features and personnel sound features based on the Mel-frequency cepstral coefficients.

[0013] By adopting the above technical solution, the acoustic sensor data is pre-emphasized and framed, which can highlight the high-frequency part and balance the contribution of each frequency component. The frame data is mapped from the time domain to the frequency domain through fast Fourier transform to obtain the spectrum signal, making the frequency characteristics of the sound more obvious. The spectrum signal is passed through a set of Mel filters to obtain the Mel spectrum, which simulates the auditory characteristics of the human ear and highlights the frequency bands related to perception. Finally, the Mel spectrum is subjected to discrete cosine transform to obtain the Mel frequency cepstrum coefficient, which captures the resonance peak information of the sound signal and forms a compact feature representation. Based on the Mel frequency cepstrum coefficient, the vehicle sound characteristics and the personnel sound characteristics can be further distinguished. In summary, the technical solution fully considers the frequency domain characteristics of the sound signal and the auditory characteristics of the human ear. The obtained vehicle sound characteristics and personnel sound characteristics can play an important role in speech recognition, voiceprint discrimination, etc., and provide strong support for the recognition of abnormal vehicle conditions and abnormal personnel conditions under acoustic modalities.

[0014] Optionally, the use of a statistical feature algorithm to extract features from the environmental sensor data to obtain temperature features, humidity features and air quality features specifically includes: normalizing and segmenting the environmental sensor data to obtain a mean, variance, maximum value and minimum value corresponding to the environmental sensor data; performing vector encoding according to the mean, variance, maximum value and minimum value to obtain corresponding temperature features, humidity features and air quality features.

[0015] By adopting the above technical solution, the temperature, humidity, and air quality data collected by the environmental sensors are normalized and segmented, and the mean, variance, maximum, and minimum values ​​of various data are calculated. The central tendency, discrete degree, and numerical range distribution characteristics of the environmental sensor database are quantitatively characterized, and the environmental conditions are reflected from a statistical perspective. These statistics are further vectorized and encoded to form feature vector representations such as temperature characteristics, humidity characteristics, and air quality characteristics, which not only reduces the data dimension but also retains the essence of environmental information.

[0016] Optionally, before inputting the target feature into a preset abnormality recognition model to determine whether there is an abnormal situation, the method also includes: acquiring historical multimodal data of visual sensors, acoustic sensors, and environmental sensors; annotating the historical multimodal data to obtain an annotated data set, wherein the annotated data set includes vehicle abnormality annotations, personnel abnormality annotations, and environmental abnormality annotations; constructing corresponding abnormality recognition sub-models for different abnormal situations, wherein the abnormality recognition sub-models include a vehicle abnormality recognition model, a personnel abnormality recognition model, and an environmental abnormality recognition model; inputting the annotated data set into each of the abnormality recognition sub-models for training, and combining each of the abnormality recognition sub-models to obtain the preset abnormality recognition model.

[0017] By adopting the above technical solution, we can obtain historical multimodal data from visual sensors, acoustic sensors, and environmental sensors, annotate these data, and build a labeled data set covering various abnormal situations of vehicles, personnel, and environment, providing sufficient and comprehensive samples for model training. For different abnormal situations, we build corresponding abnormal recognition sub-models, including vehicle abnormal recognition model, personnel abnormal recognition model, and environmental abnormal recognition model. Through training on the labeled data set, each abnormal recognition sub-model can master the judgment rules of the corresponding abnormal pattern. The trained abnormal recognition sub-models are combined to form a complete preset abnormal recognition model, realizing the unified judgment of multiple types of abnormalities.

[0018] Optionally, the target feature is input into a preset abnormality recognition model to determine whether there is an abnormal situation, specifically including: classifying the target feature according to dimension categories to obtain target sub-features, the dimension categories including vehicle dimension, personnel dimension and environment dimension; according to the dimension categories, the target sub-features are input into corresponding abnormality recognition sub-models respectively to determine whether there is an abnormal situation.

[0019] By adopting the above technical solution, the extracted target features are first classified according to the three dimensions of vehicle dimension, personnel dimension, and environmental dimension, and the complex heterogeneous features are decoupled into multiple target sub-features according to semantic attributes, simplifying the subsequent abnormal judgment logic. Using the dimension category, the target sub-features are sent to the corresponding abnormal recognition sub-model to achieve classified abnormal situation judgment. For example, the features related to the vehicle dimension are sent to the vehicle abnormal recognition model, the features related to the personnel dimension are sent to the personnel abnormal recognition model, and the features related to the environmental dimension are sent to the environmental abnormal recognition model.

[0020] In a second aspect of the present application, a mobile unmanned security device is provided, the device comprising: an acquisition module and a processing module, wherein: the acquisition module is used to move in the park according to a preset patrol route and acquire multiple sensor data; the processing module is used to trace the data source of each of the sensor data to obtain the data type corresponding to each of the sensor data; the data type includes visual sensor data, acoustic sensor data and environmental sensor data; the processing module is also used to match the target feature extraction algorithm corresponding to the target sensor data according to the target data type, the target sensor data is any one of the multiple sensor data, and the target data type is the data type corresponding to the target sensor data; the processing module is also used to extract features from the target sensor data using the target feature extraction algorithm to obtain target features; the processing module is also used to input the target features into a preset abnormality recognition model to determine whether there is an abnormal situation, the abnormal situation includes one or more of vehicle abnormal situation, personnel abnormal situation and environmental abnormal situation; the processing module is also used to trigger an alarm if it is determined that there is an abnormal situation, and send an alarm message to the park security personnel, the alarm message includes the abnormal situation type, the abnormality occurrence location, the abnormality detection time and the abnormal situation picture.

[0021] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes any one of the methods described above.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, any of the methods described above is executed.

[0023] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. Move in the park and obtain a variety of sensor data to achieve all-round and multi-angle perception of the park environment. By tracing the data source of the sensor data, the visual sensor data, acoustic sensor data and environmental sensor data can be accurately distinguished, laying the foundation for the subsequent adoption of matching feature extraction algorithms for different sensor data types. Using the matching target feature extraction algorithm, key information reflecting the target characteristics can be efficiently extracted from massive sensor data, and intelligent abnormal behavior analysis and risk assessment can be achieved by inputting a preset abnormality recognition model. Once a vehicle abnormality, personnel abnormality or environmental abnormality is found, an alarm can be automatically triggered and an alarm message containing abnormal details can be sent to the security personnel, greatly improving the real-time response capability and handling efficiency of the park security.

[0024] 2. According to different data types, targeted feature extraction algorithms are used to give full play to the data endowments of different sensors and maximize the mining of feature information related to abnormal behavior identification. For visual sensor data, a convolutional neural network model is used. For acoustic sensor data, a Mel frequency cepstral coefficient extraction algorithm is used to adaptively extract vehicle sound features and personnel sound features from sound signals by simulating the frequency selection characteristics of the human ear. For environmental sensor data, a statistical feature algorithm is used to calculate temperature features, humidity features, and air quality features, laying a solid data foundation for intelligent security decision-making.

[0025] 3. Introducing image preprocessing operations to scale and enhance the input visual sensor data can unify the image size and distribution, and improve the quality and efficiency of subsequent feature extraction. The preprocessed image is sent to the convolutional neural network. Through layer-by-layer convolution and pooling operations, the local and global features in the image data can be automatically learned, and the visual information contained in the image can be fully mined. At the end of the network, multiple fully connected layers are connected in parallel to output the appearance features of vehicles and personnel respectively, realizing the decoupling and parallel extraction of vehicle visual features and personnel visual features, which not only retains the personalized features of the two types of targets, but also takes into account the efficiency of calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of a mobile unmanned security method disclosed in an embodiment of the present application; Figure 2 It is a module schematic diagram of a mobile unmanned security device disclosed in an embodiment of the present application; Figure 3 It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present application.

[0027] Explanation of the reference numerals: 201, acquisition module; 202, processing module; 300, electronic device; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. DETAILED DESCRIPTION

[0028] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0029] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0030] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0031] This application provides a mobile unmanned security method, referring to Figure 1 , Figure 1 1 is a flow chart of a mobile unmanned security method provided in an embodiment of the present application. The method is applied to a mobile unmanned security robot and includes steps S101 to S106, which are as follows: Step S101: Move within the park according to a preset patrol route and obtain various sensor data.

[0032] In step S101, the mobile unmanned security robot has autonomous navigation and environmental perception capabilities. The robot's control module integrates algorithms such as path planning, positioning navigation, and obstacle avoidance. It can autonomously plan its route based on the pre-loaded park map information, and locate its own position in real time during movement, detect and avoid obstacles. Before performing a patrol mission, the robot will load preset patrol route information from a local storage device. The preset patrol route is planned and designed by park managers or security professionals based on the actual situation and security needs of the park. The robot uses its own navigation and positioning capabilities based on the loaded preset patrol route to move in an orderly manner along the preset patrol route in the park. During the movement, the robot will continue to collect various sensor data of the park environment through the sensor module it carries. The sensor module generally includes visual sensors, acoustic sensors, and environmental sensors. Visual sensors such as cameras can capture images and videos in the park. Acoustic sensors such as microphones can collect sound information in the park. Environmental sensors can monitor physical environmental parameters such as temperature, humidity, and smoke in the park.

[0033] Step S102: Tracing the data source of each sensor data to obtain the data type corresponding to each sensor data; the data type includes visual sensor data, acoustic sensor data and environmental sensor data.

[0034] In step S102, after the robot obtains the sensor data, it will parse and extract the sensor data to obtain the meta information of the sensor data. The meta information contains key attributes such as the source, type, format, and timestamp of the data, which are used to describe and identify the characteristics of the data. The robot can extract the corresponding meta information field from the received data packet according to the data format and encoding rules agreed in the communication protocol. According to the extracted meta information, the robot traces the data source of the sensor data to determine the source and type of the data. Specifically, the robot will analyze the fields such as the device ID and sensor ID in the meta information to identify which specific sensor device the data comes from. For example, the device ID in the meta information can uniquely identify a mobile unmanned security robot, and the sensor ID can further distinguish different sensor modules on the robot, such as cameras, microphones, temperature and humidity sensors, etc. After identifying the data source, the robot further determines the data type of the sensor data. Different types of sensor data have different data formats and characteristics. The robot can directly determine the data type according to the data type field or data format description in the meta information. The data type includes visual sensor data, acoustic sensor data, and environmental sensor data.

[0035] Step S103: matching a target feature extraction algorithm corresponding to the target sensor data according to the target data type, the target sensor data is any sensor data among a plurality of sensor data, and the target data type is a data type corresponding to the target sensor data.

[0036] In step S103, the robot obtains any type of sensor data (target sensor data), and first determines the type of the sensor data to determine whether it belongs to image, video, audio or time series data. This step can be directly judged by metadata such as file extension, data format, sensor identification, etc., or the data content can be sampled and analyzed to identify the corresponding target data type. According to the determined target data type, the robot selects a matching target feature extraction algorithm from the preset feature extraction algorithm library. The preset feature extraction algorithm library presets various common image feature extraction, audio feature extraction and time series feature extraction algorithm implementations. For different types of sensor data, one or more corresponding feature extraction algorithm options are provided to meet the needs of different application scenarios.

[0037] Step S104: extracting features from the target sensor data using a target feature extraction algorithm to obtain target features.

[0038] In step S104, if it is determined that the target data type is visual sensor data, a convolutional neural network is used to extract features from the visual sensor data to obtain vehicle appearance features and personnel appearance features; if it is determined that the target data type is acoustic sensor data, a Mel-frequency cepstral coefficient extraction algorithm is used to extract features from the acoustic sensor data to obtain vehicle sound features and personnel sound features; if it is determined that the target data type is environmental sensor data, a statistical feature algorithm is used to extract features from the environmental sensor data to obtain temperature features, humidity features and air quality features.

[0039] In one possible implementation, a convolutional neural network is used to perform feature extraction on visual sensor data to obtain appearance features, and the appearance features include vehicle appearance features and person appearance features. Specifically, the method includes: performing a preprocessing operation on the visual sensor data to obtain a preprocessed image, and the preprocessing operation includes image scaling and image enhancement; inputting the preprocessed image into a convolutional neural network, and extracting local features and global features of the preprocessed image through multi-layer convolution and pooling operations; at the end of the convolutional neural network, different fully connected layers are set to output vehicle appearance features and person appearance features according to local features and global features, respectively.

[0040] In a possible implementation, a Mel-frequency cepstral coefficient extraction algorithm is used to extract features from acoustic sensor data to obtain vehicle sound features and personnel sound features, specifically including: pre-emphasis and frame processing on the acoustic sensor data to obtain multiple frame data; fast Fourier transform on each frame data to obtain a spectrum signal; input the spectrum signal into a Mel filter group to obtain a Mel spectrum; discrete cosine transform on the Mel spectrum to obtain Mel-frequency cepstral coefficients; and determining corresponding vehicle sound features and personnel sound features according to the Mel-frequency cepstral coefficients.

[0041] In one possible implementation, a statistical feature algorithm is used to extract features from environmental sensor data to obtain temperature features, humidity features, and air quality features, specifically including: normalizing and segmenting the environmental sensor data to obtain the mean, variance, maximum, and minimum values ​​corresponding to the environmental sensor data; performing vector encoding based on the mean, variance, maximum, and minimum values ​​to obtain corresponding temperature features, humidity features, and air quality features.

[0042] Specifically, when the target data type is visual sensor data, the robot uses a convolutional neural network (CNN) to extract features from the visual image (visual sensor data) to obtain vehicle appearance features and personnel appearance features. The robot preprocesses the visual image, including image scaling and image enhancement, to obtain a preprocessed image. Image scaling unifies images of different sizes and resolutions to a fixed size, such as 224x224 pixels; image enhancement adjusts the brightness, contrast, and sharpness of the image to improve image quality and feature stability. The robot inputs the preprocessed image into the CNN model and extracts local and global features of the preprocessed image through multi-layer convolution and pooling operations. The convolution layer uses convolution kernels of different sizes and numbers to perform sliding window operations on the preprocessed image to extract local patterns such as edges, textures, and shapes; the pooling layer downsamples the convolution output to reduce the size of the feature map, improve the translation invariance and robustness of the feature, and extract more global features of the image (global features). Finally, at the end of the CNN model, the robot sets two parallel fully connected layers, corresponding to the classification tasks of vehicles and personnel, respectively. The fully connected layer performs nonlinear transformation and probability mapping on the feature vectors output by convolution and pooling to obtain vehicle appearance features and person appearance features.

[0043] When the target data type is acoustic sensor data, the robot uses the Mel Frequency Cepstrum Coefficient (MFCC) extraction algorithm to extract features from the audio signal to obtain vehicle sound features and personnel sound features. The robot performs pre-emphasis and framing on the original audio. Pre-emphasis uses a first-order high-pass filter to enhance the energy of the high-frequency part and balance the spectrum distribution; framing divides the continuous audio into overlapping frames of fixed length, such as a Hamming window with a length of 25ms and a step size of 10ms. Then, the robot performs a fast Fourier transform (FFT) on each frame data to convert the time domain signal into a frequency domain signal to obtain a spectrogram. The robot inputs the spectrogram into the Mel filter bank, performs a nonlinear mapping from frequency to Mel scale, simulates the auditory characteristics of the human ear, and obtains a Mel spectrum. Finally, the robot performs a discrete cosine transform (DCT) on the Mel spectrum to obtain the MFCC feature coefficients. The MFCC feature compresses the information of the spectrum and highlights the main components of the audio through dimensionality reduction and decorrelation. The robot constructs a sound feature vector based on the statistics and dynamic changes of the MFCC features, such as the mean and variance, to distinguish between vehicle sound features and personnel sound features.

[0044] When the target data type is environmental sensor data, the robot uses statistical feature algorithms to extract features from time series data such as temperature, humidity, and air quality. The robot normalizes and performs segmented statistical processing on the original environmental data. Normalization maps environmental sensor data of different dimensions and ranges to a unified scale, such as between 0 and 1; segmented statistics divide continuous time series data into time windows of fixed lengths, such as 1 minute, 5 minutes, and 1 hour. Then, the robot calculates the statistical features of the data in each time window to obtain the mean, variance, maximum value, and minimum value of each time window. These statistical features characterize the overall distribution and change trend of environmental data at different time scales. Finally, the robot arranges the statistical features of each time window in chronological order to form a feature vector sequence as temperature features, humidity features, and air quality features.

[0045] Taking vehicle appearance feature extraction as an example, the robot first scales the vehicle image to 224x224 pixels and performs enhancement operations such as brightness, contrast, and sharpening. Then, the robot uses a convolutional neural network model to extract a 2048-dimensional feature vector of the vehicle image through 50 layers of convolution and pooling. Next, the robot connects a 256-dimensional fully connected layer and a Softmax classifier at the end of the convolutional neural network model, fine-tunes the model parameters, and outputs vehicle appearance features such as the vehicle model and color.

[0046] Taking the extraction of human voice features as an example, the robot first performs 16kHz, 16-bit pre-emphasis and framing on the human voice clips, with each frame length of 25ms and a frame shift of 10ms. Then, the robot performs FFT and Mel filter bank transformation on each frame of voice data to obtain 24-dimensional MFCC features and first-order difference features. Next, the robot performs voiceprint modeling on the MFCC feature sequence to obtain the human voice features.

[0047] Step S105: Input the target features into a preset abnormality recognition model to determine whether there is an abnormal situation, which includes one or more of a vehicle abnormal situation, a personnel abnormal situation, and an environmental abnormal situation.

[0048] In a possible implementation, the process of constructing a preset abnormality recognition model includes: acquiring historical multimodal data of visual sensors, acoustic sensors, and environmental sensors; annotating the historical multimodal data to obtain an annotated data set, the annotated data set including vehicle abnormality annotations, personnel abnormality annotations, and environmental abnormality annotations; constructing corresponding abnormality recognition sub-models for different abnormal situations, the abnormality recognition sub-models including vehicle abnormality recognition models, personnel abnormality recognition models, and environmental abnormality recognition models; inputting the annotated data set into each abnormality recognition sub-model for training, and combining each abnormality recognition sub-model to obtain a preset abnormality recognition model.

[0049] Specifically, the robot first obtains a large amount of historical multimodal data from the visual sensors, acoustic sensors and environmental sensors of the mobile unmanned security robot, including visual sensor data, acoustic sensor data and environmental sensor data, among which the visual sensor data includes video clips and image frames; the acoustic sensor data includes audio clips; and the environmental sensor data includes temperature and humidity sequences and air quality indicators. The robot manually annotates the collected historical multimodal data to obtain an annotated data set. The annotation process requires the collaboration of professional security personnel and data annotators. According to the safety management specifications and abnormal behavior definitions of the park, each data sample is reviewed and marked one by one. The annotation content includes three aspects: vehicle anomaly annotation, personnel anomaly annotation and environmental anomaly annotation. Vehicle anomaly annotation is mainly for illegal intrusion and illegal parking of vehicles; personnel anomaly annotation is mainly for illegal intrusion, theft and robbery of personnel; environmental anomaly annotation is mainly for fire, explosion and leakage.

[0050] The robot divides different training subsets from the anomaly annotation dataset according to three categories of abnormal situations: vehicle abnormal situations, personnel abnormal situations, and environmental abnormal situations. For each type of abnormal situation, the robot constructs and trains the corresponding abnormal recognition sub-models, namely, the vehicle abnormal recognition model, the personnel abnormal recognition model, and the environmental abnormal recognition model. Taking the vehicle abnormal recognition model as an example, the robot inputs the vehicle-related abnormal annotation data into the machine learning algorithm, such as the convolutional neural network. The machine learning algorithm learns a series of abnormal discrimination rules and decision functions by analyzing the association between the vehicle appearance features and vehicle sound features of the vehicle and the abnormal labels, and forms a vehicle abnormal recognition model. The vehicle abnormal recognition model can take the vehicle appearance features and vehicle sound features of the vehicle as input and output whether the vehicle belongs to the vehicle abnormal situation in the abnormal situation. Similarly, the robot uses the personnel-related anomaly annotation dataset to train the personnel abnormal recognition model. The robot extracts the personnel-related abnormal annotation samples, feeds the personnel's image and video data into the convolutional neural network, and uses the key point detection technology to extract the personnel's appearance features; the personnel's voice clips are fed into the speech recognition engine, and the personnel's voice features are extracted using the voiceprint recognition technology. Similarly, the robot extracts anomaly-labeled samples related to the environment, extracts statistical features of environmental parameter sequences such as temperature, humidity, and air quality, such as mean, variance, maximum, and minimum values, and performs one-hot encoding on the threshold triggering conditions of environmental parameters, such as recording a temperature of 1 when it exceeds 50 degrees, and recording it as 0 otherwise. The statistical features and one-hot encoding are input into the classifier together, and the mapping rules are learned with the fire, explosion, and leakage anomaly labels to form an environmental anomaly recognition sub-model.

[0051] In step S105, the target features are classified according to the dimension categories to obtain target sub-features, and the dimension categories include vehicle dimension, personnel dimension and environment dimension. According to the dimension categories, the target self-features are input into the corresponding abnormality recognition sub-models to determine whether there is an abnormal situation.

[0052] Specifically, the robot extracts the multimodal target features and sends them to the corresponding abnormal recognition sub-models according to the three dimensions of vehicle, personnel and environment: the vehicle abnormal recognition model takes the vehicle appearance features and vehicle sound features as input, and uses the deep learning algorithm to determine whether the vehicle is illegally intruding or illegally parking. The personnel abnormal recognition model takes the personnel appearance features and personnel sound features as input, and uses the pattern recognition algorithm to determine whether the personnel is illegally intruding or has the risk of theft. The environmental abnormal recognition model takes the temperature features, humidity features and air quality features as input, and uses the threshold judgment and statistical test algorithm to determine whether a fire, explosion or leakage occurs. The robot runs the preset abnormal recognition model (three abnormal recognition sub-models), performs inference prediction on the target features, and obtains the probability score of whether the vehicle, personnel and environment are abnormal. When the probability score of any dimension exceeds the preset threshold, it is determined that the current scene has a corresponding abnormal situation (vehicle abnormal situation, personnel abnormal situation and environmental abnormal situation). The preset threshold can be set according to historical data and security requirements, and this application does not limit this.

[0053] Step S106: If it is determined that an abnormal situation exists, an alarm is triggered and an alarm message is sent to the park security personnel. The alarm message includes the type of abnormal situation, the location where the abnormal situation occurred, the abnormal situation detection time, and the abnormal situation picture.

[0054] In step S106, once the robot determines that an abnormality exists, the alarm process will be automatically triggered. The alarm process involves two key steps: generating alarm information and sending alarm information. The robot will generate alarm information containing the specific abnormality type, the location of the abnormality, the detection time, and related visual data (such as abnormal situation pictures) according to the detected abnormality type (such as illegal intrusion, illegal parking, fire, etc.). The generated alarm information is sent to the park security personnel through various methods such as short message service, email, and in-app push notification.

[0055] Reference Figure 2The present application also provides a mobile unmanned security device, which is a robot. The robot includes an acquisition module 201 and a processing module 202; the acquisition module 201 is used to move in the park according to a preset patrol route and acquire multiple sensor data; the processing module 202 is used to trace the data source of each sensor data to obtain the data type corresponding to each sensor data; the data type includes visual sensor data, acoustic sensor data and environmental sensor data; the processing module 202 is also used to match the target feature extraction algorithm corresponding to the target sensor data according to the target data type, the target sensor data is any sensor data among the multiple sensor data, and the target data type is the data type corresponding to the target sensor data; the processing module 202 is also used to extract features from the target sensor data using the target feature extraction algorithm to obtain target features; the processing module 202 is also used to input the target features into a preset abnormality recognition model to determine whether there is an abnormal situation, and the abnormal situation includes one or more of vehicle abnormality, personnel abnormality and environmental abnormality; the processing module 202 is also used to trigger an alarm if it is determined that there is an abnormal situation, and send an alarm message to the park security personnel, and the alarm message includes the abnormal situation type, the abnormality occurrence location, the abnormality detection time and the abnormal situation picture.

[0056] In one possible implementation, the processing module 202 uses a target feature extraction algorithm to perform feature extraction on the target sensor data to obtain target features, specifically including: if the processing module 202 determines that the target data type is visual sensor data, it uses a convolutional neural network to perform feature extraction on the visual sensor data to obtain vehicle appearance features and personnel appearance features; if the processing module 202 determines that the target data type is acoustic sensor data, it uses a Mel-frequency cepstral coefficient extraction algorithm to perform feature extraction on the acoustic sensor data to obtain vehicle sound features and personnel sound features; if the processing module 202 determines that the target data type is environmental sensor data, it uses a statistical feature algorithm to perform feature extraction on the environmental sensor data to obtain temperature features, humidity features and air quality features.

[0057] In one possible implementation, the processing module 202 uses a convolutional neural network to perform feature extraction on visual sensor data to obtain appearance features, specifically including: the processing module 202 performs a preprocessing operation on the visual sensor data to obtain a preprocessed image, and the preprocessing operation includes image scaling and image enhancement; the processing module 202 inputs the preprocessed image into the convolutional neural network, and extracts local features and global features of the preprocessed image through multi-layer convolution and pooling operations; the processing module 202 sets different fully connected layers at the end of the convolutional neural network, and outputs vehicle appearance features and personnel appearance features according to local features and global features, respectively.

[0058] In a possible implementation, the processing module 202 uses a Mel-frequency cepstral coefficient extraction algorithm to extract features from the acoustic sensor data to obtain vehicle sound features and personnel sound features, specifically including: the processing module 202 pre-emphasizes and frames the acoustic sensor data to obtain multiple frame data; the processing module 202 performs fast Fourier transform on each frame data to obtain a spectrum signal; the processing module 202 inputs the spectrum signal into a Mel filter group to obtain a Mel spectrum; the processing module 202 performs discrete cosine transform on the Mel spectrum to obtain Mel-frequency cepstral coefficients; the processing module 202 determines the corresponding vehicle sound features and personnel sound features according to the Mel-frequency cepstral coefficients.

[0059] In one possible implementation, the processing module 202 uses a statistical feature algorithm to extract features from the environmental sensor data to obtain temperature features, humidity features, and air quality features, specifically including: the processing module 202 normalizes and performs segmented statistics on the environmental sensor data to obtain the mean, variance, maximum value, and minimum value corresponding to the environmental sensor data; the processing module 202 performs vector encoding according to the mean, variance, maximum value, and minimum value to obtain corresponding temperature features, humidity features, and air quality features.

[0060] In a possible implementation, the processing module 202 inputs the target feature into a preset abnormality recognition model, and before determining whether an abnormal situation exists, the method further includes: the acquisition module 201 acquires historical multimodal data of visual sensors, acoustic sensors, and environmental sensors; the processing module 202 labels the historical multimodal data to obtain a labeled data set, and the labeled data set includes vehicle abnormality labels, personnel abnormality labels, and environmental abnormality labels; the processing module 202 constructs corresponding abnormality recognition sub-models for different abnormal situations, and the abnormality recognition sub-models include vehicle abnormality recognition models, personnel abnormality recognition models, and environmental abnormality recognition models; the processing module 202 inputs the labeled data set into each abnormality recognition sub-model for training, and combines each abnormality recognition sub-model to obtain a preset abnormality recognition model.

[0061] In a possible implementation, the processing module 202 inputs the target features into a preset abnormality recognition model to determine whether there is an abnormal situation, specifically including: the processing module 202 classifies the target features according to dimension categories to obtain target sub-features, and the dimension categories include vehicle dimension, personnel dimension, and environment dimension; the processing module 202 inputs the target sub-features into corresponding abnormality recognition sub-models according to the dimension categories to determine whether there is an abnormal situation.

[0062] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0063] The present application also provides an electronic device. Figure 3 , Figure 3 The electronic device 300 may include: at least one processor 301 , at least one network interface 304 , a user interface 303 , a memory 305 , and at least one communication bus 302 .

[0064] The communication bus 302 is used to realize the connection and communication between these components.

[0065] The user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0066] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0067] Among them, the processor 301 may include one or more processing cores. The processor 301 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Optionally, the processor 301 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 301 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 301, and it can be implemented separately through a chip.

[0068] Among them, the memory 305 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may optionally also be at least one storage device located away from the aforementioned processor 301. Refer to Figure 3 , the memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module and an application program of a mobile unmanned security method.

[0069] exist Figure 3In the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call the application program of a mobile unmanned security method stored in the memory 305. When executed by one or more processors 301, the electronic device 300 executes one or more of the methods described in the above embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0070] The present application also provides a computer-readable storage medium, which stores instructions. When executed by one or more processors 301, the electronic device 300 executes one or more of the methods described in the above embodiments.

[0071] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0072] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0073] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0074] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0075] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0076] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure.

[0077] This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art not described in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A mobile unmanned security method, characterized in that: The method comprises: Move around the park according to preset patrol routes and acquire a variety of sensor data; Tracing the data source of each of the sensor data to obtain the data type corresponding to each of the sensor data; the data type includes visual sensor data, acoustic sensor data, and environmental sensor data; According to the target data type, a target feature extraction algorithm corresponding to the target sensor data is matched, the target sensor data is any one of the multiple sensor data, and the target data type is a data type corresponding to the target sensor data; Using the target feature extraction algorithm to extract features from the target sensor data to obtain target features; Input the target feature into a preset abnormality recognition model to determine whether there is an abnormal situation, wherein the abnormal situation includes one or more of a vehicle abnormal situation, a personnel abnormal situation, and an environmental abnormal situation; If it is determined that there is an abnormal situation, an alarm is triggered and an alarm message is sent to the park security personnel. The alarm message includes the type of abnormal situation, the location where the abnormal situation occurred, the abnormal situation detection time and the abnormal situation picture.

2. The method according to claim 1, characterized in that The use of the target feature extraction algorithm to extract features from the target sensor data to obtain target features specifically includes: If it is determined that the target data type is visual sensor data, a convolutional neural network is used to extract features of the visual sensor data to obtain vehicle appearance features and person appearance features; If it is determined that the target data type is acoustic sensor data, a Mel frequency cepstral coefficient extraction algorithm is used to extract features of the acoustic sensor data to obtain vehicle sound features and personnel sound features; If it is determined that the target data type is environmental sensor data, a statistical feature algorithm is used to extract features from the environmental sensor data to obtain temperature features, humidity features, and air quality features.

3. The method according to claim 2, characterized in that The method of using a convolutional neural network to extract features from the visual sensor data to obtain appearance features specifically includes: Performing a preprocessing operation on the visual sensor data to obtain a preprocessed image, wherein the preprocessing operation includes image scaling and image enhancement; Inputting the preprocessed image into a convolutional neural network, and extracting local features and global features of the preprocessed image through multi-layer convolution and pooling operations; At the end of the convolutional neural network, different fully connected layers are set to output vehicle appearance features and person appearance features respectively according to the local features and the global features.

4. The method according to claim 2, characterized in that: The method of extracting features from the acoustic sensor data using a Mel frequency cepstral coefficient extraction algorithm to obtain vehicle sound features and personnel sound features specifically includes: Pre-emphasize and frame the acoustic sensor data to obtain multiple frame data; Performing fast Fourier transform on each frame data to obtain a spectrum signal; Inputting the spectrum signal into a Mel filter bank to obtain a Mel spectrum; Performing discrete cosine transform on the Mel frequency spectrum to obtain Mel frequency cepstrum coefficients; The corresponding vehicle sound features and human sound features are determined according to the Mel-frequency cepstral coefficients.

5. The method according to claim 2, characterized in that: The use of a statistical feature algorithm to extract features from the environmental sensor data to obtain temperature features, humidity features, and air quality features specifically includes: Normalizing and segmenting the environmental sensor data to obtain a mean, variance, maximum value, and minimum value corresponding to the environmental sensor data; Vector encoding is performed according to the mean, the variance, the maximum value and the minimum value to obtain corresponding temperature characteristics, humidity characteristics and air quality characteristics.

6. The method according to claim 1, characterized in that Before inputting the target feature into a preset abnormality recognition model to determine whether there is an abnormal situation, the method further includes: Obtain historical multimodal data from visual sensors, acoustic sensors, and environmental sensors; Annotating the historical multimodal data to obtain an annotated data set, wherein the annotated data set includes vehicle anomaly annotations, personnel anomaly annotations, and environment anomaly annotations; According to different abnormal situations, corresponding abnormality recognition sub-models are constructed, wherein the abnormality recognition sub-models include a vehicle abnormality recognition model, a personnel abnormality recognition model and an environmental abnormality recognition model; The annotated data set is input into each of the anomaly recognition sub-models for training, and each of the anomaly recognition sub-models is combined to obtain the preset anomaly recognition model.

7. The method according to claim 6, characterized in that The step of inputting the target feature into a preset abnormality recognition model to determine whether there is an abnormality specifically includes: Classifying the target features according to dimension categories to obtain target sub-features, wherein the dimension categories include vehicle dimension, personnel dimension, and environment dimension; According to the dimension category, the target sub-features are respectively input into the corresponding abnormality recognition sub-model to determine whether there is an abnormal situation.

8. A mobile unmanned security device, characterized in that: The device comprises an acquisition module (201) and a processing module (202), wherein: The acquisition module (201) is used to move within the park according to a preset patrol route and acquire multiple sensor data; The processing module (202) is used to trace the data source of each of the sensor data to obtain the data type corresponding to each of the sensor data; the data type includes visual sensor data, acoustic sensor data and environmental sensor data; The processing module (202) is further used to match a target feature extraction algorithm corresponding to the target sensor data according to a target data type, wherein the target sensor data is any one of the plurality of sensor data, and the target data type is a data type corresponding to the target sensor data; The processing module (202) is further used to extract features from the target sensor data using the target feature extraction algorithm to obtain target features; The processing module (202) is further used to input the target feature into a preset abnormality recognition model to determine whether there is an abnormal situation, wherein the abnormal situation includes one or more of a vehicle abnormal situation, a personnel abnormal situation and an environmental abnormal situation; The processing module (202) is also used to trigger an alarm if it is determined that an abnormal situation exists, and send an alarm message to the park security personnel, wherein the alarm message includes the type of abnormal situation, the location where the abnormal situation occurs, the time when the abnormal situation is detected, and a picture of the abnormal situation.

9. An electronic device, characterized in that: The electronic device (300) comprises a processor (301), a memory (305), a user interface (303) and a network interface (304), wherein the memory (305) is used to store instructions, the user interface (303) and the network interface (304) are used to communicate with other devices, and the processor (301) is used to execute the instructions stored in the memory (305) so that the electronic device (300) executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is executed.

Images