A digital monitoring site prompt system
By designing a digital monitoring site prompt system including edge video recognition and voice recognition modules, the problem of insufficient security of video information and difficulty in real-time analysis is solved, local real-time analysis and video security guarantee are realized, and service quality is improved and suitable Mobile on-site monitoring.
Patent Information
- Application Number
- CN202410747730.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-11
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-06-11
AI Technical Summary
In the process of on-site service of monitoring technicians, the prior art problems are insufficient security of video information and difficulty in real-time analysis, especially when the bandwidth of wireless networks is limited.
A digital monitoring on-site prompt system is designed, including a camera module, a microphone module, a control module, a positioning module, a wireless module, a prompt module, a storage module, an edge video recognition module and a voice recognition module. The system prevents video and audio streams from being processed locally and uses custom MobileNet networks and pre-trained CNN models for real-time identification and analysis, avoiding video and audio uploads, thereby ensuring video security.
Real-time analysis of video and audio is realized locally, ensuring video security, preventing technicians from making fraud, improving service quality, and reducing the computing pressure of the equipment, making the equipment smaller and more portable, suitable for mobile on-site monitoring.
Smart Images

Figure CN118570979B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of monitoring technology, and in particular to a digital monitoring site prompting system. Background Art
[0002] With the improvement of living standards, people need more convenience in life. When encountering problems, they also hope to reduce their own costs. Therefore, they hope that the corresponding problem solvers can provide door-to-door services. Therefore, door-to-door services are gradually becoming popular among people.
[0003] In order to improve the quality of service, operating companies have strict requirements on the speech and manners of technicians during on-site service. Among them, operating companies mainly learn about the professional qualities of technicians from customer feedback, which is information lag and is not conducive to the development of the company. Some methods use video recorders to record the service process of technicians, and administrators remotely monitor the services of technicians to promptly correct the service attitude and service of technicians. This method requires uploading videos to the server, and video recorders may record the privacy of customers (faces), which will be leaked for various reasons when uploaded to the server. At the same time, due to the bandwidth cost of wireless networks, videos may not be uploaded to the server in real time, and such analysis also lags or even interrupts. Summary of the invention
[0004] In view of the above technical problems, the present invention provides a digital monitoring site prompt system to provide a monitoring system that can ensure video security, perform real-time analysis and prompt and correct users.
[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.
[0006] The present invention discloses a digital monitoring site prompt system, the system comprises a camera module, a microphone module, a control module, a positioning module, a wireless module, a prompt module, a storage module, an edge video recognition module and a voice recognition module, the positioning module is used to obtain the current position, the wireless module is used to connect with an external server, wherein:
[0007] The control module is used to obtain the video stream from the camera module and the audio stream of the user from the microphone module, and to pre-process the video stream and the audio stream and send them to the edge video recognition module and the voice recognition module respectively;
[0008] The storage module is used to locally store the video stream and the audio stream;
[0009] The edge video recognition module is used to: recognize the face in the preprocessed video stream in real time based on a locally built-in pre-trained custom MobileNet network, wherein the custom MobileNet network includes a hybrid deep convolution unit, a full convolution layer, a global average pooling layer, a fully connected layer, and a classification layer composed of an activation function, wherein the hybrid deep convolution unit is composed of two point-by-point convolutions and one deep convolution, and the activation functions of the classification layer include Mish, TanhExp, and ReLU;
[0010] The speech recognition module is used to: perform speech recognition and emotion classification on mixed audio features in real time based on a pre-trained classification model, obtain the speech content and emotion post-record of the audio, the mixed audio features are fused by the extracted MFCC features and corresponding time domain features in the pre-processed audio, the MFCC features are obtained by performing frame segmentation, Hamming window processing, discrete Fourier transform, obtaining Mel spectrum and applying discrete cosine transform on the pre-processed audio, and the time domain features are multiple of the maximum value, minimum value, mean value, median, mode, standard deviation, variance, covariance, root mean square and percentile of a partial area of the matrix composed of the MFCC features;
[0011] The prompt module is used to be triggered by the control module to interactively remind the user when the edge video recognition module recognizes a face but the voice recognition module fails to recognize the first specific voice, recognizes the second specific voice, or detects negative emotions.
[0012] Furthermore, the control module preprocesses the video stream, including: adjusting the frame rate of the video stream, adjusting each frame of the video stream to n*m pixels, and normalizing the pixel value of each frame of the video stream.
[0013] Furthermore, the control module pre-processes the audio stream, including:
[0014] Removing a silent portion of the audio stream;
[0015] Based on a finite impulse response high-pass filter, the high-frequency component in the audio stream is enhanced;
[0016] Signal normalization is performed on the audio stream.
[0017] Furthermore, the number of layers of the custom MobileNet network is less than that of the standard MobileNet V1 network.
[0018] Furthermore, the system also includes an encryption module, which is used to extract frames in the video stream where faces are recognized, perform face encryption on the extracted frames of the video stream, and then return them to the video stream to obtain the encrypted video stream.
[0019] Furthermore, the encryption module specifically includes:
[0020] A face region detection unit, the face region detection unit performs face detection on frames in which faces are recognized in the video stream, and sequentially selects the video streams in which faces are detected to form an image data set;
[0021] A virtual face feature generating unit, the virtual face feature generating unit is used to generate a virtual random number for the image data set based on a random number generating function, the virtual random number corresponds to the recording sequence specified by the face region detecting unit, and the virtual face feature generating unit generates virtual face feature parameters according to original face feature parameters of real face information extracted from the face region by adding a noise value based on the virtual random number;
[0022] A virtual face feature vector and data generation unit, wherein the virtual face feature vector and data generation unit is used to extract a virtual face feature vector from the virtual face feature parameters, search for virtual face information from a database based on principal component analysis and linear discriminant analysis techniques, insert the virtual face information into the face area of the corresponding original image in the image data set to obtain an encrypted face image, and encrypt the real face information and record it in a blockchain database.
[0023] Furthermore, the system further includes a decryption module, and the decryption module includes:
[0024] A facial feature restoration unit, the facial feature restoration unit being used to add a restored random number obtained based on a restoration random generation function to a predefined similarity value to reversely calculate the noise value used to generate the virtual facial feature parameters, and after deducting the reversely calculated noise value, generate restored facial feature parameters corresponding to the original facial feature parameters;
[0025] A facial image restoration unit, the facial image restoration unit is used to vectorize the restored facial feature parameters, and use the vectorized values to generate a restored facial feature vector, search for the real facial information in the database of the blockchain according to the restored facial feature vector, insert the real facial information into the facial area of the encrypted facial image, and obtain the restored original image.
[0026] Furthermore, the classification model is a CNN model.
[0027] Furthermore, the prompt module may provide interactive reminders to the user based on one or more of the following methods:
[0028] Loudspeaker announcements;
[0029] Sending a text message to the user's mobile phone;
[0030] Sending alerts to the administrator platform of an external server so that the administrator can remind the user;
[0031] Send reminder messages to wired or wirelessly connected headphones;
[0032] Send prompt information to the display screen to drive the vibrator to vibrate;
[0033] Make the LED flash.
[0034] The technical solution disclosed in this disclosure has the following beneficial effects:
[0035] Based on the edge video recognition module and the voice recognition module, edge analysis of video and audio can be performed locally without uploading the video and audio, which greatly ensures video security. At the same time, through the analysis of video and audio, face analysis can prevent technicians from cheating, and audio analysis can monitor the service attitude of technicians, which can be corrected through the prompt module, greatly improving the service quality. When recognizing faces, based on the simplified custom MobileNet network, the computing pressure of the device can be greatly reduced, so that the device carrying the edge video recognition module and the voice recognition module can be smaller and more portable, and thus more suitable for portable mobile on-site monitoring. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 This is a structural block diagram of a digital monitoring site prompt system in an embodiment of this specification;
[0037] Figure 2 This is a comparison diagram of the depthwise separable convolution unit and the hybrid depthwise convolution unit in the embodiments of this specification. DETAILED DESCRIPTION
[0038] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.
[0039] In addition, the accompanying drawings are only schematic illustrations of the present disclosure. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.
[0040] like Figure 1 As shown, an embodiment of the present specification provides a digital monitoring site prompt system, the system comprising: a camera module 101, a microphone module 102, a control module 103, a positioning module 104, a wireless module 105, a prompt module 106, a storage module 107, an edge video recognition module 108 and a voice recognition module 109, the positioning module 104 is used to obtain the current positioning, the wireless module 105 is used to connect to an external server, the control module 103 is used to obtain the video stream from the camera module 101 and the user's audio stream from the microphone module 102, and is used to pre-process the video stream and the audio stream and send them to the edge video recognition module 108 and the voice recognition module 109 respectively, and the storage module 107 is used to locally store the video stream and the audio stream.
[0041] Among them, the combination of the above-mentioned camera module 101, microphone module 102, control module 103, positioning module 104, wireless module 105, prompt module 106, and storage module 107 can exist in the form of a mobile phone, tablet or other mobile device with a GPU, and the edge video recognition module 108 and the voice recognition module 109 can exist in the form of hardware or software.
[0042] The edge video recognition module 108 is used to: recognize faces in the preprocessed video stream in real time based on a locally built-in pre-trained custom MobileNet network, wherein the custom MobileNet network includes a hybrid deep convolution unit, a full convolution layer, a global average pooling layer, a fully connected layer, and a classification layer composed of an activation function, wherein the hybrid deep convolution unit is composed of two point-by-point convolutions and one deep convolution, and the activation functions of the classification layer include Mish, TanhExp, and ReLU.
[0043] Among them, the MobileNet network is based on an optimized version of CNN with a compact size. It is implemented by using a depthwise separable convolution unit (dsc), a full convolution layer, a global average pooling layer, a fully connected layer, and a classification layer using a softmax function. The depthwise separable convolution unit consists of a depthwise convolution (Dwcv) and a one-to-one pointwise convolution (Pwcv). In the MobileNet-V1 network, each input channel needs to be filtered separately in Dwcv and then filtered in Pwcv. The purpose of using Pwcv is to linearly combine all outputs of Dwcv. The MobileNet-V1 network model has hyperparameters for width multiplier and resolution multiplier, with values ranging from 0 to 1. The size of the model can be adjusted by adjusting these two hyperparameters, but this will result in a significant drop in the recognition rate. In this embodiment, a hybrid depthwise convolution unit is used to replace the depthwise separable convolution unit, thereby improving accuracy and reducing the number of layers of the model. Specifically, as Figure 2 As shown in the figure, the hybrid depth convolution unit is composed of two edge Pwcv and one middle Dwcv into one layer, instead of defining Pwcv and Dwcv as separate layers. This keeps the function of Dsc unchanged, but relative to the 28 layers of the standard MobileNet-V1 network, using a hybrid depth convolution unit to replace the depth-separable convolution unit can reduce the number of layers to 14, greatly reducing the size of the model so that it can be used on mobile devices.
[0044] In addition, in the custom MobileNet network, Mish, TanhExp and ReLU are used as activation functions. ReLU is a standard activation function, which is widely used in major architectures, but it has the "ReLU dead zone" problem, that is, some neurons may stop outputting results other than 0. Therefore, this embodiment also uses nonlinear functions Mish and TanhExp with similar shapes as activation functions, and selectively switches them when used, which enhances the overall accuracy of the custom MobileNet network.
[0045] In the custom MobileNet network, without affecting the accuracy, all repeated and unnecessary layers can be deleted, such as unnecessary layers with the same output shape as (4,4,512), that is, 5 layers are deleted from the 9-13 layers of the standard MobileNet network. Deleting these layers with insignificant effects can significantly reduce the amount of computation and the number of parameters of the network. The slight performance loss caused by deleting the layers is compensated by the replacement of the hybrid depth convolution unit and the addition of the Mish and TanhExp activation functions. Finally, in order to further reduce the amount of computation, the channel depth of the last depth convolution of the hybrid depth convolution unit can be reduced from 1024 to 780 to optimize the model performance and the use of hardware resources.
[0046] The edge video recognition module 108 specifically runs on the mobile device body, performs recognition locally in real time, and uses a trained recognition model based on a custom MobileNet network to track faces in the video stream. The training process of the recognition model can refer to the existing CNN training, that is, marking face annotations on the training data and inputting them into the custom MobileNet network to train the recognition model. The application scenario of the edge video recognition module 108 is generally for the user to arrive at the customer's designated work area, the work order takes effect, and the user communicates with the customer face to face. The camera module 101 records the video stream including the customer's face, and the edge video recognition module 108 recognizes the face in the video stream, and then feeds it back to the control module 103, and the control module 103 allows the work order to take effect. In addition, the edge video recognition module 108 can also be used to recognize the user's face, and the punch-in is allowed only after the recognition is passed.
[0047] The speech recognition module 109 is used to: perform speech recognition and emotion classification on mixed audio features in real time based on a pre-trained classification model, and obtain the speech content and emotion record of the audio, wherein the mixed audio features are fused by the extracted MFCC features and the corresponding time domain features in the pre-processed audio, and the MFCC features are obtained by performing frame segmentation, Hamming window processing, discrete Fourier transform, obtaining Mel spectrum and applying discrete cosine transform on the pre-processed audio, and the time domain features are multiple types of the maximum value, minimum value, mean value, median, mode, standard deviation, variance, covariance, root mean square and percentile of a partial area of the matrix composed of the MFCC features.
[0048] Among them, the preferred application scenario of the present invention is the door-to-door service of users (technicians, housekeepers), for which the service attitude of users is particularly important. However, the service actions of users are difficult to monitor, so starting from voice recognition, it is possible to monitor whether users follow the standard process, such as monitoring whether the opening and closing remarks of users during the service process are read according to the speech template, whether uncivilized language is used, and whether the conversation with the employer contains negative emotions.
[0049] The speech recognition module 109 can be run on the mobile device body to perform speech recognition locally, or it can be run in an external server. Since the volume of the audio stream is very small relative to the video stream, the time delay in remote data transmission and analysis is very small, and real-time performance can be achieved. The classification model contained in the speech recognition module 109 needs to be pre-trained, and the classification model can be a type of CNN. When training the classification model, a data set representing different speech templates, a data set representing different emotions, and a data set representing uncivilized language are used. The speech template can be "hello, goodbye, thank you, etc.", and the emotion can be anger, boredom, disgust, fear, happiness, neutrality, and sadness, etc., and then the data set is pre-processed to extract MFCC features and time domain features. After the MFCC features and time domain features are fused, they are trained with the labels of each data set to obtain a classification model. When the classification model performs speech recognition, the MFCC features and time domain features in the pre-processed audio are first extracted, and then the mixed audio features that fuse the MFCC features and time domain features are recognized, and the labels corresponding to the audio are output. These labels exist at the same time, such as speech template 1, happy.
[0050] Specifically, when extracting MFCC features, in order to prevent information loss, each audio segment is divided into n millisecond frames, and there is an overlap of m milliseconds between consecutive frames. The number of frames of a single audio can be expressed as , T is the total number of frames, N is the number of samples, and S is the sampling rate. After the frame segmentation is completed, each individual frame will be processed by the Hamming window to ensure the smooth edge of the frame. The specific calculation formula is as follows:
[0051] ; is the total number of samples per frame.
[0052] Then, the magnitude spectra of all frames are calculated by discrete Fourier transform, which are passed to the Mel filter bank for calculation to represent the perceived frequency. During the calculation, the scale frequency warping is performed based on the following formula:
[0053] ;
[0054] ;
[0055] in, Indicates the actual frequency, represents the corresponding perceived frequency, represents a triangular filter, represents the weight assigned to the Kth region of the spectrum, which has an The output bands contribute to represents the amplitude spectrum, This means that the Mel spectrum is obtained by multiplying all the magnitude spectra by a triangular filter.
[0056] Finally, the MFCC features are obtained by performing discrete cosine transform on the Mel spectrum of each frame, and the calculation is as follows:
[0057] ;
[0058] C is the MFCC feature.
[0059] After obtaining the MFCC features, the time domain features need to be calculated. The time domain features first need to be processed by binning the MFCC features, and each box contains X rows of data in each column. Different time domain features are extracted from all MFCC feature boxes. These features include minimum, maximum, mean, median, mode, standard deviation, variance, covariance, root mean square, 25%, 50% and 75% quantiles of each box and other statistics. MFCC features and time domain features can be organized into main feature vectors as input to the CNN-based classification model, which will output the corresponding sentiment classification and speech label.
[0060] The prompt module 106 is used to be triggered by the control module to interactively remind the user when the edge video recognition module recognizes a face but the voice recognition module fails to recognize the first specific voice, recognizes the second specific voice, or detects negative emotions.
[0061] Among them, the first specific voice refers to the words in the speech template, the second specific voice refers to uncivilized words, and negative emotions refer to anger, sadness, disgust and other emotions. After the user arrives at the work area (the employer's preset location), the user starts the system and triggers interactive buttons such as "Start Service", "Start Work", "Start Recording", etc., the camera module, microphone module and other modules start working, the user communicates face to face with the employer, the employer's face is recognized and tracked, and the user uses the speech template during the communication process, which is a standard process and does not require prompts. If the classification model in the speech recognition module does not recognize the content of the speech template, or recognizes the user's uncivilized words, or recognizes negative emotions until the face disappears or after the preset time, the user needs to be prompted to maintain good service quality.
[0062] When the prompt module 106 interactively reminds the user, it is based on one or more of the following methods: speaker broadcast; sending text messages to the user's mobile phone; sending alarms to the administrator platform of an external server so that the administrator can remind the user; sending prompt information to wired or wireless headphones; sending prompt information to the display screen to drive the vibrator to vibrate; driving the LED to flash.
[0063] In one embodiment, the control module preprocesses the video stream, including: adjusting the frame rate of the video stream, adjusting each frame of the video stream to n*m pixels, and normalizing the pixel value of each frame of the video stream. According to mobile devices with different computing capabilities, the specific values of the frame rate and the pixel are different, such as the pixel can be 300*300 and the frame rate can be 10-30.
[0064] In one embodiment, the control module preprocesses the audio stream, including: removing silent parts in the audio stream; enhancing high-frequency components in the audio stream based on a high-pass filter of a finite impulse response; and performing signal normalization on the audio stream.
[0065] Among them, for classification models that cannot tolerate background noise or silence, speech preprocessing is an extremely important stage to facilitate the subsequent extraction of features from the audio, and most of the spoken parts in the audio contain emotion-related features. To achieve this goal, this embodiment adopts silence elimination and pre-enhancement methods. Pre-enhancement is to increase the power of the high-frequency part of the audio while keeping the low frequency unchanged to improve the signal-to-noise ratio. Pre-enhancement can be achieved by using a high-pass filter with a finite impulse response. The following formula:
[0066] ;
[0067] Pre-enhancement using a high-pass filter with a finite impulse response may result in changes in the energy distribution between frequencies and changes in the overall energy level, which can have a significant impact on energy-related features. Therefore, it is necessary to normalize the audio stream to ensure that the audio is comparable regardless of how the amplitude changes. The normalization formula is as follows:
[0068] ;
[0069] in, Indicates the audio part, and represent the mean and standard deviation respectively, Represents the normalized audio part.
[0070] In one embodiment, the system further includes an encryption module, which is used to extract frames from the video stream where faces are recognized, perform face encryption on the extracted frames of the video stream, and return them to the video stream to obtain the encrypted video stream.
[0071] Although the above embodiment proposes that the edge video recognition module analyzes the video stream locally and stores the video stream locally to prevent privacy leakage caused by interception of the video stream by others after uploading it to the server, it is still necessary to extract the video from the mobile device in situations where video is needed as evidence, such as when an item is damaged, property is stolen, or service evaluation occurs with the employer. Encrypting the face in the video stream can effectively prevent privacy leakage.
[0072] The encryption module specifically includes a face region detection unit, a virtual face feature generation unit, a virtual face feature vector and a data generation unit. Specifically, the face region detection unit performs face detection on frames in which faces are recognized in the video stream, and sequentially selects the video streams in which faces are detected to form an image data set.
[0073] The virtual facial feature generation unit is used to generate a virtual random number for the image data set based on a random number generation function, and the virtual random number corresponds to the record order specified by the face area detection unit. The virtual facial feature generation unit generates virtual facial feature parameters based on the original facial feature parameters of the real facial information extracted from the face area by adding a noise value based on the virtual random number.
[0074] Among them, the pseudo random number can be expressed as , is the random number generation function, Is the smooth issuance or sequence of random numbers, which can be simulated by software, is a predefined seed value. When the predefined seed value and the random number generation range are the same, a series of random numbers are generated, that is, the virtual random numbers are obtained. Assume that the random number generation range is between 0 and 100, and the generation cycle is two times. At the same time, assume that the "S" of the first round and the second round are 0.2 and 0.5 respectively. In this embodiment, the first round randomly generates random numbers in the range of 0 to 100, and multiplies the number by a fixed seed to generate random numbers within the range expected by the user. Therefore, the random numbers generated in the first round will be generated within the range expected by the user by multiplying the fixed seed value by 0.2, and the range will be limited to 0 to 20. Therefore, in the second round, random numbers will be generated from the range of 0 to 50. The obtained virtualized random numbers will be used to generate noise values. The method can be to add the difference between the median values of the predefined virtualized random number generation range to the predefined similarity value, as shown in the following formula:
[0075] ;
[0076] represents the generated noise value, represents a predefined similarity value, represents a pseudo random number, Indicates the pre-set range of pseudo random number generation. The similarity value is inversely proportional to the noise value. The higher the similarity value, the closer the reproducibility of the original data. On the contrary, the lower the similarity value, the more opposite the reproducibility of the original data.
[0077] The virtual face feature vector and data generation unit is used to extract the virtual face feature vector from the virtual face feature parameters, search for virtual face information from the database based on principal component analysis and linear discriminant analysis techniques, insert the virtual face information into the face area of the corresponding original image in the image data set to obtain an encrypted face image, and encrypt the real face information and record it in the database of the blockchain.
[0078] As a supplement, the system further includes a decryption module, which includes a facial feature recovery unit and a facial image recovery unit. The decryption module is used to decrypt the above-mentioned encrypted facial image.
[0079] Specifically, the facial feature recovery unit is used to add the restored random number obtained based on the recovery random generation function to a predefined similarity value to reversely calculate the noise value used to generate the virtual facial feature parameters, and after deducting the reversely calculated noise value, generate restored facial feature parameters corresponding to the original facial feature parameters.
[0080] The recovery random number can be expressed as , is the random number generation function, is a random number distribution sequence. is a predefined seed value. The noise value is calculated by adding the similarity value used to generate the encrypted face image to the recovered random number, using the random number function used for encryption In The value of allows recovery of random numbers with the above The restored facial feature parameters can be generated by adding the virtual facial feature parameters to the noise value. The process of reverse calculation of the noise value is as follows:
[0081] ;
[0082] in, Restore the noise value of the face.
[0083] A facial image restoration unit, the facial image restoration unit is used to vectorize the restored facial feature parameters, and use the vectorized values to generate a restored facial feature vector (i.e., a result of the vectorization of the restored facial feature parameters), search for the real facial information in the database of the blockchain according to the restored facial feature vector, insert the search result (real facial information) into the facial area of the encrypted facial image, and obtain the restored original image.
[0084] The encryption module and decryption module can be deployed on a mobile device. The encryption module encrypts the face of the video stream on the storage module and replaces the original video stream. The encryption module and decryption module can also be deployed on a server. When the server receives the video stream, the face in it is encrypted. When the user or other third party needs to access the video stream on the server, what they see is the face-encrypted video. If the video needs to be decrypted, it is decrypted after the administrator's authorization.
[0085] Based on the above encryption module and decryption module, the faces in the video stream can be encrypted, which can effectively prevent privacy leakage. In situations where the real identity needs to be provided, the faces can be decrypted and restored, which can protect the customer's privacy while maintaining the authenticity of the surveillance video.
[0086] Beneficial effects:
[0087] The present invention is based on an edge video recognition module and a voice recognition module, and can perform edge analysis on video and audio locally without uploading the video and audio, thereby greatly ensuring video security; at the same time, through the analysis of video and audio, face analysis can prevent technicians from cheating, and audio analysis can monitor the service attitude of technicians, which can be corrected through the prompt module, thereby greatly improving the service quality; when recognizing faces, based on a simplified custom MobileNet network, the computing pressure of the device can be greatly reduced, so that the device carrying the edge video recognition module and the voice recognition module can be smaller and more portable, and thus more suitable for portable mobile on-site monitoring
[0088] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.
Claims
1. A digital monitoring site prompt system, characterized in that: The system includes a camera module, a microphone module, a control module, a positioning module, a wireless module, a prompt module, a storage module, an edge video recognition module and a voice recognition module. The positioning module is used to obtain the current location, and the wireless module is used to connect to an external server, wherein: The control module is used to obtain the video stream from the camera module and the audio stream of the user from the microphone module, and to pre-process the video stream and the audio stream and send them to the edge video recognition module and the voice recognition module respectively; The storage module is used to locally store the video stream and the audio stream; The edge video recognition module is used to: recognize the face in the preprocessed video stream in real time based on a locally built-in pre-trained custom MobileNet network, wherein the custom MobileNet network includes a hybrid deep convolution unit, a full convolution layer, a global average pooling layer, a fully connected layer and a classification layer, wherein the classification layer is composed of an activation function, and the hybrid deep convolution unit is composed of two point-by-point convolutions and one deep convolution, and the activation functions of the classification layer include Mish, TanhExp and ReLU; The speech recognition module is used to: perform speech recognition and emotion classification on mixed audio features in real time based on a pre-trained classification model to obtain the speech content and emotion post-record of the audio stream, wherein the mixed audio features are fused by the extracted MFCC features and corresponding time domain features in the pre-processed audio stream, wherein the MFCC features are obtained by performing frame segmentation, Hamming window processing, discrete Fourier transform, obtaining Mel spectrum and applying discrete cosine transform on the pre-processed audio stream, and the time domain features are multiple of the maximum value, minimum value, mean value, median, mode, standard deviation, variance, covariance, root mean square and percentile of a partial area of the matrix composed of the MFCC features; The prompt module is used to be triggered by the control module to interactively remind the user when the edge video recognition module recognizes a face but the voice recognition module fails to recognize the first specific voice, recognizes the second specific voice, or detects negative emotions.
2. A digital monitoring site prompt system according to claim 1, characterized in that: The control module preprocesses the video stream, including: adjusting the frame rate of the video stream, adjusting each frame of the video stream to n*m pixels, and normalizing the pixel value of each frame of the video stream.
3. A digital monitoring site prompt system according to claim 1, characterized in that: The control module preprocesses the audio stream, including: Removing a silent portion of the audio stream; Based on a finite impulse response high-pass filter, the high-frequency component in the audio stream is enhanced; Signal normalization is performed on the audio stream.
4. A digital monitoring site prompt system according to claim 1, characterized in that: The custom MobileNet network has fewer layers than the standard MobileNet V1 network.
5. A digital monitoring site prompt system according to claim 1, characterized in that: The system also includes an encryption module, which is used to extract frames from the video stream in which faces are recognized, perform face encryption on the extracted frames of the video stream, and then return them to the video stream to obtain the encrypted video stream.
6. A digital monitoring site prompt system according to claim 5, characterized in that: The encryption module specifically includes: A face region detection unit, the face region detection unit performs face detection on frames in which faces are recognized in the video stream, and sequentially selects the video streams in which faces are detected to form an image data set; A virtual face feature generating unit, the virtual face feature generating unit is used to generate a virtual random number for the image data set based on a random number generating function, the virtual random number corresponds to the recording sequence specified by the face region detecting unit, and the virtual face feature generating unit generates virtual face feature parameters according to original face feature parameters of real face information extracted from the face region by adding a noise value based on the virtual random number; A virtual face feature vector and data generation unit, wherein the virtual face feature vector and data generation unit is used to extract a virtual face feature vector from the virtual face feature parameters, search for virtual face information from a database based on principal component analysis and linear discriminant analysis techniques, insert the virtual face information into the face area of the corresponding original image in the image data set to obtain an encrypted face image, and encrypt the real face information and record it in a blockchain database.
7. A digital monitoring site prompt system according to claim 6, characterized in that: The system also includes a decryption module, the decryption module including: A facial feature restoration unit, the facial feature restoration unit being used to add a restored random number obtained based on a restoration random generation function to a predefined similarity value to reversely calculate the noise value used to generate the virtual facial feature parameters, and after deducting the reversely calculated noise value, generate restored facial feature parameters corresponding to the original facial feature parameters; A facial image restoration unit, the facial image restoration unit is used to vectorize the restored facial feature parameters, and use the vectorized values to generate a restored facial feature vector, search for the real facial information in the database of the blockchain according to the restored facial feature vector, insert the real facial information into the facial area of the encrypted facial image, and obtain the restored original image.
8. A digital monitoring site prompt system according to claim 1, characterized in that: The classification model is a CNN model.
9. A digital monitoring site prompt system according to claim 1, characterized in that: The prompt module interactively reminds the user based on one or more of the following methods: Loudspeaker announcements; Send SMS to user's mobile phone; Sending alerts to the administrator platform of an external server so that the administrator can remind the user; Send reminder messages to wired or wirelessly connected headphones; Send prompt information to the display screen; driving the vibrator to vibrate; Make the LED flash.
Citation Information
Patent Citations
Facial expression recognition method of improved MobileNet model
CN111860046A
AI intelligent edge computing system and method based on 5G
CN115601904A
