A multi-modal construction site robot safety protection method

Abstract

The application provides a multimodal construction site robot safety protection method, which comprises a pre-stored feature searching step, a depth detection step, a face depth feature extraction step, a face RGB feature extraction step, a sound feature extraction step, a feature fusion step and an authentication step; before feature extraction, the depth detection is performed to exclude the suspicion of forgery; in the feature fusion, the face depth feature vector is considered in addition to the common face RGB feature vector and the sound feature vector; the face depth feature vector is not easily affected by the local shielding of external objects on the face, is less limited by the site, and the features fused with the face depth feature vector can enhance the stability of the authentication process; the IRFL network is used to increase the face RGB feature vector, identify the regions irrelevant to the identity features and apply noise to the regions. The application can ensure that the operator of the construction robot has qualifications, and protect the code and structural design of the robot from being stolen by unqualified personnel.

Description

TECHNICAL FIELD

[0001] The present application relates to identity recognition technology, in particular to a multi-modal construction site robot safety protection technology. BACKGROUND

[0002] Taking an industrial plastering robot as an example, the robot has independent intellectual property rights, and the internal code, design structure and working mode need to be protected during use to avoid malicious theft by irrelevant personnel. In order to determine the qualification of the operator, the following methods can be used at present:

[0003] 1. Assign special personnel to supervise, but this method requires a special supervisor for each robot, and the personnel cost increases linearly as the number of robots increases.

[0004] 2. A simple and direct method of identifying the operator's identity card is used, but this method only identifies the correctness and error of the information on the card and the qualification of the card registrant. If someone steals the card, they can still impersonate by swiping the card, thereby achieving the purpose of malicious online.

[0005] 3. Use face recognition technology. Face recognition technology is now very mature and is applied in many situations. However, due to the special nature of the industrial plastering robot's working environment, the environment in which the operator works is generally complex, for example, there is a lot of dust, and the operator also wears appropriate construction protective equipment such as a helmet, which will affect the stability and accuracy of the recognition to some extent.

[0006] 4. Use voiceprint recognition technology. By recognizing the voiceprint of the operator, visual problems can be avoided to some extent, but due to the working environment of the construction site, there may be a lot of continuous background noise, which affects the accuracy and stability of the recognition. SUMMARY

[0007] The technical problem to be solved by the present application is to provide a method for identifying the identity of an operator of a construction site robot in a noisy and complex construction site environment.

[0008] The technical solution adopted by the present application to solve the above technical problem is a multi-modal construction site robot safety protection method, comprising the steps of:

[0009] Pre-stored feature searching step: matching the card information of the operator with the database to find the feature vector Y corresponding to the card information;

[0010] Depth detection step: collecting the depth image of the operator through the front camera, and performing depth detection to determine whether it is normal, and if it is normal, proceeding to the subsequent feature extraction step;

[0011] Facial depth feature extraction steps: Use a trained deep feature network to extract facial depth feature vectors from the operator's depth image;

[0012] The facial RGB feature extraction steps are as follows: A trained RGB feature network is used to extract the RGB feature vector of the operator's face from the RGB image captured by the front-facing camera. This vector is then enhanced by an identity-related feature learning (IRFL) network to obtain the final facial RGB feature vector. The IRFL network is used to identify regions in the head region that are unrelated to identity features, and noise is added to these regions whose probability density function conforms to a normal distribution. These unrelated regions are areas obscured by objects.

[0013] Voice feature extraction steps: Use the trained voice feature network to perform triangular filtering on the voiceprint feature information that is not related to the text to obtain the voice feature vector;

[0014] Feature fusion steps: Expand the facial depth feature vector, facial RGB feature vector, and voice feature vector from the 1-dimensional data space to 2-dimensional, and then perform a Cartesian product to obtain the fused feature V.

[0015]

[0016] Where V1 is the facial depth feature vector, V2 is the facial RGB feature vector, and V3 is the voice feature vector. Represents the Cartesian product;

[0017] Authentication steps: After reducing the dimensionality of the fused feature V, input the dimensionality-reduced fused feature V into the authentication network. The authentication network outputs the authentication result. When the authentication is successful, operation permissions are granted.

[0018] Before feature extraction, this invention first uses depth detection to eliminate forgery suspicions. During feature fusion, in addition to considering the conventional facial RGB feature vectors and voice feature vectors, facial depth feature vectors are also considered. Facial depth feature vectors are less affected by partial occlusion of the face by external objects and are less restricted by the scene; fusing facial depth feature vectors enhances the stability of the authentication process. An IRFL network is used to augment the facial RGB feature vector extraction, aiming to create a cleaner facial feature representation. The IRFL network identifies regions unrelated to identity features, such as parts of the face obscured by goggles, helmets, etc. To prevent interference from non-identity-related factors, noise is added to these unrelated regions, rendering them non-specific and preventing them from affecting the final authentication result.

[0019] The beneficial effects of this invention are that it ensures that the operators of the industrial robot are qualified and protects the robot's code and structural design from being stolen by unqualified personnel. Detailed Implementation

[0020] Step 1: Collect card information by having operators swipe their cards, match it with the database, and find the corresponding operator and their feature vector Y.

[0021] Step 2: Collect depth images of the operator using the front-facing camera and perform three samplings at different angles to obtain three depth images D1, D2, and D3;

[0022] Calculate the variance of depth in the depth image within the headshot frame for each depth image:

[0023]

[0024] Where x i Let x be the depth value of each depth point, x be the expected value of the entire depth image, and n be the total number of depth points in the image.

[0025] If this variance s 2 If the depth is less than the preset threshold δ, which is preset to 2.5, it indicates that the depth fluctuation is not obvious and there is suspicion of forgery. 2 If the variance is greater than the threshold δ, it indicates that the depth detection is normal. When the variance of all three depth images is greater than δ, we can proceed to the next step.

[0026] Step 3: Use a CNN network to extract facial depth feature vectors from the operator's depth image. Since depth images and RGB images are highly aligned, it is desirable to retain feature representations similar to those in the RGB image within the feature vectors. Therefore, a CNN network is also used for feature extraction.

[0027] Step 3-1: Feature extraction is performed using a neural network model modified from Zeiler & Fergus. To obtain a 128-dimensional feature vector, a new convolutional layer is added after the last layer of the Zeiler & Fergus network. The kernel size of this convolutional layer is 1*1*128, and the stride and pooling size are both set to 2.

[0028] Step 3-2: The network uses a triplet loss function, which includes two positive examples and one negative example. The goal is to maximize the Euclidean distance between the positive and negative examples while minimizing the Euclidean distance between the positive examples.

[0029]

[0030] Where xa This refers to the images used in training, x p finger and x a The same image of an individual, x n finger and x a For images of different individuals, the subscript i indicates the depth point number.

[0031] Step 4: Capture the operator's RGB image using the front-facing camera and extract its features.

[0032] Step 4-1: Feed the RGB image into the pooling layer, and iterate through all the RGB pixels using the sliding window method. Set the stride to 1, the window size to 4*4, and the pooling strategy to max-pooling, so that the windows can overlap and fully preserve the feature information in the original image.

[0033]

[0034]

[0035] Where W' and H' are the width and height of the sliding window, W and H are the width and height of the image, S is the step size of the sliding window, F is the size of the window, and the image after pooling is denoted as X;

[0036] Step 4-2: X will simultaneously learn an IRFL (Identity Related Feature Learning) network through a CNN network and identity-related features. In the CNN, this will be a deep residual value network (ResNet). By adding shortcuts to the deep neural network, the degradation problem of deep neural networks is avoided. This network has a total of 34 layers of neurons, with a shortcut added every two layers. That is, every two layers of neurons form a residual unit. By learning the residual H(x)-x, where x is the input data and H(x) is the learned feature, when the residual is 0, the network only performs an identity mapping, so it does not increase the learning difficulty of the network. The added shortcuts allow the input to skip the residual unit and directly affect the input of the next residual unit.

[0037] y l =x l +f(x l ,θ)

[0038] x l+1 =R(y l )

[0039] Where x l It is the output of the previous residual unit and the input of the current residual unit. f is the mapping learned by this residual unit, and y is the input of the current residual unit. lLet θ be the intermediate value, where θ is the set of parameters in the mapping, and x is the intermediate value. l+1 R is the output of this residual unit, and R is the ReLU function.

[0040] The ResNet algorithm will produce a feature vector X. The main function of this network is to recognize faces and form a feature representation; it is a recognition network.

[0041] The function g maps X to a set of feature vectors using the learned parameters θ, and then immediately performs subsequent recognition work.

[0042]

[0043] From parameters The obtained features are compared with Y through the network to give a final judgment. However, this method still needs to wait for the IRFL network to be enhanced.

[0044]

[0045] The recognition network is trained by minimizing this mean squared error loss function. The difference between IRFL and ordinary face recognition lies in the network's purpose: to serve the overall network framework of multimodal identity authentication and recognition. IRFL aims to form a cleaner facial feature representation. Therefore, in the process of minimizing the mean squared error loss function, the network learns how to identify regions unrelated to identity features, such as hats, helmets, and scarves. The output of this step is a series of bounding boxes stored in a vector with a specific format, used to mark the approximate locations of identity-independent regions.

[0046] [X',Y',W',H']

[0047] Where X' represents the X-coordinate of the center position of the calibration box, Y' represents the Y-coordinate of the center position of the calibration box, W' represents the width of the calibration box, and H' represents the height of the calibration box. These four parameters allow us to locate the position of a region. The output of this step will be a matrix containing data from several calibration boxes; let's call this matrix M.

[0048]

[0049] Step 4-3: With the M matrix, we can distinguish between important and distracting parts of the image. To avoid interference from non-identity-related factors, we need to add noise to each labeled region in M. To ensure that the noisy image regions no longer have special characteristics, the probability density function of the noise in the region needs to conform to a normal distribution. For each pixel in a region, perform the following operation: pixel = pixel + k*GS(mu, sigma).

[0050] Where pixel is the value of each pixel, k is the coefficient (set to 256 in this example), GS is a function that generates numbers that follow an N(0,1) distribution, mu is the mean (set to 0), and sigma is the variance (set to 1).

[0051] This gives us the processed RGB portrait data, which we'll call P.

[0052] Step 4-4: Input the image into a 34-layer ResNet34. The image first passes through a convolutional layer with 64 7x7 kernels, a stride of 2, and padding of 3. Then it passes through a pooling layer with 3x3 kernels, a stride of 2, and padding of 1. After that, it passes through 6 consecutive convolutional layers with the same parameters: 3x3 kernels, 64 kernels, a stride of 1, and padding of 1. Shortcuts are added to every two layers as a residual block. Next, it passes through a convolutional layer with 128 3x3 kernels, a stride of 2, and padding of 1, followed by a similar convolutional layer with a stride of 1. Then, it passes through eight consecutive convolutional layers with 128 3x3 kernels, a stride of 1, and padding of 1, each pair forming a residual block with a shortcut added. Next, it passes through a convolutional layer with 256 3x3 kernels, a stride of 2, and padding of 1, and another convolutional layer with identical parameters but a stride of 1. Then, it passes through ten consecutive convolutional layers with 256 3x3 kernels, a stride of 1, and padding of 1, each pair forming a residual block. The network consists of several sub-blocks, each with a shortcut added. The sub-blocks pass through a convolutional layer with 512 3x3 kernels, a stride of 2, and padding of 1, and another convolutional layer with identical parameters but a stride of 1. This is followed by four more convolutional layers with 512 3x3 kernels, a stride of 1, and padding of 1. Each sub-block is paired with a residual block, and a shortcut is added. Finally, the sub-blocks pass through an average pooling layer, connecting to a fully connected layer with 1000 neurons. The output layer is then a final fully connected layer with 128 neurons. This is a facial feature extraction network for visual face recognition. It is initially trained for face recognition purposes, using the following loss function:

[0053]

[0054] After training, only the parameters of the part other than the final classification output layer are saved, which are then used to extract facial feature vectors. The feature vectors will be 128-dimensional vectors.

[0055] Step 5-1: The main purpose of step 5 is to process and abstract acoustic features. Since the final goal is for identity verification, the primary objective is to extract text-independent voiceprint features from the short text. In actual use, the received sound information enters a pre-trained deep neural network (DNN). This neural network aims to extract and represent text-independent voiceprint features from the sound signal. The structure of the deep neural network used to extract the speaker feature vector (d-vector) is as follows: There are five hidden layers, each with 512 units. The first three layers do not use regularization strategies, while the last two layers use a random dropout strategy with a 50% inactivation probability. During training, a softmax layer is added as the output layer to output a one-hot vector representing the speaker's identity. However, only the output of the fifth layer is needed for feature extraction. This is also a Maxout Network to better optimize individual activation functions. The activation function of the hidden layers is the ReLU function, with a learning rate initialized to 0.001 and decaying exponentially.

[0056] Step 5-2: The collected sound signal first needs to be processed into 40-dimensional Filter Banks features, which reduces the DCT (Discrete Cosine Transform) operation compared to MFCC. To process the sound signal into Filter Banks features, the sound is first sampled at a sampling frequency of 16kHz. Then, the resulting numerical signal is pre-emphasized to enhance the high-frequency components and reduce the influence of the vocal cords and lips during phonation. The result of the pre-emphasis is...

[0057] s(xs)=s(xs)+k*s(xs-1)

[0058] Where s is the sound signal, xs is the time-domain variable, and k is the pre-enhancement coefficient, which is 0.97.

[0059] Step 5-3: Frame the pre-enhanced signal, dividing the variable-length signal into equal segments. Set the length of each segment to 25ms and the frame shift to 10ms. Set the frame shift to prevent information loss at both ends of the frame after windowing.

[0060] Step 5-4: Window each frame of the signal and multiply each frame by a smooth window function to make the signal change less drastic, reduce the intensity of the side lobes after the Fourier transform, and thus obtain better results. Here, the Hamming window is selected as the window function w(n).

[0061]

[0062] Step 5-5: At this point, the signal still contains a mixture of high and low frequencies and is a time-domain signal. A Fast Fourier Transform (FFT) is used to convert the signal to the frequency domain, separating the various frequencies to facilitate neural network training. Because it is a digital signal, a Discrete Fourier Transform (DFT) is used. A 512-point FFT is performed on each frame to calculate the spectrum, followed by a Short-Time Fourier Transform (SFT).

[0063]

[0064] To generate 40 filter banks, 40 triangular filters are used for filtering, where the relationship between frequency f and Mel frequency m is as follows:

[0065]

[0066] f = 700(10) m / 2595 -1)

[0067] The equation for the triangular filter function is as follows:

[0068]

[0069] H m (k) represents the amplitude of frequency k on the m-th triangular filter function.

[0070] Steps 5-6: Normalize the obtained filter banks to balance the spectrum and improve the signal-to-noise ratio by subtracting the average value of all frames in each frame. This results in 40-dimensional filter bank features.

[0071] Step Six: The characteristic of multimodal information lies in the mutual enhancement and complementarity of modalities. After obtaining 40-dimensional voice features, 128-dimensional facial RGB features, and facial depth features, the three features need to be fused. Here, the original 1-dimensional data space is expanded to 3 dimensions by performing a Cartesian product after expansion.

[0072]

[0073] ∧ represents logical AND.

[0074] At the same time, it preserves the internal interrelationships of features without discarding the external correlations of features.

[0075]

[0076] Where V1 is the facial depth feature vector, V2 is the facial RGB feature vector, and V3 is the voice feature vector. To represent the Cartesian product, first expand V1, V2, and V3 with 1s, resulting in vectors of 41, 129, and 129 dimensions respectively. Then, perform a Cartesian product on these three vectors. The final fusion feature matrix V is obtained, where V is a 41*129*129 matrix. Thus, the matrix will include the external feature relations generated by the Cartesian product and the original internal feature relations generated by the expansion element 1.

[0077] Step 7: The feature fusion matrix is ​​too large, with 682,281 features. Since the feature matrix is ​​generated by the Cartesian product, a lot of redundancy will inevitably be generated in the matrix, which will increase the difficulty and accuracy of model training. Before actually entering the certification DNN, it is necessary to reduce the feature dimension.

[0078] Step 7-1: Decentering. There are currently three modalities: sound V1, depth V2, and RGB V3. Decentering is performed on each of the three modalities. First, the mean of the vector for each modality is calculated.

[0079]

[0080] Then, each feature is subtracted from its mean, so that the center of the feature returns to the zero position, where n is the dimension of each modality vector.

[0081] Step 7-2: Calculate the covariance matrix of each mode.

[0082]

[0083] Step 7-3: Calculate the eigenvalues ​​of the covariance matrix C, denoted as a1, a2, a3…ap (p<=n), where p is the dimension of the eigenvalues;

[0084] Step 7-4: Calculate the importance of each feature value and determine the feature dimensions after dimensionality reduction. The method for calculating the degree of importance is as follows:

[0085]

[0086] Features with an importance score greater than 0.9 are defined as important features and retained. The number of target features P after dimensionality reduction is obtained.

[0087] Step 7-5: Create a projection matrix, calculate the eigenvectors bi corresponding to the first P largest eigenvalues, and then form the projection matrix M:

[0088] M=[b1, b2, b3,..., bP]

[0089] Step 7-6: Combine the projection matrix M and the original eigenvector Vi to obtain the dimensionality-reduced eigenvector W.i :

[0090] W i =M T ×V i

[0091] Step 8: Input the feature matrix into the final authentication DNN, which is a neural network with two hidden layers and one output layer. The network receives two matrices: one is the feature matrix of the authentication initiator collected on-site, and the other is the feature matrix corresponding to the object claimed by the authentication initiator, stored in the database. Expand the two matrices into vectors and then input them into the authentication neural network. The neural network has two hidden layers: the first layer has 128 units, the second layer has 64 units, and finally, there is an output unit. The activation function of all hidden layers is set to the ReLU function, and the activation function of the output unit is the sigmoid function, used for binary classification. An output result of 1 indicates that the two matrices match and authentication is successful; otherwise, authentication fails. The learning rate of the authentication network is set to 5e4, the optimizer is Adam, and random deactivation is used for regularization with a deactivation probability of 0.15.

[0092] Step 8: If the returned result is 1, it means that the authentication was successful and the robot has been granted operation permissions. If the returned result is 0, it means that the authentication failed and you can apply for re-authentication.

Claims

1. A multimodal safety protection method for construction site robots, characterized in that, Includes the following steps: Pre-stored feature lookup steps: Receive the operator's card information and match it with the database to find the feature vector Y corresponding to the card information; Depth detection steps: Collect depth images of the operator using the front-facing camera and perform three samplings at different angles to obtain three depth images; calculate the variance of the depth in the head region of the three depth images, and calculate the mean variance s² of the three depth images. If the mean variance s² is greater than or equal to the preset threshold δ, it indicates that the depth detection is normal; otherwise, it indicates that the depth fluctuation is not obvious and the subsequent steps will not be performed. Facial depth feature extraction steps: Use a trained deep feature network to extract facial depth feature vectors from the operator's depth image; The steps for extracting RGB features of the face are as follows: The trained RGB feature network is used to extract the RGB feature vector of the operator's face from the RGB image captured by the front-facing camera, and then the RGB feature vector is enhanced by the IRFL network to obtain the final RGB feature vector of the face. The IRFL network is used to identify regions unrelated to identity features in the head region, and to add noise to the probability density function of these unrelated regions, which follows a normal distribution; the regions unrelated to identity features are those occluded by objects. Voice feature extraction steps: Use the trained voice feature network to perform triangular filtering on the voiceprint feature information that is not related to the text to obtain the voice feature vector; Feature fusion steps: Expand the facial depth feature vector, facial RGB feature vector, and voice feature vector from the 1-dimensional data space to 2-dimensional, and then perform a Cartesian product to obtain the fused features. : ； in, This is the facial depth feature vector. This is the RGB feature vector of the face. For sound feature vectors, Represents the Cartesian product; , and They represent , and The vector expanded with 1; Authentication steps: merging features After dimensionality reduction, the fused features are... The system inputs an authentication network, which outputs the authentication result. Once authentication is successful, operation permissions are granted.

2. The method as described in claim 1, characterized in that, Both the deep feature network and the RGB feature network are convolutional neural networks (CNNs). The sound feature network is a deep neural network (DNN).

3. The method as described in claim 2, characterized in that, The threshold δ is preset to 2.5.

Images