Sign language alphabet spelling recognition method based on convolutional neural network

Abstract

The method for recognizing sign language letter spelling based on convolutional neural network extracts hand depth map features by using convolutional neural network and recognizes sign language letter spelling. After a depth camera acquires sign language pictures and depth pictures, the camera sends the pictures to a target detection network to extract accurate hand target pictures and accurate depth pictures. After the hand target is extracted, the depth picture is segmented by using a linear transformation based on gray value pseudo color and a color gamut segmentation algorithm to segment the accurate sign language gesture target, and a color fusion algorithm is used to supplement the lost sign language gesture information. After segmentation, the picture is pixel processed into a single channel binary image by using gray scale and local area binary to reduce the network input parameter quantity, and the picture preprocessing is completed. Finally, the preprocessed sign language gesture picture is sent into a convolutional neural network for feature extraction, the extracted features are connected with a full connection layer, and classification is performed by using a softmax classifier. After training, the network model is saved and used for sign language letter spelling recognition.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image recognition processing technology, specifically relating to a sign language letter spelling recognition method based on convolutional neural networks. Background Technology

[0002] Gestures are the most effective and natural way for people to convey information through physical movements. Although gestures are not traditional language communication, they have become an effective medium for communication between deaf and hard-of-hearing individuals. The National Institute on Deafness and Other Communication Disorders defines American Sign Language (ASL) as a complete and complex language that uses gestures to express thoughts and communicate information among deaf or hard-of-hearing individuals. Recent sign language recognizers rely on hard-coded algorithms, such as edge detection. Furthermore, these algorithms require preprocessing, such as image segmentation, to remove unwanted data, or morphological filters to remove defects (dilation, erosion). Some authors have investigated the application of artificial intelligence-based algorithms in sign language image recognition. Islam et al. used a feedforward artificial neural network (ANN) trained with backpropagation using 30 feature vectors. They combined the K-curvature algorithm and the convex hull algorithm for feature extraction, using fingertip detection as a preprocessing stage. Pansare et al. proposed an ASL gesture recognizer based on edge orientation histograms, employing methods such as region and feature extraction, feature matching, and pattern recognition to preprocess sign language images and improve recognition accuracy. Recent advancements in Artificial Intelligence (AI), such as deep learning and transfer learning, can help bridge this communication gap while improving the classification accuracy and computational power required for the inference stage, increasing the value of automation, pattern recognition, feature extraction, and neural learning, and reducing software development and maintenance costs.

[0003] Most current work is based on RGB image segmentation. However, RGB image segmentation technology has many drawbacks. For example, when the ambient light and the user's skin tone are similar, the segmentation effect is very poor. Furthermore, classifiers trained on RGB images have poor performance in classifying different users, which greatly limits the generalization ability of the model. Research shows that current sign language gesture segmentation work is not perfect, and the current classification models have poor robustness in low-light scenes, noise interference, and target-irrelevant scenes. In real life, sign language environments are complex and varied, which poses a huge challenge to sign language spelling recognition. In addition, most works use large models with many training parameters, resulting in high training costs. Summary of the Invention

[0004] To address the above problems, this invention proposes a sign language spelling recognition method based on convolutional neural networks. It uses a target detection algorithm to segment the hand target from images captured by a depth camera to obtain accurate sign language gesture images. Precise hand contour information is extracted using gesture segmentation and image processing algorithms, facilitating hand separation from the background and enabling feature extraction by the convolutional neural network. The convolutional neural network is then used to extract sign language spelling gesture features and train the network. After training, it can recognize sign language spellings and improve recognition accuracy. The proposed lightweight convolutional neural network enhances recognition accuracy in complex scenes such as those with changing lighting, noise interference, and unrelated targets, improving the system's robustness. The use of a lightweight neural network reduces network structure complexity and training overhead.

[0005] To achieve the above objectives, the technical solution proposed in this invention is a sign language spelling recognition method based on convolutional neural networks. Sign language spelling recognition mainly includes three processes: hand target detection, gesture segmentation, and network training. The hand target detection stage mainly involves: generating regions of interest where targets exist, and extracting features to filter targets. The gesture segmentation stage mainly involves: visualizing depth maps based on distance information, color gamut segmentation and color fusion, and feature channel conversion. The network training stage mainly involves: extracting features to train the network.

[0006] The main contents of this method are as follows:

[0007] Step 1: Hand target detection. Acquire color and depth images of sign language using a depth camera. Use a target detection network to cut out the hand from the image. Simultaneously, cut the depth image based on the size information of the cut hand image and the intrinsic parameters of the depth camera.

[0008] Step 2: Normalize the acquired local depth image to convert the depth information into grayscale information, and perform a pseudo-color linear transformation based on the grayscale information to visualize the depth image.

[0009] Step 3: Obtain the visualized depth map and segment the gestures according to the color gamut. At the same time, judge the segmented gesture results according to the gesture fusion rules and perform color fusion.

[0010] Step 4: Perform grayscale and binarization on the generated sign language segmentation gestures, and perform pixel inversion on the processed results to concentrate the image pixel information. After processing, feed the image into a convolutional neural network for training.

[0011] Step 5: During training, the segmentation depth map size is normalized and fed into a convolutional neural network for feature extraction. The output features are connected to a fully connected layer, and finally, a softmax function is used to output 24 classification probabilities. After setting the number of training rounds, the trained model is saved and loaded for testing. Four of the five-person datasets (sign language) are used for training, and one person is used for testing. Five cross-training iterations are performed to verify the model's generalization ability and save the best-performing training model.

[0012] Furthermore, in step 1, an image pyramid is generated based on the original sign language color image to accommodate objects of different sizes. Several regions of interest (ROIs) containing potential targets are obtained from the image pyramid using a selective search algorithm. These ROIs are scaled to 227*227 pixels and input into a convolutional neural network (CNN) to extract features. The output of the fully connected layer of the CNN is used as the input to an SMV classifier to classify the input image. Boundary regression is performed on the ROIs classified by the SVM. The regression values ​​of the bounding boxes are used to correct the original proposal windows, generating prediction window coordinates and prediction probabilities. The result with the highest prediction probability is selected for output and saved.

[0013] Furthermore, in step 2, the raw sign language gesture depth information acquired by the depth camera is 16 bits, with a pixel range of (0 to 65535). First, the 16-bit depth information needs to be normalized to 8 bits. Assuming the original depth information matrix H is normalized to J, the following formula applies:

[0014]

[0015] After normalization, all pixel information is normalized to the range of (0~255). At this time, the depth information is converted into grayscale information, and pixels at the same distance will have the same grayscale value.

[0016] A 16-bit depth map is normalized and then converted to an 8-bit depth map. Distance information is converted into grayscale information. Image grayscale information represents the local brightness and darkness of the image. For a given image's bit depth and its value range, a grayscale-to-color image conversion can be performed to make the differences within the image more obvious, thereby achieving gesture segmentation. First, a pseudo-color linear transformation based on grayscale information needs to be performed on the original depth data so that the minimum and maximum values ​​at the original depth correspond to the minimum and maximum values ​​at the converted bit depth, respectively. Assuming Src is a three-dimensional array representing the image with dimensions height, width, and dimension, Dst is the output array after the linear transformation, scale is a scaling factor, and shift is the offset, the transformation is performed using the following linear transformation formula:

[0017] Dst=Deal(Src)×scale+(shift)

[0018] The pixel inverse transformation matrix Deal(src) is derived from the following linear transformation formula:

[0019]

[0020]

[0021]

[0022] Where R(x,y), G(x,y), and B(x,y) represent the color values ​​of the R, G, and B channels, respectively, f(x,y) represents the gray value of a specific point in the grayscale image, and f is the gray value of the selected grayscale image.

[0023] After inputting the image, the array is scaled according to the scaling factor (scale) and the elements are offset by the offset amount (shift). After scaling, the image depth and pixel information change accordingly, thus changing the color.

[0024] The scale factor is determined based on the distance between the hand and the camera, and is calculated using the following formula:

[0025] D×scale=255

[0026] Where D is the distance from the region of interest to the camera, and different regions of interest can be highlighted with different colors by setting different regions of interest.

[0027] Furthermore, in step 3, to facilitate the modification of image pixels, the image needs to be converted from the RGB image space to the HSV color space. HSV consists of three components: Hue, Saturation, and Value. First, the value of H is modified to determine the color to be segmented, and then the values ​​of S and V are dynamically adjusted to determine the range of colors to be segmented.

[0028] To segment the green color, the range of the green color space is determined according to the color space table as (35, 43, 46) to (77, 255, 255). At the same time, a mask space (img, lowerb, upperb) is generated according to the original image size, where img is an HSV format image, and lowerb and upperb are the mask space ranges.

[0029] Copy the HSV image and perform a mask operation on the image pixels, changing the image pixel values ​​within the mask space to 255 (white) and the remaining image pixel values ​​to 0 (black). Finally, perform an AND operation between the original image and the image processed according to the mask space (removing black and keeping white) to obtain the original image mask position area, and the segmentation is complete.

[0030] Because depth cameras cannot perfectly preserve sign language distance information, when the distance span between the preceding and following sign language movements is large, two colors will appear when visualizing the hand image in the region of interest, even if the region of interest is pre-defined. In order to maintain the integrity of the sign language gestures, it is necessary to record multiple color matrices in the region of interest and merge them.

[0031] The image, after undergoing pseudo-linear grayscale color transformation, is divided into two image matrices, Img_matrix_1 and Img_matrix_2, based on a color gamut segmentation method. The fusion thresholds Eff_low_1 and Eff_low_2 are manually set empirically, and fusion is performed using the following discrimination rules:

[0032] Rule ①: If the pixel value of Img_matrix_1 (or Img_matrix_2) is less than Eff_low_1, the current sign language segmentation image is designated as Img_matrix_2 (or Img_matrix_1).

[0033] Rule ②: If the pixel value of Img_matrix_1 (or Img_matrix_2) is greater than Eff_low_2, designate the current sign language segmentation image as Img_matrix_1 (or Img_matrix_2).

[0034] Rule ③: If the pixel value of Img_matrix_1 (or Img_matrix_2) is greater than Eff_low_1 and less than Eff_low_2, then the image matrix is ​​fused using the following fusion formula:

[0035] result=255-(255-Img_matrix_1)×(255-Img_matrix_2)÷128

[0036] The result is the image after image fusion. This formula can be used to merge two images of different colors to display a complete sign language gesture image.

[0037] Furthermore, in step 4, the grayscale of the color image is converted using a weighted method, with the R, G, B ratio being 3:6:1. Assuming the color of a certain point is RGB (red: R, green: G, blue: B), the following calculation formula applies:

[0038] Gray = R × 0.3 + G × 0.59 + B × 0.11

[0039] Where R, G, and B are the three primary colors of the image, representing red, green, and blue respectively, and Gray is the grayscale value of the image. The coefficients are the values ​​obtained after weighted transformation. In order to segment the gesture target from an image with uneven brightness, a uniform threshold cannot be used to filter the global target. Instead, we consider starting from local pixels and gradually calculating the threshold with the current pixel as the center. To highlight the pixel features of the image, the threshold needs to be inverted. First, the image is binarized using a local thresholding binarization method. Then, the thresholding result is inverted using the following formula:

[0040] Reverse = 255-binary

[0041] Where Reverse is the flipped image and binary is a single-channel binary image. This method flips the image to make its features more apparent.

[0042] Furthermore, in step 5, to effectively extract the feature information of sign language, a lightweight convolutional neural network is proposed. The network consists of three convolutional modules: conv1, conv2, and conv3. To extract features more effectively, the first layer is set to have a 32-channel convolutional kernel, the second layer to have 64 layers, and the third layer to have 128 layers. The convolution operation is performed using the following formula:

[0043]

[0044] Where N is the size of the output feature map after convolution, W is the size of the input network image / feature map, F is the kernel size, P is the dilation coefficient, and S is the convolution stride.

[0045] To accelerate computation and prevent gradient vanishing, the ReLU non-linear activation function is used. Max pooling is performed after each convolutional layer to reduce the number of parameters. To accelerate network convergence, the stride of each convolutional layer is set to 3. Dropout is implemented to prevent overfitting. During training, the segmentation depth map size is normalized and fed into the convolutional neural network for feature extraction. The output features are concatenated with fully connected layers, and finally, a softmax function is used to output 24 classification probabilities. After setting the number of training epochs, the trained model is saved and loaded for testing. A five-person dataset is used, with four people's sign language data for training and one person for testing. This is repeated five times in a cross-training cycle to verify the model's generalization ability and save the best-performing training model.

[0046] After the model training is complete, the model is loaded and sign language recognition is performed. First, the camera acquires depth images of human sign language and segments them using the depth image segmentation algorithm proposed in this invention. Then, the segmented depth images are fed into the network for prediction, and the prediction results are output as text. Finally, the prediction results are recorded and compared with the real labels to verify the model's sign language recognition performance.

[0047] Compared with existing technologies, the beneficial effects of this invention are as follows:

[0048] (1) In this invention, the computational overhead is reduced and the computational efficiency is improved by using a lightweight network.

[0049] (2) Based on visual depth information, this invention avoids the influence of environmental factors such as skin color and light, and has high accuracy and low error in sign language gesture segmentation.

[0050] (3) Based on the traditional method, this scheme makes gesture segmentation more accurate by performing grayscale pseudo-color linear transformation on image depth information, thereby improving the accuracy of sign language gesture segmentation.

[0051] (4) The present invention can significantly improve the accuracy of sign language recognition with fewer training rounds. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating sign language letter spelling recognition in an embodiment of the present invention.

[0053] Figure 2 This is a flowchart illustrating the hand target detection stage in an embodiment of the present invention.

[0054] Figure 3 This is a flowchart illustrating the process of visualizing a depth map using depth information in an embodiment of the present invention.

[0055] Figure 4 This is a flowchart illustrating the process of color gamut segmentation and color fusion stages for gesture segmentation in an embodiment of the present invention.

[0056] Figure 5 This is a flowchart illustrating the feature channel conversion process for gesture segmentation in an embodiment of the present invention.

[0057] Figure 6 This is a flowchart illustrating the process of extracting features and training the network in an embodiment of the present invention.

[0058] Figure 7 This is a flowchart illustrating the workflow of the convolutional neural network in an embodiment of the present invention. Detailed Implementation

[0059] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0060] This research on sign language spelling recognition based on deep learning uses a depth camera to capture RGB-D images of sign language. The depth information is used to segment the gestures from the background to obtain accurate sign language gestures. A convolutional neural network model is then trained to train the model, and finally, the sign language is recognized and classified.

[0061] First, the target sits at a certain distance in front of the Kinect depth camera. After stabilizing, they begin performing sign language gestures at a fixed distance. Simultaneously, the Kinect captures images of the sign language gestures at pre-set time intervals and saves them according to category. 1000 images are captured for each sign language gesture. After capturing these images, the Kinect depth camera automatically generates an RGB color image and a 16-bit depth image. The color image stores the pixel information of the sign language gesture, while the depth image stores the distance information from the sign language gesture to the depth camera. Next, the captured sign language images undergo target detection processing. This process includes two steps: generating regions of interest (ROIs) containing the target, and feature extraction and target selection. First, the Faster R-CNN object detection network detects the gesture target. Using a sliding window, ROIs are extracted from the original image according to a pre-set aspect ratio bounding box. After extraction, the target selection phase begins. The RPN network extracts features from the ROIs and trains the RPN network, ultimately obtaining the most accurate gesture proposal. The RGB color image and depth image are then cropped using the gesture proposal, resulting in a precise RGB color image and depth image of the sign language gesture. After target detection is complete, sign language gesture segmentation begins. The main tasks in this stage include: depth image normalization, depth map visualization based on distance information, color gamut segmentation, color fusion, and grayscale binarization. These operations accurately separate the gestures from the background. Finally, in the network training stage, the accurate sign language gesture images are fed into the network for training and sign language recognition and classification. In the sign language gesture segmentation stage, this invention utilizes depth information, converting it to color information and segmenting it using color gamut information. This allows for accurate segmentation of sign language gestures, enabling the network to better extract sign language features. This method is unaffected by factors such as skin color and ambient light, and significantly improves the accuracy of sign language spelling recognition.

[0062] This method mainly consists of the following steps:

[0063] Step 1: Hand target detection, cutting out the hand from the entire captured image. The workflow diagram is as follows. Figure 2 As shown:

[0064] Step 1.1) Take RGB color images and depth maps of the sign language with the camera.

[0065] Step 1.2) Extract the region of interest using the RPN object detection network.

[0066] Step 1.3) The region proposal is fed into a convolutional neural network with 13 convolutional layers, 13 non-linear activation layers, and 4 pooling layers for feature extraction.

[0067] Step 1.4) Combine the obtained region of interest with the features extracted by the convolutional neural network. Figure 1 The data is fed into the region of interest pooling layer for feature processing, bounding box regression is performed and fully connected to the fully connected layer, and the network predicted coordinates are output. The size of the region of interest is then corrected to obtain an accurate sign language gesture map.

[0068] Step 1.5) Select a specified number of data points from the output set of region of interest data (output data consists of images and probability information) in descending order of probability, and simultaneously cut the depth image according to the image size information of the cutting gesture and the intrinsic parameters of the depth camera.

[0069] Step 2: Perform a pseudo-color linear transformation based on grayscale information on the acquired local depth image to achieve depth image visualization. The workflow diagram is as follows: Figure 3 As shown:

[0070] Step 2.1) Set the effective depth range in the depth image.

[0071] Step 2.2) Select color allocation strategy: In order of increasing depth, the shallower the depth, the darker the color, and the deeper the depth, the lighter the color.

[0072] Step 2.3) Record the depth map color gamut information and image pixel information to ensure that the range of image color changes is within a certain range and dynamically adjust the color allocation strategy.

[0073] Step 2.4) Output and save the visualized depth map.

[0074] Step 3: Obtain the visualized depth map, segment the gestures according to the color gamut, and perform color fusion. The flowchart is as follows: Figure 4 As shown:

[0075] Step 3.1) Read the color gamut and depth information of the hand image and determine the color gamut range P of the hand image as [pixela, pixelb].

[0076] Step 3.2) For each pixel in the image, determine whether it is included in the color gamut range P. If it is, copy the pixel value to a new blank image of the same size. If it is not included, clear the pixel value to zero.

[0077] Step 3.3) Add up the pixels from different color gamuts to generate a new sign language segmentation image and save it.

[0078] Step 4: Perform feature channel conversion on the generated sign language segmentation gestures to facilitate network training. The workflow diagram is as follows: Figure 5 As shown:

[0079] Step 4.1) Input the sign language segmentation image and convert it to grayscale.

[0080] Step 4.2) Set a threshold Ex based on experience. If the pixel value is less than or equal to Ex, adjust the pixel value to 0. If the pixel value is greater than Ex, change the value to 255 to obtain the binary image of the sign language gesture.

[0081] Step 4.3) Output a single-channel binary image of sign language gestures.

[0082] Step 5: Through, as Figure 7 The convolutional neural network shown extracts features from segmented gestures and trains and tests the network. The workflow diagram is as follows: Figure 6 As shown:

[0083] Step 5.1) Feed the sign language gesture images into a convolutional neural network to extract features.

[0084] Step 5.2) Update the weights, bias parameters and other parameters, and output the loss function.

[0085] Step 5.3) After the network training is complete, save the training model and test the network according to the test samples to verify the network's effectiveness and accuracy.

[0086] As can be seen from the above description, compared with existing technologies, the deep learning-based sign language spelling recognition algorithm proposed in this invention accurately segments sign language gestures from the background by utilizing sign language depth information. It uses a lightweight convolutional neural network for feature extraction and training, reducing training time and improving network feature utilization. This makes the network model highly adaptable to factors such as skin color, ambient light, and noise interference, improving recognition performance and achieving high-precision sign language spelling recognition.

[0087] The above description is only a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. Any equivalent modifications or changes made by those skilled in the art based on the content disclosed in the present invention should be included within the scope of protection set forth in the claims.

Claims

1. A method for recognizing hand alphabet spelling based on a convolutional neural network, characterized in that: The method comprises the following steps: Step 1, hand target detection, obtain sign language color image and depth image through depth camera, cut out hand image through target detection network, and cut depth image according to hand cutting image size information and depth camera internal parameter; Step 2, normalize the obtained local depth image, convert the depth image into a gray image, and perform pseudo-color linear transformation on the image based on the gray image to realize depth image visualization; Step 3, obtain the visualized depth image and segment the hand gesture according to the color gamut, and discriminate and perform color fusion on the segmented hand gesture result according to the hand gesture fusion rule; In step 3, the image is converted from RGB image space to HSV color space; HSV is composed of hue H, saturation S and value V; first, modify the value of H to determine the color to be segmented, and then dynamically adjust the values of S and V to determine the color range to be segmented; According to the color space table, determine the spatial range of the color, and generate a mask space (img, lowerb, upperb) according to the original image size, wherein img is an hsv format image, lowerb and upperb are mask space ranges; Copy the hsv image and perform mask operation on the image pixels, change the image pixel values in the mask space range to 255 (white), and change the remaining image pixel values to 0 (black), finally perform AND operation on the original gray image and the image processed according to the mask space, that is, remove black and keep white, obtain the mask position area of the original image, and the segmentation is completed; In step 3, in order to maintain the integrity of the sign language gesture, record the color matrix in the area and fuse them together; The image subjected to the gray value pseudo-color linear transformation is saved in two image matrices Img_matrix_1 and Img_matrix_2 through the color gamut segmentation method, and the fusion threshold Eff_low_1 and Eff_low_2 are manually set according to experience and fused through the following discrimination rules: Rule 1, if the pixel value of Img_matrix_1 or Img_matrix_2 is less than Eff_low_1, the current sign language segmentation image is designated as Img_matrix_2 or Img_matrix_1; Rule 2, if the pixel value of Img_matrix_1 or Img_matrix_2 is greater than Eff_low_2, the current sign language segmentation image is designated as Img_matrix_1 or Img_matrix_2; Rule 3, if the pixel value of Img_matrix_1 or Img_matrix_2 is greater than Eff_low_1 and less than Eff_low_2, the image matrix is fused through the following fusion formula: result=255-(255-Img_matrix_1)×(255-Img_matrix_2)÷128 Wherein result is the image after image fusion, through this formula, two colors of images are fused together to show the complete sign language gesture map; Step 4, the generated segmented gesture result is grayed and binarized, and the processing result is pixel inverted, and the image after pixel inversion is sent into the convolutional neural network for training; Step 5, the image after pixel inversion is normalized in size and sent into the convolutional neural network for feature extraction, the output features are connected with the full connection layer, and finally the 24 classification probabilities are output through softmax; after the network training is set for a number of rounds, the training model is saved and loaded for testing, the training set and the test set are divided for the collected data set, five cross-training is carried out, the model generalization is verified, and the best training model is saved.

2. The method of recognizing hand alphabet spelling based on a convolutional neural network according to claim 1, characterized in that: In step 1, the image pyramid is generated according to the sign language color image, so as to meet the needs of different size objects, and a plurality of regions of interest in which the target may exist are obtained from the image pyramid through the selective search algorithm; the region of interest is scaled to 227*227 and input into the convolutional neural network to extract features, the output of the full connection layer of the convolutional neural network is taken as the input of the SVM classifier, the sign language color image is classified, the region of interest classified by the SVM is subjected to bounding box regression, the regression value of the regression box is used to correct the initial window, the prediction window coordinates and the prediction probability are generated, the result with the maximum prediction probability is selected and output and saved. 3.The sign language letter spelling recognition method based on a convolutional neural network according to claim 1, characterized in that: In step 2, the information of the depth image obtained by the depth camera is 16 bits, and the pixel range is 0 to 65535; first, the 16-bit depth information needs to be normalized to 8-bit depth, assuming that the original depth information matrix H is normalized to J, and the following formula is used: After normalization, all pixel information is normalized to the range of 0~255, at this time the depth information is converted into gray information, and pixels with the same distance have the same gray value.

4. The sign language letter spelling recognition method based on a convolutional neural network according to claim 3, characterized in that: In step 2, the data under the original depth is subjected to pseudo-color linear transformation based on the gray information, so that the minimum value under the original depth corresponds to the minimum value under the converted depth, and the maximum value under the original depth corresponds to the maximum value under the converted depth, assuming that Src represents a three-dimensional array of an image, the dimensions are height, width and dimension, Dst is the output array after linear transformation, scale is the scale factor, and shift is the offset, and the linear transformation formula is as follows: Dst=Deal(Src)×scale+(shift) Wherein the pixel inverse transformation matrix Deal(src) is obtained by the following linear transformation formula: Wherein R(x,y), G(x,y) and B(x,y) represent the color values of R, G and B channels respectively, and f(x,y) represents the gray value of a specific point gray image, and f is the gray value of the selected gray image; The array is scaled according to the scale factor scale and the elements are offset by shift, the depth information and pixel information of the scaled image are changed accordingly, and the color is changed; The scale factor is determined according to the distance of the hand from the camera, and the scale factor is determined by the following formula: D×scale=255 Where D is the distance from the region of interest to the camera, different region colors are highlighted by setting different regions of interest.

5. The method of claim 1, wherein: In step 4, the gray scale of the color image is converted by the weighted method, and the ratio of R, G and B is 3:6:

1. Assuming that the color of a certain point is RGB, the following calculation formula is obtained: Gray = R x 0.3 + G x 0.59 + B x 0.11 Where R, G and B are the three primary colors of the image representing red, green and blue respectively, Gray is the gray value of the image, and the coefficient is the value obtained after weighted conversion; considering starting from local pixel points, the threshold value is calculated step by step with the current pixel point as the center. In order to highlight the image pixel features, the image needs to be thresholded. First, the image is binarized by the local threshold binarization method, and then the thresholding result is inverted by the following formula: Reverse = 255-binary Where Reverse is the inverted image, and binary is a single-channel binary image. By this method, the image is inverted to make the image features obvious.

6. The method of claim 1, wherein: In step 5, a lightweight convolutional neural network is proposed, which consists of three convolutional modules, namely conv1, conv2 and conv3. The first layer is set to 32 channels, the second layer is set to 64 layers, and the third layer is set to 128 layers. Use ReLU nonlinear activation function, perform maximum pooling after each convolution to reduce the number of parameters, and set the convolution step of each convolution layer to 3; set dropout to prevent network overfitting.

7. The method of recognizing sign language alphabet spelling based on a convolutional neural network according to claim 6, characterized in that: In step 5, the pixel inverted image is normalized in size and sent to the convolutional neural network for feature extraction during training. The output features are connected to the fully connected layer, and finally the 24 classification probabilities are output by softmax.

8. The method of recognizing sign language alphabet spelling based on a convolutional neural network according to claim 7, characterized in that: In step 5, after the model is trained, load the model and perform sign language recognition; first, the camera acquires the sign language depth image and performs segmentation, then sends the segmented sign language depth image to the network for prediction, and outputs the prediction result in text form, finally records the prediction result and compares it with the true label to verify the performance of the model sign language recognition.

Images