A signboard semantic information extraction method for the blind aid field

Abstract

The application discloses a signboard semantic information extraction method for the field of helping the blind, and particularly relates to the fields of image processing and deep learning technology. First, an image description model is designed, wherein a visual transformer is used as an encoder to perform image feature extraction, and a plurality of transformer structures are used as a decoder to generate semantic description. Second, various indoor scene images containing signboards are collected by a blind aid device, and a data set is prepared to fine-tune and test a pre-trained image description model. Third, a target recognition module is used to extract signboard images from images captured by the blind aid device and perform tilt correction. Finally, the signboard images are divided into blocks, linearly projected into fixed-dimension vectors as inputs of the encoder, and input into the image description model to generate semantic information description of the signboard. The application converts the signboard images captured by the blind aid device into natural language description, and provides support for the blind to independently walk in an unfamiliar environment.

Description

Technical Field

[0001] This invention relates to the fields of image processing and deep learning technologies, specifically a method for extracting semantic information from signs for the blind. Background Technology

[0002] Currently, research in the field of assistive devices for the blind is still incomplete and immature, far from achieving the goal of helping visually impaired individuals walk independently in unfamiliar environments. Assistive devices primarily focus on helping visually impaired individuals navigate and avoid obstacles. Existing research relies on technologies such as computer vision, LiDAR, and multi-sensor fusion for obstacle avoidance, while navigation depends heavily on prior maps. When dealing with unfamiliar environments, a map must be constructed first, which reduces its practical applicability and widespread adoption. To address this issue and enable visually impaired individuals to walk more easily in unfamiliar, large-scale indoor environments, this invention proposes a method for extracting semantic information from signposts in the field of assistive devices for the blind. This method allows visually impaired individuals to determine their location in unfamiliar environments by reading signpost information, just like sighted individuals.

[0003] Computer vision, a simulation of biological vision using computers and related equipment, is an important part of artificial intelligence. Its main task is to process acquired images or videos to obtain information about the corresponding scene. The emergence and application of deep learning have greatly improved computer vision in many aspects, such as image classification, object detection, object tracking, and image segmentation. In tasks like image classification and object detection, it has even reached or surpassed human recognition accuracy. In recent years, research on image description has also increased significantly, enabling detailed descriptions of images that conform to semantic information. With the enhanced computing power provided by deep learning systems, computers are able to recognize and react to everything they see, a significant advancement already achieved. This provides greater support for computer vision-based research in assistive technologies for the blind.

[0004] The semantic information extraction method for signs in the field of assistive visual impairment proposed in this invention is based on image description methods. Since Oriol Vinyals et al. proposed the image description model in 2014, using convolutional networks as image encoders and recurrent neural networks as decoders to generate descriptions, image description technology has made a qualitative leap. Currently, the latest methods use transformer-based multimodal models for image description tasks, achieving state-of-the-art results. This provides support for the semantic information extraction method for signs in the field of assistive visual impairment proposed in this patent.

[0005] Prior to this, image description models had been applied in the field of assistive visually impaired individuals. However, describing all assistive images to help blind people understand their environment actually reduced accuracy. This invention more specifically applies image description methods to the extraction of semantic information from signs, making the task more clearly defined. Based on this, this invention provides a method for extracting semantic information from signs using image description technology in the field of assistive visually impaired individuals. Furthermore, a search of relevant technologies revealed that there is currently no method for extracting semantic information from signs using image description technology specifically for the field of assistive visually impaired individuals. Summary of the Invention

[0006] To enable visually impaired individuals to navigate unfamiliar, large-scale indoor environments more autonomously, this invention proposes a method for extracting semantic information from signage in the field of assistive technology for the blind. This method helps blind individuals obtain environmental information and guides them to their desired destinations. Based on image description technology, the method extracts signage from images, encodes the images, and decodes them to generate natural language descriptions that the signage can understand, thereby guiding the blind individual's actions.

[0007] This invention proposes a method for extracting semantic information from signs in the field of assistive visually impaired individuals, comprising the following steps:

[0008] An image description model with an encoder-decoder structure is constructed. The basic computer vision model based on visual Transformer is used as the encoder for image feature extraction, and multiple standard Transformer modules are used to form the decoder for generating semantic descriptions of the image.

[0009] Pre-training and fine-tuning of the image description model; pre-training a Transformer-based multimodal model using a large-scale public dataset of image-text pairs, which learns more features compared to convolutional networks. Training is performed using the loss function of a language model; for input image I, it generates sentences {y1, y2, ..., y1, y2, ..., y3, ..., y4, ..., y5, ..., y6, ..., y7, ..., y8, ..., y9, ..., y1, y2 ...1, y2, ..., y1, y2, ..., y1, y2, N The loss function is as follows:

[0010]

[0011] Where CE is the cross-entropy loss function with label smoothing of 0.1, I represents the input image, y_i is the word predicted at the current time step, y_j (j=0,…,i-1) are the words that have already been generated, p(y_i|I,{y_j,j=0,…,i-1}) is the probability of generating word y_i given that i-1 words have already been generated, and L is the length of the description sentence.

[0012] Dataset Creation. A large number of images of various indoor signs were collected using assistive devices for the visually impaired. The YOLOv5 model was used for recognition, and the signs were cropped to avoid inputting too many non-sign features. Edge detection was performed on the segmented signs to calculate their apex corner coordinates, and affine transformation was used for correction. Finally, each sign was labeled with a semantically relevant descriptive statement.

[0013] For an input sign image X∈R W×H×3 (R W×H×3 Let X represent a set of real numbers with dimensions (W, H, 3), where W and H represent the width and height of the image, respectively, and the third dimension is the number of channels. First, the image needs to be converted into a queue format that conforms to the model input. After passing through the image segmentation module, the image is converted into a 2D sequence X. p ∈R N×(P2×3) The size of each image block is (,P), where P=4 and N=HW / P. 2 The number of image blocks; feature extraction involves four stages. Stage 1, input sequence X p After a linear embedding layer, Convert to a fixed dimension C, output z0∈R N×C Then, using two consecutive Swing Transformer blocks with the dimensions unchanged, the calculation for the two consecutive Swing Transformer blocks is as follows:

[0014]

[0015]

[0016]

[0017]

[0018] Among them, W_MSA is a window-based self-attention module. Compared with the traditional self-attention module, the window-based self-attention module divides the input into non-overlapping windows, each containing M×M blocks, and performs self-attention calculation within each window, reducing computational complexity and improving execution speed. SW_ is a moving window self-attention module. The moving window attention module divides the input using windows of different sizes, solving the problem of lacking cross-window connections in the window-based self-attention module and improving modeling capabilities. LN is the layer normalization operation. and z l These are the outputs of W_MSA and MLP in the l-th Swing Transformer block, respectively; in stage two, adjacent 2×2 blocks are merged through the block merging module, and the dimension is transformed to... After two consecutive Swing Transformer blocks; in stage 4, adjacent 2×2 blocks are merged through the block merging module, and the dimension is transformed. After six consecutive Swin Transformer blocks; in stage four, adjacent 2×2 blocks are merged using the block merging module, and the dimension is transformed. After passing through two consecutive SwinTransformer blocks, the final image features of the sign are obtained.

[0019] The sign feature map is decoded using a standard Transformer. The Transformer block contains two multi-head self-attention blocks. The first multi-head self-attention block performs a mask operation; its input is the output of the previous Transformer block. The resulting self-attention matrix is ​​multiplied by the mask matrix to prevent the last generated word from knowing information after the (i+1)th word. The second multi-head self-attention block takes the sign feature map as input, adds its output to the output of the mask multi-head self-attention block, normalizes the result, and then passes it through a feedforward layer to obtain the Transformer block's output. The output of the last Transformer block passes through a linear layer and then undergoes Softmax to obtain the final predicted output and the probability of each word at each position in the sentence.

[0020] Error backpropagation updates the model parameters. The loss function is calculated the same as during pre-training.

[0021]

[0022] The Adam Optimizer was used as the optimization algorithm, with an exponential decay rate β1 of 0.9 for the first-order moment estimation and β2 of 0.999 for the second-order moment estimation. The learning rate lrate was adjusted throughout the training process according to the following formula:

[0023]

[0024] Where d model For each sub-layer of the Transformer, step_ is the number of training steps, and warmup_ represents the number of "warm-up steps" before the learning rate reaches its maximum value. The given value is 3000-4000 (the total number of training steps can be determined according to the size of the training set and the size of the epoch, and the "warm-up steps" can be adjusted accordingly).

[0025] Indoor scene images are obtained using assistive devices for the visually impaired. The YOLOv5 method is used to detect the presence of signs in the images. If a sign is detected and its proportion in the image exceeds a given threshold (typically 10%, but can be 10%-15%), the sign is segmented from the original image based on its target bounding box. The Canny operator is used to perform edge detection on the segmented sign, and the coordinates of the four vertices of the sign are calculated based on the edge detection results. Then, an affine transformation method is used to correct the tilted sign. The corrected sign is input into the image description model to obtain the predicted output of the semantic description and the confidence probability of each word in the sentence. If the confidence probability of each word is greater than a given threshold, the generated semantic information description is output. Attached Figure Description

[0026] The accompanying drawings, which form part of this invention, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation of this invention.

[0027] Figure 1 This is a flowchart of the semantic information extraction method for signs according to the present invention;

[0028] Figure 2 This is a schematic diagram of the semantic information extraction method for signs according to the present invention;

[0029] Figure 3 This is a schematic diagram of the sign described in this invention; Detailed Implementation

[0030] To better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. Note that the aspects described below with reference to the accompanying drawings and specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0031] A flowchart of a method for extracting semantic information from signs for the blind is shown below. Figure 1 As shown in the diagram, the method is illustrated in the figure below. Figure 2 As shown, the main components include the creation of a dataset of signs for assistive visually impaired scenarios, the design of a Transformer-based image description model, model training, and model inference (usage), as detailed below:

[0032] Step 1: Acquire a large number of images of various indoor scenes using a camera on an assistive device (located at the user's eye). Filter the acquired images; if a sign is detected in an image and its proportion exceeds a threshold, the sign is segmented based on the bounding box of the detected object. This avoids excessive input of non-sign features during training and inference, which could distract attention from the sign. After filtering and processing the acquired images, a large number of sign images are obtained. Each sign image is manually annotated, and five different, detailed semantic sentences are written for each sign.

[0033] Step 2: The image description model adopts an encoder-decoder structure. The encoder uses a computer vision basic model based on SwinTransformer, which can be used to extract features from images or sentences and is compatible with most downstream tasks, including image description. It includes an image segmentation module and four feature transformation modules. The decoder is used to decode image features to generate sentence features, and uses a standard Transformer structure as the encoder.

[0034] Step 3: Model pre-training. The encoder can be trained using ImageNet-22K. Considering the massive amount of data and the extremely high hardware requirements, we directly load the pre-trained weights provided by Microsoft, which have been trained on hundreds of millions of image-text pairs.

[0035] Step 4: Model fine-tuning, forward propagation of input. The pre-trained model has learned feature information from various images and texts. Fine-tuning is then performed for the downstream task of image description, making it more targeted and effective on a specific type of data. For the encoder, the input image is divided into 4×4 blocks, and their features are directly concatenated to make the input image X∈R. W×H×3 Convert to sequence X p ∈R N×(16×3) Where N = HW / 16 is the number of image blocks; sequence X p The final feature map is obtained after four feature transformation stages. Stage 1: Input sequence X p After a linear embedding layer, Convert to a fixed dimension C, output z0∈R N×C Then, using two consecutive Swing Transformer blocks with the dimensions unchanged, the calculation for the two consecutive Swing Transformer blocks is as follows:

[0036]

[0037]

[0038]

[0039]

[0040] Among them, W_MSA is a window-based self-attention module. Compared with other self-attention modules, the window-based self-attention module divides the input into non-overlapping windows, with each window containing M×M blocks, and performs self-attention calculations within each window, reducing computational complexity and improving execution speed. SW_ is a moving window self-attention module. The moving window attention module divides the input using windows of different sizes, solving the problem of lacking cross-window connections in the window-based self-attention module and improving modeling capabilities. and z l These are the outputs of W_MSA and W_MSA in the l-th Swing Transformer block, respectively; in stage two, adjacent 2×2 blocks are merged through the block merging module, and the dimension is transformed to... After two consecutive Swing Transformer blocks; in stage 4, adjacent 2×2 blocks are merged through the block merging module, and the dimension is transformed. After six consecutive Swin Transformer blocks; in stage four, adjacent 2×2 blocks are merged using the block merging module, and the dimension is transformed. After passing through two consecutive SwinTransformer blocks, the final sign image features are obtained. The sign feature map is then decoded using a standard Transformer. Each Transformer block contains two multi-head self-attention blocks. The first multi-head self-attention block performs a mask operation; its input is the output of the previous Transformer block. The resulting self-attention matrix is ​​multiplied by the mask matrix to prevent the last generated word from knowing information after the (i+1)th word. The second multi-head self-attention block takes the sign feature map as input, adds its output to the output of the mask multi-head self-attention block, normalizes the result, and then passes through a feedforward layer to obtain the Transformer block output. The output of the last Transformer block passes through a linear layer and then undergoes Softmax to obtain the final predicted output and the probability of each word at each position in the sentence.

[0041] Step 5: Model fine-tuning, calculate the loss function, and update the model. Backpropagation of the error updates the model parameters. The loss function is calculated as follows:

[0042]

[0043] The Adam Optimizer is used as the optimization algorithm to update the model weights, with hyperparameters β1 = 0.9, β2 = 0.999, ∈ = 10. -8Meanwhile, the learning rate lrate is adjusted throughout the training process according to the following formula:

[0044]

[0045] Where d model The output dimension of each sub-layer of the Transformer is denoted by step_num, the number of training steps is denoted by warmup_steps, which represents the number of "warm-up steps" before the learning rate reaches its maximum value, and is given as 4000.

[0046] Step 6: Model Testing (Inference). An indoor scene image is acquired using an assistive device for the visually impaired. The YOLOv5 model is used to detect the presence of signs in the image and whether the signs occupy more than 30% of the image. If so, the signs are segmented from the original image based on their bounding boxes. The Canny operator is used to detect the edges of the segmented signs, and the coordinates of the four vertices are calculated based on the edge detection results. Then, an affine transformation method is used to correct the tilted signs. The corrected signs are used as input to the image description model to generate a description of the sign information and the confidence probability of each word in the sentence. If the confidence probability of each word is not less than 0.6, the semantic information description of the sign is output.

Claims

1. A method for extracting semantic information from signs in the field of assistive visually impaired individuals, characterized in that, Includes the following steps: An image description model with an encoder-decoder structure is constructed. The basic computer vision model based on visual Transformer is used as the encoder for image feature extraction, and multiple standard Transformer modules are used to form the decoder for generating semantic descriptions of the image. Pre-training and fine-tuning of the image description model; collecting over 5,000 images of various indoor signs using assistive devices and creating a dataset for model fine-tuning; Indoor scene images are obtained through assistive devices for the visually impaired. The YOLOv5 method is used to detect whether there are signs in the images. If a sign is detected and its proportion in the image exceeds a given threshold, the sign in the image is extracted and corrected. Finally, a semantic information description of the sign is generated through an image description model. The fine-tuning process is as follows: Step 1: Dataset creation; Collect over 5000 images of various indoor signs using assistive devices for the visually impaired, identify and crop the signs using the YOLOv5 model to avoid inputting too many non-sign features; Perform edge detection on the segmented signs to calculate the apex coordinates of the signs, and correct them using affine transformation; Finally, label each sign with a descriptive statement that conforms to semantic information. Step 2: For an input sign image X∈R W×H×3 R W×H×3 Let W represent a set of real numbers with dimensions (W, H, 3); W and H represent the width and height of the image, respectively, and the third dimension is the number of channels. First, the image needs to be converted into a queue format that conforms to the model input. After passing through the image segmentation module, the image is converted into a 2D sequence. The size of each image block is (P, P), where P = 4 and N = HW / P. 2 The number of image blocks; feature extraction goes through four stages; stage one, input sequence X p After a linear embedding layer, Convert to a fixed dimension C, output z0∈R N×C Then, using two consecutive Swing Transformer blocks with the dimensions unchanged, the calculation for the two consecutive Swing Transformer blocks is as follows: Among them, W_MSA is a window-based self-attention module. Compared with the self-attention module, the window-based self-attention module divides the input into non-overlapping windows. Each window contains M×M blocks, and self-attention is calculated for each window. SW_MSA is a moving window self-attention module. The moving window attention module divides the input using windows of different sizes. LN is a layer normalization operation. and z l These are the outputs of W_MSA and MLP in the l-th Swing Transformer block, respectively; in stage two, adjacent 2×2 blocks are merged through the block merging module, and the dimension is transformed to... After passing through two consecutive Swing Transformer blocks; in stage three, the block merging module merges adjacent 2×2 blocks, transforming the dimension to... After six consecutive Swin Transformer blocks; in stage four, adjacent 2×2 blocks are merged using the block merging module, and the dimension is transformed. After passing through two consecutive SwinTransformer blocks, the final image features of the sign are obtained; Step 3: The sign feature map is decoded using a standard Transformer. The Transformer block contains two multi-head self-attention blocks. The first multi-head self-attention block performs a Mask operation, and its input is the output of the previous Transformer block. After obtaining the self-attention matrix, it is multiplied by the Mask matrix to prevent the last generated i-th word from knowing the information after the (i+1)-th word. The input of the second multi-head self-attention block is the sign feature map. Its output is added to the output of the Mask multi-head self-attention block, and then normalized and passed through a feedforward layer to obtain the output of the Transformer block. The output of the last Transformer block is passed through a linear layer and then Softmax to obtain the final predicted output and the probability of each word at each position in the sentence. Step 4: Backpropagate the error and update the model parameters; the loss function calculation is the same as before training. Where I represents the input image, y i For the word predicted at the current time step, y j (j = 0, ..., i-1) represents the words that have already been generated, p(y i |I,{y j The expression {j = 0, ..., i-1} represents the process of generating word y given that i-1 words have already been generated. i The probability is given by L, where L is the length of the description statement; adaptive matrix estimation is used as the optimization algorithm, with the exponential decay rate β1 = 0.9 for the first-order moment estimation and β2 = 0.999 for the second-order moment estimation; the learning rate lrate is changed throughout the training process according to the following formula: Where d model The output dimension of each sub-layer of the Transformer is denoted by step_num, the number of training steps is denoted by warmup_steps, which represents the number of "warm-up steps" before the learning rate reaches its maximum value, and is given as 3000-4000.

2. The method for extracting semantic information from signs in the field of assistive visually impaired persons according to claim 1, characterized in that, The image description model that constructs the encoder-decoder structure includes: The encoder based on the visual Transformer consists of an image segmentation module and four feature transformation modules based on the Swing Transformer. The Swing Transformer is derived by replacing the multi-head self-attention layer in the standard Transformer with an attention layer based on a shift window. Feature transformation module 1 consists of a linear embedding layer and two Swing Transformer blocks. Feature transformation modules 2, 3, and 4 each consist of a block merging module and several Swing Transformer blocks. Feature transformation modules 2 and 4 contain 2 Swing Transformer blocks, and feature transformation module 3 contains 6 Swing Transformer blocks. The decoder consists of standard Transformer modules. Each standard Transformer consists of a multi-head self-attention layer and a feedforward layer. Each layer is normalized before it is completed, and residual connections are used between every two sub-layers.

3. The method for extracting semantic information from signs in the field of assistive visually impaired persons according to claim 1, characterized in that, The pre-training process of the image description model is as follows: A Transformer-based multimodal model is pre-trained using a language model loss function. For an input image I, the model generates a sentence {y}. 1, y2,…, N The loss function is as follows: Where CE is the cross-entropy loss function with label smoothing of 0.

1.

4. The method for extracting semantic information from signs in the field of assistive visually impaired persons according to claim 1, characterized in that, The process involves obtaining indoor scene images using assistive devices for the visually impaired, detecting the presence of signs in the images using the YOLOv5 method, and extracting and correcting signs if their proportion in the image exceeds a given threshold. Finally, generating semantic information descriptions of the signs using an image description model specifically includes: Indoor scene images are obtained using assistive devices for the visually impaired. The YOLOv5 method is used to detect whether there are signs in the images. If a sign is detected and its proportion in the image exceeds a given threshold of 10%, the sign is segmented from the original image based on its target bounding box. The Canny operator is used to perform edge detection on the segmented signs. The coordinates of the four vertices of the signs are calculated based on the edge detection results. Then, the affine transformation method is used to correct the tilted signs. The corrected sign is input into the image description model to obtain the predicted output of the semantic description and the confidence probability of each word in the sentence. If the confidence probability of each word is greater than the given threshold of 0.6, the generated semantic information description is output.

Images