Image description method for multi-feature fast reasoning in blind assistance scene

Abstract

The application relates to an image description method for multi-feature fast reasoning in a scene for the blind, and relates to the technical field of image description.The application comprises the following steps: firstly, grid features of a scene image are extracted using a deep convolution layer, and geometric relationships between each grid are calculated; secondly, shallow patch features are obtained by encoding the flattened scene image using a Transformer; thirdly, an adaptive shared Transformer block is constructed to fuse the grid features and the shallow patch features and to adaptively calculate the contribution degree of each feature; finally, a natural language description of the image is generated by decoding the final visual representation and the task adaptive vector in combination.The application solves the problems of lack of detailed expression and slow reasoning speed of existing image description methods in a scene for the blind, improves the performance of the model while lightening the model and improving the reasoning speed, and is beneficial to the deployment and application of the model in a scene for the blind.

Description

Technical Field

[0001] This invention relates to the field of image description technology, specifically to an image description method for rapid inference based on multiple features in assistive visually impaired scenarios. Background Technology

[0002] Currently, research in the field of assistive technology for the visually impaired is still incomplete and immature. Existing methods mainly focus on using technologies such as computer vision, lidar, and multi-sensor fusion to achieve obstacle avoidance and navigation, but they cannot enable visually impaired individuals to fully understand environmental information. Image description technology, on the other hand, can automatically generate a natural language description based on scene images, improving visually impaired individuals' understanding of their environment. Therefore, this technology has broad practical application value and prospects in the field of assistive technology for the visually impaired.

[0003] The key to image captioning technology lies in mining, extracting, and analyzing semantic information within images. Currently, commonly used image captioning techniques employ an encoder-decoder structure, where the encoder uses a pre-trained visual feature extractor to encode the image, extracting its semantic information, which is then fed into the decoder along with pre-generated partial text information to generate a text description. With continuous improvements in visual feature extractors, their performance is gradually increasing. However, relying solely on the development of powerful visual feature extractors is insufficient to enhance the contribution of visual content. Therefore, some techniques have proposed methods to enhance visual information by predicting text modal information containing high-level semantics (such as the subject, inter-subject relationships, and subject attributes, as shown in the image). However, this involves two technical challenges: first, since text and visual information are heterogeneous, high-quality text cannot be directly predicted using only image features; second, fusing text features with high-level semantics with image features is not easy, requiring a trade-off between fusion quality and complexity.

[0004] Existing image captioning techniques have achieved excellent performance under large-scale data training, but they lack consideration for practical applications. Most image captioning models employ the fusion of region features and other visual features to enhance visual representation and obtain more refined text descriptions. Therefore, a key challenge in the field of image captioning technology is that the application of multiple features leads to a significant increase in the number of model parameters and floating-point operations, reducing the model's inference speed and hindering practical deployment. In assistive visually impaired scenarios, image captioning technology often needs to consider real-time inference while ensuring excellent performance.

[0005] Based on this, the present invention provides an image description method for fast inference with multiple features in assistive visual impairment scenarios, which improves image description performance while reducing model size and floating-point operation count. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides an image description method for rapid inference based on multiple features in assistive visual impairment scenarios.

[0007] To address the aforementioned technical problems, this invention employs the following technical solution: a multi-feature fast reasoning image description method for assistive visually impaired scenarios, which generates corresponding text descriptions through an image description network model. The training process of the image description model includes the following steps:

[0008] Step 1: Use the deep convolutional network ResNeXt152 to extract image grid features and calculate grid geometric features; sum the grid features and geometric features and flatten them from 2D to obtain grid features X with relative positional information. G ;

[0009] Step 2: Shallow Patch Feature Extraction of the Image: Divide the original input image into windows of fixed size. Each window is flattened into a one-dimensional vector. All vectors constitute the patch feature X of the image. p The shallow patch feature X is obtained by encoding it using a linear layer and a Transformer block. S ;

[0010] Step 3: Feature fusion and encoding to obtain the final visual representation Z: Use adaptive shared Transformer blocks to fuse grid features and shallow patch features with relative position information, and use multiple regular Transformers to encode the fused features to obtain the final visual feature Z;

[0011] Step 4: Predict the output: Concatenate a task-adaptive vector S with the visual feature Z to obtain the output. Initialize semantic vector T, and combine visual features The input to the decoder learns the shallow relationship between visual and semantic information through a self-attention mechanism, and adds it to the semantic vector T to obtain high-level semantic features. visual features and high-level semantic features As input to the Transformer decoding block, text descriptions L = [l1, l2, ..., l] are generated step by step. n ], where l i It represents the i-th word in the sentence.

[0012] Step 5: Loss Function Calculation: Define the image description loss function as follows:

[0013]

[0014] Where CE represents the cross-entropy loss, l i It is the i-th word in the sentence, n represents the length of the sentence, and p(li |l 1：i-1 ) represents the probability of generating the i-th word.

[0015] Specifically, the geometric feature calculation steps for the mesh features in step 1 include: for the i-th mesh, calculating its center coordinates (x, y, y) based on its top-left and bottom-right corner coordinates. i y i The width of the grid is w. i and height h i Then, the positional relationship vector between each grid is calculated based on the center coordinates. The calculation process is as follows:

[0016]

[0017] G ij =FC(r) ij )

[0018] Where r ij Indicates the positional relationship between grid i and grid j, (x i y i ) and (x j y j The coordinates of the i-th and j-th grids are given by , and FC is a fully connected layer. Then, the relationship vectors between all grids are concatenated and represented as G, which is input into a convolutional layer to obtain the geometric features X of the grids. r =Conv(G);

[0019] Specifically, in step 2, X p The execution process of inputting into Transformer is as follows:

[0020] X S =LN(MHA(X) p X p X p )+X p );

[0021] LN is the layer normalization operation in the Transformer block, and MHA is the multi-head attention mechanism operation in the Transformer block.

[0022] Specifically, step 3, which involves fusing multiple features through adaptive sharing of Transformer blocks, includes: grid features X with relative position information. G and shallow patch feature X S After applying the same shared multi-head attention mechanism and normalization, two features with the same distribution are obtained. After undergoing independent layer normalization, two features X with independent properties are obtained. G ′, XS The inputs are then simultaneously fed into a shared feedforward layer and normalized by independent layers to obtain the filtered features. Grid feature X with relative position information G and shallow patch feature X S After addition, the contributions of the two features are calculated using a gating mechanism consisting of a linear layer and a sigmoid function; the filtered features... The fused feature Z is obtained by summing the Hadamard products of the contribution values ​​and the summation of the products. The entire calculation process is as follows:

[0023]

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030] g = Sigmoid(L(X) G +X S )),

[0031]

[0032] Where CLN represents the shared layer normalization operation, CMHA represents the shared multi-head attention mechanism operation, CFF represents the shared feedforward layer operation, LN represents the layer normalization operation, MHA represents the multi-head attention mechanism operation, L represents the linear layer operation, and ⊙ represents the matrix dot product.

[0033] Specifically, in step 4, high-level semantic features are calculated. hour:

[0034]

[0035] Where LN is the layer normalization operation in Transformer, and MHA is the multi-head attention mechanism computation in Transformer.

[0036] Specifically, in step 4, when generating the text description L step by step, the calculation process of the decoding block is as follows:

[0037]

[0038]

[0039]

[0040]

[0041] Where LN is the layer normalization operation in Transformer, MHA is the multi-head attention mechanism computation in Transformer, and MMHA is the multi-head attention mechanism operation with masking. This represents the semantic features after conversion. The feature is the result of combining visual and semantic information, where l represents the word prediction.

[0042] Compared with existing technologies, the beneficial effects of this invention are: this invention avoids the use of complex visual features similar to regional features, which is conducive to end-to-end model training and dataset calibration; this invention calculates additional geometric information on the basis of grid features to make up for the loss of positional information after the grid features are flattened, and at the same time, it integrates shallow patch features to enhance the fine-grained information in visual features; this invention constructs an adaptive shared Transformer block, which effectively avoids the additional model parameters and floating-point calculations brought about by multiple features, and improves the model inference speed. Attached Figure Description

[0043] The accompanying drawings, which form part of this invention, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation of this invention.

[0044] Figure 1 This is a structural diagram of the image description model in this invention;

[0045] Figure 2 This is a comparison of the number of parameters and floating-point calculations between the adaptive shared feature fusion method in this invention and other fusion methods. Detailed Implementation

[0046] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings. Note that the aspects described below with reference to the drawings are merely exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0047] Image description tasks are typically defined as follows: given an image, identify objects in the image, analyze the relationships between objects, understand the content of the image, and finally generate a corresponding natural and fluent text description.

[0048] The image description model proposed in this invention is as follows: Figure 1As shown, it includes the following three parts: (1) image feature extraction module; (2) feature fusion module; and (3) text prediction module.

[0049] (1) Image feature extraction module: First, the size of any input image is uniformly converted to 224×224. The deep convolutional network ResNeXt152 is used to extract the image grid features. Then, the obtained two-dimensional image grid visual features are unfolded into one-dimensional flattened grid features. in Let N be the i-th one-dimensional grid feature in the image obtained by the visual feature extractor, and let N be the total number of grid features.

[0050] Then, the relative position information of the grid features is calculated and converted into geometric features. For each grid, its top-left corner coordinates are defined as follows: The coordinates of the bottom right corner are Let the top-left corner of the first grid feature be (0,0) and the bottom-right corner be (1,1). Calculate the center coordinates, width, and height of the i-th grid:

[0051]

[0052]

[0053]

[0054] The positional relationship vector between each grid is calculated based on the center coordinates. The calculation process is as follows:

[0055]

[0056] G ij =FC(r) ij ).

[0057] Where r ij Indicates the positional relationship between grid i and grid j, (x i y i ) and (x j y j The coordinates of the i-th and j-th grids are given by , and FC is a fully connected layer. Then, the relationship vectors between all grids are concatenated and represented as G, which is input into a convolutional layer to obtain the geometric features X of the grids. r =Conv(G);

[0058] Flattened mesh feature X g Geometric features X of mesh features r The summation yields a grid feature X with relative positional information. G ;

[0059] Shallow patch feature extraction of images: The original input image is divided into windows of a fixed size of 32×32. Each window is flattened into a one-dimensional vector, and all vectors constitute the patch feature X of the image. p The shallow patch feature X is obtained by encoding it using a linear layer and a Transformer block. S ;

[0060] X p The execution process of inputting into Transformer is as follows:

[0061] X S =LN(MHA(X) p X p X p )+X p );

[0062] Where LN is the layer normalization operation in the Transformer block, and MHA is the multi-head attention mechanism operation in the Transformer block.

[0063] (2) Feature fusion module. Mesh features X with relative position information. G and shallow patch feature X S After applying the same shared multi-head attention mechanism and normalization, two features with the same distribution are obtained. After undergoing independent layer normalization, two features X with independent properties are obtained. G ′, X S The inputs are then simultaneously fed into a shared feedforward layer and normalized by independent layers to obtain the filtered features. Grid feature X with relative position information G and shallow patch feature X S After addition, the contributions of the two features are calculated using a gating mechanism consisting of a linear layer and a sigmoid function; the filtered features... The fused feature Z is obtained by summing the Hadamard products of the contribution values ​​and the summation of the products. The entire calculation process is as follows:

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070]

[0071] g = Sigmoid(L(X) G +X S )),

[0072]

[0073] Where CLN represents the shared layer normalization operation, CMHA represents the shared multi-head attention mechanism operation, CFF represents the shared feedforward layer operation, LN is the layer normalization operation, MHA is the multi-head attention mechanism operation in Transformer, L is the linear layer operation, and ⊙ represents the matrix dot product.

[0074] (3) Text prediction module. Based on the obtained fused visual features Z, the text modal information containing high-level semantics is first predicted, and then it is jointly predicted with the contour-enhanced image features to predict the text description.

[0075] This invention first initializes a set of task-adaptive vectors S to learn the contribution of visual features and semantic information to text generation, and then concatenates these vectors with the fused visual features.

[0076] Initialize semantic vector T, and combine visual features The input to the decoder learns the shallow relationship between visual and semantic information through a self-attention mechanism, and adds it to the semantic vector T to obtain high-level semantic features. High-level semantic features The calculation is as follows:

[0077]

[0078] Where LN is the layer normalization operation in Transformer, and MHA is the multi-head attention mechanism computation in Transformer.

[0079] visual features and high-level semantic features As input to the Transformer decoding block, text descriptions L = [l1, l2, ..., l] are generated step by step. n ], where l i This represents the i-th word in the sentence. The calculation process for the decoding block is as follows:

[0080]

[0081]

[0082]

[0083]

[0084] LN is the layer normalization operation in Transformer, MHA is the multi-head attention mechanism computation in Transformer, and MMHA is the multi-head attention mechanism operation with masking. This represents the semantic features after conversion. The feature is the result of combining visual and semantic information, where l represents the word prediction.

[0085] Next, we define the loss function for the image description task:

[0086]

[0087] Where CE represents the cross-entropy loss, l i It is the i-th word in the sentence, n represents the length of the sentence, and p(l i |l 1：i-1 ) represents the probability of generating the i-th word.

[0088] The method proposed in this invention enables end-to-end training and achieves higher performance by fusing multiple features to improve visual representation, while also offering faster inference speed. Compared to existing methods, this invention avoids using complex visual features similar to region features, improving the model's inference speed and the ease of dataset creation, making it more suitable for practical applications. The combination of grid features and geometric features compensates for the loss of positional information after feature flattening, while the fusion of shallow patch features provides more fine-grained information, thus enhancing visual representation.

[0089] The adaptive shared Transformer block constructed in this invention avoids the additional model parameters and floating-point operations caused by multi-feature fusion through its shared structure. Furthermore, its gating mechanism adaptively calculates the contribution of each feature to obtain a better visual representation. Thus, the model achieves stronger visual features through feature fusion while reducing the number of model parameters and floating-point operations, which is beneficial for its deployment in practical applications.

[0090] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0091] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A multi-feature fast reasoning image description method for assistive visually impaired scenarios, which generates corresponding text descriptions through an image description network model; the training process of the image description network model is characterized by, Includes the following steps: Step 1: Use the deep convolutional network ResNeXt152 to extract image grid features and calculate the geometric features of the grid; Summing the mesh features and geometric features and flattening them from two dimensions to one dimension yields the mesh feature X with relative position information. G ; Step 2: Shallow Patch Feature Extraction of the Image: Divide the original input image into windows of fixed size. Each window is flattened into a one-dimensional vector. All vectors constitute the patch feature X of the image. p The shallow patch feature X is obtained by encoding it using a linear layer and a Transformer block. S ; Step 3: Feature fusion and encoding to obtain the final visual representation Z: Use adaptive shared Transformer blocks to fuse grid features and shallow patch features with relative position information, and use multiple regular Transformers to encode the fused features to obtain the final visual feature Z; Step 4: Predict the output: Concatenate a task-adaptive vector S with the visual feature Z to obtain the output. Initialize semantic vector T, and combine visual features The input to the decoder learns the shallow relationship between visual and semantic information through a self-attention mechanism, and adds it to the semantic vector T to obtain high-level semantic features. visual features and high-level semantic features As input to the Transformer decoding block, text description L = [l1, l2, ..., l] is generated step by step. n ], where l i It is the i-th word in the sentence, and n represents the length of the sentence; Step 5: Loss Function Calculation: Define the image description loss function as follows: Where CE represents the cross-entropy loss, l i It is the i-th word in the sentence, n represents the length of the sentence, and p(l i |l 1:i-1 () represents the probability of generating the i-th word; Step 3 includes: Grid feature X with relative position information G and shallow patch feature X S After applying the same shared multi-head attention mechanism and normalization, two features with the same distribution are obtained. After undergoing independent layer normalization, two features X with independent properties are obtained. G ′, X S The inputs are then simultaneously fed into a shared feedforward layer and normalized by independent layers to obtain the filtered features. Grid feature X with relative position information G and shallow patch feature X S After addition, the contributions of the two features are calculated using a gating mechanism consisting of a linear layer and a sigmoid function; the filtered features... The fused feature Z is obtained by summing the Hadamard products of the contribution values ​​and the summation of the products. The entire calculation process is as follows: g = Sigmoid(L(X) G +X S )), Where CLN represents the shared layer normalization operation, CMHA represents the shared multi-head attention mechanism operation, CFF represents the shared feedforward layer operation, LN represents the layer normalization operation, MHA represents the multi-head attention mechanism operation, L represents the linear layer operation, and ⊙ represents the matrix dot product.

2. The image description method for multi-feature fast reasoning in assistive visual impairment scenarios according to claim 1, characterized in that, The steps for calculating the geometric features of the mesh in step 1 include: For the i-th grid, calculate its center coordinates (x, y, y) based on its top-left and bottom-right corner coordinates. i ,y i The width of the grid is w. i and height h i Then, the positional relationship vector between each grid is calculated based on the center coordinates. The calculation process is as follows: G ij =FC(r ij ) Where r ij Indicates the positional relationship between grid i and grid j, (x i ,y i ) and (x j ,y j The coordinates of the i-th and j-th grids are given by , respectively; FC is a fully connected layer; then the relationship vectors between all grids are concatenated together as G, which is input into a convolutional layer to obtain the geometric features X of the grid. r =Conv(G).

3. The image description method for multi-feature fast reasoning in assistive visual impairment scenarios according to claim 1, characterized in that, In step 2, X p The execution process of inputting into Transformer is as follows: X S =LN(MHA(X p ,X p ,X p )+X p ); LN is the layer normalization operation in the Transformer block, and MHA is the multi-head attention mechanism operation in the Transformer block.

4. The image description method for multi-feature fast reasoning in assistive visual impairment scenarios according to claim 1, characterized in that, In step 4, high-level semantic features The calculation is as follows: Where LN is the layer normalization operation in Transformer, and MHA is the multi-head attention mechanism computation in Transformer.

5. The image description method for multi-feature fast reasoning in assistive visual impairment scenarios according to claim 1, characterized in that, In step 4, when generating the text description L step by step, the calculation process of the decoding block is as follows: Where LN is the layer normalization operation in Transformer, MHA is the multi-head attention mechanism computation in Transformer, and MMHA is the multi-head attention mechanism operation with masking. This represents the semantic features after conversion. The feature is the result of combining visual and semantic information, where l represents the word prediction.

Images