Image description method based on deep learning
Through deep learning methods, combined with CNN and RNN, the attention mechanism is used to generate image descriptions, which solves the ambiguity and inconsistency of image descriptions in the prior art, achieves more accurate and rich image descriptions, and improves the degree of automation and efficiency .
Patent Information
- Application Number
- CN202411199476.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-27
AI Technical Summary
The prior art lacks in-depth understanding of image semantics in image description tasks, resulting in ambiguity and inaccuracy of description, difficulty in capturing image details and key information, the generated text description is inconsistent with the image content, and it is impossible to fully reflect image features and meanings. .
Using deep learning-based image description method, image semantic features are extracted through convolutional neural networks (CNNs), and combined with recurrent neural networks (RNNs) and attention mechanisms, text descriptions related to image content are generated, using large-scale images and text The dataset is model-trained and the model is optimized for improved accuracy and consistency.
Achieve more accurate and richer image descriptions, which can better capture image details and key information, and the generated text description is more consistent with the image content, which improves the degree of automation and efficiency, and significantly improves the accuracy, richness and consistency.
Smart Images
Figure CN119206434A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image description method based on deep learning. Background Art
[0002] Deep learning, computer vision and natural language processing are important branches in the field of artificial intelligence, which aim to enable computers to understand and process images and languages. Traditional computer vision methods have limitations in image description tasks, while deep learning-based image description technology can automatically convert images into text descriptions by combining deep learning models and natural language generation models, thereby achieving accurate descriptions of image content.
[0003] Existing technical solutions:
[0004] 1. Convolutional Neural Networks (CNN): A deep learning model for image feature extraction that can identify different features in an image and capture its semantic information.
[0005] 2. Recurrent Neural Networks (RNN): A deep learning model for natural language generation that can generate continuous text sequences based on contextual information.
[0006] 3. Attention Mechanism: Used in deep learning models to help the model focus on relevant parts of the image when generating text descriptions to improve the accuracy of the description.
[0007] 4. Image-to-Text Generation Models: Using deep learning methods, taking images as input, and training models to generate text descriptions related to the image content.
[0008] Current image description technologies have the following problems when generating text descriptions:
[0009] (1) Lack of in-depth understanding of image semantics leads to ambiguity and inaccuracy in description;
[0010] (2) It is difficult to capture the details and key information in the image, resulting in a lack of richness and accuracy in the generated description;
[0011] (3) The generated text description is poorly consistent with the image content and is insufficient to fully reflect the characteristics and meaning of the image.
[0012] Therefore, the prior art has defects and needs to be improved. Summary of the invention
[0013] The technical problem to be solved by the present invention is to provide an image description method based on deep learning to solve the technical problem that the prior art has limitations in image description tasks and cannot accurately convert image content into natural language descriptions.
[0014] The technical solution of the present invention is as follows: a deep learning-based image description method is provided, comprising the following steps.
[0015] S1: Collect large-scale image datasets. The image datasets should meet the diversity requirements. The image datasets should include various image types such as natural scenes, objects, and people. Ensure the diversity and coverage of the datasets to improve the generalization ability of the model.
[0016] S2: Preprocess the collected image data, including image denoising, resizing, and color normalization; these preprocessing steps are designed to improve the robustness and accuracy of the model for the input images.
[0017] S3: Use convolutional neural networks (CNN) and recurrent neural networks (RNN) to train an image description generation model; the model takes the input image as a source for feature extraction and uses RNN to generate text descriptions related to the image content.
[0018] S4: During the training process, CNN is used to extract semantic feature representations of images; these semantic features will be passed as input to RNN for generating text descriptions.
[0019] S5: During the training process, RNN is used as the sequence generation model; RNN will receive the semantic feature representation extracted by CNN and gradually generate a text sequence corresponding to the image content.
[0020] S6: Optimize and adjust the image description generation model through back-propagation algorithm and optimization algorithm; achieve the description accuracy and consistency of the image description generation model, and reduce the deviation and error of the generated text.
[0021] S7: Evaluate the generated text descriptions using evaluation metrics. The performance of the model in terms of accuracy and semantic consistency is evaluated by comparing with manually annotated reference descriptions.
[0022] S8: Deploy the trained model to the actual application environment to realize the automatic description function of the image.
[0023] The present invention first uses a convolutional neural network (CNN) to extract image features, and extracts different levels of feature representation in the image through multi-layer convolution and pooling operations. Next, a recurrent neural network (RNN) is used in combination with an attention mechanism to model the extracted image features and generate a text sequence. The attention mechanism can dynamically adjust the attention to different areas in the image, so that the generated text description is more accurate and consistent. In addition, the present invention also combines large-scale image and text data sets to train a deep learning model to improve the performance and generalization ability of the model. The present invention can achieve an accurate description of the image content and generate natural language text related to the image content. This technology can be applied to image search, automatic image annotation, image tagging and other fields, and provides a powerful tool for the combination of computer vision and natural language processing.
[0024] In step S4, semantic features are extracted from the input image through convolution and pooling operations of CNN.
[0025] In step S5, it also includes: using the RNN to generate a coherent text description using context information in a recursive manner.
[0026] In step S6, the optimization algorithm is a stochastic gradient descent algorithm.
[0027] In step S7, the evaluation index is: BLEU evaluation method or METEOR evaluation method.
[0028] In step S8, the system is integrated into an existing computer vision system through an API interface so that the system can read the image and generate a corresponding text description.
[0029] The image description generation model includes: an input module, a feature extraction module, a stacked channel attention module, and an image description generation module.
[0030] The trained image description generation model describes the image in the following steps.
[0031] 101: The input module inputs the image to be tested into the pre-trained image description generation model.
[0032] 102: A feature extraction module extracts a first feature of the image to be tested.
[0033] 103: The stacked channel attention module inputs the first feature into the local channel attention layer twice for processing, adds the two outputs and processes them through the sigmoid activation function, and performs dot product with the first feature to obtain the second feature, wherein the specific processing process in the local channel attention layer is: reducing the number of input feature channels and normalizing them, restoring the number of channels after processing through the relu activation function, and normalizing them again to obtain the output of the local channel attention layer.
[0034] 104: The stacked channel attention module inputs the second feature into the local channel attention layer and the global channel attention layer for processing respectively, adds the two outputs and processes them through the sigmoid activation function, and performs a dot product with the first feature to obtain a third feature, wherein the global attention layer is composed of a local attention layer plus a global average pooling layer.
[0035] 105: The image description generation module adds the third feature and the position information code and inputs the result into the Transformer encoder-decoder to predict and generate the description of the image to be tested
[0036] By adopting the above scheme, the present invention provides an image description method based on deep learning. Compared with the prior art, the scheme of the present application has the following technical advantages:
[0037] 1. More accurate image description: The present invention uses a deep learning-based method to more deeply understand the semantic information of the image, thereby generating more accurate and descriptive text. Compared with traditional methods, the image description of the present invention can more accurately capture the details and key information in the image.
[0038] 2. Improve the richness of description: The technical solution of the present invention combines convolutional neural networks and recurrent neural networks to generate richer and more vivid image descriptions by modeling image features and generating text sequences. Compared with traditional methods, the text descriptions generated by the present invention are more detailed and specific, and can better reflect the characteristics and meaning of the image.
[0039] 3. Improve the consistency between image and text: The present invention introduces an attention mechanism that can dynamically adjust the attention paid to different areas in the image, so that the generated text description is more consistent with the image content. Through the application of the attention mechanism, the present invention can better capture the important areas in the image and accurately convert them into corresponding text descriptions.
[0040] 4. Improved automation and efficiency: The present invention uses a deep learning model to automatically generate image descriptions, reducing the need for manual intervention. Compared with manually writing image descriptions, the present invention can improve work efficiency and has higher consistency and stability.
[0041] 5. Supported by experimental results: The technical solution of the present invention has been scientifically analyzed and tested by training and testing on large-scale image and text data sets. The experimental results show that compared with the prior art, the image description technology of the present invention has achieved significant improvements in accuracy, richness and consistency. Specific data and experimental results can be provided in the patent application to further support the description of the technical advantages. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 Generate model functional framework diagram for image description;
[0043] Figure 2 Flowchart for generating image descriptions for the image description generation model. DETAILED DESCRIPTION
[0044] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0045] The present invention provides an image description method based on deep learning, comprising the following steps.
[0046] S1: Collect large-scale image datasets. The image datasets should meet the diversity requirements. The image datasets should include various image types such as natural scenes, objects, and people. Ensure the diversity and coverage of the datasets to improve the generalization ability of the model.
[0047] S2: Preprocess the collected image data, including image denoising, resizing, and color normalization; these preprocessing steps are designed to improve the robustness and accuracy of the model for the input images.
[0048] S3: Use convolutional neural networks (CNN) and recurrent neural networks (RNN) to train an image description generation model; the model takes the input image as a source for feature extraction and uses RNN to generate text descriptions related to the image content.
[0049] S4: During the training process, CNN is used to extract semantic feature representations of images; these semantic features will be passed as input to RNN for generating text descriptions.
[0050] S5: During the training process, RNN is used as the sequence generation model; RNN will receive the semantic feature representation extracted by CNN and gradually generate a text sequence corresponding to the image content.
[0051] S6: Optimize and adjust the image description generation model through back-propagation algorithm and optimization algorithm; achieve the description accuracy and consistency of the image description generation model, and reduce the deviation and error of the generated text.
[0052] S7: Evaluate the generated text descriptions using evaluation metrics. The performance of the model in terms of accuracy and semantic consistency is evaluated by comparing with manually annotated reference descriptions.
[0053] S8: Deploy the trained model to the actual application environment to realize the automatic description function of the image.
[0054] The present invention first uses a convolutional neural network (CNN) to extract image features, and extracts different levels of feature representation in the image through multi-layer convolution and pooling operations. Next, a recurrent neural network (RNN) is used in combination with an attention mechanism to model the extracted image features and generate a text sequence. The attention mechanism can dynamically adjust the attention to different areas in the image, so that the generated text description is more accurate and consistent. In addition, the present invention also combines large-scale image and text data sets to train a deep learning model to improve the performance and generalization ability of the model. The present invention can achieve an accurate description of the image content and generate natural language text related to the image content. This technology can be applied to image search, automatic image annotation, image tagging and other fields, and provides a powerful tool for the combination of computer vision and natural language processing.
[0055] In step S4, semantic features are extracted from the input image through convolution and pooling operations of CNN.
[0056] In step S5, it also includes: using the RNN to generate a coherent text description using context information in a recursive manner.
[0057] In step S6, the optimization algorithm is a stochastic gradient descent algorithm.
[0058] In step S7, the evaluation index is: BLEU evaluation method or METEOR evaluation method.
[0059] In step S8, the system is integrated into an existing computer vision system through an API interface so that the system can read the image and generate a corresponding text description.
[0060] See also Figure 1 The image description generation model includes: an input module 100, a feature extraction module 200, a stacked channel attention module 300, and an image description generation module 400.
[0061] See also Figure 2 ,The trained image description generation model describes the image including the following steps.
[0062] 101: The input module inputs the image to be tested into the pre-trained image description generation model.
[0063] 102: A feature extraction module extracts a first feature of the image to be tested.
[0064] 103: The stacked channel attention module inputs the first feature into the local channel attention layer twice for processing, adds the two outputs and processes them through the sigmoid activation function, and performs dot product with the first feature to obtain the second feature, wherein the specific processing process in the local channel attention layer is: reducing the number of input feature channels and normalizing them, restoring the number of channels after processing through the relu activation function, and normalizing them again to obtain the output of the local channel attention layer.
[0065] 104: The stacked channel attention module inputs the second feature into the local channel attention layer and the global channel attention layer for processing respectively, adds the two outputs and processes them through the sigmoid activation function, and performs a dot product with the first feature to obtain a third feature, wherein the global attention layer is composed of a local attention layer plus a global average pooling layer.
[0066] 105: The image description generation module adds the third feature and the position information code and inputs the result into the Transformer encoder-decoder to predict and generate the description of the image to be tested
[0067] In summary, the present invention provides an image description method based on deep learning. Compared with the prior art, the solution of the present application has the following technical advantages:
[0068] 1. More accurate image description: The present invention uses a deep learning-based method to more deeply understand the semantic information of the image, thereby generating more accurate and descriptive text. Compared with traditional methods, the image description of the present invention can more accurately capture the details and key information in the image.
[0069] 2. Improve the richness of description: The technical solution of the present invention combines convolutional neural networks and recurrent neural networks to generate richer and more vivid image descriptions by modeling image features and generating text sequences. Compared with traditional methods, the text descriptions generated by the present invention are more detailed and specific, and can better reflect the characteristics and meaning of the image.
[0070] 3. Improve the consistency between image and text: The present invention introduces an attention mechanism that can dynamically adjust the attention paid to different areas in the image, so that the generated text description is more consistent with the image content. Through the application of the attention mechanism, the present invention can better capture the important areas in the image and accurately convert them into corresponding text descriptions.
[0071] 4. Improved automation and efficiency: The present invention uses a deep learning model to automatically generate image descriptions, reducing the need for manual intervention. Compared with manually writing image descriptions, the present invention can improve work efficiency and has higher consistency and stability.
[0072] 5. Supported by experimental results: The technical solution of the present invention has been scientifically analyzed and tested by training and testing on large-scale image and text data sets. The experimental results show that compared with the prior art, the image description technology of the present invention has achieved significant improvements in accuracy, richness and consistency. Specific data and experimental results can be provided in the patent application to further support the description of the technical advantages.
[0073] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A deep learning-based image description method, characterized in that: The following steps are involved: S1: Collect large-scale image datasets. The image datasets should be diverse. S2: preprocessing the collected image data, including image denoising, size adjustment and color standardization; S3: Use convolutional neural networks and recurrent neural networks to train an image description generation model; the model takes the input image as a source for feature extraction and uses RNN to generate text descriptions related to the image content; the abbreviation of convolutional neural network is CNN and the abbreviation of recurrent neural network is RNN; S4: During the training process, CNN is used to extract semantic features of the image; these semantic features will be passed as input to the RNN to generate text descriptions; S5: During the training process, RNN is used as a sequence generation model; RNN will receive the semantic features extracted by CNN and gradually generate a text sequence corresponding to the image content; S6: Optimize and adjust the parameters of the image description generation model through back propagation algorithm and optimization algorithm; achieve the description accuracy and consistency of the image description generation model, and reduce the deviation and error of the generated text; S7: Use evaluation metrics to evaluate the generated text description; S8: Deploy the trained image description generation model to the actual application environment to realize the automatic description function of the image.
2. The image description method based on deep learning according to claim 1, characterized in that: Image datasets include natural scenes, objects, and people.
3. The image description method based on deep learning according to claim 1, characterized in that: In step S4, semantic features are extracted from the input image through convolution and pooling operations of CNN.
4. The image description method based on deep learning according to claim 1, characterized in that: In step S5, it also includes: using the context information in a recursive manner, the RNN generates a coherent text description.
5. The image description method based on deep learning according to claim 1, characterized in that: In step S6, the optimization algorithm is a stochastic gradient descent algorithm.
6. The image description method based on deep learning according to claim 1, characterized in that: In step S7, the evaluation index is: BLEU evaluation method or METEOR evaluation method.
7. The image description method based on deep learning according to claim 1, characterized in that: In step S7, it also includes: evaluating the performance of the model in terms of accuracy and semantic consistency by comparing with manually annotated reference descriptions.
8. The image description method based on deep learning according to claim 1, characterized in that: In step S8, the system is integrated into an existing computer vision system through an API interface so that the system can read the image and generate a corresponding text description.
9. The image description method based on deep learning according to claim 1, characterized in that: The image description generation model includes: an input module, a feature extraction module, a stacked channel attention module, and an image description generation module.
10. The image description method based on deep learning according to claim 8, characterized in that: The trained image description generation model describes the image in the following steps: 101: The input module inputs the image to be tested into the pre-trained image description generation model; 102: A feature extraction module extracts a first feature of the image to be tested; 103: The stacked channel attention module inputs the first feature into the local channel attention layer twice for processing, adds the two obtained outputs, processes them through the sigmoid activation function, and performs dot product with the first feature to obtain the second feature, wherein the specific processing process in the local channel attention layer is: reducing the number of input feature channels and performing normalization processing, restoring the number of channels after processing through the relu activation function, and performing normalization processing again to obtain the output of the local channel attention layer; 104: The stacked channel attention module inputs the second feature into the local channel attention layer and the global channel attention layer for processing respectively, adds the two outputs, processes them through a sigmoid activation function, and performs a dot product with the first feature to obtain a third feature, wherein the global attention layer is composed of a local attention layer plus a global average pooling layer; 105: The image description generation module adds the third feature and the position information code and inputs the result into the Transformer encoder-decoder to predict and generate the description of the image to be tested.