A method, device, equipment and medium for image recognition of a large visual language model

Abstract

This application discloses an image recognition method, device, equipment and medium of a large visual language model. First, a news image data set for image recognition is constructed, and then a visual encoding classifier model is trained using the news image data set. The predicted image is input into the visual encoding classifier model, and the enhanced prompt text is identified in combination with the prompt text. Finally, the enhanced prompt text is input into the text decoder to obtain the classification result. All relevant information is extracted and integrated through the decoding process to ensure that the output text not only accurately reflects the content of the image, but also contains a deeper level of contextual understanding. The ability of the large visual language model in image recognition tasks is improved, especially when processing content in specific fields such as news images, showing higher recognition accuracy and understanding depth.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to an image recognition method, device, equipment and medium for a large visual language model. Background Art

[0002] Image classification is a very basic but important task in the field of computer vision, which aims to enable computers to understand the information of input images and classify images. The Visual Language Model is an advanced artificial intelligence technology that combines the capabilities of computer vision and natural language processing to understand and generate language descriptions related to visual content. This model is trained through deep learning algorithms and can recognize objects, scenes, and activities in images, and provide detailed information in the form of natural language or perform complex language tasks.

[0003] Large visual language models can achieve functions such as image description generation, visual question answering, image content analysis, etc. However, although large visual language models perform well in understanding and generating complex language descriptions related to visual content, they may face challenges when performing some simple image classification tasks. These models are usually designed to handle more advanced language understanding and generation tasks rather than directly targeting image classification. Therefore, they may not be as efficient or accurate as models specifically optimized for classification in some basic classification scenarios. Summary of the invention

[0004] The purpose of this application is to provide an image recognition method, device, equipment and medium for a large visual language model in order to overcome the existing technical defects, extract and integrate all relevant information through the decoding process, and ensure that the output text not only accurately reflects the content of the image, but also contains a deeper level of contextual understanding.

[0005] The purpose of this application is achieved through the following technical solutions:

[0006] In a first aspect, the present application proposes an image recognition method based on a large visual language model, the method comprising:

[0007] Build a news image dataset for image recognition;

[0008] Using the news image dataset to train a visual encoding classifier model;

[0009] The predicted image is input into the visual encoding classifier model and combined with the prompt text to identify the enhanced prompt text;

[0010] Input the enhanced prompt text into the text decoder to obtain the classification result.

[0011] In a possible implementation, the step of constructing a news image dataset for image recognition includes:

[0012] Collecting a news video material set and de-framing the video in the news video material set to obtain an image material set;

[0013] Performing similarity deduplication on the image material set to obtain a similarity deduplicated image material set;

[0014] The image material set after similarity removal is divided according to the number of categories to obtain the corresponding news image dataset.

[0015] In a possible implementation, the step of performing similarity deduplication on the image material set to obtain a similarity deduplicated image material set includes:

[0016] Select any two images in the image material set and extract hue, saturation and brightness respectively;

[0017] Calculate the product of one dimension of the hue channel of one image and the corresponding dimension of the saturation or brightness channel of another image to get the total number of pixels of the hue channel;

[0018] The hue channel difference of the two images is calculated based on the total number of pixels in the hue channel, and the saturation channel difference and brightness channel difference of the two images are also calculated;

[0019] The channel average difference is calculated based on the hue channel difference, saturation channel difference and brightness channel difference;

[0020] When the average channel difference is greater than a preset threshold, it indicates that the two images are not similar, and similarity deduplication is performed to obtain a set of image materials after similarity deduplication.

[0021] In a possible implementation, the visual encoding classifier model includes a block embedding module, a position embedding module, a self-attention module, a multi-layer perception module, and a classifier module. The step of training the visual encoding classifier model using the news image dataset includes:

[0022] The images in the news image dataset are input into the block embedding module for convolution to obtain the first feature;

[0023] Input the first feature into the position embedding module for mapping to obtain the second feature;

[0024] Input the second feature into the self-attention module to obtain the third feature;

[0025] Input the third feature into the first normalization layer to obtain the fourth feature;

[0026] Input the fourth feature into the multi-layer perception module to obtain the fifth feature;

[0027] Input the fifth feature into the second normalization layer to obtain the sixth feature;

[0028] After replacing the second feature with the sixth feature, the feature is sequentially input into the self-attention module, the first normalization layer, the multi-layer perception module, and the second normalization layer, and the seventh feature is obtained when the number of cycles reaches a preset threshold;

[0029] Inputting the seventh feature into the classifier module to obtain the eighth feature;

[0030] Input the eighth feature and category information into the cross entropy loss function to calculate the loss score;

[0031] Use the gradient descent method to update the loss score to the parameters of the visual encoding classifier model to complete the training of the visual encoding classifier model.

[0032] In a possible implementation, the step of inputting the predicted image into a visual encoding classifier model and combining the predicted image with the prompt text to identify and enhance the prompt text includes:

[0033] Input the predicted image into the visual encoding classifier model to obtain the classification result;

[0034] Select the classification results according to the confidence level to form a result list;

[0035] The constructed prompt text and the result list are concatenated and then input into the text encoder to obtain the text encoding features;

[0036] The text encoding feature and the seventh feature are fused to obtain the enhanced prompt text.

[0037] In one possible implementation, the multi-layer perception module includes an activation function and two linear layers. The activation function The calculation formula is: , For input, is the cumulative distribution function of the standard normal distribution.

[0038] In a possible implementation, the calculation formula of the first normalization layer is: ,in For the third feature, is the scaling parameter, is the translation parameter, is the mean of the input features, is the standard deviation of the input features, is a constant.

[0039] In a second aspect, the present application proposes an image recognition device based on a large visual language model, the device comprising:

[0040] A data set construction module, used to construct a news image data set, wherein the news image data set is used for image recognition;

[0041] A training module, used to train a visual encoding classifier model using the news image data set to obtain classification information;

[0042] A text recognition module is used to combine the classification information with the text prompt to identify and enhance the prompt text;

[0043] The decoding module is used to combine the enhanced prompt text with the image features and decode it into image classification text.

[0044] In a third aspect, the present application further proposes a computer device, comprising a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the image recognition method as described in any one of the first aspects.

[0045] In a fourth aspect, the present application further proposes a computer-readable storage medium, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the image recognition method as described in any one of the first aspects.

[0046] The above main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application; and in the present application, (non-conflicting options) options and other options can also be freely combined. After understanding the scheme of the present application, those skilled in the art can understand that there are many combinations based on the existing technology and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.

[0047] This application discloses an image recognition method, device, equipment and medium of a large visual language model. First, a news image data set for image recognition is constructed, and then a visual encoding classifier model is trained using the news image data set. The predicted image is input into the visual encoding classifier model, and the enhanced prompt text is identified in combination with the prompt text. Finally, the enhanced prompt text is input into the text decoder to obtain the classification result. All relevant information is extracted and integrated through the decoding process to ensure that the output text not only accurately reflects the content of the image, but also contains a deeper level of contextual understanding. The ability of the large visual language model in image recognition tasks is improved, especially when processing content in specific fields such as news images, showing higher recognition accuracy and understanding depth. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A flow chart of an image recognition method based on a large visual language model proposed in an embodiment of the present application is shown.

[0050] Figure 2 The structure diagram of the visual encoding classifier model proposed in the embodiment of the present application is shown. DETAILED DESCRIPTION

[0051] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0052] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of this application.

[0053] In the existing technology, large visual language models can achieve functions such as image description generation, visual question answering, and image content analysis. However, although large visual language models perform well in understanding and generating complex language descriptions related to visual content, they may face challenges when performing some simple image classification tasks. These models are usually designed to handle more advanced language understanding and generation tasks rather than directly targeting image classification. Therefore, in some basic classification scenarios, they may not be as efficient or accurate as models specifically optimized for classification.

[0054] Therefore, in order to solve the above technical problems, the embodiment of this application proposes an image recognition method, device, equipment and medium of a large visual language model, which extracts and integrates all relevant information through the decoding process to ensure that the output text not only accurately reflects the content of the image, but also contains a deeper level of contextual understanding. The ability of the large visual language model in image recognition tasks has been improved, especially when processing content in specific fields such as news images, showing higher recognition accuracy and understanding depth, which will be described in detail below.

[0055] Please refer to Figure 1 , Figure 1 A schematic diagram of a process of an image recognition method based on a large visual language model proposed in an embodiment of the present application is shown, and the method includes:

[0056] Step S100: construct a news image dataset for image recognition.

[0057] Build a news image dataset, collect and organize a large number of news-related images as training samples. News images often contain rich contextual information and diversity, which are suitable for training models to understand and identify elements in various news scenes. Through professional annotation and classification, ensure the quality and pertinence of the dataset.

[0058] Step S100 includes:

[0059] Step S110: collect a news video material set and deframe the video in the news video material set to obtain an image material set.

[0060] Collecting news video materials involves collecting a series of news-related video materials (NDi), with index i ranging from 1 to N, where N represents the total number of videos and is a positive integer. This step builds a high-quality image dataset, and then splits each video NDi to extract single-frame images, thereby forming a large image material collection INDimga, where the index range of imga is from 1 to M_All, where M_All represents the total number of images after splitting, which is also a positive integer, greatly increasing the number of image samples available for training.

[0061] Step S120: performing similarity deduplication on the image material set to obtain a similarity deduplicated image material set.

[0062] Image similarity deduplication In order to ensure the quality of the data set and avoid redundancy, the images in the image material set INDimga are analyzed for similarity and duplicate content is removed. Through the similarity calculation method, the deduplicated image set SINDimgqc (imgqc=1,2,...,M_QC) is obtained, where M_QC is the total number of deduplicated images, which is a positive integer and less than the original total number M_All, which can ensure the diversity and uniqueness of the images in the data set. Step S120 efficiently realizes image deduplication through detailed color feature analysis and improves image recognition of the large visual language model.

[0063] Step S120 includes:

[0064] Step S121: select any two images in the image material set, and extract hue, saturation and brightness respectively.

[0065] Select any two images Ia and Ib from the image material set, and extract the three basic attributes of their color space: hue ( , ), Saturation( , ),brightness( , ), which is represented as a two-dimensional array whose dimensions are determined by the actual width of the image ( , ) and height( , )Decide, , , , They are all positive integers, ensuring that every pixel in the image can be traversed.

[0066] Step S122: Calculate the product of one dimension of the hue channel of one image and the corresponding dimension of the saturation or brightness channel of another image to obtain the total number of pixels of the hue channel.

[0067] Determine the total number of channel pixels by calculating one dimension of the hue channel ( ) to the corresponding dimension of the saturation or brightness channel of another image ( ) to get the total number of pixels in the hue channel , The value of is a positive integer, and are the dimensions of the image's a and b channels. Based on the assumption, the total number of pixels in the saturation and brightness channels is also the same ( , in is the total number of pixels in the saturation channel, is the total number of pixels in the brightness channel), which simplifies the uniformity of subsequent difference calculations.

[0068] Step S123: Calculate the hue channel difference between the two images according to the total number of pixels in the hue channel, and simultaneously calculate the saturation channel difference and the brightness channel difference between the two images.

[0069] Calculate the difference in hue, saturation, and brightness channels for each pair of images. The difference is obtained by taking the absolute value, summing it, and then dividing it by the total number of pixels to get the hue difference , To take the absolute value operation, Σ represents the sum operation of the values ​​of each element in the two-dimensional array. is a positive integer. Similarly, the saturation difference can be obtained And brightness difference , the above operation quantifies the inconsistency between the two images in terms of each color attribute.

[0070] Step S124: Calculate the channel average difference according to the hue channel difference, the saturation channel difference and the brightness channel difference.

[0071] Taking into account the differences in the three color channels, the average value of the hue channel difference, saturation channel difference, and brightness channel difference is calculated. ,As an indicator of the overall similarity of images, it helps to balance the contribution of different color attributes to the determination of image similarity.

[0072] Step S125: When the average channel difference is greater than a preset threshold, it indicates that the two images are not similar, and similarity deduplication is performed to obtain a set of image materials after similarity deduplication.

[0073] Set a threshold Y, if the average difference calculated If it is greater than the threshold Y, images Ia and Ib are considered to be dissimilar, and deduplication is performed accordingly. This step filters out duplicate or highly similar images based on preset criteria to ensure the diversity and quality of the data set.

[0074] Step S130: Divide the image material set after similarity removal according to the number of categories to obtain a corresponding news image data set.

[0075] Classification of image subsets according to a predefined number of categories , the deduplicated image set SINDimgqc is divided into Divide into Image subset , each subset corresponds to a specific type of news image content. Number of categories is a positive integer, , indexes sub1 to subclsn and num1 to numclsn respectively mark the number of images within each category, both of which are positive integers. They not only organize the data but also prepare for subsequent classification training.

[0076] Step S200: training a visual encoding classifier model using a news image dataset.

[0077] Using the constructed news image dataset, a visual encoding classifier model is trained. The core of this model is to learn how to convert images into high-dimensional feature vectors (visual encodings) and learn to map these encodings to specific categories. This process enables the model to understand the image content and classify it.

[0078] The visual encoding classifier model includes a block embedding module, a position embedding module, a self-attention module, a multi-layer perception module and a classifier module. Step S200 includes:

[0079] The images in the news image dataset are input into the block embedding module for convolution to obtain the first feature;

[0080] Input the first feature into the position embedding module for mapping to obtain the second feature;

[0081] Input the second feature into the self-attention module to obtain the third feature;

[0082] Input the third feature into the first normalization layer to obtain the fourth feature;

[0083] Input the fourth feature into the multi-layer perception module to obtain the fifth feature;

[0084] Input the fifth feature into the second normalization layer to obtain the sixth feature;

[0085] After replacing the second feature with the sixth feature, the feature is sequentially input into the self-attention module, the first normalization layer, the multi-layer perception module, and the second normalization layer, and the seventh feature is obtained when the number of cycles reaches a preset threshold;

[0086] Inputting the seventh feature into the classifier module to obtain the eighth feature;

[0087] Input the eighth feature and category information into the cross entropy loss function to calculate the loss score;

[0088] Use the gradient descent method to update the loss score to the parameters of the visual encoding classifier model to complete the training of the visual encoding classifier model.

[0089] Please refer to Figure 2 , Figure 2 The structure diagram of the visual coding classifier model proposed in the embodiment of the present application is shown. The visual coding classifier model is first constructed, and then the constructed visual coding classifier model is classified. The visual coding classifier model is constructed in the following manner: a block embedding module PE1, a position embedding module PE2, a self-attention module SA, a multi-layer perception module MP and a classifier module FC_CLS are constructed in sequence, wherein the structure of the block embedding module PE1 is a 2D convolution structure, the number of input channels is 3, the number of output channels is 1152, the convolution kernel size is 14*14, the step size is 14, and the padding dimension is 0. The role of the position embedding module PE2 is to map discrete indexes to a continuous high-dimensional vector space, with an input index of up to 4900 and an output dimension of 1152. The self-attention module SA is divided into four linear layers, where the input dimension of the linear layer k_p is 1152 and the output dimension is 1152; the input dimension of the linear layer v_p is 1152 and the output dimension is 1152; the input dimension of the linear layer q_p is 1152 and the output dimension is 1152; the linear layer o_p has an input dimension of 1152 and an output dimension of 1152. The multi-layer perception module includes an activation function and two linear layers. The activation function The calculation formula is: , For input, is the cumulative distribution function of the standard normal distribution, the dimension of x is 4304, the input dimension of the linear layer FC1 is 1152, and the output dimension is 4304. The input dimension of the linear layer FC2 is 4304, and the output dimension is 4096. The input dimension of the classifier module FC_CLS is 4096, and the output dimension is the number of categories clsn.

[0090] After the visual encoding classifier model is constructed, the visual encoding classifier model is trained using the news image dataset. The training process is as follows: for the image Img with the category information subi in the news image dataset, where (i=1,2,...,clsn), it is input to the block embedding module PE1 to obtain the first feature F1, and the first feature F1 is input to the position embedding module PE2 to obtain the second feature F2, and the second feature F2 is input to the self-attention module SA to obtain the third feature F3. The third feature F3 is passed through the first normalization layer LN1 to obtain the fourth feature F4. The calculation formula of the first normalization layer LN1 is: ,in For the third feature, is the scaling parameter, is the translation parameter, is the mean of the input features, is the standard deviation of the input features, is a constant used to prevent the denominator from being zero.

[0091] The fourth feature F4 is input into the layer perception module MP to obtain the fifth feature F5, and the fifth feature F5 is passed through the second normalization layer LN2 to obtain the sixth feature F6, wherein the calculation formula of the second normalization layer LN2 is the same as that of the first normalization layer LN1, and the embodiment of the present application is not repeated here. The sixth feature F6 is used to replace the original second feature and the intermediate step of generating the sixth feature F6 is re-executed until the preset threshold (28 times) is reached to obtain the seventh feature F7. The seventh feature F7 is input into the classifier module FC_CLS to obtain the eighth feature F8. The eighth feature F8 and the category information subi are input into the cross entropy loss function to obtain the score , is the cross entropy loss function. During the back propagation of model parameters, the model parameters are optimized by gradient descent, so that the score SCORE gradually decreases and the model training is completed.

[0092] Step S300: input the predicted image into the visual encoding classifier model, and combine it with the prompt text to identify and enhance the prompt text.

[0093] In order to construct visual recognition enhanced prompt text and further improve the accuracy and meticulousness of recognition, this application introduces the construction of enhanced prompt text, combining the classification information learned by the model in step S200 with carefully designed text prompts to form a guided context to help the model use additional language clues during the recognition process to understand the context and details in the image.

[0094] In a possible implementation, the step of inputting the predicted image into a visual encoding classifier model and combining the predicted image with the prompt text to identify and enhance the prompt text includes:

[0095] Input the predicted image into the visual encoding classifier model to obtain the classification result;

[0096] Select the classification results according to the confidence level to form a result list;

[0097] The constructed prompt text and the result list are concatenated and then input into the text encoder to obtain the text encoding features;

[0098] The text encoding feature and the seventh feature are fused to obtain the enhanced prompt text.

[0099] For any given prediction image , first use the previously defined model processing flow to extract its deep features to obtain This feature vector captures the key visual information of the image and lays the foundation for subsequent classification and text fusion. The image is input into the visual encoding classifier model, and the model outputs a probability distribution about the possible categories to which the image may belong. The top five categories with the highest probability are selected as the prediction results and form a list. , is the category name (i=1,2,3,4,5). The model makes the most confident guesses about the image content, which improves the practicality and flexibility of the classification results.

[0100] In order to further improve the performance of the model on a specific task, a prompt text PROMPT is constructed. This text contains the context of the task, specific requirements, and expected output format, and is intended to guide the model on how to better understand and generate output. For example, in the news image classification scenario, it may emphasize paying attention to specific elements or contextual clues in the image.

[0101] Combine the constructed prompt text PROMPT with the predicted category list Concatenate and then pass through the text encoder Convert it into a continuous vector text encoding feature The role of the text encoder is to map natural language text into a dense vector in a high-dimensional space so that the model can understand and process text information.

[0102] Finally, the image features Text encoding features Perform feature fusion to generate (The encoded visual recognition enhances the prompt text information). This fusion operation ( ) involve addition, concatenation, or other advanced feature combining techniques.

[0103] Step S400: input the enhanced prompt text into a text decoder to obtain a classification result.

[0104] The visual recognition enhancement hint generated in step S300 is combined with the image features and converted into the final classification or description text through a decoding process. This decoding process is designed to extract and integrate all relevant information to ensure that the output text not only accurately reflects the content of the image, but also contains a deeper level of contextual understanding.

[0105] The visual recognition enhanced prompt text information VET is passed through the text decoder to obtain the classification result , A text decoder.

[0106] A possible implementation of an image recognition device based on a large visual language model is provided below, which is used to execute the various execution steps and corresponding technical effects of the image recognition method shown in the above embodiments and possible implementations. The device includes:

[0107] A data set construction module, used to construct a news image data set, wherein the news image data set is used for image recognition;

[0108] A training module, used to train a visual encoding classifier model using the news image data set to obtain classification information;

[0109] A text recognition module is used to combine the classification information with the text prompt to identify and enhance the prompt text;

[0110] The decoding module is used to combine the enhanced prompt text with the image features and decode it into image classification text.

[0111] This preferred embodiment provides a computer device, which can implement the steps in any embodiment of the image recognition method provided in the embodiments of the present application. Therefore, the beneficial effects of the image recognition method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0112] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present application provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps of any embodiment of the image recognition method provided in the embodiment of the present application.

[0113] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0114] Since the instructions stored in the storage medium can execute the steps in any image recognition method embodiment provided in the embodiments of the present application, the beneficial effects that can be achieved by any image recognition method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0115] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An image recognition method based on a large visual language model, characterized in that: The method comprises: Build a news image dataset for image recognition; Using the news image dataset to train a visual encoding classifier model; The visual coding classifier model includes a block embedding module, a position embedding module, a self-attention module, a multi-layer perception module and a classifier module. The structure of the block embedding module is a 2D convolution structure. The self-attention module is divided into four linear layers. The multi-layer perception module includes an activation function and two linear layers. The steps of training the visual coding classifier model using the news image dataset include: The images in the news image dataset are input into the block embedding module for convolution to obtain the first feature; Input the first feature into the position embedding module for mapping to obtain the second feature; Input the second feature into the self-attention module to obtain the third feature; Input the third feature into the first normalization layer to obtain the fourth feature; Input the fourth feature into the multi-layer perception module to obtain the fifth feature; Input the fifth feature into the second normalization layer to obtain the sixth feature; After replacing the second feature with the sixth feature, the feature is sequentially input into the self-attention module, the first normalization layer, the multi-layer perception module, and the second normalization layer, and the seventh feature is obtained when the number of cycles reaches a preset threshold; Inputting the seventh feature into the classifier module to obtain the eighth feature; Input the eighth feature and category information into the cross entropy loss function to calculate the loss score; Use the gradient descent method to update the loss score to the parameters of the visual encoding classifier model to complete the training of the visual encoding classifier model; Input the predicted image into the visual encoding classifier model to obtain the classification result; Select the classification results according to the confidence level to form a result list; The constructed prompt text and the result list are concatenated and then input into the text encoder to obtain the text encoding features; The text encoding feature and the seventh feature are fused to obtain the enhanced prompt text; Input the enhanced prompt text into the text decoder to obtain the classification result.

2. The image recognition method according to claim 1, characterized in that: The steps to build a news image dataset for image recognition include: Collecting a news video material set and de-framing the video in the news video material set to obtain an image material set; Performing similarity deduplication on the image material set to obtain a similarity deduplicated image material set; The image material set after similarity removal is divided according to the number of categories to obtain the corresponding news image dataset.

3. The image recognition method according to claim 2, characterized in that: The step of performing similarity deduplication on the image material set to obtain a similarity deduplicated image material set includes: Select any two images in the image material set and extract hue, saturation and brightness respectively; Calculate the product of one dimension of the hue channel of one image and the corresponding dimension of the saturation or brightness channel of another image to get the total number of pixels of the hue channel; The hue channel difference of the two images is calculated based on the total number of pixels in the hue channel, and the saturation channel difference and brightness channel difference of the two images are also calculated; The channel average difference is calculated based on the hue channel difference, saturation channel difference and brightness channel difference; When the average channel difference is greater than a preset threshold, it indicates that the two images are not similar, and similarity deduplication is performed to obtain a set of image materials after similarity deduplication.

4. The image recognition method according to claim 1, characterized in that: Activation Function The calculation formula is: , For input, is the cumulative distribution function of the standard normal distribution.

5. The image recognition method according to claim 1, characterized in that: The calculation formula of the first normalization layer is: ,in For the third feature, is the scaling parameter, is the translation parameter, is the mean of the input features, is the standard deviation of the input features, is a constant.

6. An image recognition device based on a large visual language model, characterized in that: The device comprises: A data set construction module, used to construct a news image data set, wherein the news image data set is used for image recognition; A training module, used to train a visual coding classifier model using the news image data set to obtain classification information, wherein the visual coding classifier model includes a block embedding module, a position embedding module, a self-attention module, a multi-layer perception module, and a classifier module, wherein the structure of the block embedding module is a 2D convolution structure, the self-attention module is divided into four linear layers, and the multi-layer perception module includes an activation function and two linear layers; The images in the news image dataset are input into the block embedding module for convolution to obtain the first feature; Input the first feature into the position embedding module for mapping to obtain the second feature; Input the second feature into the self-attention module to obtain the third feature; Input the third feature into the first normalization layer to obtain the fourth feature; Input the fourth feature into the multi-layer perception module to obtain the fifth feature; Input the fifth feature into the second normalization layer to obtain the sixth feature; After replacing the second feature with the sixth feature, the feature is sequentially input into the self-attention module, the first normalization layer, the multi-layer perception module, and the second normalization layer, and the seventh feature is obtained when the number of cycles reaches a preset threshold; Inputting the seventh feature into the classifier module to obtain the eighth feature; Input the eighth feature and category information into the cross entropy loss function to calculate the loss score; Use the gradient descent method to update the loss score to the parameters of the visual encoding classifier model to complete the training of the visual encoding classifier model; The text recognition module is used to input the predicted image into the visual encoding classifier model to obtain the classification result; Select the classification results according to the confidence level to form a result list; The constructed prompt text and the result list are concatenated and then input into the text encoder to obtain the text encoding features; The text encoding feature and the seventh feature are fused to obtain the enhanced prompt text; The decoding module is used to combine the enhanced prompt text with the image features and decode it into image classification text.

7. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the image recognition method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is loaded and executed by a processor to implement the image recognition method according to any one of claims 1 to 5.

Images