Image data rapid screening method and device based on large model

Abstract

The invention relates to an image data rapid screening method and device based on a large model. The method comprises the steps that image data and a preset cue word template are obtained, a BERT model is called to conduct pre-reasoning on the cue word template, word segmentation marking is conducted on the cue word template, and corresponding cue word feature vectors are generated. And cyclically reading the image data and the cue word feature vector through a cyclic reasoning code, taking the image data and the cue word feature vector as the input of the VisualGLM-6b multi-mode large model, and outputting visual features and semantic information. A matching sequence of image data and text data is calculated based on visual features and semantic information through an image text matching technology, the matching sequence is processed through post-processing codes, and matching scores and confidence scores of the image data and cues in the matching sequence are analyzed. And when the matching score exceeds a preset dynamic threshold value and the confidence score exceeds a first threshold value, screening and determining image data containing the target prompt word.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and big data processing technology, and in particular to a method and device for quickly screening image data based on a large model. Background Art

[0002] With the explosive growth of video and image data, it becomes particularly important to quickly filter out content of interest from massive data.

[0003] At present, most data screening methods mainly rely on traditional image processing technology or manual screening, which is difficult to handle complex and diverse scenes. With the development of deep learning and large model technology, the accuracy of image recognition and classification has been significantly improved. This is because large models have more parameters and rich training data, which have sufficient advantages in solving zero-shot problems. Therefore, the method of using large models for efficient image data screening has very sufficient practical significance. For deep learning models, the importance of data is self-evident. High-quality data has a huge benefit to the model, but the current cost of data acquisition is huge. For the current training data of millions of levels, it is too time-consuming, labor-intensive and investment-intensive to process from beginning to end.

[0004] However, the traditional method of data screening through large models is usually a single image model or language model. This means that when encountering a large amount of image data and text data at the same time during the data screening process, the application of the two models not only requires a lot of manpower and material resources to create a rich data set, but its cross-modal understanding and generalization capabilities also need to be further improved, which also reduces the accuracy of data screening to a certain extent. Summary of the invention

[0005] Based on this, it is necessary to provide a method and device for rapid screening of image data based on a large model, which can save manpower and material resources, improve the model's cross-modal understanding and generalization capabilities, and thus improve the accuracy of data screening.

[0006] The present invention provides a method for rapid screening of image data based on a large model, the method comprising:

[0007] Acquire image data and a preset prompt word template, and call the BERT model to perform pre-inference on the prompt word template to segment and mark the prompt word template and generate a corresponding prompt word feature vector;

[0008] The image data and the cue word feature vector are read cyclically through the cyclic reasoning code, and the image data and the cue word feature vector are used as inputs of the VisualGLM-6b multimodal large model to output corresponding visual features and semantic information;

[0009] Calculating a matching sequence of the image data and the text data based on the visual features and the semantic information by using an image-text matching technology, and processing the matching sequence by using a post-processing code to parse a matching score and a confidence score between the image data and the prompt word in the matching sequence;

[0010] When the matching score exceeds a preset dynamic threshold and the confidence score exceeds a first threshold, the image data containing the target prompt word is screened and determined.

[0011] In one embodiment, the acquiring of image data and a preset prompt word template, and calling a BERT model to perform pre-inference on the prompt word template to segment and mark the prompt word template and generate a corresponding prompt word feature vector include:

[0012] Performing word segmentation processing on the text data corresponding to the prompt word template to split the words or phrases in the text data into tokens in the BERT model vocabulary;

[0013] Convert the token corresponding to each word segment into a fixed-length word embedding vector to convert the text data into a numerical representation suitable for processing by the BERT model;

[0014] Among them, the tag includes a first tag and a second tag, the first tag is used to mark the beginning of the text data to classify the text data, the second tag is used to segment different paragraphs in the text data and the end of the text data, and the word embedding vector includes a vocabulary embedding vector, a paragraph embedding vector and a position embedding vector.

[0015] In one embodiment, the acquiring of image data and a preset prompt word template, and calling a BERT model to perform pre-inference on the prompt word template to segment and mark the prompt word template and generate a corresponding prompt word feature vector, further includes:

[0016] Calling the BERT model to perform multi-layer encoding on the text data after word segmentation processing, and generating the prompt word feature vector according to the word embedding vector;

[0017] The prompt word feature vector is saved as npy format data through the Numpy library;

[0018] The prompt word feature vector includes semantic information and context information of each word segment, and the word segment with the first mark is the overall representation of the text data, and the other word segments represent the context information between each word segment in the text data.

[0019] In one embodiment, the looping inference code loops through the image data and the cue word feature vector, and uses the image data and the cue word feature vector as inputs of the VisualGLM-6b multimodal large model to output corresponding visual features and semantic information, including:

[0020] Processing the plurality of image data in blocks, and caching the image data after the block processing, so as to cyclically read the image data after the block processing from the cache;

[0021] The npy format data is read from the Numpy library in a loop, and the npy format data is parsed into the corresponding prompt word feature vector, so as to be input into the VisualGLM-6b multimodal large model together with the read image data.

[0022] In one embodiment, the image-text matching technology is used to calculate the matching sequence of the image data and the text data based on the visual features and the semantic information, and the matching sequence is processed by a post-processing code to parse the matching score and confidence score between the image data and the prompt word in the matching sequence, which also includes:

[0023] Calling an image encoder and a text encoder to encode the image data and the text data respectively, so as to focus on a target object and a target area in the image data and the text data, and generate corresponding visual feature vectors and text feature vectors;

[0024] The visual feature vector and the text feature vector are used as a Q-Former network to perform feature fusion to generate a fused multimodal feature vector, where the multimodal feature vector combines the visual features and semantic information.

[0025] In one embodiment, the calculating a matching sequence of the image data and the text data based on the visual features and the semantic information by the image-text matching technology, and processing the matching sequence by a post-processing code to parse a matching score and a confidence score between the image data and the prompt word in the matching sequence, includes:

[0026] Calling Image-Text Matching image text matching technology to calculate the cosine similarity of the image data and the text data according to the multimodal feature vector, wherein the cosine similarity is used to characterize the size of the angle between the visual feature vector and the text feature vector;

[0027] A similarity matrix is ​​obtained based on the cosine similarity, and a matching sequence of the image data and the text data is generated according to the similarity matrix through the VisualGLM-6b multimodal large model, wherein the matching sequence is used to characterize the correspondence between the target area in the image data and the segmentation words in the text data.

[0028] In one embodiment, the method further comprises: calculating a matching sequence of the image data and the text data based on the visual features and the semantic information by using the image-text matching technology, and processing the matching sequence by a post-processing code to parse a matching score and a confidence score between the image data and the prompt word in the matching sequence.

[0029] Parsing the similarity score in the matching sequence through a post-processing code and obtaining the preset dynamic threshold, wherein the similarity score is obtained based on the cosine similarity;

[0030] The corresponding target areas, target objects and prompt words in the matching sequence are aggregated by weighted averaging, and the confidence score of each matching result in the matching sequence is calculated according to the matching score.

[0031] The present invention also provides a device for rapid screening of image data based on a large model, the device comprising:

[0032] A model pre-inference module is used to obtain image data and a preset prompt word template, and call the BERT model to perform pre-inference on the prompt word template to segment and mark the prompt word template and generate a corresponding prompt word feature vector;

[0033] A loop reasoning module, used for looping through the loop reasoning code to read the image data and the prompt word feature vector, and use the image data and the prompt word feature vector as inputs of the VisualGLM-6b multimodal large model to output corresponding visual features and semantic information;

[0034] A post-processing module, used to calculate a matching sequence of the image data and the text data based on the visual features and the semantic information by using an image-text matching technology, and to process the matching sequence by using a post-processing code to parse a matching score and a confidence score between the image data and the prompt word in the matching sequence;

[0035] The image data screening module is used to screen and determine the image data containing the target prompt word when the matching score exceeds a preset dynamic threshold and the confidence score exceeds a first threshold.

[0036] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method for rapid screening of image data based on a large model as described above is implemented.

[0037] The present invention also provides a computer storage medium storing a computer program, wherein when the computer program is executed by a processor, the method for rapid screening of image data based on a large model as described above is implemented.

[0038] The above-mentioned large-model-based image data rapid screening method and device obtain image data and a preset prompt word template, and call the BERT model to perform pre-reasoning on the prompt word template, so as to perform word segmentation marking on the prompt word template and generate a corresponding prompt word feature vector. Then, the image data and the prompt word feature vector are read in a loop through the loop reasoning code, and the image data and the prompt word feature vector are used as the input of the VisualGLM-6b multimodal large model to output the corresponding visual features and semantic information. Subsequently, the matching sequence of the image data and the text data is calculated based on the visual features and semantic information through the image-text matching technology, and the matching sequence is processed through the post-processing code to parse the matching score and confidence score between the image data and the prompt word in the matching sequence. Finally, when the matching score exceeds the preset dynamic threshold and the confidence score exceeds the first threshold, the image data containing the target prompt word is screened and determined. This method finds the required target data in massive image data or video data based on prompt keywords. For example, if you want to collect a data set about various animals, you only need to make inferences and predictions in a large number of pictures based on the prompt words "there is a zebra in the picture" or "there is a giraffe in the picture", and extract the pictures whose prediction results are greater than the set threshold. The extracted pictures contain a large number of related animals, which provides a rich basis for creating a data set, thereby saving a lot of manpower and material resources and creating a high-quality data set. Compared with a single image model or language model, the multimodal large model has stronger cross-modal understanding ability and better generalization ability, which improves the accuracy of data recognition to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 One of the flow charts of the large model-based image data rapid screening method provided by the present invention;

[0041] Figure 2 A schematic diagram of the overall process of image data screening based on a large model fast screening method in a specific embodiment of the present invention;

[0042] Figure 3 The second flow chart of the method for rapid screening of image data based on a large model provided by the present invention;

[0043] Figure 4 The third flow chart of the method for rapid screening of image data based on a large model provided by the present invention;

[0044] Figure 5 The fourth flowchart of the method for rapid screening of image data based on a large model provided by the present invention;

[0045] Figure 6 The fifth flowchart of the method for rapid screening of image data based on a large model provided by the present invention;

[0046] Figure 7 The sixth flow chart of the method for rapid screening of image data based on a large model provided by the present invention;

[0047] Figure 8 The seventh flowchart of the method for rapid screening of image data based on a large model provided by the present invention;

[0048] Fig. 9 A schematic diagram of the structure of a large-model-based image data rapid screening device provided by the present invention;

[0049] Fig.10 This is a diagram of the internal structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Combine the following Figures 1 to 10 The invention describes a method and device for rapid screening of image data based on a large model.

[0052] like Figure 1 As shown, in one embodiment, a method for rapid screening of image data based on a large model comprises the following steps:

[0053] Step S110, acquiring image data and a preset prompt word template, and calling the BERT model to perform pre-inference on the prompt word template to segment and mark the prompt word template and generate a corresponding prompt word feature vector.

[0054] Specifically, the server obtains a large amount of image data to be screened and a pre-designed prompt word template, and calls the BERT model to perform pre-inference on the prompt word template to segment and mark the prompt word template and generate a corresponding prompt word feature vector.

[0055] Combination Figure 2 As shown, in a specific embodiment, the large model-based image data fast screening method provided by the present invention first designs a prompt word template, such as "Is there ** in the picture? If so, please answer yes or no.", and sets the interested tags ["zebra", "elephant", "giraffe", "cheetah", "lion", ...]. Subsequently, in order to save reasoning resources, it is necessary to use the BERT model to pre-infer the prompt words in advance, and the pre-inference process is as follows:

[0056] (1) Input processing:

[0057] The input prompt word text is first tokenized, which means splitting the words or phrases in the text into tokens in the BERT model's vocabulary.

[0058] After word segmentation, the input text is tagged with special tags [CLS] and [SEP]. The [CLS] tag indicates the beginning of the text and is usually used for text-level classification tasks, while the [SEP] tag is used to segment different texts (or paragraphs) and mark the end of the text.

[0059] (2) Word embedding generation:

[0060] Each token after word segmentation will be converted into a fixed-length vector (embedding). The word embedding of the BERT model not only considers the vocabulary itself, but also combines positional encoding, so that the BERT model can understand the order information of words in the text.

[0061] The input token is converted into the sum of three embedding vectors: Token Embedding, Segment Embedding, and Position Embedding. These embedding vectors convert the input text into a numerical representation suitable for processing by the BERT model.

[0062] (3) Feature vector generation:

[0063] After each word segment passes through the multi-layer encoder of the BERT model, word vectors are finally generated. These vectors not only contain the semantic information of the word segment, but also combine the dependencies in the context, making them more generalizable to various language tasks (such as classification, question answering, etc.). In particular, the vector of the [CLS] tag will be used as the overall representation of the text, while the vectors of other word segments represent the contextual information of each word in the sentence.

[0064] (4) Feature vector preservation

[0065] The generated feature vector usually exists as 768-dimensional data for one word. The specific semantic information represented by the feature vector is implicit and difficult to be clearly analyzed. Therefore, the Numpy library is used to save the feature vector as npy format data, which can be directly read when used.

[0066] Step S120, cyclically reading the image data and the prompt word feature vector through the cyclic reasoning code, and using the image data and the prompt word feature vector as the input of the VisualGLM-6b multimodal large model to output the corresponding visual features and semantic information.

[0067] Specifically, the server reads a large amount of pre-cached image data and prompt word feature vectors through a loop inference code, and uses the read image data and prompt word feature vectors as input to the VisualGLM-6b multimodal large model to call the VisualGLM-6b multimodal large model to process the image data and prompt word feature vectors and output corresponding visual features and semantic information.

[0068] Combination Figure 2 As shown, in a specific embodiment, the large model-based fast screening method for image data provided by the present invention first selects the VisualGLM-6b multimodal large model, which can support multi-round dialogues with the advantages of high accuracy, fast reasoning speed, and small memory usage. It is also necessary to design a loop reasoning code, which can divide millions of pictures into blocks, multi-threaded processing, block-level data preloading, etc., in order to accelerate the image reading speed and preprocessing process.

[0069] In the present embodiment, due to the large number of prompt words and image data, a loop logic is designed here to accelerate reasoning. The previously divided image data is directly read from the cache in a loop, and the npy format data generated by the prompt word is read in a loop again, and parsed into the corresponding prompt word feature vector, and the image data and the prompt word feature vector are input into the VisualGLM-6b multimodal large model, wherein the image data and the prompt word feature vector are not the same. For example, to input the feature vector of "Is there a zebra in the picture? Please answer yes or no", the image cache in the block-level data needs to be input in a loop, and the next prompt word feature vector is input in a loop after the block-level data is looped, and the block-level image data cache is released after the prompt word loop is completed, and the next block-level image data is loaded until the loop ends.

[0070] Step S130, calculating a matching sequence of image data and text data based on visual features and semantic information through image-text matching technology, and processing the matching sequence through a post-processing code to parse the matching score and confidence score between the image data and the prompt word in the matching sequence.

[0071] Specifically, the server calculates the matching sequence of the image data and the text data based on the visual features and semantic information obtained in step S120 through the image-text matching technology, and processes the matching sequence through the post-processing code to further parse out the matching score and confidence score between the image data and the prompt word in the matching sequence.

[0072] Combination Figure 2 As shown, in a specific embodiment, the large-model-based image data rapid screening method provided by the present invention uses an Image Encoder to encode the image. The encoder focuses on the significant areas or important objects in the image, and does not encode each pixel or insignificant details in the image in detail. The visual feature vector generated by the image encoding and the text feature vector generated by the text encoder are feature fused through the Q-Former network. Among them, Q-Former is a self-attention mechanism for combining the representation of images and texts to learn the correlation between the two. The purpose of the feature fusion process is to allow the Q-Former network to understand the relationship between the visual information and the text information in the image, and generate a fused multimodal feature vector. The multimodal feature vector is a comprehensive understanding of the image and text information, combining the visual features from the image encoder and the semantic information of the text encoder.

[0073] In this embodiment, the image-text matching sequence is calculated using the Image-Text Matching technology for the fused features. The matching process is as follows:

[0074] (1) The cosine similarity of the image and text is calculated based on the multimodal feature vector to represent the angle between the two vectors, thereby measuring their similarity.

[0075] (2) The VisualGLM-6b multimodal model generates a matching sequence of images and texts through a similarity matrix. This sequence can represent the correspondence between the target area in the image and the words or phrases in the text. For example, given a picture and a description text, the model may output a sequence indicating that certain words in the text (such as "zebra") match certain parts of the image (such as the zebra area in the image).

[0076] (3) Finally, the model outputs a prediction sequence indicating the degree of match between the image and the text, i.e., the matching sequence.

[0077] Step S140: When the matching score exceeds a preset dynamic threshold and the confidence score exceeds a first threshold, the image data containing the target prompt word is screened and determined.

[0078] Specifically, when the matching score obtained in step S130 exceeds the preset dynamic threshold and the confidence score exceeds the set first threshold, the screening of the image data is completed, and the screened and determined image data contains the corresponding target prompt word.

[0079] Combination Figure 2 As shown, in a specific embodiment, the image data fast screening method based on a large model provided by the present invention obtains a matching sequence between the previously obtained image data and text data, and needs to use a post-processing code to process the answer result. During the processing, it is first necessary to parse the similarity score in the matching sequence between the image and the text, and then define a dynamic threshold, which is used to determine whether the target object exists in the image according to the matching score. Subsequently, more details are extracted from the matching results, such as specific regional matching and the frequency of occurrence of key prompt words, etc., and the extracted information can also be aggregated using weighted average to improve the accuracy of the judgment. Finally, the confidence score of the matching result is calculated, which is used to evaluate the reliability of the model judgment result based on the mean and standard deviation of the matching score. If the matching score is higher than the set dynamic threshold and the confidence is high enough (higher than the set threshold, such as the first threshold), it is determined that the image contains the target object, that is, the screening of the image data is completed.

[0080] In this embodiment, the image data finally screened out can be further effectively screened to eliminate invalid data and noise data, and the screened data can be further organized into high-quality data. If the quality of the data screened initially is poor, a small amount of valid data can be used to fine-tune the model, and the above steps can be repeated to further screen the image data on the basis of improving the model screening accuracy.

[0081] The above-mentioned image data fast screening method based on the large model, by obtaining image data and preset prompt word template, and calling BERT model to perform pre-reasoning on the prompt word template, to perform word segmentation marking on the prompt word template, and generate corresponding prompt word feature vector. Then read the image data and prompt word feature vector in a loop through the loop reasoning code, and use the image data and prompt word feature vector as the input of the VisualGLM-6b multimodal large model, to output corresponding visual features and semantic information. Subsequently, the matching sequence of image data and text data is calculated based on visual features and semantic information by image text matching technology, and the matching sequence is processed by post-processing code to parse the matching score and confidence score between image data and prompt words in the matching sequence. Finally, when the matching score exceeds the preset dynamic threshold and the confidence score exceeds the first threshold, screen and determine the image data containing the target prompt word. This method finds the required target data in massive image data or video data based on prompt keywords. For example, if you want to collect a data set about various animals, you only need to make inferences and predictions in a large number of pictures based on the prompt words "there is a zebra in the picture" or "there is a giraffe in the picture", and extract the pictures whose prediction results are greater than the set threshold. The extracted pictures contain a large number of related animals, which provides a rich basis for creating a data set, thereby saving a lot of manpower and material resources and creating a high-quality data set. Compared with a single image model or language model, the multimodal large model has stronger cross-modal understanding ability and better generalization ability, which improves the accuracy of data recognition to a certain extent.

[0082] like Figure 3 As shown, in one embodiment, the large model-based fast screening method for image data provided by the present invention obtains image data and a preset prompt word template, and calls the BERT model to perform pre-inference on the prompt word template to segment and mark the prompt word template, and generate a corresponding prompt word feature vector, which specifically includes the following steps:

[0083] Step S112, performing word segmentation processing on the text data corresponding to the prompt word template to split the words or phrases in the text data into tokens in the BERT model vocabulary.

[0084] Step S114, converting the token corresponding to each word segment into a fixed-length word embedding vector to convert the text data into a numerical representation suitable for processing by the BERT model.

[0085] Among them, the tag includes a first tag and a second tag, the first tag is used to mark the beginning of the text data to classify the text data, the second tag is used to segment different paragraphs in the text data and the end of the text data, and the word embedding vector includes a vocabulary embedding vector, a paragraph embedding vector and a position embedding vector.

[0086] like Figure 4 As shown, in one embodiment, the large model-based fast screening method for image data provided by the present invention obtains image data and a preset prompt word template, and calls the BERT model to perform pre-inference on the prompt word template to segment and mark the prompt word template, and generate a corresponding prompt word feature vector, which specifically also includes the following steps:

[0087] Step S116, calling the BERT model to perform multi-layer encoding on the text data after word segmentation processing, and generating a prompt word feature vector based on the word embedding vector.

[0088] Step S118, saving the prompt word feature vector as npy format data through the Numpy library.

[0089] The prompt word feature vector contains the semantic information and context information of each word segment, and the word segment with the first mark is the overall representation of the text data, and the other word segments represent the context information between each word segment in the text data.

[0090] like Figure 5 As shown, in one embodiment, the large model-based image data fast screening method provided by the present invention reads image data and prompt word feature vectors through a loop reasoning code loop, and uses the image data and prompt word feature vectors as inputs of the VisualGLM-6b multimodal large model to output corresponding visual features and semantic information, specifically including the following steps:

[0091] Step S122, performing block processing on the plurality of image data, and caching the block-processed image data, so as to cyclically read the block-processed image data from the cache.

[0092] Step S124, cyclically read npy format data from the Numpy library, and parse the npy format data into corresponding prompt word feature vectors, and input them into the VisualGLM-6b multimodal large model together with the read image data.

[0093] like Figure 6 As shown, in one embodiment, the large model-based image data fast screening method provided by the present invention calculates the matching sequence of image data and text data based on visual features and semantic information through image-text matching technology, and processes the matching sequence through post-processing code to parse the matching score and confidence score between the image data and the prompt word in the matching sequence, and also includes the following steps:

[0094] Step S610, calling an image encoder and a text encoder to encode the image data and the text data respectively, so as to focus on the target object and the target area in the image data and the text data, and generate corresponding visual feature vectors and text feature vectors.

[0095] Step S620, the visual feature vector and the text feature vector are used as a Q-Former network to perform feature fusion to generate a fused multimodal feature vector, which combines visual features and semantic information.

[0096] like Figure 7 As shown, in one embodiment, the large model-based image data fast screening method provided by the present invention calculates the matching sequence of image data and text data based on visual features and semantic information through image-text matching technology, and processes the matching sequence through post-processing code to parse the matching score and confidence score between the image data and the prompt word in the matching sequence, specifically including the following steps:

[0097] Step S132, calling the Image-Text Matching technology to calculate the cosine similarity between the image data and the text data according to the multimodal feature vector, where the cosine similarity is used to characterize the angle between the visual feature vector and the text feature vector.

[0098] Step S134, obtain a similarity matrix based on cosine similarity, and generate a matching sequence of image data and text data according to the similarity matrix through the VisualGLM-6b multimodal large model, and the matching sequence is used to characterize the correspondence between the target area in the image data and the segmentation in the text data.

[0099] like Figure 8 As shown, in one embodiment, the large model-based image data fast screening method provided by the present invention calculates the matching sequence of image data and text data based on visual features and semantic information through image-text matching technology, and processes the matching sequence through post-processing code to parse the matching score and confidence score between the image data and the prompt word in the matching sequence, and specifically includes the following steps:

[0100] Step S136, parse the similarity score in the matching sequence through the post-processing code, and obtain a preset dynamic threshold, where the similarity score is obtained based on cosine similarity.

[0101] Step S138: performing aggregation processing on the corresponding target regions, target objects and prompt words in the matching sequence by weighted averaging, and calculating the confidence score of each matching result in the matching sequence according to the matching score.

[0102] The following is a description of the image data rapid screening device based on a large model provided by the present invention. The image data rapid screening device based on a large model described below and the image data rapid screening method based on a large model described above can be referenced to each other.

[0103] like Fig. 9 As shown, in one embodiment, a device for rapid screening of image data based on a large model includes a model pre-reasoning module 910, a loop reasoning module 920, a post-processing module 930 and an image data screening module 940.

[0104] The model pre-inference module 910 is used to obtain image data and a preset prompt word template, and call the BERT model to perform pre-inference on the prompt word template to segment and mark the prompt word template and generate a corresponding prompt word feature vector.

[0105] The loop reasoning module 920 is used to loop through the loop reasoning code to read the image data and the prompt word feature vector, and use the image data and the prompt word feature vector as the input of the VisualGLM-6b multimodal large model to output the corresponding visual features and semantic information.

[0106] The post-processing module 930 is used to calculate the matching sequence of image data and text data based on visual features and semantic information through image-text matching technology, and process the matching sequence through post-processing code to parse the matching score and confidence score between the image data and the prompt word in the matching sequence.

[0107] The image data screening module 940 is used to screen and determine the image data containing the target prompt word when the matching score exceeds a preset dynamic threshold and the confidence score exceeds a first threshold.

[0108] In this embodiment, the large model-based image data rapid screening device provided by the present invention, the model pre-reasoning module 910 is specifically used for:

[0109] The text data corresponding to the prompt word template is segmented to split the words or phrases in the text data into tokens in the BERT model vocabulary.

[0110] The token corresponding to each word segment is converted into a fixed-length word embedding vector to convert the text data into a numerical representation suitable for processing by the BERT model.

[0111] Among them, the tag includes a first tag and a second tag, the first tag is used to mark the beginning of the text data to classify the text data, the second tag is used to segment different paragraphs in the text data and the end of the text data, and the word embedding vector includes a vocabulary embedding vector, a paragraph embedding vector and a position embedding vector.

[0112] In this embodiment, the large model-based image data rapid screening device provided by the present invention, the model pre-reasoning module 910 is specifically used for:

[0113] The BERT model is called to perform multi-layer encoding on the text data after word segmentation, and a prompt word feature vector is generated based on the word embedding vector.

[0114] The prompt word feature vector is saved as npy format data through the Numpy library.

[0115] The prompt word feature vector contains the semantic information and context information of each word segment, and the word segment with the first mark is the overall representation of the text data, and the other word segments represent the context information between each word segment in the text data.

[0116] In this embodiment, the large model-based image data rapid screening device provided by the present invention, the loop reasoning module 920 is specifically used for:

[0117] The plurality of image data are processed in blocks, and the image data after the block processing are cached, so that the image data after the block processing are cyclically read from the cache.

[0118] The npy format data is read from the Numpy library in a loop, and the npy format data is parsed into the corresponding prompt word feature vector, which is input into the VisualGLM-6b multimodal large model together with the read image data.

[0119] In this embodiment, the large model-based image data rapid screening device provided by the present invention further includes a feature fusion module for:

[0120] The image encoder and the text encoder are called to encode the image data and the text data respectively, so as to focus on the target object and the target area in the image data and the text data, and generate corresponding visual feature vectors and text feature vectors.

[0121] The visual feature vector and the text feature vector are used as a Q-Former network for feature fusion to generate a fused multimodal feature vector, which combines visual features and semantic information.

[0122] In this embodiment, the large model-based image data rapid screening device provided by the present invention, the post-processing module 930 is specifically used for:

[0123] The Image-Text Matching technology is called to calculate the cosine similarity of the image data and the text data based on the multimodal feature vector. The cosine similarity is used to characterize the angle between the visual feature vector and the text feature vector.

[0124] The similarity matrix is ​​obtained based on cosine similarity, and the matching sequence of image data and text data is generated according to the similarity matrix through the VisualGLM-6b multimodal large model. The matching sequence is used to characterize the correspondence between the target area in the image data and the segmentation in the text data.

[0125] In this embodiment, the large model-based image data rapid screening device provided by the present invention, the post-processing module 930 is specifically used for:

[0126] The similarity score in the matching sequence is parsed through the post-processing code, and the preset dynamic threshold is obtained. The similarity score is obtained based on the cosine similarity.

[0127] The corresponding target regions, target objects, and prompt words in the matching sequence are aggregated by weighted averaging, and the confidence score of each matching result in the matching sequence is calculated based on the matching score.

[0128] Fig.10 An example of a physical structure diagram of an electronic device is shown. The electronic device may be a smart terminal, and its internal structure diagram may be as follows: Fig.10 As shown. The electronic device includes a processor, an internal memory and a network interface connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for rapid screening of image data based on a large model is implemented, and the method includes:

[0129] Obtain image data and a preset prompt word template, and call the BERT model to perform pre-inference on the prompt word template to segment and mark the prompt word template and generate a corresponding prompt word feature vector;

[0130] The image data and the cue word feature vector are read in a loop through the loop reasoning code, and the image data and the cue word feature vector are used as the input of the VisualGLM-6b multimodal large model to output the corresponding visual features and semantic information;

[0131] The matching sequence of image data and text data is calculated based on visual features and semantic information through image-text matching technology, and the matching sequence is processed through post-processing code to parse the matching score and confidence score between the image data and the prompt word in the matching sequence;

[0132] When the matching score exceeds a preset dynamic threshold and the confidence score exceeds a first threshold, the image data containing the target prompt word is screened and determined.

[0133] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present invention, and does not constitute a limitation on the electronic device to which the scheme of the present invention is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0134] On the other hand, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements a method for rapid screening of image data based on a large model, the method comprising:

[0135] Obtain image data and a preset prompt word template, and call the BERT model to perform pre-inference on the prompt word template to segment and mark the prompt word template and generate a corresponding prompt word feature vector;

[0136] The image data and the cue word feature vector are read in a loop through the loop reasoning code, and the image data and the cue word feature vector are used as the input of the VisualGLM-6b multimodal large model to output the corresponding visual features and semantic information;

[0137] The matching sequence of image data and text data is calculated based on visual features and semantic information through image-text matching technology, and the matching sequence is processed through post-processing code to parse the matching score and confidence score between the image data and the prompt word in the matching sequence;

[0138] When the matching score exceeds a preset dynamic threshold and the confidence score exceeds a first threshold, the image data containing the target prompt word is screened and determined.

[0139] In another aspect, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, a method for rapid screening of image data based on a large model is implemented, the method comprising:

[0140] Obtain image data and a preset prompt word template, and call the BERT model to perform pre-inference on the prompt word template to segment and mark the prompt word template and generate a corresponding prompt word feature vector;

[0141] The image data and the cue word feature vector are read in a loop through the loop reasoning code, and the image data and the cue word feature vector are used as the input of the VisualGLM-6b multimodal large model to output the corresponding visual features and semantic information;

[0142] The matching sequence of image data and text data is calculated based on visual features and semantic information through image-text matching technology, and the matching sequence is processed through post-processing code to parse the matching score and confidence score between the image data and the prompt word in the matching sequence;

[0143] When the matching score exceeds a preset dynamic threshold and the confidence score exceeds a first threshold, the image data containing the target prompt word is screened and determined.

[0144] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0145] By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0146] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for rapid screening of image data based on a large model, characterized in that: The method comprises: Acquire image data and a preset prompt word template, and call the BERT model to perform pre-inference on the prompt word template to segment and mark the prompt word template and generate a corresponding prompt word feature vector; The image data and the cue word feature vector are read cyclically through the cyclic reasoning code, and the image data and the cue word feature vector are used as inputs of the VisualGLM-6b multimodal large model to output corresponding visual features and semantic information; Calculating a matching sequence of the image data and the text data based on the visual features and the semantic information by using an image-text matching technology, and processing the matching sequence by using a post-processing code to parse a matching score and a confidence score between the image data and the prompt word in the matching sequence; When the matching score exceeds a preset dynamic threshold and the confidence score exceeds a first threshold, the image data containing the target prompt word is screened and determined.

2. The method for rapid screening of image data based on a large model according to claim 1, characterized in that: The acquiring of image data and a preset prompt word template, and calling a BERT model to perform pre-inference on the prompt word template to segment and mark the prompt word template, and generate a corresponding prompt word feature vector, includes: Performing word segmentation processing on the text data corresponding to the prompt word template to split the words or phrases in the text data into tokens in the BERT model vocabulary; Convert the token corresponding to each word segment into a fixed-length word embedding vector to convert the text data into a numerical representation suitable for processing by the BERT model; Among them, the tag includes a first tag and a second tag, the first tag is used to mark the beginning of the text data to classify the text data, the second tag is used to segment different paragraphs in the text data and the end of the text data, and the word embedding vector includes a vocabulary embedding vector, a paragraph embedding vector and a position embedding vector.

3. The method for rapid screening of image data based on a large model according to claim 2, characterized in that: The step of acquiring the image data and the preset prompt word template, and calling the BERT model to perform pre-inference on the prompt word template to segment and mark the prompt word template and generate a corresponding prompt word feature vector, further includes: Calling the BERT model to perform multi-layer encoding on the text data after word segmentation processing, and generating the prompt word feature vector according to the word embedding vector; The prompt word feature vector is saved as npy format data through the Numpy library; The prompt word feature vector includes semantic information and context information of each word segment, and the word segment with the first mark is the overall representation of the text data, and the other word segments represent the context information between each word segment in the text data.

4. The method for rapid screening of image data based on a large model according to claim 3, characterized in that: The loop inference code loop reads the image data and the prompt word feature vector, and uses the image data and the prompt word feature vector as inputs of the VisualGLM-6b multimodal large model to output corresponding visual features and semantic information, including: Processing the plurality of image data in blocks, and caching the image data after the block processing, so as to cyclically read the image data after the block processing from the cache; The npy format data is read from the Numpy library in a loop, and the npy format data is parsed into the corresponding prompt word feature vector, so as to be input into the VisualGLM-6b multimodal large model together with the read image data.

5. The method for rapid screening of image data based on a large model according to claim 4, characterized in that: The method further includes: calculating a matching sequence of the image data and the text data based on the visual features and the semantic information by using the image-text matching technology, and processing the matching sequence by a post-processing code to parse a matching score and a confidence score between the image data and the prompt word in the matching sequence. Calling an image encoder and a text encoder to encode the image data and the text data respectively, so as to focus on a target object and a target area in the image data and the text data, and generate corresponding visual feature vectors and text feature vectors; The visual feature vector and the text feature vector are used as a Q-Former network to perform feature fusion to generate a fused multimodal feature vector, where the multimodal feature vector combines the visual features and semantic information.

6. The method for rapid screening of image data based on a large model according to claim 5, characterized in that: The method of calculating a matching sequence of the image data and the text data based on the visual features and the semantic information by using the image-text matching technology, and processing the matching sequence by using a post-processing code to parse a matching score and a confidence score between the image data and the prompt word in the matching sequence, includes: Calling Image-Text Matching image text matching technology to calculate the cosine similarity of the image data and the text data according to the multimodal feature vector, wherein the cosine similarity is used to characterize the size of the angle between the visual feature vector and the text feature vector; A similarity matrix is ​​obtained based on the cosine similarity, and a matching sequence of the image data and the text data is generated according to the similarity matrix through the VisualGLM-6b multimodal large model, wherein the matching sequence is used to characterize the correspondence between the target area in the image data and the segmentation words in the text data.

7. The method for rapid screening of image data based on a large model according to claim 6, characterized in that: The method further includes calculating a matching sequence of the image data and the text data based on the visual features and the semantic information by using the image-text matching technology, and processing the matching sequence by using a post-processing code to parse a matching score and a confidence score between the image data and the prompt word in the matching sequence. Parsing the similarity score in the matching sequence through a post-processing code and obtaining the preset dynamic threshold, wherein the similarity score is obtained based on the cosine similarity; The corresponding target areas, target objects and prompt words in the matching sequence are aggregated by weighted averaging, and the confidence score of each matching result in the matching sequence is calculated according to the matching score.

8. A device for rapid screening of image data based on a large model, characterized in that: The device comprises: A model pre-inference module is used to obtain image data and a preset prompt word template, and call the BERT model to perform pre-inference on the prompt word template to segment and mark the prompt word template and generate a corresponding prompt word feature vector; A loop reasoning module, used for looping through the loop reasoning code to read the image data and the prompt word feature vector, and use the image data and the prompt word feature vector as inputs of the VisualGLM-6b multimodal large model to output corresponding visual features and semantic information; A post-processing module, used to calculate a matching sequence of the image data and the text data based on the visual features and the semantic information by using an image-text matching technology, and to process the matching sequence by using a post-processing code to parse a matching score and a confidence score between the image data and the prompt word in the matching sequence; The image data screening module is used to screen and determine the image data containing the target prompt word when the matching score exceeds a preset dynamic threshold and the confidence score exceeds a first threshold.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the large model-based image data rapid screening method described in any one of claims 1 to 7 are implemented.

10. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the large model-based image data rapid screening method according to any one of claims 1 to 7 are implemented.

Images