Methods, devices, media and equipment for detecting and alerting defects in ID photos

Abstract

This invention discloses a method for detecting and alerting defects in ID photos based on image-text fusion, comprising: constructing and training an ID photo defect detection model, wherein the ID photo defect detection model includes a text encoding network and an image encoding network; inputting the ID photo defect and alert text information into the text encoding network to obtain corresponding text encoding information, and constructing a text feature database; acquiring an image to be processed, and performing feature encoding processing on the image to be processed using the image encoding network in the ID photo defect detection model to obtain image encoding information; retrieving the text encoding information corresponding to the image encoding information from the text feature database; and retrieving the corresponding ID photo defect and alert text information based on the text encoding information. This invention solves the problems of low efficiency and accuracy, high cost, and rigid detection results in existing technologies for detecting ID photo defects.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, medium, and equipment for detecting and alerting defects in ID photos based on image-text fusion. Background Technology

[0002] Thanks to technological advancements, modern life is becoming increasingly convenient. For example, self-service document processing machines can be used to take ID photos, eliminating the need for users to specifically seek out photo studios; moreover, the instant photo-taking function of these machines avoids the need for users to make a second trip to a photo studio to collect their ID photos.

[0003] Passport photos typically have certain compliance requirements for the photographer. To obtain a satisfactory passport photo, self-service document processing machines usually have specific requirements regarding the machine environment, the user's clothing, posture, appearance, and facial expressions. However, many users are unaware of these posture and clothing requirements before using self-service machines, making it possible for their photos to fail the inspection standards for ID cards, driver's licenses, and entry / exit permits. Therefore, it is necessary to perform defect detection and correction or provide prompts for passport photos obtained from self-service machines.

[0004] When taking ID photos using self-service document processing equipment, the equipment generally has certain requirements regarding the machine environment or the user's clothing, posture, appearance, and facial expressions. Since many users have unpredictable postures, clothing, and appearances, there is often a certain probability that they will not pass the inspection standards for ID cards, driver's licenses, entry and exit permits, and other types of photos.

[0005] In existing technologies, 2020113997343 provides a method and apparatus for detecting defects in document images. This involves acquiring document images from at least one angle under multiple light sources, detecting the defect status of the document images according to defect types, and employing a method that matches the defect type. However, when there are multiple detection items, multiple defect detection functions need to be chained together, resulting in high development costs and long inference times. It is evident that traditional deep learning detection methods lack flexibility, frequently resulting in false detections, and require a large amount of manually labeled data, leading to low detection efficiency, high costs, and rigid detection results. Summary of the Invention

[0006] This invention provides a method, apparatus, medium, and equipment for detecting and alerting defects in ID photos based on image-text fusion, in order to solve the problems of low efficiency and accuracy, high cost, and rigid detection results in the existing technology when detecting defects in ID photos.

[0007] A method for detecting and alerting defects in ID photos based on image-text fusion, the method comprising:

[0008] A defect detection model for ID photos is constructed and trained, the defect detection model for ID photos including a text encoding network and an image encoding network;

[0009] The defects in the ID photo and the prompt text information are input into the text encoding network to obtain the corresponding text encoding information, and a text feature database is constructed.

[0010] The image to be processed is acquired, and the image encoding network in the ID photo defect detection model is used to perform feature encoding processing on the image to be processed to obtain image encoding information.

[0011] Obtain the text encoding information corresponding to the image encoding information from the text feature database;

[0012] The corresponding ID photo defect and prompt text information are obtained based on the text encoding information.

[0013] Optionally, the image encoding network adopts a CLIP architecture with dual cascaded attention modules (CCAs), which are connected in series before the ViT-B / 32 model of the image encoder.

[0014] Optionally, each attention module (CCA) has the same structure;

[0015] In the attention module, the input image is first convolved and max-pooled using a preset convolution kernel to obtain an initial feature map; then, three 1*1 convolution kernels are used to extract features from the initial feature map to obtain a first feature map, a second feature map, and a third feature map; after performing an association operation on the first and second feature maps, normalization is performed to obtain a fourth feature map; and finally, an aggregation operation is performed on the third and fourth feature maps to obtain the target feature map.

[0016] Optionally, in the ViT-B / 32 model, classification token information is first added to the input image, and then a linear transformation is performed to divide the linearly transformed image into several image blocks of a specified size; each image block is converted into a low-dimensional feature vector; a position encoder is used to assign a corresponding position embedding vector to each image block and add it to the low-dimensional vector feature; a multi-layer encoder is used to extract features from the low-dimensional feature vector and position embedding vector of each image block to obtain image encoding information.

[0017] Optionally, the text encoding network employs the pre-trained word embedding model BERT;

[0018] In the pre-trained word embedding model BERT, the defect markers of ID photos are first mapped to high-dimensional real vectors. Based on the position of the defect markers in the text, a position embedding vector is generated. A multi-layer encoder is used to extract features from the high-dimensional real vectors and their corresponding position embedding vectors to obtain feature encoding. The feature encoding is then projected into a low-dimensional space to obtain text encoding information. Each five-layer structure in the multi-layer encoder contains a self-attention mechanism and a feedforward neural network.

[0019] Optionally, the training model for detecting defects in ID photos includes:

[0020] Obtain image samples, label the image samples with ID photo defects and prompt text information, and construct image-text pairs based on the image samples and ID photo defects and prompt text information;

[0021] During training, a preset number of image-text pairs are input as a batch into the ID photo defect detection model for training, and the image-text contrast loss is obtained through a preset objective optimization function.

[0022] The SGD optimizer is used to optimize the objective function of the ID photo defect detection model for each batch and backpropagate until the image contrast loss decreases to the preset accuracy, at which point the iteration stops.

[0023] Optionally, obtaining the text encoding information corresponding to the image encoding information from the text feature database includes:

[0024] Traverse each text encoding information in the text feature database and calculate the similarity between the image encoding information and the text encoding information;

[0025] Obtain the text encoding information with the highest similarity.

[0026] A device for detecting and alerting defects in ID photos based on image-text fusion, the device comprising:

[0027] The model training module is used to build and train a defect detection model for ID photos, which includes a text encoding network and an image encoding network.

[0028] The database construction module is used to input the defective ID photo and prompt text information into the text encoding network to obtain the corresponding text encoding information and construct a text feature database.

[0029] The image encoding module is used to acquire the image to be processed and to perform feature encoding processing on the image to be processed using the image encoding network in the ID photo defect detection model to obtain image encoding information.

[0030] The encoding acquisition module is used to acquire the text encoding information corresponding to the image encoding information from the text feature database;

[0031] The prompt message acquisition module is used to obtain the corresponding ID photo defect and prompt message text information based on the text encoding information.

[0032] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the image-text fusion-based method for detecting and alerting defects in ID photos as described above.

[0033] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the image-text fusion-based method for detecting and alerting defects in ID photos as described above.

[0034] The ID photo defect detection model constructed and trained in this embodiment includes a text encoding network and an image encoding network. The text encoding network is used to input ID photo defects and prompt text information to obtain text encoding information, thus constructing a text feature database. During inference, an image to be processed is acquired, and the image encoding network in the ID photo defect detection model is used to perform feature encoding processing on the image to obtain image encoding information. The text encoding information corresponding to the image encoding information is obtained from the text feature database. Based on the text encoding information, the corresponding ID photo defects and prompt text information are obtained. This invention, through image detection combined with text prediction, effectively improves the efficiency and accuracy of ID photo defect detection without reducing the quality of ID photo shooting. It overcomes the problem of low detection flexibility in existing technologies, and the generated prompts are more intuitive and user-acceptable, greatly improving the convenience for users to take standard ID photos. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart of a method for detecting and alerting defects in ID photos based on image-text fusion, provided in an embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram of the structure of a defect detection model for ID photos provided in an embodiment of the present invention;

[0038] Figure 3This is a schematic diagram of the structure of an attention module provided in an embodiment of the present invention;

[0039] Figure 4 This is a schematic diagram of the structure of a defect detection and alert device for ID photos based on image-text fusion provided in an embodiment of the present invention;

[0040] Figure 5 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] The image-text fusion-based ID photo defect detection and prompting method provided in this invention first constructs and trains an ID photo defect detection model. Then, the ID photo defects and prompting text information are encoded by the ID photo defect detection model and stored in a text feature database. During ID photo defect detection, an ID photo image is input, and the ID photo defect detection model performs image encoding on the image image to obtain image encoding information. The text encoding information corresponding to the image encoding information is retrieved from the text feature database, and the corresponding textual ID photo defect and prompting text information is returned to assist the user in adjusting the photo. Through image-text fusion, the efficiency and accuracy of ID photo defect detection are effectively improved, overcoming the problem of low detection flexibility in existing technologies. The generated prompts are more intuitive and user-friendly, greatly improving the convenience for users to take standard ID photos.

[0043] The following is a detailed description of the image-text fusion-based defect detection and alert method for ID photos provided in this embodiment, such as... Figure 1 As shown, the method for detecting and alerting defects in ID photos based on image-text fusion includes:

[0044] Step S101: Construct and train an ID photo defect detection model, wherein the ID photo defect detection model includes a text encoding network and an image encoding network;

[0045] Step S102: Input the defective ID photo and prompt text information into the text encoding network to obtain the corresponding text encoding information and construct a text feature database;

[0046] Step S103: Obtain the image to be processed, and use the image coding network in the ID photo defect detection model to perform feature coding processing on the image to be processed to obtain image coding information;

[0047] Step S104: Obtain the text encoding information corresponding to the image encoding information from the text feature database;

[0048] Step S105: Obtain the corresponding ID photo defect and prompt text information based on the text encoding information.

[0049] The ID photo defect detection model is used to detect ID photo defects in the input image and output the detection results. ID photo defects refer to ID photo rules that the input image does not meet, including but not limited to: hair covering eyebrows, unacceptable hair color, wearing colored contact lenses, excessive face rotation angle, excessive side rotation angle, excessive pitch angle, clothing color similar to the background color, wearing glasses, glasses with glare, wearing a necklace, head height not meeting specifications, hair covering ears, wearing badges, etc.

[0050] Figure 2 This is a schematic diagram of the structure of the ID photo defect detection model provided in an embodiment of the present invention. The ID photo defect detection model includes a text encoding network and an image encoding network. The text encoding network is used to encode the input ID photo defects and prompt text information to obtain corresponding text encoding information. The image encoding network is used to detect ID photo defects in the input image information and output the corresponding image encoding information.

[0051] As an example, in step S101, this embodiment constructs and trains an ID photo defect detection model. The image encoding network employs a CLIP architecture with dual cascaded attention modules (CCAs), which are connected in series before the ViT-B / 32 model of the image encoder. The text encoding network uses the pre-trained word embedding model BERT. CLIP (Contrastive Language–Image Pre-training) is a large-scale image-text pre-training model based on contrastive learning, open-sourced by OpenAI. ViT, short for Vision Transformer, is a visual processing model based on the Transformer architecture. During training, ID photo defect and prompt labels are added to sample images, and the sample images, ID photo defects, and prompt labels are used as training samples input into the ID photo defect detection model for training.

[0052] The text feature database is a collection of text-encoded information corresponding to defects in ID photos and their accompanying prompts.

[0053] As an example, in step S102, after the ID photo defect detection model is trained, this embodiment collects relevant ID photo defects and prompt text information, encodes them through the text encoding network in the ID photo defect detection model, and stores the corresponding text encoding information in the text feature database. The text feature database is used to extract text encoding information based on image encoding information, which helps to improve the accuracy and reliability of extracting text from images.

[0054] The image encoding information is obtained by feature encoding processing through the image encoding network in the ID photo defect detection model, reflecting the ID photo defects in the image to be processed.

[0055] As an example, in step S103, during the inference process using the ID photo defect detection model and the text feature database, the image to be processed is first acquired and input into the ID photo defect detection model. The image encoding network in the model performs feature encoding processing on the image to be processed to obtain image encoding information.

[0056] As an example, in step S104, to avoid false detections by the ID photo defect detection model, this embodiment further obtains the corresponding text encoding information from the text feature database based on the image encoding information. Since the text encoding information contained in the text feature database all correspond to ID photo defects and their prompt text information, more accurate and reliable encoding information can be obtained.

[0057] As an example, in step S105, after obtaining the text encoding information, this embodiment obtains the corresponding ID photo defect and prompt text information. The ID photo defect refers to the non-compliance information of the image to be processed as an ID photo. At the same time, the ID photo defect and prompt text information are output to guide the user to make adjustments.

[0058] In summary, the image-text fusion-based ID photo defect detection and prompting method provided in this embodiment constructs and trains an ID photo defect detection model, including a text encoding network and an image encoding network; and builds a text feature database. When detecting ID photo defects, an ID photo image is input, and the ID photo defect detection model encodes the image image to obtain image encoding information. Then, the text feature database retrieves the corresponding text encoding information based on the image encoding information and returns the corresponding textual information of the ID photo defect and prompts to assist users in adjusting their photos. By using image-text fusion and image-to-text search, the efficiency and accuracy of ID photo defect detection are effectively improved, and the problem of low detection flexibility in existing technologies is overcome. The generated prompts are more intuitive and user-friendly, greatly improving the convenience for users to take standard ID photos.

[0059] Optionally, as an example, the ID photo defect detection model includes a text encoding network and an image encoding network. The image encoding network adopts a CLIP architecture with a dual-cascaded attention module (CCA), which is connected in series before the ViT-B / 32 model of the image encoder. In this embodiment, there are many subtle ID photo defects during the self-service ID photo taking process, which cannot be covered by a single sparse CCA module. Therefore, this embodiment adds a Recurrent Criss-Cross Attention Module before the original Image Encoder in the Clip framework to form a dual-cascaded attention module (CCA). This improves density, obtains dense image context information, eliminates long-distance spatial dependencies, and reduces time and space complexity to O((H+W)-1*(H*W)).

[0060] Optionally, as an example, each attention module CCA has the same structure; in the attention module, the input image is first convolved and max-pooled according to a preset convolution kernel to obtain an initial feature map; then, three 1*1 convolution kernels are used to extract features from the initial feature map to obtain a first feature map, a second feature map, and a third feature map respectively; after performing an association operation on the first feature map and the second feature map, normalization processing is performed to obtain a fourth feature map; and an aggregation operation is performed on the third feature map and the fourth feature map to obtain the target feature map.

[0061] Figure 3This is a schematic diagram of the attention module provided in an embodiment of the present invention. In the attention module, the size of the input image is 448*448*3, and the preset convolution kernel is 3*3*24. After convolution and max pooling of the input image with a 3*3*24 convolution kernel, an initial feature map H with a size of 224*224*24 is obtained. H is then processed by three 1*1 convolutional layers to generate a first feature map Q, a second feature map K, and a third feature map V, respectively. The size of the first feature map Q and the second feature map K is 224*224*8, while the size of the third feature map V remains unchanged and is the same as the size of the initial feature map H, which is still 224*224*24. An affinity operation is performed on the first feature map Q and the second feature map K, that is, a fourth feature map D with a size of 224*224*447 is obtained by multiplying a pixel in the first feature map Q with the corresponding point in the second feature map K and the pixels in the corresponding row and column. The fourth feature map D, after being normalized by softmax, is then aggregated with the third feature map V to obtain the target feature map H' with a size of 224*224*24. After passing through a cascaded attention module CCA, it is fed into the Clip's original image encoder ViT-B / 32 model.

[0062] Optionally, as an example, in the ViT-B / 32 model, classification token information is first added to the input image, and then a linear transformation is performed to divide the linearly transformed image into several image blocks of a specified size; each image block is converted into a low-dimensional feature vector; a position encoder is used to assign a corresponding position embedding vector to each image block and add it to the low-dimensional feature vector; a multi-layer encoder is used to extract features from the low-dimensional feature vector and position embedding vector of each image block to obtain image encoding information.

[0063] In this embodiment, the input image of the ViT-B / 32 model is the output signal of the dual-cascaded attention module. First, the ViT-B / 32 model adds a special classification token (CLS token) to the beginning of the input image. A linear transformation scales the pixel values ​​of the input image to the range [-1, 1]. Then, the transformed image is divided into several image patches of a specified size, such as 16x16 fixed-size patches. Each patch is a 32x32 pixel block, flattened into a vector, and then a low-dimensional feature vector (patch embedding) is obtained through a linear transformation. To capture positional information in the image, the ViT-B / 32 model employs a position encoder. The position encoder assigns a position embedding vector to each position (i.e., each patch). The position embedding vector is added to the low-dimensional feature vector (patch embedding) to fuse positional information and visual features. In this embodiment, the ViT-B / 32 model uses a multi-layer Transformer encoder to process the low-dimensional feature vector and the position embedding vector. Each Transformer encoder consists of multiple self-attention heads and a feed-forward neural network. The attention heads capture the relationships between global and local data, while the feed-forward neural network extracts features and performs non-linear transformations. The low-dimensional feature vector and location embedding vector of the classification token (CLS token) are processed by the Transformer encoder to represent the semantic features of the entire image. The final output of the ViT-B / 32 model is the vector representation of the CLS token processed by the Transformer encoder.

[0064] Optionally, as an example, the text encoding network uses a pre-trained word embedding model BERT. In the pre-trained word embedding model BERT, the defect markers of the ID photo are first mapped to high-dimensional real vectors. Based on the position of the defect markers in the text, a position embedding vector is generated. A multi-layer encoder is used to extract features from the high-dimensional real vectors and their corresponding position embedding vectors to obtain feature encoding. The feature encoding is then projected into a low-dimensional space to obtain text encoding information. Each five-layer structure in the multi-layer encoder includes a self-attention mechanism and a feedforward neural network.

[0065] This embodiment employs the pre-trained word embedding model BERT to acquire text encoding information. Image defect markers, including but not limited to user hair covering eyebrows, hair color, wearing colored contact lenses, excessive face rotation angle, excessive side rotation angle, excessive pitch angle, clothing color similar to the background color, wearing glasses, glasses glare, wearing necklaces, head height not meeting specifications, hair covering ears, wearing badges, etc., are mapped to a high-dimensional real-valued vector representation to capture the semantic information of the markers. To preserve the order information of the markers in the text, positional embeddings are introduced to represent the position of each defect marker in the text, and corresponding positional embedding vectors are generated using sine and cosine functions. A multi-layer Transformer Encoder is designed, with each layer consisting of multiple Transformer encoder layers with the same structure, all including a self-attention mechanism and a feedforward neural network. The self-attention mechanism is used to establish the correlation between markers in the marker sequence, and the feedforward neural network is used to enhance the non-linearity of feature representation. Then, a trainable projection operation is applied to the output of the last encoder layer to map the vector representation of each defect marker to a lower-dimensional space for encoding, thus obtaining the text encoding information.

[0066] Optionally, as an example, the training of the ID photo defect detection model in step S101 includes:

[0067] Step S1011: Obtain image samples, label the image samples with ID photo defects and prompt text information, and construct image-text pairs based on the image samples and ID photo defects and prompt text information;

[0068] In step S1012, during training, a preset number of image-text pairs are input as a batch into the ID photo defect detection model for training, and the image-text contrast loss is obtained through a preset objective optimization function.

[0069] Step S1013: Use the SGD optimizer to optimize the objective optimization function of the ID photo defect detection model for each batch and backpropagate until the image contrast loss decreases to the preset accuracy and the iteration stops.

[0070] The image-text pair is used to train the ID photo defect detection model, and consists of the image sample and the labeled ID photo defect and prompt text information.

[0071] As an example, in step S1011, this embodiment collects several photos of multiple volunteers taken by the self-service certificate processing equipment as image samples. For example, 20 qualified and 20 unqualified photos of 1000 volunteers in different states, then there are a total of 40,000 photos, denoted as [X1, X2, X3, ..., Xn].

[0072] For the collected image samples, add corresponding ID photo defect and prompt text information to the image samples. For example, the user's hair may cover the eyebrows, hair color, wearing colored contact lenses, excessive face rotation angle, excessive side rotation angle, excessive tilt angle, the user's clothing color may be similar to the background color, wearing glasses, glasses may reflect light, wearing a necklace, the distance between the top of the head and the specifications may not meet the requirements, hair may cover the ears, wearing a necklace, wearing a badge, etc., and corresponding adjustment instructions. One image sample corresponds to a set of ID photo defect and prompt text information, denoted as [Y1, Y2, Y3, ..., Yn].

[0073] Image samples [X1, X2, X3, ..., Xn] and their corresponding ID photo defects and prompt text information [Y1, Y2, Y3, ..., Yn] are combined into image-text pairs [X1-Y1, X2-Y2, X3-Y3, ..., Xn-Yn]. Optionally, as a preferred example of the present invention, the above image sample pairs are divided into training set, validation set, and test set in a ratio of 7:2:1.

[0074] Each batch can select 128 image-text pairs. The image-text contrast loss refers to the cosine similarity between the image encoding information extracted from the image sample and the text encoding information extracted from the ID photo defect and prompt text information. The larger the cosine similarity, the stronger the correspondence between the image and the text, and vice versa.

[0075] As an example, in step S1012, the image-text pair is input into the ID photo defect detection model for training. Image samples are processed by an image encoding network to extract image encoding information, and ID photo defects and prompt text information are processed by a text encoding network to extract text encoding information. Then, the contrast loss between the image encoding information and the text encoding information, i.e., cosine similarity, is calculated. Finally, the model is optimized using a preset objective optimization function. Optionally, the objective optimization function maximizes the cosine similarity of positive samples and minimizes the cosine similarity of negative samples. Positive samples are image-text pairs consisting of an image sample and its corresponding ID photo defect and prompt text information, while negative samples are image-text pairs consisting of an image sample and non-corresponding ID photo defects and prompt text information. This embodiment maximizes the cosine similarity of N positive samples and minimizes N by training the parameters of the text encoder and image encoder. 2 The cosine similarity of -N negative samples. The objective function formula is:

[0076]

[0077] in, Image encoding information for the i-th image sample. The text encoding information for the j-th text sample. for and The cosine similarity. Here, the text samples refer to the text information regarding defects in ID photos and accompanying notices.

[0078] As an example, in step S1013, this embodiment uses the SGD optimizer to optimize the loss function of the ID photo defect detection model for each batch and performs backpropagation. Iteration stops when the loss cost of the ID photo defect detection model decreases to a specified accuracy. During training each batch, 64 image-text pairs are used as a batch input to train and test the ID photo defect detection model to observe the accuracy and recall of the test set.

[0079] Optionally, as an example, step S104, namely obtaining the text encoding information corresponding to the image encoding information from the text feature database, includes:

[0080] Step S1041: Traverse each text encoding information in the text feature database and calculate the similarity between the image encoding information and the text encoding information;

[0081] Step S1042: Obtain the text encoding information with the highest similarity.

[0082] The similarity is the cosine similarity between image encoding information and text encoding information. To avoid false detections in the ID photo defect detection model, this embodiment further obtains the corresponding text encoding information from the text feature database based on the image encoding information, achieving the effect of detecting text from images. Here, this embodiment traverses each piece of text encoding information in the text feature database, calculates the cosine similarity between the image encoding information and the text encoding information, and then selects the text encoding information with the highest similarity as the text encoding information corresponding to the image encoding information. Since the text encoding information contained in the text feature database all correspond to ID photo defects and their prompt text information, more standardized ID photo defect detection results and prompt information can be obtained, improving the accuracy of the ID photo defect detection model. It also overcomes the problem of low detection flexibility in existing technologies, and the generated prompts are more intuitive and user-friendly, greatly improving the convenience for users to take standard ID photos.

[0083] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0084] In one embodiment, the present invention also provides a device for detecting and alerting defects in ID photos based on image-text fusion, which corresponds one-to-one with the method for detecting and alerting defects in ID photos based on image-text fusion described in the above embodiments. For example... Figure 4 As shown, the image-text fusion-based ID photo defect detection and alert device includes a model training module 41, a database construction module 42, an image encoding module 43, an encoding acquisition module 44, and an alert message acquisition module 45. Detailed descriptions of each functional module are as follows:

[0085] Model training module 41 is used to construct and train a defect detection model for ID photos, wherein the defect detection model for ID photos includes a text encoding network and an image encoding network;

[0086] Database construction module 42 is used to input the defective ID photo and prompt text information into the text encoding network to obtain the corresponding text encoding information and construct a text feature database;

[0087] Image encoding module 43 is used to acquire the image to be processed and to perform feature encoding processing on the image to be processed using the image encoding network in the ID photo defect detection model to obtain image encoding information;

[0088] Encoding acquisition module 44 is used to acquire text encoding information corresponding to the image encoding information from the text feature database;

[0089] The prompt acquisition module 45 is used to acquire the corresponding ID photo defect and prompt text information based on the text encoding information.

[0090] Optionally, the image encoding network adopts a CLIP architecture with dual cascaded attention modules (CCAs), which are connected in series before the ViT-B / 32 model of the image encoder.

[0091] Optionally, each attention module (CCA) has the same structure;

[0092] In the attention module, the input image is first convolved and max-pooled using a preset convolution kernel to obtain an initial feature map; then, three 1*1 convolution kernels are used to extract features from the initial feature map to obtain a first feature map, a second feature map, and a third feature map; after performing an association operation on the first and second feature maps, normalization is performed to obtain a fourth feature map; and finally, an aggregation operation is performed on the third and fourth feature maps to obtain the target feature map.

[0093] Optionally, in the ViT-B / 32 model, classification token information is first added to the input image, and then a linear transformation is performed to divide the linearly transformed image into several image blocks of a specified size; each image block is converted into a low-dimensional feature vector; a position encoder is used to assign a corresponding position embedding vector to each image block and add it to the low-dimensional vector feature; a multi-layer encoder is used to extract features from the low-dimensional feature vector and position embedding vector of each image block to obtain image encoding information.

[0094] Optionally, the text encoding network employs the pre-trained word embedding model BERT;

[0095] In the pre-trained word embedding model BERT, the defect markers of ID photos are first mapped to high-dimensional real vectors. Based on the position of the defect markers in the text, a position embedding vector is generated. A multi-layer encoder is used to extract features from the high-dimensional real vectors and their corresponding position embedding vectors to obtain feature encoding. The feature encoding is then projected into a low-dimensional space to obtain text encoding information. Each five-layer structure in the multi-layer encoder contains a self-attention mechanism and a feedforward neural network.

[0096] Optionally, the model training module 41 includes:

[0097] The sample pair acquisition unit is used to acquire image samples, label the image samples with ID photo defects and prompt text information, and construct image-text pairs based on the image samples and ID photo defects and prompt text information.

[0098] The training unit is used to input a preset number of image-text pairs as a batch into the ID photo defect detection model for training each time, and obtain the image-text contrast loss through a preset objective optimization function.

[0099] The optimization unit is used to optimize the objective function of the ID photo defect detection model for each batch using the SGD optimizer and backpropagate until the image contrast loss decreases to a preset accuracy and the iteration stops.

[0100] Optionally, the encoding acquisition module 44 includes:

[0101] The calculation unit is used to traverse each text encoding information in the text feature database and calculate the similarity between the image encoding information and the text encoding information;

[0102] The acquisition unit is used to obtain the text encoding information with the highest similarity.

[0103] Specific limitations regarding the image-text fusion-based ID photo defect detection and alert device can be found in the above-described limitations of the image-text fusion-based ID photo defect detection and alert method, and will not be repeated here. Each module in the aforementioned image-text fusion-based ID photo defect detection and alert device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in memory as software, so that the processor can call and execute the corresponding operations of each module.

[0104] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for detecting and alerting defects in ID photos based on image-text fusion.

[0105] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0106] A defect detection model for ID photos is constructed and trained, the defect detection model for ID photos including a text encoding network and an image encoding network;

[0107] The defects in the ID photo and the prompt text information are input into the text encoding network to obtain the corresponding text encoding information, and a text feature database is constructed.

[0108] The image to be processed is acquired, and the image encoding network in the ID photo defect detection model is used to perform feature encoding processing on the image to be processed to obtain image encoding information.

[0109] Obtain the text encoding information corresponding to the image encoding information from the text feature database;

[0110] The corresponding ID photo defect and prompt text information are obtained based on the text encoding information.

[0111] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0112] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0113] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for detecting and alerting defects in ID photos based on image-text fusion, characterized in that, include: A defect detection model for ID photos is constructed and trained. The defect detection model includes a text encoding network and an image encoding network. The image encoding network adopts a CLIP architecture with dual cascaded attention modules (CCA), which are connected in series before the ViT-B / 32 model of the image encoder. The defects in the ID photo and the prompt text information are input into the text encoding network to obtain the corresponding text encoding information, and a text feature database is constructed. The image to be processed is acquired, and the image encoding network in the ID photo defect detection model is used to perform feature encoding processing on the image to be processed to obtain image encoding information. Obtain the text encoding information corresponding to the image encoding information from the text feature database; The corresponding ID photo defect and prompt text information are obtained based on the text encoding information.

2. The method for detecting and alerting defects in ID photos based on image-text fusion as described in claim 1, characterized in that, Each attention module (CCA) has the same structure; In the attention module, the input image is first convolved and max-pooled using a preset convolution kernel to obtain an initial feature map; then, three 1*1 convolution kernels are used to extract features from the initial feature map to obtain a first feature map, a second feature map, and a third feature map; after performing an association operation on the first and second feature maps, normalization is performed to obtain a fourth feature map; and finally, an aggregation operation is performed on the third and fourth feature maps to obtain the target feature map.

3. The method for detecting and alerting defects in ID photos based on image-text fusion as described in claim 2, characterized in that, In the ViT-B / 32 model, classification token information is first added to the input image, and then a linear transformation is performed to divide the linearly transformed image into several image blocks of a specified size. Each image block is converted into a low-dimensional feature vector. A position encoder is used to assign a corresponding position embedding vector to each image block and add it to the low-dimensional feature vector. A multi-layer encoder is used to extract features from the low-dimensional feature vector and position embedding vector of each image block to obtain image encoding information.

4. The method for detecting and alerting defects in ID photos based on image-text fusion as described in claim 1, characterized in that, The text encoding network uses the pre-trained word embedding model BERT; In the pre-trained word embedding model BERT, the defect markers of ID photos are first mapped to high-dimensional real vectors. Based on the position of the defect markers in the text, a position embedding vector is generated. A multi-layer encoder is used to extract features from the high-dimensional real vectors and their corresponding position embedding vectors to obtain feature encoding. The feature encoding is then projected into a low-dimensional space to obtain text encoding information. Each five-layer structure in the multi-layer encoder contains a self-attention mechanism and a feedforward neural network.

5. The method for detecting and alerting defects in ID photos based on image-text fusion as described in claim 1, characterized in that, The training model for detecting defects in ID photos includes: Obtain image samples, label the image samples with ID photo defects and prompt text information, and construct image-text pairs based on the image samples and ID photo defects and prompt text information; During training, a preset number of image-text pairs are input as a batch into the ID photo defect detection model for training, and the image-text contrast loss is obtained through a preset objective optimization function. The SGD optimizer is used to optimize the objective function of the ID photo defect detection model for each batch and backpropagate until the image contrast loss decreases to the preset accuracy, at which point the iteration stops.

6. The method for detecting and alerting defects in ID photos based on image-text fusion as described in claim 1, characterized in that, The step of obtaining the text encoding information corresponding to the image encoding information from the text feature database includes: Traverse each text encoding information in the text feature database and calculate the similarity between the image encoding information and the text encoding information; Obtain the text encoding information with the highest similarity.

7. A device for detecting and alerting defects in ID photos based on image-text fusion, characterized in that, The device includes: The model training module is used to build and train a defect detection model for ID photos, which includes a text encoding network and an image encoding network. The database construction module is used to input the defective ID photo and prompt text information into the text encoding network to obtain the corresponding text encoding information and construct a text feature database. The image encoding module is used to acquire the image to be processed, and to perform feature encoding processing on the image to be processed using the image encoding network in the ID photo defect detection model to obtain image encoding information; the image encoding network adopts a CLIP architecture with dual cascaded attention modules CCA, and the attention modules CCA are connected in series before the ViT-B / 32 model of the image encoder. The encoding acquisition module is used to acquire the text encoding information corresponding to the image encoding information from the text feature database; The prompt message acquisition module is used to obtain the corresponding ID photo defect and prompt message text information based on the text encoding information.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the image-text fusion-based method for detecting and alerting defects in ID photos as described in any one of claims 1 to 6.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image-text fusion-based method for detecting and alerting defects in ID photos as described in any one of claims 1 to 6.

Images