Certificate anti-counterfeiting detection model training method and device, computer readable storage medium, electronic equipment and computer program product

Abstract

The embodiment of the invention discloses a training method and device of a certificate anti-counterfeiting detection model, a computer readable storage medium, electronic equipment and a computer program product. The training method of the certificate anti-counterfeiting detection model can comprise the following steps: training an initial certificate feature extraction model comprising a video encoder and a query converter of which at least part of parameters are in a frozen state by utilizing a first training sample comprising a first certificate image sequence and a description text for a first certificate sample, and obtaining a target certificate feature extraction model capable of learning visual representation most related to the certificate description text. Training an initial certificate anti-counterfeiting detection model containing a target certificate feature extraction model and a large language model of which the parameters are in a frozen state by utilizing a second certificate image sequence containing a second certificate sample and a second training sample of a preset prompt word, and obtaining a target certificate anti-counterfeiting detection model capable of generating a certificate anti-counterfeiting detection result in a text form.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a training method, device, computer-readable storage medium, electronic device and computer program product for a document anti-counterfeiting detection model. Background Art

[0002] In recent years, with the continuous development of image processing technology, online document detection functions have become more automated, accurate and efficient, and users' acceptance of online document detection functions has also increased. Therefore, service providers have begun to widely use online document detection functions to improve the efficiency of user identity verification and protect the rights and interests of all parties. However, in different business scenarios, when service providers use online document detection functions to generate document anti-counterfeiting detection results, the collection conditions and number of user document images required, as well as the conditions for identifying the authenticity of the documents required may be different, so that service providers often need to train multiple document anti-counterfeiting detection models in a targeted manner to meet the actual needs of various business scenarios. The operation is complicated and time-consuming.

[0003] Based on this, how to efficiently generate a target document anti-counterfeiting detection model with good accuracy and versatility, so as to improve the accuracy, flexibility and convenience of the document anti-counterfeiting detection solution, has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of this specification provide a training method, device, computer-readable storage medium, electronic device and computer program product for a certificate anti-counterfeiting detection model, which can efficiently generate a target certificate anti-counterfeiting detection model with good accuracy and versatility, thereby improving the accuracy, flexibility and convenience of certificate anti-counterfeiting detection based on the target certificate anti-counterfeiting detection model, and has good practicality.

[0005] The embodiment of this specification provides a method for training a document anti-counterfeiting detection model, the method comprising:

[0006] Acquire a first training sample generated based on a first certificate sample; the first training sample includes a first certificate image sequence collected for the first certificate sample, and a description text for the first certificate sample;

[0007] The initial document feature extraction model is trained using the first training sample to obtain a target document feature extraction model; wherein the initial document feature extraction model comprises: a video encoder whose at least some parameters are in a frozen state and a query converter connected in sequence;

[0008] Acquire a second training sample generated based on the second certificate sample; the second training sample includes a second certificate image sequence collected for the second certificate sample, and a preset prompt word for guiding the large language model to output a certificate anti-counterfeiting detection result for the second certificate sample;

[0009] The initial document anti-counterfeiting detection model is trained using the second training sample to obtain a target document anti-counterfeiting detection model; wherein the initial document anti-counterfeiting detection model includes: the target document feature extraction model connected in sequence, and the large language model whose parameters are in a frozen state.

[0010] The present specification also provides a document anti-counterfeiting detection method, the method comprising:

[0011] Obtain the document image sequence of the user's target document;

[0012] The document image sequence is processed using a target document anti-counterfeiting detection model to obtain a document anti-counterfeiting detection result for the target document of the user; wherein the target document anti-counterfeiting detection model is a model trained using the above-mentioned document anti-counterfeiting detection model training method.

[0013] The embodiment of this specification also provides a training device for a certificate anti-counterfeiting detection model, including:

[0014] A first acquisition module is used to acquire a first training sample generated based on a first certificate sample; the first training sample includes a first certificate image sequence collected for the first certificate sample, and a description text for the first certificate sample;

[0015] A first training module, used to train an initial document feature extraction model using the first training sample to obtain a target document feature extraction model; wherein the initial document feature extraction model includes: a video encoder whose at least some parameters are in a frozen state and a query converter connected in sequence;

[0016] A second acquisition module is used to acquire a second training sample generated based on a second certificate sample; the second training sample includes a second certificate image sequence collected for the second certificate sample, and a preset prompt word for guiding the large language model to output a certificate anti-counterfeiting detection result for the second certificate sample;

[0017] The second training module is used to train the initial certificate anti-counterfeiting detection model using the second training sample to obtain a target certificate anti-counterfeiting detection model; wherein the initial certificate anti-counterfeiting detection model includes: the target certificate feature extraction model connected in sequence, and the large language model whose parameters are in a frozen state.

[0018] The embodiment of this specification also provides a document anti-counterfeiting detection device, including:

[0019] The certificate image acquisition module is used to acquire a sequence of certificate images of the user's target certificate;

[0020] The anti-counterfeiting detection module is used to process the document image sequence using a target document anti-counterfeiting detection model to obtain a document anti-counterfeiting detection result for the target document of the user; wherein the target document anti-counterfeiting detection model is a model trained using the above-mentioned document anti-counterfeiting detection model training method.

[0021] The embodiments of this specification also provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0022] An embodiment of the present specification also provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the above method.

[0023] The embodiments of the present specification also provide a computer program product, on which at least one instruction is stored, and when the at least one instruction is executed by a processor, the steps of the above method are implemented.

[0024] In the embodiment of the present specification, a video encoder and a query converter whose parameters are at least partially frozen can be sequentially connected as an initial document feature extraction model, and the initial document feature extraction model is trained by using a first training sample including a first document image sequence collected for a first document sample and a description text for the first document sample, so that the generated target document feature extraction model can learn the visual representation most relevant to the document description text, thereby improving the target document feature extraction model's ability to accurately extract the document visual features required by the large language model. In addition, the target document feature extraction model and the large language model whose parameters are frozen can also be sequentially connected as an initial document anti-counterfeiting detection model, and the initial document anti-counterfeiting detection model is trained by using a second document image sequence including a second document sample and a second training sample of a preset prompt word, so that the generated target document anti-counterfeiting detection model can generate a document anti-counterfeiting detection result in the form of text with good accuracy according to the document visual features extracted by the target document feature extraction model, so as to ensure the accuracy of the document anti-counterfeiting detection scheme implemented based on the target document anti-counterfeiting detection model.

[0025] In addition, during the model training process, at least some parameters of the video encoder in the target document anti-counterfeiting detection model can be frozen, and the parameters of the large language model in the target document anti-counterfeiting detection model can also be frozen, thereby improving the model training efficiency and reducing the resources required for model training.

[0026] In addition, since the target certificate anti-counterfeiting detection model can be used to process certificate image sequences, and can conveniently configure the certificate description text and preset prompt words in the training samples, so that the target certificate anti-counterfeiting detection model can meet the detection needs of various types of certificates, it has good versatility and flexibility, which is conducive to improving the versatility and ease of use of the certificate anti-counterfeiting detection solution implemented based on the target certificate anti-counterfeiting detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of an application scenario of a document anti-counterfeiting detection solution provided in an embodiment of this specification;

[0028] Figure 2 A flowchart of a method for training a document anti-counterfeiting detection model provided in an embodiment of this specification;

[0029] Figure 3 A schematic diagram of the structure of an initial document feature extraction model provided in the embodiments of this specification;

[0030] Figure 4 A schematic diagram of the structure of an initial document anti-counterfeiting detection model provided in the embodiments of this specification;

[0031] Figure 5 A flowchart of a document anti-counterfeiting detection method provided in an embodiment of this specification;

[0032] Figure 6 A schematic diagram of the structure of a training device for a certificate anti-counterfeiting detection model provided in an embodiment of this specification;

[0033] Figure 7 A schematic diagram of the structure of a document anti-counterfeiting detection device provided in an embodiment of this specification;

[0034] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0036] Before specifically describing the document anti-counterfeiting detection solution provided in the embodiments of this specification, the relevant technical background is first described.

[0037] When providing services to users, service providers can use the online certificate detection function to detect and process the certificate images submitted by users, so as to obtain the certificate anti-counterfeiting detection results for users conveniently and quickly, which is conducive to protecting the rights and interests of all parties. However, in different business scenarios, when performing online certificate detection for users, the acquisition conditions and number of user certificate images required to be used may be different, and the conditions for the authenticity identification of the certificates required to be judged may also be different. For example, some anti-counterfeiting detection tasks may only require the use of a single certificate image collected from a frontal perspective, while some anti-counterfeiting detection tasks may require the use of multiple certificate images collected under various conditions such as under and without illumination of a flash device. In addition, some anti-counterfeiting detection tasks may require the detection of problems such as certificate clarity, certificate occlusion, and certificate punching, while some anti-counterfeiting detection tasks may require the detection of whether the certificate anti-counterfeiting mark is included, or whether the user uses printed images, certificates displayed on electronic screens, etc. to cheat.

[0038] At present, service providers usually train different certificate anti-counterfeiting detection models according to the certificate anti-counterfeiting detection requirements of various business scenarios to meet the actual needs of various business scenarios. Obviously, this certificate anti-counterfeiting detection solution requires the training of multiple certificate anti-counterfeiting detection models with poor versatility, which makes the operation complicated, consumes more resources and time, and cannot meet the new certificate anti-counterfeiting detection requirements in time when the certificate anti-counterfeiting detection requirements change, which is also easy to affect the security of business operations.

[0039] Based on this, how to efficiently generate a target document anti-counterfeiting detection model with good accuracy and versatility, so as to improve the accuracy, flexibility and convenience of the document anti-counterfeiting detection solution, has become an urgent problem to be solved.

[0040] See also Figure 1 , Figure 1 A schematic diagram of an application scenario of a document anti-counterfeiting detection solution provided in an embodiment of this specification.

[0041] like Figure 1As shown, the server device 101 can usually use the training samples to train the target document feature extraction model in advance. Specifically, the server device 101 can obtain a first training sample generated based on the first document sample; wherein the first training sample can include a first document image sequence collected for the first document sample,

[0042] The first training sample is used to train an initial document feature extraction model to obtain a target document feature extraction model; wherein the initial document feature extraction model may include: a video encoder connected in sequence, at least some of whose parameters are in a frozen state, and a query converter.

[0043] The server device 101 may also obtain a second training sample generated based on the second certificate sample; the second training sample may include a second certificate image sequence collected for the second certificate sample, and a preset prompt word for guiding the large language model to output the certificate anti-counterfeiting detection result for the second certificate sample. The initial certificate anti-counterfeiting detection model is trained using the second training sample to obtain a target certificate anti-counterfeiting detection model; wherein the initial certificate anti-counterfeiting detection model may include: the target certificate feature extraction model connected in sequence, and the large language model whose parameters are in a frozen state.

[0044] Subsequently, when the user uses the terminal device 102 to perform online document anti-counterfeiting detection on the target document 103, if the user has obtained the use and transmission authority of the document image sequence of the target document in advance, the terminal device 102 can send the collected document image sequence of the target document to the server device 101, so as to use the target document feature extraction model at the server device 101 to process the document image sequence of the target document, thereby obtaining the document anti-counterfeiting detection result for the user's target document. Alternatively, the terminal device 102 can also obtain the target document feature extraction model from the server device 101, and then use the target document feature extraction model to process the document image sequence of the target document, thereby obtaining the document anti-counterfeiting detection result for the user's target document. There is no specific limitation on this.

[0045] See also Figure 2 , is a flow chart of a method for training a document anti-counterfeiting detection model provided in an embodiment of this specification. The execution subject of the process can be a model training device, or an application installed on the model training device. Figure 2 The process shown is described in detail, and the training method of the document anti-counterfeiting detection model may specifically include the following steps:

[0046] Step 202: Acquire a first training sample generated based on a first certificate sample; the first training sample includes a first certificate image sequence collected for the first certificate sample, and a description text for the first certificate sample.

[0047] In the embodiment of this specification, when there is a need to perform document anti-counterfeiting detection on a preset type of document, a first document image sequence of a first document sample of the preset type and a description text for the first document sample can be obtained to generate a first training sample.

[0048] Among them, the preset category to which the first certificate sample belongs can be determined according to actual needs of the equipment. For example, it can include but is not limited to: a certificate used to reflect the identity of the user, a certificate used to reflect the educational background of the user, a certificate used to reflect the professional qualifications of the user, a certificate used to reflect the assets of the user, etc., and there is no specific limitation on this.

[0049] The first document image sequence may include a plurality of document images collected for the first document sample according to preset image collection conditions. The preset image collection conditions may include, but are not limited to: collecting document images under illumination by a flash device, collecting document images without illumination by a flash device, collecting document images from a frontal viewing angle, collecting document images from a non-frontal viewing angle, etc., and no specific limitation is made thereto.

[0050] The description text for the first certificate sample may be: text data that introduces the features of the first certificate sample or the first certificate image sequence from different angles. For example, the description text for the first certificate sample may include, but is not limited to: information reflecting the certificate type to which the first certificate sample belongs, information reflecting the material of the first certificate sample, information reflecting the clarity, acquisition angle, reflection, overexposure, user blocking of the certificate, and certificate incompleteness of the first certificate image sequence, without specific limitation.

[0051] Step 204, using the first training sample to train the initial document feature extraction model to obtain a target document feature extraction model; wherein the initial document feature extraction model includes: a video encoder connected in sequence, at least some parameters of which are in a frozen state, and a query converter.

[0052] In the embodiment of this specification, the target certificate anti-counterfeiting detection model to be trained may include two parts: a target certificate feature extraction model and a large language model (LLM). The target certificate feature extraction model can be used to extract visual feature data of a certificate image sequence of a user certificate, and the large language model can generate a certificate anti-counterfeiting detection result for the user certificate based on the visual feature data of the certificate image sequence extracted by the target certificate feature extraction model.

[0053] Since the target document feature extraction model is required to learn the visual representation that is most relevant to the document description text, so as to meet the requirements for the document visual features that need to be input into the large language model, and in order to save the resources consumed by model training and improve the model training efficiency, based on this, the initial document feature extraction model can include: a video encoder whose parameters are at least partially frozen and connected in sequence, and a query converter whose parameters are not frozen.

[0054] The video encoder can receive a sequence of ID images and extract an initial ID visual feature vector from the ID image sequence to input into the query converter. The query converter (Q-Former) can usually be a lightweight transformer model. By pre-training the query converter using the ID image feature-ID description text, the image converter can extract the ID visual representation that is most relevant to the ID description text and input it into the large language model, so as to serve as a bridge to connect the large language model and the video encoder.

[0055] In the embodiment of this specification, the model training process can be divided into two stages. The first stage can be used to train the target document feature extraction model. The second stage can be used to train the target document feature extraction model and the large language model with frozen parameters to obtain the target document anti-counterfeiting detection model.

[0056] In practical applications, both the initial document feature extraction model and the target document feature extraction model may include: a video encoder whose at least some parameters are in a frozen state, and a query converter whose parameters are in an unfrozen state. It can be seen that in the above two model training stages, the parameters of the query converter can usually be optimized, and some parameters of the video encoder can be optimized in any model training stage, or not optimized in any model training stage, which has good flexibility.

[0057] Step 206, obtaining a second training sample generated based on the second document sample; the second training sample includes a second document image sequence collected for the second document sample, and a preset prompt word for guiding the large language model to output a document anti-counterfeiting detection result for the second document sample.

[0058] In the embodiments of the present specification, the second certificate sample and the second certificate sample can be the same sample or different samples. However, under normal circumstances, the second certificate sample and the second certificate sample can be of the same type, and the acquisition method of the second certificate image sequence and the first certificate image sequence can also be consistent, which is beneficial to improving the accuracy of the certificate anti-counterfeiting detection results generated by the trained target certificate anti-counterfeiting detection model.

[0059] In the embodiments of this specification, prompts can help indicate the direction and topic of the conversation, ensure that the large language model can understand the specific needs of the user, and guide the direction of the conversation, improve the relevance and accuracy of the large language model's response, and reduce misunderstandings and invalid answers. In addition, the language style and output format can be adjusted so that the large language model can better meet the needs of different scenarios. Based on this, when it is necessary to use a large language model for document anti-counterfeiting detection, a preset prompt word can be carried in the second training sample to guide the large language model to output the document anti-counterfeiting detection result for the second document sample. Among them, the specific content and format of the preset prompt word can be set according to actual needs, and no specific limitation is made to this.

[0060] Optionally, the preset prompt word can be specifically used to guide the large language model to output a preset type of certificate description information for the second certificate sample; and the certificate description information can be used to generate a final certificate anti-counterfeiting detection result for at least one anti-counterfeiting detection task for the second certificate sample.

[0061] For ease of understanding, an example is given to illustrate this. When the anti-counterfeiting detection task is used to detect whether the document is obscured, the preset prompt word can be: the second document sample is not obscured, or the upper right corner area of ​​the second document sample is obscured, or the face image area in the second document sample is obscured, etc. When the anti-counterfeiting detection task is used to detect whether the document has a perforation problem, the preset prompt word can be: the second document sample is not perforated, or the lower edge of the second document sample has a circular hole, or the right edge of the second document sample has 3 trapezoidal holes, etc., without specific limitation.

[0062] Step 208, using the second training sample to train the initial certificate anti-counterfeiting detection model to obtain a target certificate anti-counterfeiting detection model; wherein the initial certificate anti-counterfeiting detection model includes: the target certificate feature extraction model connected in sequence, and the large language model whose parameters are in a frozen state.

[0063] In the embodiment of the specification, since the training cost of the large language model is high and the pre-trained large language model has good generalization ability, the parameters of the pre-trained large language model can be directly frozen during the model training process. And since the second training sample can include the second document image sequence of the second document sample and the preset prompt words for guiding the large language model to output the document anti-counterfeiting detection result for the second document sample, when the initial document anti-counterfeiting detection model is trained using the second training sample, the model training efficiency can be improved on the basis of ensuring that the target document anti-counterfeiting detection model obtained by training can generate a document anti-counterfeiting detection result with good accuracy.

[0064] Figure 2 In the method, by using the first training sample to train the initial document feature extraction model, the generated target document feature extraction model can learn the visual representation most relevant to the document description text, thereby improving the target document feature extraction model's ability to accurately extract the document visual features required by the large language model. And, by using the second training sample to train the initial document anti-counterfeiting detection model, the generated target document anti-counterfeiting detection model can generate a document anti-counterfeiting detection result in the form of text with good accuracy based on the document visual features extracted by the target document feature extraction model, so as to ensure the accuracy of the document anti-counterfeiting detection solution implemented based on the target document anti-counterfeiting detection model.

[0065] In addition, during the model training process, at least some parameters of the video encoder in the target document anti-counterfeiting detection model can be frozen, and the parameters of the large language model in the target document anti-counterfeiting detection model can also be frozen, thereby improving the model training efficiency and reducing the resources required for model training.

[0066] In addition, since the target certificate anti-counterfeiting detection model can be used to process certificate image sequences, and can conveniently configure the certificate description text and preset prompt words in the training samples, so that the target certificate anti-counterfeiting detection model can meet the detection needs of various types of certificates, it has good versatility and flexibility, which is conducive to improving the versatility and ease of use of the certificate anti-counterfeiting detection solution implemented based on the target certificate anti-counterfeiting detection model.

[0067] based on Figure 2 The method in this specification also provides some specific implementation plans of the method, which are described below.

[0068] In a feasible implementation, the query converter in the initial document feature extraction model may include: an image converter and a text converter; the video encoder in the initial document feature extraction model may be connected to the image converter, and the image converter may be connected to the text converter.

[0069] Correspondingly, step 204, using the first training sample to train the initial document feature extraction model to obtain the target document feature extraction model, may include:

[0070] The first document image sequence in the first document sample is input into the video encoder to obtain a first document visual feature vector output by the video encoder.

[0071] The description text for the first certificate sample is input into the text converter to obtain a text semantic feature vector output by the text converter.

[0072] According to the first preset model training loss, based on the first query embedding vector generated by the image converter based on the first document visual feature vector and the text semantic feature vector, parameters of the initial document feature extraction model are optimized to obtain a target document feature extraction model.

[0073] Figure 3 A schematic diagram of the structure of an initial document feature extraction model provided in the embodiment of this specification; for ease of understanding, combined with Figure 3 The training principle of the initial document feature extraction model is explained. Figure 3 As shown, the initial document feature extraction model may include: a video encoder 31 and a query converter 32. The query converter 32 may include: an image converter 304 and a text converter 305. Specifically, the video encoder 31 may be connected to the image converter 304, and the image converter 305 may be connected to the text converter 305.

[0074] Since the large language model is not involved in the stage of training the target document feature extraction model, the large language model may not be included in the initial document feature extraction model, that is, there is no need to connect the image converter 305 with the large language model.

[0075] In the embodiments of this specification, the core idea of ​​the query converter can be to improve the representation ability and information retrieval effect of the model by introducing a query mechanism. It can include three key parts: Input Embedding: converting the input image data and text data into vector representation; Query Generation: generating query embedding vectors for retrieval; Interaction Layer: based on the attention mechanism, realizing the interaction between the query embedding vector and the feature vector of the input data, thereby generating the final output representation.

[0076] Specifically, the input of the text converter 305 in the query converter 32 may be the description text for the first certificate sample. The text converter 305 may serve as both a text encoder and a text decoder to extract the text semantic feature vector of the description text for the first certificate sample.

[0077] The input of the image converter 304 can be a series of learnable query embedding vectors. By training the loss according to the first preset model, the model parameters are optimized by combining the query embedding vector, the first document visual feature vector extracted by the video encoder 31 for the first document image sequence, and the text semantic feature vector extracted by the query converter 32, so that the image converter 304 can extract the document visual feature vector that is most similar to the description text of the first document sample from the first document image sequence, thereby serving as the first query embedding vector.

[0078] In practical applications, the first preset model training loss may include: at least one of: image-text contrastive learning (ITC) loss, image-text matching (ITM) loss and image-grounded text generation (ITG) loss.

[0079] Among them, the image-text contrastive learning loss can be used to learn to align image representations and text representations so that their mutual information is maximized. In practical applications, this can be achieved by comparing the similarity of positive image-text pairs and negative image-text pairs. Image-based text generation losses are often used to train query transformers to generate text given an input image. Image-text matching losses usually aim to learn fine-grained alignment between image and text representations, which can be viewed as a binary classification loss, requiring the model to predict whether an image-text pair is positive (matching) or negative (mismatching).

[0080] In practical applications, a positive image-text pair may refer to a first training sample including a first ID card image sequence of a first ID card sample and a description text for the first ID card sample having a high degree of match, while a negative image-text pair may refer to a first training sample including a first ID card image sequence of a first ID card sample and a description text for the first ID card sample having a low degree of match, and there is no specific limitation on this.

[0081] In a feasible implementation, the video encoder may include: an embedding layer in a video visual self-attention model, a spatial encoder in the video visual self-attention model, and a pre-trained image encoder whose parameters are in a frozen state.

[0082] Wherein, the embedding layer may be connected to the spatial encoder, the spatial encoder may be connected to the pre-trained image encoder, and the pre-trained image encoder may be connected to the image converter in the query converter.

[0083] The embedding layer may be configured to receive the first document image sequence and output an embedding vector of the first document image sequence.

[0084] The spatial encoder may be used to perform spatial encoding processing on the embedded vector to obtain an initial visual feature vector of the first document image sequence.

[0085] The pre-trained image encoder may be used to perform encoding processing on the initial visual feature vector to obtain the first document visual feature vector of the first document image sequence.

[0086] For ease of understanding, combined Figure 3 Explain the working principle of the video encoder. Figure 3 As shown, the video encoder 31 may include: an embedding layer 301 in the video visual self-attention model, a spatial encoder 302 in the video visual self-attention model, and a pre-trained image encoder 303 whose parameters are in a frozen state, which are connected in sequence. When the first document image sequence is input into the video encoder 31, it is processed by the embedding layer 301, the spatial encoder 302 and the pre-trained image encoder 303 in sequence, so that the first document visual feature vector corresponding to the first document image sequence can be extracted. By connecting the pre-trained image encoder 303 with the image converter 304 in the query converter 32, the image converter 304 can obtain the first document visual feature vector from the pre-trained image encoder 303, and then generate the first query embedding vector, which will not be described in detail.

[0087] In the embodiments of this specification, the video vision self-attention model (Video Vision Transformer, ViViT) is a Transformer-based video processing model, which can capture the spatiotemporal dependencies in the video by utilizing the Transformer's self-attention mechanism, so that the ViViT model performs well in video processing tasks. In practical applications, the video vision self-attention model can usually include: an embedding layer, a spatial encoder (SpatialTransformer Encoder), a temporal encoder Temporal Transformer Encoder, and a classification module connected in sequence.

[0088] Since it is necessary to ensure that relatively accurate visual features of the document image are extracted in the embodiments of this specification, and it is also necessary to save resources consumed in the model training process and reduce the time consumption of model training, the embedding layer and spatial encoder in the video visual self-attention model can be used to extract the initial visual feature vector of the first document image sequence, and then the pre-trained image encoder is used to further process the initial visual feature vector to obtain the first document visual feature vector of the first document image sequence.

[0089] In practical applications, pre-trained embedding layers and spatial encoders can also be used to build the video encoder in the initial document feature extraction model. At this time, the embedding layer and spatial encoder in the video encoder can be in a parameter frozen state to improve the model training efficiency; or they can be in a parameter unfrozen state so that their parameters can be optimized during the model training process. There is no specific limitation on this.

[0090] Optionally, the pre-trained image encoder may be a pre-trained image encoder in a pre-trained multimodal visual-textual large language model, the query converter may be a trainable module in the pre-trained multimodal visual-textual large language model, and the large language model may be a pre-trained large language model in the pre-trained multimodal visual-textual large language model.

[0091] In the embodiments of this specification, the pre-trained multimodal visual-textual large language model (BLIP-2) is a general and efficient visual language pre-training model, which aims to freeze the parameters of the pre-trained image encoder and the large language model, and use the query converter (Q-Former) to extract useful visual features from the frozen image encoder using a set of learnable query embedding vectors, and use them as the input required for the large language model to output text, thereby connecting the information bottleneck between the frozen image encoder and the frozen large language model to achieve image-text alignment. Since the generalization ability and model performance of the pre-trained multimodal visual-textual large language model are good, and by freezing the parameters of the image encoder and the large language model, the computational cost can be effectively reduced, therefore, the pre-trained multimodal visual-textual large language model (BLIP-2) can be used to build a target document feature extraction model and a target document anti-counterfeiting detection model, which will not be elaborated on.

[0092] In a feasible implementation manner, the initial document anti-counterfeiting detection model may further include: a preset fully connected layer.

[0093] The output layer of the image converter in the target document feature extraction model can be connected to the preset fully connected layer, and the preset fully connected layer can also be connected to the input layer of the large language model.

[0094] Correspondingly, step 208: using the second training sample to train the initial document anti-counterfeiting detection model to obtain the target document anti-counterfeiting detection model may include:

[0095] The second certificate image sequence in the second certificate sample is input into the video encoder in the target certificate feature extraction model to obtain a second certificate visual feature vector output by the video encoder.

[0096] The second document visual feature vector is input into the image converter in the target document feature extraction model to obtain a second query embedding vector output by the image converter in the target document feature extraction model.

[0097] The second query embedding vector is input into the preset fully connected layer to obtain a target document feature vector of a preset dimension output by the preset fully connected layer.

[0098] According to the second preset model training loss, according to the document anti-counterfeiting detection result generated by the large language model based on the target document feature vector and the preset prompt words in the second training sample, parameters of the initial document anti-counterfeiting detection model are optimized to obtain the target document anti-counterfeiting detection model.

[0099] Figure 4This is a schematic diagram of the structure of an initial certificate anti-counterfeiting detection model provided in the embodiment of this specification. Figure 4 The content in this article explains the training principle of the initial document anti-counterfeiting detection model. Figure 4 The initial document anti-counterfeiting detection model can mainly include three parts, namely: a target document feature extraction model 01, a preset fully connected layer 02 and a large language model 03.

[0100] Among them, the target document feature extraction model 01 may include: a video encoder 41 and a query converter 42. The video encoder 41 may specifically include: an embedding layer 401 in a video visual self-attention model, a spatial encoder 402, and a pre-trained image encoder 403 in a pre-trained multimodal visual-text large language model. The query converter 404 may be a trainable module in a pre-trained multimodal visual-text large language model. In practical applications, since the query converter 404 does not have text input in the current model training stage, the query converter 404 may only include the trained image converter 404, or may also include the trained query converter 404 and the text converter ( Figure 4 ), which is not specifically limited.

[0101] When the second certificate image sequence in the second training sample is input into the target certificate feature extraction model, and is processed in sequence by the embedding layer 401, the spatial encoder 402, the pre-trained image encoder 403 and the image converter 404, the second query embedding vector extracted by the image converter 404 can be input into the preset fully connected layer for feature dimension transformation, thereby meeting the feature requirements of the large language model for the input data, so that the large language model can combine the received certificate visual feature data with the preset prompt words to generate corresponding certificate anti-counterfeiting detection results.

[0102] In practical applications, the second preset model training loss can be used to measure the difference between the document anti-counterfeiting detection results generated by the large language model and the benchmark document anti-counterfeiting detection results in the preset prompt words. Alternatively, it can be set according to actual needs. It is only necessary to ensure that the target document anti-counterfeiting detection model obtained by optimizing the parameters of the initial document anti-counterfeiting detection model according to the second preset model training loss has good performance and can generate document anti-counterfeiting detection results with better accuracy. There is no specific limitation on this.

[0103] Optionally, the preset prompt word can be used to guide the large language model to output a preset type of document description information for the second document sample; the document description information can be used to generate a final document anti-counterfeiting detection result for at least one anti-counterfeiting detection task for the second document sample. Since the implementation principle of generating the final document anti-counterfeiting detection result of each anti-counterfeiting detection task according to the document description information will be explained in detail in the subsequent embodiments of the document anti-counterfeiting detection method, it will not be repeated here.

[0104] Optionally, at least some parameters of the video encoder in the target document feature extraction model are in a frozen state; and parameters of the image converter in the target document feature extraction model are in an unfrozen state.

[0105] It can be understood that when the second training sample is used to optimize the parameters of the initial document anti-counterfeiting detection model, it is usually mainly used to optimize the parameters of the image converter at the target document feature extraction model within the initial document anti-counterfeiting detection model. Therefore, the parameters of the image converter can usually be in an unfrozen state.

[0106] Also, since the parameters of the video encoder at the target document feature extraction model within the initial document anti-counterfeiting detection model have been optimized when training the initial document feature extraction model, the parameters of the video encoder can be frozen at this time to improve the model training efficiency. Of course, some of its parameters can also be unfrozen, which is beneficial to improving model performance. There is no specific limitation on this.

[0107] Based on the same idea, the embodiment of this specification also provides a document anti-counterfeiting detection method. Figure 5 , is a flow chart of a document anti-counterfeiting detection method provided in an embodiment of this specification. The execution subject of the process can be a user's terminal device or a server device, or an application installed on the user's terminal device or the server device. Figure 5 The process shown is described in detail, and the document anti-counterfeiting detection method may specifically include the following steps:

[0108] Step 502: Obtain a sequence of ID images of the user's target ID.

[0109] In the embodiment of this specification, when it is necessary to perform anti-counterfeiting detection on the target certificate of the user, a certificate image sequence obtained by collecting images of the target certificate can be obtained. Figure 2The target certificate anti-counterfeiting detection model trained by the method processes the certificate image sequence of the target certificate. Therefore, the certificate image sequence of the target certificate in step 502 can be made consistent with the acquisition conditions of the certificate image sequence contained in the training samples in steps 202 and 206; for example, it can include but is not limited to: images of the target certificate acquired under the illumination of a flash device, images of the target certificate acquired without being illuminated by a flash device, images of the target certificate acquired from a frontal perspective, images of the target certificate acquired from a non-frontal perspective, etc., without specific limitation.

[0110] In practical applications, when Figure 5 When the execution subject of the solution is the user's terminal device, step 502 can be implemented by using the image acquisition device at the terminal device; Figure 5 When the executor of the scheme is a server device, it can communicate with the terminal device, thereby obtaining the document image sequence of the target document from the terminal device to implement step 502; of course, step 502 can also be implemented in other ways, which are not specifically limited.

[0111] Step 504, using the target certificate anti-counterfeiting detection model to process the certificate image sequence to obtain the certificate anti-counterfeiting detection result for the target certificate of the user; wherein the target certificate anti-counterfeiting detection model is a model trained by the above-mentioned certificate anti-counterfeiting detection model training method.

[0112] In the embodiment of this specification, the document image sequence of the target document can be input into the use Figure 2 The target certificate anti-counterfeiting detection model trained by the method can easily and quickly obtain the certificate anti-counterfeiting detection results of the target certificate for the user with good accuracy; and because the target certificate anti-counterfeiting detection model has good versatility, the target certificate anti-counterfeiting detection model can be made to specifically target the processing capabilities of various anti-counterfeiting detection tasks through the model training stage, so that a single target certificate anti-counterfeiting detection model can be used to meet the processing requirements of multiple anti-counterfeiting detection tasks, which is conducive to improving the versatility, flexibility and ease of use of the certificate anti-counterfeiting detection solution.

[0113] In a feasible implementation, step 504, using the target certificate anti-counterfeiting detection model to process the certificate image sequence to obtain the certificate anti-counterfeiting detection result for the target certificate of the user, may include:

[0114] The document image sequence is processed using a target document anti-counterfeiting detection model to obtain document description information of a preset type for the target document of the user.

[0115] According to the preset type of certificate description information, a final certificate anti-counterfeiting detection result of at least one anti-counterfeiting detection task for the target certificate of the user is generated.

[0116] Send at least one of the preset type of certificate description information and the final certificate anti-counterfeiting detection result to the user's terminal device.

[0117] In the embodiments of this specification, since the large language model can understand the meaning of language texts, learn the statistical laws and patterns of languages ​​through deep learning, and has strong generalization capabilities, it can generate coherent and accurate information of text types such as dialogues and articles. In addition, since it is necessary to use a single target certificate anti-counterfeiting detection model to meet the processing requirements of multiple anti-counterfeiting detection tasks, and the preset certificate authenticity identification conditions required for different anti-counterfeiting detection tasks may also be different, and in the process of business operations, the preset certificate authenticity identification conditions required for anti-counterfeiting detection tasks may also change, therefore, it is not necessary to make the target certificate anti-counterfeiting detection model directly output the final certificate anti-counterfeiting detection results of each anti-counterfeiting detection task, but it is possible to make the target certificate anti-counterfeiting detection model output the certificate description information of the preset type of the user's target certificate.

[0118] In practical applications, the preset categories of the document description information that the target document anti-counterfeiting detection model can output for the user's target document can be realized by configuring the preset prompt words used in the training process of the target document anti-counterfeiting detection model according to the actual situation, so as not to specifically limit the preset categories, meanings and forms of the document description information. For ease of understanding, an example is given to illustrate this. For example, the document description information of the preset types of the target document output by the target document anti-counterfeiting detection model may include but is not limited to: This is a sequence of images collected for a credible bachelor's degree certificate according to the front view and the view angles of 15 degrees and 30 degrees upward, the quality is clear, the lower left corner is blocked by a finger, and the document is not incomplete. Or, this is a sequence of images collected for the picture of the bachelor's degree certificate displayed on the electronic screen, the image clarity score is 70 points, the document has no punching problem, the document is not blocked by a finger, there is a risk of forged documents, etc., and no specific limitation is made to this.

[0119] Optionally, generating a final document anti-counterfeiting detection result of at least one anti-counterfeiting detection task for the target document of the user according to the document description information of the preset type may include:

[0120] For any of the anti-counterfeiting detection tasks, first document description information related to a preset document authenticity identification condition required for determining the anti-counterfeiting detection task is obtained from the preset type of document description information.

[0121] According to the first certificate description information, it is determined whether the preset certificate authenticity identification condition is met to obtain a first determination result.

[0122] A final document anti-counterfeiting detection result for the target document of the user under the anti-counterfeiting detection task is generated according to the first judgment result.

[0123] In the embodiments of this specification, the preset document authenticity identification conditions required for each anti-counterfeiting detection task can be pre-configured according to the implementation requirements, as well as the document description information involved in each preset document authenticity identification condition. In addition, it is also necessary to pre-train the model so that the target document anti-counterfeiting detection model has the ability to output the document description information involved in each preset document authenticity identification condition, so that it can be used in Figure 5 During the implementation of the middle scheme, the first document description information related to the preset document authenticity identification conditions required for the anti-counterfeiting detection task can be extracted from the preset type of document description information output by the target document anti-counterfeiting detection model according to the pre-configured operations and pre-configured information, and then the final document anti-counterfeiting detection result of the user's target document under the anti-counterfeiting detection task can be generated based on the first document description information, which is convenient and quick.

[0124] It is understandable that in order to improve the scalability of the anti-counterfeiting detection scheme, the target certificate anti-counterfeiting detection model, in addition to the ability to output the certificate description information involved in the currently set preset certificate authenticity identification conditions, can also have the additional ability to output richer certificate description information such as type, content, and form, so that comprehensive certificate description information can be used in the future to promptly meet new or changed certificate anti-counterfeiting detection needs without the need to re-train the target certificate anti-counterfeiting detection model, which is conducive to improving the flexibility and versatility of the certificate anti-counterfeiting detection scheme, reducing the cost of implementing the scheme, and having good practicality.

[0125] In actual applications, in order to facilitate the user to know the anti-counterfeiting detection progress of an individual's target certificate, the preset type of certificate description information generated by the target certificate anti-counterfeiting detection model for the certificate image sequence of its target certificate, and / or at least one of the final certificate anti-counterfeiting detection results corresponding to the anti-counterfeiting detection task for the user's target certificate can also be sent to the user's terminal device, which is conducive to improving the user experience.

[0126] In a feasible implementation manner, after generating the final document anti-counterfeiting detection result for the target document of the user under the anti-counterfeiting detection task according to the first judgment result, the method may further include:

[0127] Obtain an update request for preset document authenticity identification conditions required for the anti-counterfeiting detection task.

[0128] According to the update request, the authenticity identification condition of the updated document required for judging the anti-counterfeiting detection task is determined.

[0129] From the preset type of certificate description information, obtain the second certificate description information involved in the updated certificate authenticity identification condition.

[0130] According to the second certificate description information, it is determined whether the updated certificate authenticity identification condition is met to obtain a second determination result.

[0131] According to the second judgment result, an updated document anti-counterfeiting detection result is generated for the target document of the user under the anti-counterfeiting detection task.

[0132] In a feasible implementation manner, after generating the final document anti-counterfeiting detection result of at least one anti-counterfeiting detection task for the target document of the user according to the document description information of the preset type, the following may also be included:

[0133] Get a new request for the anti-counterfeiting detection task.

[0134] According to the newly added request, the target document authenticity identification conditions required for the newly added anti-counterfeiting detection task are determined.

[0135] The third certificate description information involved in the target certificate authenticity identification condition is obtained from the preset type of certificate description information.

[0136] According to the third certificate description information, it is determined whether the target certificate authenticity identification condition is met to obtain a third determination result.

[0137] According to the third judgment result, a new document anti-counterfeiting detection result is generated for the target document of the user under the new anti-counterfeiting detection task.

[0138] In the embodiments of the present specification, after the document image sequence of the user's target document is processed using the target document anti-counterfeiting detection model to obtain the document description information of a preset type for the user's target document, in addition to being able to immediately generate the current document anti-counterfeiting detection result for the user's target document based on the document description information of the preset type, the document description information of the preset type for the user's target document can also be stored to facilitate changes in the preset document authenticity identification conditions required for the anti-counterfeiting detection task to be determined, or when there is a new anti-counterfeiting detection task, the document description information of the preset type for the user's target document generated and stored in the early stage can be called to execute the latest version of the anti-counterfeiting detection task in real time, so that in different business scenarios, different document anti-counterfeiting detection strategies can be efficiently and conveniently configured to quickly and in real time identify and defend risks, which is conducive to ensuring the security of business operations.

[0139] Based on the same idea, the embodiments of this specification also provide a device corresponding to the above method. Figure 6 , is a schematic diagram of the structure of a training device for a certificate anti-counterfeiting detection model provided in an embodiment of this specification. Figure 6 As shown, the training device 6 of the certificate anti-counterfeiting detection model can be implemented as all or part of the electronic device through software, hardware or a combination of both. According to some embodiments, the training device 6 of the certificate anti-counterfeiting detection model may include a first acquisition module 61, a first training module 62, a second acquisition module 63 and a second training module 64, wherein:

[0140] The first acquisition module 61 is used to acquire a first training sample generated based on a first certificate sample; the first training sample includes a first certificate image sequence collected for the first certificate sample, and a description text for the first certificate sample.

[0141] The first training module 62 is used to train the initial document feature extraction model using the first training sample to obtain a target document feature extraction model; wherein the initial document feature extraction model includes: a video encoder connected in sequence, at least some parameters of which are in a frozen state, and a query converter.

[0142] The second acquisition module 63 is used to acquire a second training sample generated based on the second certificate sample; the second training sample includes a second certificate image sequence collected for the second certificate sample, and a preset prompt word for guiding the large language model to output the certificate anti-counterfeiting detection result for the second certificate sample.

[0143] The second training module 64 is used to train the initial certificate anti-counterfeiting detection model using the second training sample to obtain a target certificate anti-counterfeiting detection model; wherein the initial certificate anti-counterfeiting detection model includes: the target certificate feature extraction model connected in sequence, and the large language model whose parameters are in a frozen state.

[0144] Optionally, the query converter in the initial document feature extraction model includes: an image converter and a text converter. The video encoder in the initial document feature extraction model is connected to the image converter, and the image converter is connected to the text converter;

[0145] The first training module 62 may be specifically used for:

[0146] The first document image sequence in the first document sample is input into the video encoder to obtain a first document visual feature vector output by the video encoder.

[0147] The description text for the first certificate sample is input into the text converter to obtain a text semantic feature vector output by the text converter.

[0148] According to the first preset model training loss, based on the first query embedding vector generated by the image converter based on the first document visual feature vector and the text semantic feature vector, parameters of the initial document feature extraction model are optimized to obtain a target document feature extraction model.

[0149] Optionally, the video encoder may include: an embedding layer in a video visual self-attention model, a spatial encoder in the video visual self-attention model, and a pre-trained image encoder whose parameters are in a frozen state.

[0150] Wherein, the embedding layer is connected to the spatial encoder, the spatial encoder is connected to the pre-trained image encoder, and the pre-trained image encoder is connected to the image converter in the query converter.

[0151] The embedding layer is used to receive the first document image sequence and output an embedding vector of the first document image sequence.

[0152] The spatial encoder is used to perform spatial encoding processing on the embedded vector to obtain an initial visual feature vector of the first document image sequence.

[0153] The pre-trained image encoder is used to perform encoding processing on the initial visual feature vector to obtain the first document visual feature vector of the first document image sequence.

[0154] Optionally, the pre-trained image encoder is a pre-trained image encoder in a pre-trained multimodal visual-textual large language model, the query converter is a trainable module in the pre-trained multimodal visual-textual large language model, and the large language model is a pre-trained large language model in the pre-trained multimodal visual-textual large language model.

[0155] Optionally, the first preset model training loss may include: at least one of: image-text contrast learning loss, image-text matching loss and image-based text generation loss.

[0156] Optionally, the initial document anti-counterfeiting detection model may further include: a preset fully connected layer.

[0157] The output layer of the image converter in the target document feature extraction model is connected to the preset fully connected layer, and the preset fully connected layer is also connected to the input layer of the large language model.

[0158] The second training module 64 may include:

[0159] The second certificate image sequence in the second certificate sample is input into the video encoder in the target certificate feature extraction model to obtain a second certificate visual feature vector output by the video encoder.

[0160] The second document visual feature vector is input into the image converter in the target document feature extraction model to obtain a second query embedding vector output by the image converter in the target document feature extraction model.

[0161] The second query embedding vector is input into the preset fully connected layer to obtain a target document feature vector of a preset dimension output by the preset fully connected layer.

[0162] According to the second preset model training loss, according to the document anti-counterfeiting detection result generated by the large language model based on the target document feature vector and the preset prompt words in the second training sample, parameters of the initial document anti-counterfeiting detection model are optimized to obtain the target document anti-counterfeiting detection model.

[0163] Optionally, the preset prompt word is specifically used to guide the large language model to output a preset type of certificate description information for the second certificate sample; the certificate description information is used to generate a final certificate anti-counterfeiting detection result of at least one anti-counterfeiting detection task for the second certificate sample.

[0164] Optionally, at least some parameters of the video encoder in the target document feature extraction model are in a frozen state; and parameters of the image converter in the target document feature extraction model are in an unfrozen state.

[0165] Based on the same idea, the embodiments of this specification also provide a device corresponding to the above method. Figure 7 , is a schematic diagram of the structure of a document anti-counterfeiting detection device provided in the embodiments of this specification. Figure 7 As shown, the document anti-counterfeiting detection device 7 can be implemented as all or part of the electronic device through software, hardware or a combination of both. According to some embodiments, the document anti-counterfeiting detection device 7 may include a document image acquisition module 71 and an anti-counterfeiting detection module 72, wherein:

[0166] The certificate image acquisition module 71 is used to acquire a sequence of certificate images of a user's target certificate.

[0167] The anti-counterfeiting detection module 72 is used to process the document image sequence using the target document anti-counterfeiting detection model to obtain the document anti-counterfeiting detection result for the target document of the user; wherein the target document anti-counterfeiting detection model is a model trained using the above-mentioned document anti-counterfeiting detection model training method.

[0168] Optionally, the anti-counterfeiting detection module 72 may include:

[0169] The image sequence processing unit is used to process the document image sequence using the target document anti-counterfeiting detection model to obtain document description information of a preset type for the target document of the user.

[0170] The detection result generating unit is used to generate a final document anti-counterfeiting detection result of at least one anti-counterfeiting detection task for the target document of the user according to the document description information of the preset type.

[0171] Optionally, the detection result generating unit may be specifically used for:

[0172] For any of the anti-counterfeiting detection tasks, first document description information related to a preset document authenticity identification condition required for determining the anti-counterfeiting detection task is obtained from the preset type of document description information.

[0173] According to the first certificate description information, it is determined whether the preset certificate authenticity identification condition is met to obtain a first determination result.

[0174] A final document anti-counterfeiting detection result for the target document of the user under the anti-counterfeiting detection task is generated according to the first judgment result.

[0175] Optional, Figure 7 The device described in the above method may further include:

[0176] The update request acquisition module is used to obtain an update request for the preset document authenticity identification conditions required for the anti-counterfeiting detection task.

[0177] The first determination module is used to determine the authenticity identification conditions of the updated certificate required for the anti-counterfeiting detection task according to the update request.

[0178] The first screening module is used to obtain the second certificate description information involved in the updated certificate authenticity identification condition from the preset type of certificate description information.

[0179] The second judgment result generating module is used to judge whether the authenticity identification condition of the updated certificate is met according to the second certificate description information, and obtain a second judgment result.

[0180] The detection result updating module is used to generate an updated document anti-counterfeiting detection result for the target document of the user under the anti-counterfeiting detection task according to the second judgment result.

[0181] Optional, Figure 7 The device described in the above method may further include:

[0182] The newly added request acquisition module is used to obtain the newly added request for the anti-counterfeiting detection task.

[0183] A second determination module is used to determine the target certificate authenticity identification conditions required for the newly added anti-counterfeiting detection task according to the newly added request;

[0184] A second screening module is used to obtain the third certificate description information involved in the target certificate authenticity identification condition from the preset type of certificate description information;

[0185] A third judgment result generating module, used to judge whether the target certificate authenticity identification condition is met according to the third certificate description information, and obtain a third judgment result;

[0186] The detection result adding module is used to generate a new document anti-counterfeiting detection result for the target document of the user under the new anti-counterfeiting detection task according to the third judgment result.

[0187] Optional, Figure 7 The device described in the above method may further include:

[0188] A generating module is used to send at least one of the preset type of certificate description information and the final certificate anti-counterfeiting detection result to the user's terminal device.

[0189] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0190] The present specification also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above Figure 2 and Figure 5 For the method shown, the specific execution process can refer to the specific description of the relevant embodiments of the above method, which will not be repeated here.

[0191] The present specification also provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded and executed by the processor as described above. Figure 2 and Figure 5 For the method shown, the specific execution process can refer to the specific description of the relevant embodiments of the above method, which will not be repeated here.

[0192] The embodiments of this specification also provide Figure 8 The structural diagram of the electronic device shown in FIG. Figure 8At the hardware level, the electronic device may include a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 2 and Figure 5 For the method shown, the specific execution process can refer to the specific description of the relevant embodiments of the above method, which will not be repeated here.

[0193] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0194] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for computer-readable storage media, computer program products and Figure 8 As for the electronic device shown, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0195] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0196] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0197] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0198] For the convenience of description, the above device is described by dividing it into various units according to its functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0199] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0200] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0201] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0202] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0203] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0204] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0205] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0206] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0207] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0208] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media including storage devices.

[0209] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0210] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for training a document anti-counterfeiting detection model, the method comprising: Obtaining a first training sample generated based on the first certificate sample; The first training sample includes a first document image sequence collected for the first document sample, and a description text for the first document sample; The initial document feature extraction model is trained using the first training sample to obtain a target document feature extraction model; wherein the initial document feature extraction model comprises: a video encoder whose at least some parameters are in a frozen state and a query converter connected in sequence; Acquire a second training sample generated based on the second certificate sample; the second training sample includes a second certificate image sequence collected for the second certificate sample, and a preset prompt word for guiding the large language model to output a certificate anti-counterfeiting detection result for the second certificate sample; The initial document anti-counterfeiting detection model is trained using the second training sample to obtain a target document anti-counterfeiting detection model; wherein the initial document anti-counterfeiting detection model includes: the target document feature extraction model connected in sequence, and the large language model whose parameters are in a frozen state.

2. According to the method of claim 1, the query converter in the initial document feature extraction model comprises: Image converter and text converter; The video encoder in the initial document feature extraction model is connected to the image converter, and the image converter is connected to the text converter; The step of training the initial document feature extraction model using the first training sample to obtain a target document feature extraction model includes: Inputting the first document image sequence in the first document sample into the video encoder to obtain a first document visual feature vector output by the video encoder; Inputting the description text for the first certificate sample into the text converter to obtain a text semantic feature vector output by the text converter; According to the first preset model training loss, based on the first query embedding vector generated by the image converter based on the first document visual feature vector and the text semantic feature vector, parameters of the initial document feature extraction model are optimized to obtain a target document feature extraction model.

3. The method according to claim 2, wherein the video encoder comprises: An embedding layer in a video vision self-attention model, a spatial encoder in the video vision self-attention model, and a pre-trained image encoder with parameters in a frozen state; wherein the embedding layer is connected to the spatial encoder, the spatial encoder is connected to the pre-trained image encoder, and the pre-trained image encoder is connected to the image converter in the query converter; The embedding layer is used to receive the first document image sequence and output an embedding vector of the first document image sequence; The spatial encoder is used to perform spatial encoding processing on the embedded vector to obtain an initial visual feature vector of the first document image sequence; The pre-trained image encoder is used to perform encoding processing on the initial visual feature vector to obtain the first document visual feature vector of the first document image sequence.

4. According to the method of claim 3, the pre-trained image encoder is a pre-trained image encoder in a pre-trained multimodal visual-textual large language model, the query converter is a trainable module in the pre-trained multimodal visual-textual large language model, and the large language model is a pre-trained large language model in the pre-trained multimodal visual-textual large language model.

5. According to the method of claim 2, the first preset model training loss comprises: At least one of image-text contrastive learning loss, image-text matching loss, and image-based text generation loss.

6. According to the method according to any one of claims 2 to 5, the initial document anti-counterfeiting detection model further comprises: Preset fully connected layer; The output layer of the image converter in the target document feature extraction model is connected to the preset fully connected layer, and the preset fully connected layer is also connected to the input layer of the large language model; The step of training the initial document anti-counterfeiting detection model using the second training sample to obtain a target document anti-counterfeiting detection model includes: Inputting the second certificate image sequence in the second certificate sample into the video encoder in the target certificate feature extraction model to obtain a second certificate visual feature vector output by the video encoder; Inputting the second document visual feature vector into the image converter in the target document feature extraction model to obtain a second query embedding vector output by the image converter in the target document feature extraction model; Inputting the second query embedding vector into the preset fully connected layer to obtain a target document feature vector of a preset dimension output by the preset fully connected layer; According to the second preset model training loss, according to the document anti-counterfeiting detection result generated by the large language model based on the target document feature vector and the preset prompt words in the second training sample, parameters of the initial document anti-counterfeiting detection model are optimized to obtain the target document anti-counterfeiting detection model.

7. According to the method of claim 6, the preset prompt word is specifically used to guide the large language model to output a preset type of certificate description information for the second certificate sample; the certificate description information is used to generate a final certificate anti-counterfeiting detection result for at least one anti-counterfeiting detection task for the second certificate sample.

8. According to the method of claim 6, at least part of the parameters of the video encoder in the target document feature extraction model are in a frozen state; and the parameters of the image converter in the target document feature extraction model are in an unfrozen state.

9. A document anti-counterfeiting detection method, the method comprising: Obtain the document image sequence of the user's target document; The document image sequence is processed using a target document anti-counterfeiting detection model to obtain a document anti-counterfeiting detection result for the target document of the user; wherein the target document anti-counterfeiting detection model is a model trained using the document anti-counterfeiting detection model training method described in any one of claims 1-8.

10. The method according to claim 9, wherein the step of processing the document image sequence using the target document anti-counterfeiting detection model to obtain the document anti-counterfeiting detection result for the target document of the user comprises: Processing the document image sequence using a target document anti-counterfeiting detection model to obtain document description information of a preset type for the target document of the user; According to the preset type of certificate description information, a final certificate anti-counterfeiting detection result of at least one anti-counterfeiting detection task for the target certificate of the user is generated.

11. The method according to claim 10, wherein generating a final document anti-counterfeiting detection result of at least one anti-counterfeiting detection task for the target document of the user according to the document description information of the preset type comprises: For any of the anti-counterfeiting detection tasks, obtaining first document description information related to a preset document authenticity identification condition required for judging the anti-counterfeiting detection task from the preset type of document description information; According to the first certificate description information, determine whether the preset certificate authenticity identification condition is met to obtain a first determination result; According to the first judgment result, a final document anti-counterfeiting detection result for the target document of the user under the anti-counterfeiting detection task is generated.

12. The method according to claim 11, after generating the final document anti-counterfeiting detection result for the target document of the user under the anti-counterfeiting detection task according to the first judgment result, further comprising: Obtaining an update request for preset document authenticity identification conditions required for the anti-counterfeiting detection task; Determining, according to the update request, the authenticity identification condition of the updated document required for the anti-counterfeiting detection task; Acquire second certificate description information related to the updated certificate authenticity identification condition from the preset type of certificate description information; According to the second certificate description information, determine whether the updated certificate authenticity identification condition is met to obtain a second determination result; According to the second judgment result, an updated document anti-counterfeiting detection result is generated for the target document of the user under the anti-counterfeiting detection task.

13. The method according to claim 10 or 11, after generating a final document anti-counterfeiting detection result of at least one anti-counterfeiting detection task for the target document of the user according to the document description information of the preset type, further comprising: Obtaining a new request for the anti-counterfeiting detection task; According to the newly added request, determining the target document authenticity identification conditions required for the newly added anti-counterfeiting detection task; Acquire the third certificate description information involved in the target certificate authenticity identification condition from the preset type of certificate description information; According to the third certificate description information, determine whether the target certificate authenticity identification condition is met to obtain a third determination result; According to the third judgment result, a new document anti-counterfeiting detection result is generated for the target document of the user under the new anti-counterfeiting detection task.

14. The method according to claim 10 or 11, further comprising: Send at least one of the preset type of certificate description information and the final certificate anti-counterfeiting detection result to the user's terminal device.

15. A training device for a document anti-counterfeiting detection model, comprising: A first acquisition module, used to acquire a first training sample generated based on a first certificate sample; The first training sample includes a first document image sequence collected for the first document sample, and a description text for the first document sample; A first training module, used to train an initial document feature extraction model using the first training sample to obtain a target document feature extraction model; wherein the initial document feature extraction model includes: a video encoder whose at least some parameters are in a frozen state and a query converter connected in sequence; A second acquisition module is used to acquire a second training sample generated based on a second certificate sample; the second training sample includes a second certificate image sequence collected for the second certificate sample, and a preset prompt word for guiding the large language model to output a certificate anti-counterfeiting detection result for the second certificate sample; The second training module is used to train the initial certificate anti-counterfeiting detection model using the second training sample to obtain a target certificate anti-counterfeiting detection model; wherein the initial certificate anti-counterfeiting detection model includes: the target certificate feature extraction model connected in sequence, and the large language model whose parameters are in a frozen state.

16. A document anti-counterfeiting detection device, comprising: The certificate image acquisition module is used to acquire the certificate image sequence of the user's target certificate; An anti-counterfeiting detection module is used to process the document image sequence using a target document anti-counterfeiting detection model to obtain a document anti-counterfeiting detection result for the target document of the user; wherein the target document anti-counterfeiting detection model is a model trained using the document anti-counterfeiting detection model training method described in any one of claims 1-8.

17. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 14.

18. An electronic device, comprising: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method as claimed in any one of claims 1 to 14.

19. A computer program product having at least one instruction stored thereon, wherein when the at least one instruction is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

Images