Method and system for self-service document signing based on fusion verification

By adopting an identity fusion verification neural network in the self-service document signing system, combining two-way iterative delivery and selective focus strategies, extracting features in the face image, the problem of insufficient identity verification security and accuracy of existing systems is solved, and the implementation of the problem of realizing the security and accuracy of authentication is achieved. More efficient and accurate authentication and user experience.

CN118658193BActive Publication Date: 2024-11-26SICHUAN ZHONGHE YUESHENG TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411132682.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-11-26
Estimated Expiration
2044-08-19

AI Technical Summary

Technical Problem

The existing self-service document sign-up system based on converged verification has insufficient security and accuracy in identity verification. Especially under complex sign-up environments and diversified needs, a single identity verification method is difficult to meet efficient and accurate needs, and at the same time lacks people. In-depth mining and analysis of key information in face images.

Method used

Using an identity fusion verification neural network method, the global and local features in the face image are extracted through two-way iterative transmission and selective focus strategies, and combined with the characterization vector extraction module, filtering module, downsampling module and dense connection module to perform identity Verification to improve the accuracy and security of identity verification.

Benefits of technology

Significantly improve the accuracy and system security of identity verification, ensuring that subsequent face image collection and verification will be carried out only when the target identity ID is consistent with the identity ID in the request, effectively preventing identity impersonation and illegal signing, and improving user experience and system performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118658193B_ABST
    Figure CN118658193B_ABST
Patent Text Reader

Abstract

The present application provides a method and system for self-service document signing based on fusion verification. When the user ID verification is passed, the face image to be verified is loaded into the identity fusion verification neural network to perform face identity detection operation. When debugging the network, the face image learning sample is loaded into a pre-prepared neural network, which includes a first component and a second component. The first component is used to extract and debug the image content representation vector of the face image learning sample to obtain the first face identity information; the second component is used to extract and debug the focus image representation vector of the face image learning sample based on the selective focusing strategy to obtain the second face identity information; the preset face identity information is determined through the first face identity information and the second face identity information; the error between the preset face identity information and the face identity prior information is used to tune the pre-prepared neural network to obtain the final network, and the accuracy of the identity fusion verification neural network is greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the fields of image processing and machine learning, and more specifically, to a method and system for self-service document signing based on fusion verification. Background Art

[0002] With the rapid development of information technology and the acceleration of digitalization, document signing, as an indispensable part of daily life and work, is gradually moving towards self-service and intelligentization. However, in the existing document self-service signing system based on fusion verification, there are still some problems that need to be solved.

[0003] The traditional method of document signing usually relies on manual review and confirmation, which is not only inefficient but also easily affected by human factors, resulting in a cumbersome and error-prone signing process. In order to improve the efficiency and accuracy of signing, some self-service signing systems have begun to introduce automation technology in recent years, such as using scanning devices to automatically read document information, or completing the signing process through simple identity verification methods. However, these existing self-service signing systems often have limitations in identity verification. Some systems may only rely on simple passwords, fingerprints, or a single biometric recognition technology for identity verification, and there is still room for improvement in terms of security and accuracy. In particular, when faced with complex signing environments and diverse signing requirements, a single identity verification method may seem inadequate.

[0004] In addition, when processing facial images, existing self-service sign-in systems often lack in-depth mining and analysis of key information in images. As facial recognition technology becomes increasingly popular, how to extract richer and more accurate identity information from facial images through more advanced algorithms and technologies is the key to improving the performance of self-service sign-in systems. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for self-service document signing based on fusion verification. The technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for self-service document signing based on fusion verification, which is applied to a server in a self-service document signing system based on fusion verification, wherein the self-service document signing system based on fusion verification also includes a signing terminal that is communicatively connected to the server, and the method includes:

[0007] In response to a document signing request initiated by the signing terminal, the document signing request includes an identity ID of a target object;

[0008] When the identity ID is consistent with the target identity ID of the document to be signed, an identity verification instruction is generated, and the identity verification instruction is sent to the signing terminal, so that the signing terminal can collect the face image of the target object according to the identity verification instruction;

[0009] Receiving the facial image of the target object uploaded by the signing terminal;

[0010] Using the facial image as a facial image to be verified;

[0011] Loading the face image to be verified into the identity fusion verification neural network to perform face identity detection operation, and obtaining the target face identity information corresponding to the face image to be verified;

[0012] The debugging process of the identity fusion verification neural network includes:

[0013] Obtain learning samples of face images with prior information of face identity;

[0014] Loading the face image learning sample into a pre-prepared neural network, the pre-prepared neural network comprising a first component and a second component; bidirectional iterative transmission is adopted between hidden layers of the first component, and the second component is established by a selective focusing strategy;

[0015] Extracting and debugging the image content representation vector of the face image learning sample through the first component to obtain first face identity information;

[0016] By using the second component and according to the selective focusing strategy, extracting and debugging the focused image representation vector of the face image learning sample, and obtaining the second face identity information;

[0017] Determining preset facial identity information through the first facial identity information and the second facial identity information;

[0018] The pre-prepared neural network is tuned by the error between the preset face identity information and the face identity prior information to obtain the identity fusion verification neural network.

[0019] As an implementation manner, the first component includes a representation vector extraction module, a filtering module, a downsampling module, and a dense connection module, and the first component is used to perform image content representation vector extraction and debugging on the face image learning sample to obtain the first face identity information, including:

[0020] Extracting the image content representation vector of the face image learning sample by the representation vector extraction module to obtain a set of region representation vectors and a set of image block representation vectors;

[0021] Performing a sliding window dot product operation on the region representation vector set by the filtering module to obtain a region feature distribution set;

[0022] Performing a sliding window dot product operation on the image block representation vector set by the filtering module to obtain an image block feature distribution set;

[0023] Performing an integration operation on the regional feature distribution set and the image block feature distribution set through the downsampling module to obtain an integrated representation vector;

[0024] The integrated representation vector is classified by the dense connection module to obtain the first face identity information.

[0025] As an implementation manner, the representation vector extraction module includes a visual word bag unit, a region description extraction unit, and an image block description extraction unit, and the image content representation vector of the face image learning sample is extracted by the representation vector extraction module to obtain a region representation vector set and an image block representation vector set, including:

[0026] Extracting image features of the face image learning sample through the visual word bag unit to obtain regional image features and image block features;

[0027] Extracting the region representation vector of the face image learning sample by the region description extraction unit to obtain a region descriptor;

[0028] Extracting the image block representation vector of the face image learning sample by the image block description extraction unit to obtain an image block descriptor;

[0029] Generate the region representation vector set by using the region image features and the region descriptor;

[0030] The image block representation vector set is generated by using the image block features and the image block descriptors.

[0031] As an implementation manner, the representation vector extraction module further includes an interactive long short-term memory network unit, and the generating of the region representation vector set by using the region image features and the region descriptor includes:

[0032] Extracting the semantic representation vector of the region descriptor through the interactive long short-term memory network unit to obtain an initial region representation vector;

[0033] Generate the region representation vector set by using the region image features and the initial region representation vector;

[0034] The step of generating the image block representation vector set by using the image block features and the image block descriptors includes:

[0035] Extracting the semantic representation vector of the image block descriptor through the interactive long short-term memory network unit to obtain an initial image block representation vector;

[0036] The image block representation vector set is generated by using the image block features and the initial image block representation vector.

[0037] As an implementation manner, the integrating operation on the regional feature distribution set and the image block feature distribution set by the downsampling module to obtain an integrated representation vector includes:

[0038] Performing feature dimension reduction processing on the regional feature distribution set by the downsampling module to obtain a single region representation vector;

[0039] Performing feature dimension reduction processing on the image block feature distribution set by the downsampling module to obtain a single image block representation vector;

[0040] The single region representation vector and the single image block representation vector are integrated by the downsampling module to obtain the integrated representation vector.

[0041] As an implementation mode, the second component includes an embedding mapping module, a restoration mapping module and a classifier module, and the focused image representation vector represents the intermediate layer state of the target time step; the second component extracts and debugs the focused image representation vector of the face image learning sample according to the selective focusing strategy to obtain the second face identity information, including:

[0042] Performing embedding mapping operations on the face image learning samples at each time step through the embedding mapping module to obtain a region representation vector sequence;

[0043] Determine the hidden variable to be observed through the region representation vector sequence;

[0044] Extracting semantic representation vectors from the region representation vector sequence by using the selective focusing strategy to obtain semantic embedding representation vectors;

[0045] Performing restoration mapping on the latent variable to be observed and the semantic embedding representation vector through the restoration mapping module to determine the intermediate layer state at the target time step;

[0046] Classifying the intermediate layer state through a classifier module to obtain the second face identity information;

[0047] The step of determining the latent variable to be observed by using the region representation vector sequence includes:

[0048] Generate an initial indefinite parameter set;

[0049] Determining the expectation and discreteness corresponding to the region characterization vector sequence through the initial indefinite parameter set;

[0050] According to the expectation and the discreteness, the hidden variable to be observed of any uncertain parameter in the initial uncertain parameter set is determined.

[0051] As an implementation manner, the previously prepared neural network is tuned by the error between the preset face identity information and the face identity prior information to obtain the identity fusion verification neural network, including:

[0052] Determining a target cost by using an error between the preset face identity information and the face identity prior information;

[0053] The network parameters of the neural network prepared in advance are optimized through the target cost to obtain the identity fusion verification neural network.

[0054] As an implementation manner, the first face identity information is used to indicate that the face identity information is a first preset confidence level of a face identity information sample, and the second face identity information is used to indicate that the face identity information is a second preset confidence level of the face identity information sample, and the determining of the preset face identity information through the first face identity information and the second face identity information includes: determining a confidence mean of the first preset confidence level and the second preset confidence level; and determining the confidence mean as the preset face identity information;

[0055] Alternatively, the method further includes: obtaining environmental parameters corresponding to the face image learning sample; determining a first set influence coefficient of the first face identity information and a second set influence coefficient of the second face identity information through the environmental parameters; determining the preset face identity information through the first face identity information and the second face identity information, including: determining the first face identity information through the multiplication result of the first face identity information and the first set influence coefficient; determining the second face identity information through the multiplication result of the second face identity information and the second set influence coefficient; determining the preset face identity information through the first face identity information and the second face identity information;

[0056] Alternatively, the pre-prepared neural network also includes a third neural network, and the method also includes: extracting a target face area in the face image learning sample; the target face area is a biometric core area; performing biometric extraction and debugging on the target face area through the third neural network to obtain third face identity information; determining the preset face identity information through the first face identity information and the second face identity information includes: determining the preset face identity information through the first face identity information, the second face identity information and the third face identity information.

[0057] As an embodiment, the method further includes:

[0058] Extracting and debugging the image content representation vector of the face image learning sample through the first component to obtain a first biometric feature;

[0059] By using the second component and according to the selective focusing strategy, extracting and debugging the focused image representation vector of the face image learning sample to obtain a second biometric feature;

[0060] Integrate the first biometric feature and the second biometric feature to obtain an integrated biometric feature;

[0061] The preset facial identity information is determined through the integrated biometric feature.

[0062] In a second aspect, the present application provides a self-service document signing system based on fusion verification, comprising a server and a signing terminal communicatively connected to the server, the server comprising a memory and a processor, the memory storing a computer program executable on the processor, and the processor implementing the above-described method when executing the computer program.

[0063] The beneficial effects of the present application include at least: the present application obtains a face image learning sample carrying a priori information of a face identity; loads the face image learning sample into a pre-prepared neural network, the pre-prepared neural network comprising a first component and a second component; bidirectional iterative transmission is adopted between hidden layers of the first component, and the second component is established through a selective focusing strategy; extracts and debugs the image content representation vector of the face image learning sample through the first component to obtain the first face identity information; extracts and debugs the focus image representation vector of the face image learning sample according to the selective focusing strategy through the second component to obtain the second face identity information; The preset face identity information is determined by using the information of the preset face identity information and the face identity prior information; the pre-prepared neural network is tuned through the error between the preset face identity information and the face identity prior information to obtain the identity fusion verification neural network; the first component adopts bidirectional iterative transmission between hidden layers, so that the network can maintain the semantics of the face image at all angles, and at the same time measure the context loaded into the image, thereby improving the semantic understanding ability of the network; the second component focuses on image generation and image feature extraction based on the selective focus strategy, so as to tune the network based on generative adversarial and improve the network accuracy; after the first component and the second component are integrated, the accuracy of the identity fusion verification neural network is greatly improved.

[0064] The face identity detection operation is performed through the identity fusion verification neural network. The network combines the global image content representation of the first component and the selective focus strategy of the second component, which can more accurately extract and analyze the key features in the face image, thereby effectively improving the accuracy of identity authentication. A strict identity authentication link is added to the document signing process to ensure that only when the target identity ID is consistent with the identity ID in the request, the subsequent face image collection and verification will be carried out, which effectively prevents identity fraud and illegal signing, and greatly enhances the security of the system. Through the self-service signing system, users can complete the document signing conveniently and quickly without waiting for manual review, which greatly saves users' time costs and improves user experience. In the training process of the neural network, by adopting bidirectional iterative transmission and selective focus strategies, useful information in the image can be more effectively extracted, and the interference of redundant features can be reduced, thereby optimizing the performance of the neural network and improving the speed and accuracy of face recognition.

[0065] Other features will be described in part in the following description. Those skilled in the art will discover these features in part upon inspection of the following content and drawings, or may learn these features through production or use. The features of the present application may be implemented and obtained by practicing or using various aspects of the methods, tools, and combinations listed in the detailed examples described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0067] Figure 1 It is a schematic diagram of the composition of the document self-service signing system based on fusion verification provided in an embodiment of the present application.

[0068] Figure 2 It is a flowchart of a method for self-service document signing based on fusion verification provided in an embodiment of the present application.

[0069] Figure 3 It is a schematic diagram of the debugging process of the identity fusion verification neural network provided in the embodiment of the present application.

[0070] Figure 4 It is a schematic diagram of the composition of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0072] In the following description, references to “some embodiments”, “as an implementation / scheme”, and “in one implementation” describe a subset of all possible embodiments, but it can be understood that “some embodiments”, “as an implementation / scheme”, and “in one implementation” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0073] In the following description, the terms "first\second\third" and the like are only used to distinguish similar objects and do not represent a specific order for the objects. It is understandable that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0074] The self-service document signing method based on fusion verification provided in the embodiment of the present application can be executed by a server in a self-service document signing system based on fusion verification, wherein the server can be a laptop, tablet computer, desktop computer, mobile device (e.g., mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device) and other types of terminals, and can also be implemented as a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0075] Figure 1 Schematic diagram of the application scenario of the self-service document signing method based on fusion verification provided in the embodiment of the present application. The self-service document signing system based on fusion verification provided in the embodiment of the present application includes multiple signing terminals 100, a network 200 and a server 300, and the multiple signing terminals 100 and the server 300 are connected to each other through the network 200. The server 300 is used to execute the method provided in the embodiment of the present application. Specifically, the embodiment of the present application provides a self-service document signing method based on fusion verification, which is applied to a server, such as Figure 2 As shown, the method includes:

[0076] Step S100: responding to a document signing request initiated by the signing terminal, wherein the document signing request includes an identity ID of a target object.

[0077] In actual application scenarios, when the target object signs for a document, the operation is performed on a self-service signing terminal. This signing terminal may be a touch screen device (such as a smartphone or a personal computer) with a display interface that prompts the target object to enter an identity ID, such as an ID number, a target object number, or other unique identifier. The target object enters the identity ID and confirms it, and the signing terminal sends a document signing request to the backend server. This request contains the identity ID entered by the target object. At this point, the server receives the request and is ready to proceed to the next step of identity authentication.

[0078] Step S200: When the identity ID is consistent with the target identity ID of the document to be signed, an identity verification instruction is generated and sent to the signing terminal so that the signing terminal can collect the face image of the target object according to the identity verification instruction.

[0079] Specifically, when the server receives the document signing request from the signing terminal and extracts the identity ID of the target object contained therein, it compares this identity ID with the target identity ID specified by the document to be signed. This comparison process is to ensure that the person requesting the document to be signed is the intended recipient of the document.

[0080] If the comparison result shows that the identity ID is consistent, it means that the person requesting the signature is qualified to receive the document. At this time, the server will generate an identity authentication instruction. This instruction is a clear operation requirement, telling the signing terminal that facial recognition verification is required next to further confirm the identity of the signatory. The server then sends this identity authentication instruction to the signing terminal. After receiving the instruction, the signing terminal starts the facial recognition function and prompts the target object to perform facial recognition operations, such as standing in front of the camera to ensure that the face is clearly visible. This process ensures that the signing of the document is not only based on the matching of the identity ID, but also combines facial recognition technology, thereby greatly improving the security and accuracy of the signing.

[0081] Step S300: receiving the face image of the target object uploaded by the signing terminal.

[0082] Step S400: taking the facial image as a facial image to be verified.

[0083] In step S300, the server receives the facial image of the target object uploaded by the signing terminal. This process is performed after the signing terminal has captured the facial image of the target object. The signing terminal will transmit the captured facial image to the server via the network for subsequent identity verification. Next, in step S400, the server sets the received facial image as the facial image to be verified. This means that this image will be used in the subsequent identity verification process to confirm the true identity of the person requesting to sign the document.

[0084] Step S500: Loading the face image to be verified into the identity fusion verification neural network to perform face identity detection operation, and obtaining the target face identity information corresponding to the face image to be verified.

[0085] Specifically, when the server receives the face image to be verified, it inputs the image into the identity fusion verification neural network. This neural network is specially trained for face identity detection, and it can identify the identity of a person by analyzing the facial features in the image. In practical applications, this neural network can use deep learning technology and be trained with a large amount of image data with face identity labels, so that it has the ability to accurately identify different facial features.

[0086] For example, suppose a government agency uses this document self-service signing system based on fusion verification. When a target object signs for an important document, face recognition is performed through the signing terminal. The signing terminal sends the collected face image to the server. In step S500, the server loads this face image into the identity fusion verification neural network. The neural network automatically extracts facial features in the image, such as the shape of the eyes, the outline of the nose, the size of the mouth, etc., and compares them with the known identity information stored in the database. Through the operation of the neural network, the server finally obtains the target face identity information corresponding to the face image to be verified. If this identity information is consistent with the target identity information specified by the document to be signed, the server will confirm that the identity of the signatory is valid, thereby allowing him to continue the signing process. This process not only improves the efficiency of document signing, but also greatly enhances the security and accuracy of the signing process, ensuring that the document can be correctly delivered to the designated person.

[0087] It can be understood that the recognition performance of the identity fusion verification neural network directly affects the effect of identity authentication. In the embodiment of the present application, the debugging process of the identity fusion verification neural network is provided. Please refer to Figure 3 , Figure 3 This is a schematic diagram of a debugging process of an identity fusion verification neural network provided in an embodiment of the present application, which specifically includes the following steps:

[0088] Step S10: Obtain learning samples of face images carrying prior information of face identity.

[0089] Step S10 is the basis for building and training the neural network model. In this step, the server obtains face image learning examples (i.e., training samples) carrying face identity prior information (i.e., known face identity labels) from various sources.

[0090] In application scenarios, these labeled face image samples can be obtained in a variety of ways. For example, in the security systems of government agencies, enterprises or schools, a large number of face images of target objects, students or citizens can be collected, and these images are usually associated with corresponding identity information. These images can be used as training samples to train the identity fusion verification neural network. It should be noted that before and during the collection of relevant data for application, it is necessary to ensure compliance with relevant laws and regulations, that is, the legitimate rights and interests of the target object are not infringed and the target object is informed and agrees.

[0091] When acquiring these samples, the server needs to ensure the clarity and quality of the images so that the neural network can extract effective facial features from them. In addition, the server also needs to pre-process these images, such as cropping, scaling, and normalization, to meet the input requirements of the neural network. Through this step, the server can accumulate a rich and diverse library of training samples to provide sufficient data support for subsequent neural network training. This not only helps to improve the recognition accuracy of the neural network, but also makes it more robust and reliable in practical applications.

[0092] Step S20: Loading the face image learning sample into a pre-prepared neural network, wherein the pre-prepared neural network includes a first component and a second component; bidirectional iterative transmission is adopted between hidden layers of the first component, and the second component is established through a selective focusing strategy.

[0093] In step S20, the face image learning sample is loaded into a specific neural network structure. This neural network structure includes two main components. The first component adopts a bidirectional iterative transmission mechanism, that is, a cross-directional transmission iteration is used between hidden layers. This means that information not only flows from the input layer to the output layer in the forward propagation, but also feeds back from the output layer to the input layer in the back propagation, and the weights and biases of the network are optimized through continuous iteration. This bidirectional propagation helps the network to better learn and adapt to the characteristics of the input data, so as to more accurately identify the identity information in the face image. For example, in the document signing scenario, the server loads the collected face image learning samples into this neural network. Through bidirectional iterative transmission, the network can more effectively learn the subtle feature differences of different faces, which is crucial for accurately identifying the identity of the signatory. The first component can adopt neural network structures such as Convolutional Neural Networks (CNN), Residual Networks (ResNet), Densely Connected Convolutional Networks (DenseNet), lightweight neural networks (such as MobileNet, ShuffleNet, etc.).

[0094] The second component of the neural network uses a selective focus strategy, also known as an attention strategy. In other words, when processing an image, the neural network can automatically focus on the most informative parts of the image, namely the key feature areas of the face, such as the eyes, nose, and mouth. In this way, the neural network can more effectively extract and identify features that are critical for identity verification.

[0095] In practical applications, this attention mechanism can help the neural network ignore irrelevant background information in the image and focus on key facial features. For example, in the scene of signing for a document, even if the background of the signatory's face image is complex and changeable, the attention mechanism can ensure that the neural network can accurately identify the signatory's identity.

[0096] In summary, the key to step S20 is to load the face image learning samples into a specially designed neural network. This neural network can more accurately learn and recognize the identity information in the face image through the combination of bidirectional iterative transfer and attention mechanism, thereby providing reliable identity authentication function in scenarios such as document signing.

[0097] Step S30: extracting and debugging the image content representation vector of the face image learning sample through the first component to obtain the first face identity information.

[0098] In step S30, the first component of the neural network is used to extract and debug the image content representation vector of the face image learning sample to obtain the first face identity information.

[0099] First, the server uses the face image learning samples that have been loaded into the neural network. These samples contain various facial features and corresponding identity information, which are the basis for neural network learning and recognition. Then, the first component of the neural network begins to execute. The main task of this first component is to extract the image content representation vector, that is, to extract the semantic features in the image. These features may include the shape of the eyes, the outline of the nose, the size and shape of the mouth, etc., which together constitute the unique identification of the face. In this process, the neural network will convert the original image data into a higher-level feature representation through a series of convolution, pooling and other operations. Then, through the mechanism of bidirectional iterative transmission, information is repeatedly transmitted and optimized between the hidden layers of the neural network. This can help the network better capture the subtle features in the image and improve the accuracy of recognition. In this process, the network will continuously adjust its weights and biases to make the extracted features more accurate and effective. Finally, after debugging and training, the first component of the neural network can successfully extract the image content representation vector, that is, the semantic features of the face, from the face image learning samples. These features will be used in the subsequent identity authentication process.

[0100] Step S40: extracting and debugging the focused image representation vector of the face image learning sample according to the selective focusing strategy through the second component to obtain the second face identity information.

[0101] In step S40, the second component extracts and debugs the focused image representation vector for the face image learning sample according to the selective focus strategy. In the debugging of the identity fusion verification neural network, the second component is responsible for extracting the focused image representation vector according to the selective focus strategy. The core of this strategy is to enable the neural network to focus on the key information in the image, that is, the features that are most important for identity verification.

[0102] In the process of extracting and debugging the focused image representation vector, the second component uses advanced techniques such as attention mechanisms to enable the neural network to automatically identify and focus on the most informative areas when processing face images. For example, when processing face images, the network may pay more attention to key feature areas such as eyes, nose, and mouth, because these areas usually contain more identity information.

[0103] Next is the capture of the intermediate layer state. In the neural network, the intermediate layer state refers to the intermediate data generated by the hidden layer in the process of processing the image. These data contain the characteristic information of the image at different processing stages. By extracting these intermediate layer states, the second component can obtain the focused image representation vectors, which more accurately describe the key features of the face. Then the second face identity information is obtained. After processing by the second component, the focused image representation vectors are obtained, which are further used to extract the second face identity information. This information combined with the information extracted by the first component will provide more comprehensive and accurate face identity recognition results.

[0104] The second component can be implemented using a neural network with an attention mechanism, such as the Encoder part of a Transformer model or a combination of a convolutional neural network (CNN) and an attention mechanism. For example, an attention mechanism module is introduced based on traditional convolutional neural networks (such as VGG, ResNet, etc.). This module can be added after a certain layer of CNN to guide the network to focus on key areas in the image. For example, a spatial attention module can be added after the feature map of a CNN, which learns a set of weights that indicate the importance of different positions in the feature map. In this way, the network can dynamically focus on key parts of the face image during processing, thereby generating a more effective focused image representation vector.

[0105] Step S50: Determine preset facial identity information through the first facial identity information and the second facial identity information.

[0106] The purpose of step S50 is to determine the final preset face identity information by integrating the first face identity information and the second face identity information.

[0107] In the document signing scenario, when the target object signs for the document, the signing terminal captures the target object's facial image through the camera and processes it through the identity fusion verification neural network. This neural network contains two main components, each responsible for extracting different facial identity information. The first component obtains the first facial identity information by extracting the image content representation vector; and the second component focuses on the facial image based on the selective focusing strategy to obtain the second facial identity information. In step S50, the server receives these two pieces of identity information, which are usually expressed in the form of confidence. Confidence is a value between 0 and 1, which indicates the degree of certainty of the neural network's judgment of a certain identity. For example, if the confidence of the first facial identity information is 0.8, it means that the neural network is 80% certain that this facial image belongs to a specific identity.

[0108] In order to determine the final preset face identity information, the server fuses the confidence of the first face identity information and the second face identity information. This fusion can be done in a variety of ways, such as averaging, weighted averaging, and taking the maximum value. Taking averaging as an example, if the confidence of the first face identity information is 0.8 and the confidence of the second face identity information is 0.9, then their average value is 0.85. This average value is used as the final preset face identity information confidence. After determining the preset face identity information, the server compares it with the known identity information stored in the system. If the match is successful and the confidence exceeds the preset threshold (such as 0.7), the system considers the identity of the signatory to be valid and allows him to sign for the document. On the contrary, if the match fails or the confidence is lower than the threshold, the system will reject the signing request and issue an alarm.

[0109] In this way, the identity fusion verification neural network can provide efficient and accurate identity verification services in the document signing scenario, ensuring that only authorized target objects can sign for the document. At the same time, by combining the identity information of the two components, the neural network can also improve the accuracy and robustness of identity verification and reduce the risk of misjudgment.

[0110] Step S60: The pre-prepared neural network is tuned according to the error between the preset face identity information and the face identity prior information to obtain an identity fusion verification neural network.

[0111] Step S60 adjusts the neural network by comparing the error between the preset face identity information and the known face identity prior information to obtain a more accurate and efficient identity fusion verification neural network. First, the preset face identity information is the result obtained by combining the first face identity information and the second face identity information in step S50, which represents the neural network's preliminary identity judgment of the current face image. Face identity prior information is known and accurate face identity information, usually from a reliable database or previously verified identity record.

[0112] In step S60, the server performs an error calculation, specifically, the server compares the difference or error between the preset face identity information and the face identity prior information. Based on the calculated error, the server tunes the neural network prepared in advance. This tuning may involve adjusting the weights, biases, learning rates and other parameters of the neural network, or modifying the structure of the network to reduce the possibility of similar errors in the future. For example, in a convolutional neural network, it may be necessary to adjust the size and step size of the convolution kernel or add more convolution layers to improve the recognition ability of the network. The tuning process is an iterative process, and the server repeats the above steps many times until the performance of the neural network reaches an acceptable level. This is usually achieved by setting an error threshold or number of iterations. When the performance of the neural network reaches this threshold or the number of iterations is exhausted, the tuning process stops.

[0113] As an implementation manner, the first component includes a representation vector extraction module, a filtering module, a downsampling module and a dense connection module. The step S30, extracting and debugging the image content representation vector of the face image learning sample by the first component to obtain the first face identity information, may specifically include:

[0114] Step S31: extracting the image content representation vector of the face image learning sample through the representation vector extraction module to obtain a region representation vector set and an image block representation vector set.

[0115] In step S31, the server extracts the image content representation vector of the face image learning sample, that is, the semantic features in the image, through the representation vector extraction module. These features can describe the content in the image and are crucial for identifying the identity information of the face in the image. Specifically, the server first receives a face image learning sample, which is a facial photo of a target object. The representation vector extraction module is usually a convolutional neural network (CNN) that has been trained to extract specific features in the image.

[0116] In step S31, the representation vector extraction module first processes the entire face image and divides it into different regions and image blocks. The regions may be automatically detected based on image segmentation technology, such as the facial features (eyes, nose, mouth, etc.), while the image blocks are obtained by evenly dividing the image into several small blocks. For each region and image block, the representation vector extraction module extracts a set of eigenvalues ​​that describe the visual information in the region or image block, such as edges, textures, shapes, etc. These eigenvalues ​​are combined into a vector, called a representation vector. For regions, these vectors constitute a set of region representation vectors; for image blocks, they constitute a set of image block representation vectors.

[0117] For example, if the server is processing a photo containing the target object's face, the representation vector extraction module can identify the eye area and extract the features of the area, such as the shape, size, and position of the eyes, and encode these features into a representation vector. Similarly, similar processing will be performed on other areas and image blocks. In the end, the server obtains a set of multiple representation vectors, which represent the features of different areas and image blocks in the face image. These feature vectors will be used in subsequent steps to further analyze and identify the face identity information in the image.

[0118] Step S32: performing a sliding window dot product operation on the region representation vector set through the filtering module to obtain a region feature distribution set.

[0119] Step S32 is completed by the filtering module, which is usually a convolutional network layer that plays the role of feature mapping and conversion in the neural network. Specifically, in step S32, the server uses the filtering module to perform a sliding window dot product operation on the set of regional representation vectors, also known as a convolution operation. This process can be understood as a series of convolution kernels (also known as filters or convolution kernels) in the filtering module sliding on the set of regional representation vectors, and performing a dot product operation on the representation vectors in each sliding window. The dot product operation is a method of measuring the similarity of two vectors, and its result reflects the correlation between the two vectors. In the convolution operation, each convolution kernel is equivalent to a feature detector, which is used to detect whether there are similar features in the input data. When the convolution kernel slides to a certain position, if the feature in the input data is similar to the convolution kernel, the result of the dot product operation will be larger, otherwise it will be smaller. In this way, the server can capture the spatial relationship and feature distribution between different regions in the face image. These spatial relationship and feature distribution information are crucial for subsequent face recognition and identity authentication.

[0120] To give a specific example, suppose that when the server processes a facial photo of a target subject, a convolution kernel in the filter module may be specifically used to detect the shape of the eyes. When this convolution kernel slides to the eye area, the result of the dot product operation is larger because the features of the eye area are similar to the convolution kernel. In this way, the server can accurately identify the position and shape of the eyes and extract richer facial feature information.

[0121] Finally, through the convolution operation of the filter module, the server obtains a regional feature distribution set, that is, a feature map representing the region. This feature map reflects the spatial distribution of features in different regions of the face image, providing an important basis for subsequent face recognition and identity verification.

[0122] Step S33: performing a sliding window dot product operation on the image block representation vector set through the filtering module to obtain an image block feature distribution set.

[0123] The goal of step S33 is to perform a sliding window dot product operation on the set of image block representation vectors previously extracted from the face image learning sample through the filtering module to obtain a feature distribution set of the image block. As mentioned above, the filtering module generally refers to a convolutional network layer, which is used to further extract and refine image features. When the server executes step S33, the filtering module performs a sliding window dot product operation, i.e., a convolution operation, on each vector in the set of image block representation vectors. This operation method allows the server to capture subtle differences and local features between image blocks. Specifically, the sliding window dot product operation involves performing a dot product calculation on each image block representation vector with a set of predefined convolution kernels (or filters). These convolution kernels are designed to detect specific image features, such as edges, textures, etc. When the convolution kernel slides on the image block representation vector and performs a dot product operation, a new feature map is generated, which reflects the spatial distribution of specific features in the image block. For example, suppose that the server is processing a facial image of a target object, and the image is divided into several small image blocks. In step S33, the filter module may use a convolution kernel specifically for detecting skin texture to perform a convolution operation on each image block. In this way, the server can identify which image blocks contain specific skin texture features and generate a feature map about the distribution of these features accordingly.

[0124] Finally, through the processing of step S33, the server obtains a set of image block feature distributions, that is, a feature map representing the image block. This feature map not only contains the global feature information of the image block, but also captures more detailed local features, such as the fine texture of the skin, the shape of the pupil of the eye, etc. This information is crucial for subsequent tasks such as face recognition, identity authentication or expression analysis, because they provide rich details about facial features and help enhance the accuracy and reliability of the recognition system.

[0125] Step S34: integrating the regional feature distribution set and the image block feature distribution set through the downsampling module to obtain an integrated representation vector.

[0126] In this step, the server integrates the regional feature distribution set and the image block feature distribution set obtained in the previous step through the downsampling module to generate a more comprehensive and compact feature representation, that is, an integrated representation vector. The downsampling module is used to reduce the dimension of the data while retaining key information. In step S34, the downsampling module is used to fuse the features in the regional feature distribution set and the image block feature distribution set to extract a higher-level feature representation.

[0127] Specifically, the server first receives a set of regional feature distributions and a set of image patch feature distributions as input. These feature distribution sets contain various local and global features extracted from face images. The downsampling module selects and compresses these features to remove redundant information while emphasizing those features that are most critical for identity recognition or verification tasks.

[0128] For example, in the document signing scenario, the facial features of the target object are crucial for identity verification. The downsampling module can fuse the features of key areas such as eyes, nose, mouth, and the texture and shape features of the face as a whole to generate an integrated representation vector that can accurately represent the target object's face.

[0129] The integrated representation vector is a high-dimensional vector that integrates feature information at multiple levels, including both local details and overall structure. Such a vector can provide more accurate and robust performance in subsequent recognition, verification or search tasks. Finally, through the processing of step S34, the server obtains an integrated representation vector that integrates multi-scale features, which will serve as the basic input for subsequent identity authentication or recognition tasks.

[0130] Step S35: classifying the integrated representation vector through the dense connection module to obtain the first face identity information.

[0131] In step S35, the server will use a dense connection module, also known as a fully connected layer, to classify the integrated representation vector obtained in the previous step to determine and output the first face identity information. The dense connection module maps the previously extracted features to the final classification result. In step S35, the dense connection module receives the integrated representation vector as input, which incorporates the key feature information in the face image. The function of the dense connection module is to convert this feature information into a specific classification result, that is, to identify the identity information corresponding to the face image. Specifically, when the integrated representation vector is passed to the dense connection module, the dense connection module linearly transforms the vector through its internal weights and bias parameters, and then may introduce nonlinear factors through activation functions (such as ReLU, Sigmoid, etc.) to better fit the complex classification boundaries. This process will eventually output a probability distribution, indicating the probability that the input image belongs to each possible identity.

[0132] In the document signing scenario, this classification result is crucial because it is directly related to whether the system can accurately identify the identity of the target object signing the document. For example, when the target object signs the document, the signing terminal first collects its face image, then extracts features through the above steps and integrates them into a representation vector, and finally determines the identity information of the target object through the fully connected layer in step S35.

[0133] To explain more specifically, assume that the server has obtained an integrated representation vector containing multiple facial features through the previous steps. In step S35, this vector will be passed to the fully connected layer for classification. If the system has pre-trained a classifier corresponding to a specific target object identity (that is, the weights and biases of the fully connected layer have been optimized through training data), then the fully connected layer can output a confidence distribution representing the identity of the target object based on the input feature vector. The server selects the identity with the highest confidence as the recognition result, thereby completing the identity verification when signing the document.

[0134] As an implementation manner, the above-mentioned representation vector extraction module may include a visual word bag unit, a region description extraction unit, and an image block description extraction unit. Based on this, the step S31, extracting the image content representation vector of the face image learning sample by the representation vector extraction module to obtain a region representation vector set and an image block representation vector set, may specifically include:

[0135] Step S311: extracting image features of the face image learning sample through the bag-of-visual-words unit to obtain regional image features and image block features.

[0136] In step S311, the server extracts image features of the face image learning sample through the Bag of Visual Words (BOVW) unit. The Bag of Visual Words model is an image representation method borrowed from the Bag of Words model in natural language processing. It regards the image as a collection of a series of visual words, which are clustered from local features extracted from the image. In step S311, the server first extracts a large number of local image features from the face image learning sample. These features may include corners, edges, textures, etc., which are extracted from various regions and image blocks of the image through specific algorithms (such as SIFT, SURF, etc.). Next, the server uses these local features to build a visual vocabulary. This usually involves clustering the extracted local features, and each cluster center represents a visual vocabulary. Once the visual vocabulary is built, the server can represent the original image as a frequency histogram of these visual words, that is, the bag of visual words representation.

[0137] In the specific implementation of step S311, the server performs feature extraction and visual word bag representation for different regions and image blocks of the face image. For regional image features, the server focuses on large-scale features such as the overall contour of the face and the distribution of facial features; while for image block features, the server will focus more on the feature extraction of local details such as eyes, nose, mouth, etc.

[0138] For example, suppose the server is processing a facial image of a target object for subsequent document signing and identity verification. In step S311, the server first uses the SIFT algorithm to extract a large number of local feature points from the image, and clusters these feature points to build a visual vocabulary. Then, the server represents different regions and image blocks of the facial image as frequency histograms of visual vocabulary based on this vocabulary. These histogram data constitute regional image features and image block features, which will be used in subsequent representation vector extraction and identity verification processes. In this way, step S311 effectively extracts key image features from facial image learning samples, providing strong support for subsequent facial recognition and identity verification tasks.

[0139] Step S312: extracting the region representation vector of the face image learning sample through the region description extraction unit to obtain a region descriptor.

[0140] Specifically, the server uses the region description extraction unit to process the face image. This unit is designed to capture representative and distinguishing regional features from the image. During the processing, the region description extraction unit focuses on specific areas in the image, such as key parts such as facial features and contours, and extracts representation vectors from these areas. The representation vector is a mathematical expression that can effectively describe the characteristics of the image area. In step S312, the server converts the features of these key areas into vector form, namely, the region representation vector, through the region description extraction unit. These vectors not only contain information such as the shape and texture of the region, but also reflect the spatial relationship and interdependence between regions.

[0141] Furthermore, the server connects these region representation vectors to obtain a region descriptor (this region descriptor is, for example, a matrix). The region descriptor is a concise and effective representation of the region features, which highlights the main features of the region while ignoring minor details. This is very beneficial for subsequent classification, recognition, and verification tasks because it can help the server quickly and accurately identify key information in the face image.

[0142] Step S313: extracting the image block representation vector of the face image learning sample through the image block description extraction unit to obtain an image block descriptor.

[0143] Specifically, the server uses the image block description extraction unit to deeply analyze the local features of the face image. Unlike the region description extraction unit, the image block description extraction unit focuses more on capturing the features of smaller, continuous regions in the image. These regions may include a specific part of the face, such as the eyes, the bridge of the nose, the mouth, etc., or the subtle textures and changes between these parts. During the extraction process, the image block description extraction unit converts each selected image block into a representation vector, namely the image block representation vector. This vector contains all the key information in the image block, such as pixel values, colors, textures, etc., and expresses them in a mathematically tractable way. In this way, the server can more accurately understand and compare the subtle differences between different face images. Further, these image block representation vectors are spliced, and the server obtains the image block descriptor (this image block descriptor is, for example, a matrix). Similar to the region descriptor, the image block descriptor is also a concise and effective representation of local features. Compared with the region descriptor, the image block descriptor focuses more on capturing the local details and changes of the image, so it is particularly important in scenes that require high-precision recognition, such as face recognition. By accurately capturing and analyzing the subtle features of each part of the signatory's face, the server can generate highly discriminative image block descriptors. These descriptors will be compared with the known identity information stored in the system to ensure that the signatory's identity is authentic and valid.

[0144] Step S314: Generate the region representation vector set through the region image features and the region descriptor.

[0145] Step S315: Generate the image block representation vector set using the image block features and the image block descriptors.

[0146] Specifically, the server has extracted the regional image features of the face image through the visual word bag unit in the previous step, and also obtained the regional descriptor through the regional description extraction unit. These features and descriptors are mathematical expressions of different regions in the face image, and they describe the characteristics of the image from different angles.

[0147] In step S314, the server fuses these regional image features and regional descriptors to generate a more comprehensive and representative set of regional representation vectors. This process can be understood as integrating two different types of feature information to obtain a richer and more accurate image representation.

[0148] For example, suppose that when the server processes a facial image of a target object, it extracts image features of areas such as eyes, nose and mouth through the bag of visual words unit, and obtains the descriptors of these areas through the area description extraction unit. In step S314, the server combines these features and descriptors to generate a set of representation vectors that contain comprehensive information of areas such as eyes, nose and mouth. This set of representation vectors not only contains the local feature information of the image (such as the shape of the eyes, the height of the nose, etc.), but also contains the spatial relationship and mutual dependence between these features (such as the relative position of the eyes and nose, etc.). Such a set is extremely valuable for subsequent tasks such as face recognition and verification because it provides a comprehensive and accurate image representation.

[0149] In actual operation, the server can use some algorithms or techniques to fuse these features and descriptors, such as feature splicing, weighted summation, etc. Through these methods, the server can generate a set of regional representation vectors that contain both local features and global information, providing strong data support for subsequent face recognition and verification. Step S315 generates a set of image block representation vectors through the previously extracted image block features and image block descriptors. When executing this step, the server has obtained the features and corresponding descriptors of each image block in the face image through the previous steps. Image block features may include basic information such as color, texture, and shape, while image block descriptors are a higher-level expression of these features, which can more accurately describe the uniqueness of image blocks.

[0150] For example, suppose that when processing a facial image used for document receipt verification, the server has extracted the image block features of key parts such as eyes, nose bridge, and mouth through a specific algorithm or neural network model, and generated corresponding descriptors for each image block. In step S315, the server will use these features and descriptors to generate a set of image block representation vectors. Each vector in this set represents the comprehensive information of an image block, which not only contains the basic features of the image block, but also incorporates the higher-level information provided by the descriptor. Such a set of representation vectors is crucial for face recognition and verification because they can provide richer and more accurate information than a single feature or descriptor.

[0151] In actual operation, the server can use a specific algorithm to fuse image block features and descriptors to generate a representation vector. For example, weighted fusion, tandem fusion, or feature fusion using a deep learning model can be used. Through these technical means, the server can ensure that the generated image block representation vector set is both comprehensive and accurate, providing strong support for subsequent face recognition and verification.

[0152] As an implementation manner, the representation vector extraction module further includes an interactive long short-term memory network unit. Then, the step S314, generating the set of region representation vectors through the region image features and the region descriptors, may specifically include:

[0153] Step S3141: extracting the semantic representation vector of the region descriptor through the interactive long short-term memory network unit to obtain an initial region representation vector.

[0154] Step S3141 extracts the semantic representation vector of the region descriptor through an interactive long short-term memory network unit (LSTM), thereby obtaining an initial region representation vector. The long short-term memory network (LSTM) is a special recurrent neural network (RNN) that is suitable for processing and predicting patterns with very long intervals and delays in time series data. In the field of image recognition, LSTM can be used to understand and analyze the descriptors of different regions in an image and capture their deep semantic information. The interactive long short-term memory network unit is a bidirectional long short-term memory network. In step S3141, the server first receives a series of region descriptors, and then the server inputs these descriptors into the trained LSTM network. The LSTM network can effectively process the sequence information in these descriptors, capture the long-term dependencies in the descriptors, and understand their semantic content through its internal memory units and gating mechanisms.

[0155] After LSTM processes these descriptors, it outputs a semantic representation vector, which is the result of LSTM's deep understanding of the descriptors. It captures the key information and semantic features in the descriptors. This semantic representation vector is the initial region representation vector, which will serve as the basis for subsequent image recognition and analysis. For example, if LSTM processes a descriptor that describes the eye region, it may output a semantic representation vector containing features such as eye shape, color, and size. This vector will serve as the initial representation vector of the eye region for use in subsequent steps. In this way, the server uses the powerful sequence processing capabilities of the LSTM network to extract deep semantic information from the region descriptors, providing richer and more accurate data support for subsequent image recognition and analysis. In the document signing scenario, this helps to improve the system's accuracy and reliability in identifying user identities.

[0156] Step S3142: Generate the region representation vector set using the region image features and the initial region representation vector.

[0157] When the server performs this step, it first obtains the regional image features extracted in the previous step and the initial regional representation vector obtained through the LSTM network. The regional image features may include information directly extracted from the image region, such as color, texture, and shape, while the initial regional representation vector is the semantic information obtained after the LSTM has a deep understanding of the regional descriptor.

[0158] To generate a set of region representation vectors, the server can combine these two types of information in a variety of ways. One way is to directly calculate the multiplication of the region image features and the initial region representation vector to obtain a region representation vector that combines image features and semantic information. This process is repeated for each region in the image, and finally a set of multiple region representation vectors is formed.

[0159] Another way is to combine the two types of information by setting weights. The server assigns a first regional weight to the regional image features and a second regional weight to the initial regional representation vector. These weights can be adjusted according to the actual application scenario and data to reflect the importance of different features in the final representation. Next, the server calculates the multiplication result of the regional image features and the first regional weight (first multiplication result), and the multiplication result of the initial regional representation vector and the second regional weight (second multiplication result). Finally, the two multiplication results are added together to obtain a regional representation vector that combines weighted image features and weighted semantic information. This process is also performed for each region to generate a comprehensive set of regional representation vectors.

[0160] For example, the server extracts image features of the eyes, nose, mouth and other areas, and obtains the initial representation vectors of these areas through the LSTM network. Then, by setting appropriate weights, the server can generate a set of regional representation vectors that contain both image features and semantic information. This set can more comprehensively describe the facial features of the signatory and provide accurate data support for subsequent identity verification and other tasks.

[0161] In this way, the server can effectively combine image features and semantic information to generate a rich and accurate set of region representation vectors, which helps improve the accuracy and reliability of identity recognition and ensure the legitimacy and security of document signing.

[0162] The step S315, generating the image block representation vector set by using the image block features and the image block descriptor, may specifically include:

[0163] Step S3151: extracting the semantic representation vector of the image block descriptor through the interactive long short-term memory network unit to obtain an initial image block representation vector.

[0164] Specifically, the server receives a series of image patch descriptors, which are fed into the trained interactive long short-term memory network unit. The LSTM network analyzes the descriptor to capture its semantic information and encodes it into a vector representation. This vector, the semantic representation vector, is the result of LSTM's deep understanding of the descriptor. It captures key information and features in the descriptor, such as the shape, size, and position of facial features, and expresses this information in mathematical form. This semantic representation vector is then used as the initial image patch representation vector, providing a basis for subsequent image recognition and analysis.

[0165] In this way, the server uses the powerful sequence processing capabilities of the LSTM network to extract deep semantic information from the image block descriptor, providing more accurate data support for subsequent image recognition and analysis. In the document signing scenario, this helps improve the system's recognition accuracy of the user's facial features, thereby enhancing the accuracy and reliability of identity verification.

[0166] Step S3152: Generate the image block representation vector set through the image block features and the initial image block representation vector.

[0167] First, the server obtains the image block features and the initial image block representation vector from the previous step. The image block features are usually the underlying visual features extracted from the image, such as color, texture, shape, etc., while the initial image block representation vector is obtained by semantically analyzing the image block descriptor through the LSTM network.

[0168] In order to generate a set of image block representation vectors, the server can adopt different fusion strategies. A simple and direct method is to calculate the multiplication result of the image block feature and the initial image block representation vector to obtain an image block representation vector that combines visual features and semantic information. This process is performed for each image block in the image, and finally a set of multiple image block representation vectors is formed.

[0169] Another more sophisticated method is to fuse these two types of information by setting weights. The server assigns a first image block weight to the image block feature and a second image block weight to the initial image block representation vector. These weights can be adjusted according to the actual application scenario and data to reflect the importance of different features in the final representation. Next, the server will calculate the multiplication result of the image block feature and the first image block weight (the third multiplication result), and the multiplication result of the initial image block representation vector and the second image block weight (the fourth multiplication result). Finally, the two multiplication results are added together to obtain an image block representation vector that combines weighted visual features and weighted semantic information. This process is also performed for each image block, thereby generating a more comprehensive and accurate set of image block representation vectors. In this way, the server can effectively combine visual features and semantic information to generate a rich and accurate set of image block representation vectors. In the document signing scenario, this helps to improve the system's recognition accuracy and reliability of user signature features, thereby ensuring the legality and security of document signing.

[0170] As an implementation manner, the step S34, integrating the regional feature distribution set and the image block feature distribution set by the downsampling module to obtain an integrated representation vector, may specifically include:

[0171] Step S341: performing feature dimension reduction processing on the regional feature distribution set through the downsampling module to obtain a single region representation vector;

[0172] Step S342: performing feature dimension reduction processing on the image block feature distribution set through the downsampling module to obtain a single image block representation vector;

[0173] Step S343: integrating the single region representation vector and the single image block representation vector through the downsampling module to obtain the integrated representation vector.

[0174] The original feature distribution set (i.e., the regional feature distribution set and the image block feature distribution set) is usually high-dimensional and contains a large amount of data, which is not conducive to direct use in subsequent face recognition classification. Therefore, in steps S341 and S342, the server uses a downsampling module, such as a pooling layer, to reduce the dimensionality of the high-dimensional feature set. The function of the downsampling module is to downsample the input feature map, reduce the dimension of the data by reducing the number of features, and retain the most important feature information. In face recognition, this means that the server extracts the most representative features of each area in the face image, such as the shape of the eyes, the curvature of the eyebrows, or the outline of the mouth.

[0175] Through the processing of the downsampling module, the server can compress these high-dimensional regional features into low-dimensional single-region representation vectors. These vectors capture the most significant features of each region in the face image and provide effective data representation for subsequent face recognition. Taking maximum pooling as an example, the server slides a window on each feature map of the face image and selects the maximum value in the window as the representative feature of the region. In this way, for each key area such as the eyes, nose, and mouth, the server can obtain a low-dimensional representation vector, that is, a single-region representation vector. Each vector element in the single-region representation vector represents the feature vector of each feature in the feature set.

[0176] The way of integrating the single region representation vector and the single image block representation vector is, for example, a fusion method based on splicing to obtain an integrated representation vector.

[0177] In one implementation, the second component includes an embedding mapping module, a restoration mapping module, and a classifier module, and the focused image representation vector represents the intermediate layer state of the target time step; the step S40, extracting and debugging the focused image representation vector of the face image learning sample according to the selective focusing strategy by the second component to obtain the second face identity information, may specifically include:

[0178] Step S41: performing embedding mapping operations on the face image learning samples at each time step through the embedding mapping module to obtain a region representation vector sequence.

[0179] In step S41, the server performs an embedding mapping operation on the face image learning sample through the embedding mapping module. The embedding mapping process is the image encoding process. In other words, the embedding mapping module can be an encoder. The time step here means that when processing the face image learning sample, the face image learning sample is divided into multiple regions, and these regions are encoded in sequence in a certain order (each node in this order is regarded as a time step).

[0180] Specifically, during the face recognition process, a face image can be divided into multiple regions, such as the eye region, nose region, mouth region, etc. The server will encode these regions one by one in a predetermined order. During the encoding process, the embedding mapping module will extract the features of each region and convert these features into a region representation vector. In this way, the server can obtain a sequence of region representation vectors, which is arranged in the order of time steps and captures the feature information of different regions in the face image.

[0181] Step S42: Determine the latent variable to be observed through the region characterization vector sequence.

[0182] In the previous steps, the server has converted different regions of the face image learning samples into a series of vectors through the embedding mapping module (encoder). These vectors are arranged in the processing order of the image regions (i.e., time steps) to form a sequence of regional representation vectors. These vectors capture the characteristic information of different regions in the image. Next, the server will use these regional representation vectors to determine the latent variables to be observed. Hidden variables are variables that are not directly observed in the model, but they play a key role in explaining the observed data (regional representation vectors). In this application, the latent variables, for example, represent identity-related information. In order to determine these latent variables, the server can use specific algorithms and models. For example, probabilistic graph models, autoencoders in deep learning, or other unsupervised learning methods can be used to infer the values ​​of latent variables from a sequence of regional representation vectors. These methods usually involve statistical analysis and pattern recognition of vectors to reveal the structure hidden behind the observed data.

[0183] Specifically, during the training process, the server tries to find a set of latent variables that can best explain the observed sequence of region representation vectors. This usually involves an optimization problem, that is, adjusting the values ​​of the latent variables to maximize the probability of the observed data. Once this set of optimal latent variables is found, they can be used as important features for subsequent identity verification tasks.

[0184] For example, the server obtains the representation vectors of the eyes, nose, and mouth through the encoder. The server then uses these vectors to determine the latent variables related to identity, such as the shape of the eyes, the width of the nose, or the outline of the mouth. Although these latent variables are not directly observed from the image, they are crucial for distinguishing different facial features.

[0185] As an implementation mode, the step S42 determines the hidden variables to be observed through the region characterization vector sequence, specifically including: generating an initial indefinite parameter set; determining the expectation and discreteness corresponding to the region characterization vector sequence through the initial indefinite parameter set; and determining the hidden variables to be observed of any indefinite parameter in the initial indefinite parameter set based on the expectation and discreteness.

[0186] The server generates an initial set of uncertain parameters. These uncertain parameters are pre-distributed random parameters used to initialize the uncertainty of the model. In neural network training, these parameters may represent the initial values ​​of network weights or other hyperparameters related to model training. For example, the server can randomly initialize these parameters or generate them using a predefined distribution (such as a normal distribution).

[0187] Next, the server uses these initial uncertainty parameters to calculate the expectation (i.e., mean) and dispersion (i.e., variance) of the region representation vector sequence. The expectation reflects the central tendency of the vector sequence, while the dispersion measures the degree of variation of each vector in the vector sequence.

[0188] For example, suppose the server has obtained a sequence of regional representation vectors of multiple face images through an encoder. The server calculates the average of these vectors to obtain the expectation, which can be regarded as the common feature points of these face images. At the same time, the server also calculates the variance of these vectors to measure the difference or range of variation between them. Finally, the server determines the observed latent variables of any uncertain parameter in the initial uncertain parameter set based on the calculated expectation and discreteness. These latent variables represent potential features or attributes inferred from the sequence of regional representation vectors, which may be crucial for identity authentication.

[0189] Step S43: extracting semantic representation vectors from the region representation vector sequence using the selective focusing strategy to obtain semantic embedding representation vectors.

[0190] In the previous steps, the server has divided the input face image learning samples into multiple regions, and encoded each region to generate a series of regional representation vectors. These vectors are arranged in the order of processing the image regions to form a sequence of regional representation vectors. Next, the server will adopt a selective focus strategy, that is, an attention mechanism, to further process these regional representation vectors. The core idea of ​​the attention mechanism is to imitate the selective attention behavior of humans when processing information. In step S43, the server uses the attention mechanism to process the sequence of regional representation vectors to extract the semantic representation vectors therein. Specifically, the server calculates the importance weights of each regional representation vector, which reflect the importance of the vector to the final task. Then, the server will perform a weighted summation of the vectors according to these weights to obtain a semantic embedding representation vector that incorporates all important information.

[0191] In this way, the server extracts the key semantic features in the region representation vector sequence and generates a more compact and effective semantic embedding representation vector. This vector will be used as the input of the subsequent classifier or verifier to complete tasks such as face recognition. Step S43 uses the attention mechanism to extract semantic features from the region representation vector sequence. Through this process, the server can more accurately capture the key information in the face image and improve the accuracy and efficiency of identity authentication.

[0192] Step S44: Perform restoration mapping on the latent variable to be observed and the semantic embedding representation vector through the restoration mapping module to determine the intermediate layer state of the target time step.

[0193] Step S45: classify the intermediate layer state through a classifier module to obtain the second face identity information.

[0194] In step S44, the server uses a restoration mapping module (i.e., decoder) to restore the observed latent variables and the semantic embedding representation vector. The purpose of this process is to recover more original information from the encoded and extracted features for subsequent classification and recognition. Specifically, the restoration mapping module receives the observed latent variables and the semantic embedding representation vector as input, and then transforms these highly abstract features back into a representation closer to the original input data through a series of calculations and operations. This representation is called the intermediate state of the target time step. This intermediate state contains some key information of the original input data and also incorporates the learning results of the model in the process of encoding and extracting features. Taking face recognition as an example, the observed latent variables may represent some potential features of the face, such as face shape, eye shape, etc.; while the semantic embedding representation vector captures the key semantic information in the face image. Through restoration mapping, the server can transform these features into an intermediate state that is easier to classify and recognize.

[0195] After determining the state of the middle layer, the server will use the classifier module to classify and identify this state. The classifier module is a machine learning model specifically used to distinguish different categories. It can determine the category to which the input data belongs based on the information in the middle layer state.

[0196] In the face recognition scenario, the task of the classifier module is to determine the identity of the face image based on the state of the intermediate layer. The server will input the state of the intermediate layer into the classifier module, and then the classifier will output a judgment result about the face identity, that is, the second face identity information. The classifier module can be a classifier such as SVM, Naive Bayes, Random Forest, Softmax network, etc. Steps S44 and S45 convert highly abstract features back to the representation of the original data through the two links of restoration mapping and classification recognition, and finally achieve accurate classification and recognition of the input data.

[0197] As an implementation mode, the step S60, by using the error between the preset face identity information and the face identity prior information, tunes the pre-prepared neural network to obtain the identity fusion verification neural network, which may specifically include: determining the target cost by using the error between the preset face identity information and the face identity prior information; optimizing the network parameters of the pre-prepared neural network by using the target cost to obtain the identity fusion verification neural network.

[0198] The purpose of step S60 is to tune the pre-prepared neural network by comparing the error between the preset face identity information and the face identity prior information, so as to obtain a more accurate identity fusion verification neural network. First, the server obtains the preset face identity information and the face identity prior information. The face identity prior information is a known and accurate face identity label, while the preset face identity information is the face identity information predicted by the network model. The error between the two reflects the accuracy of the network model in face identity recognition. Next, the server uses this error to determine the target cost. The target cost is an indicator to measure the prediction accuracy of the network model, which indicates the gap between the prediction result of the network model and the actual result. Generally, the smaller the target cost, the higher the prediction accuracy of the network model.

[0199] In order to optimize the network model, the server adjusts the network parameters of the neural network prepared in advance according to the target cost. These network parameters include network weights, biases, etc., which directly affect the prediction results of the network model. By continuously adjusting these parameters, the prediction results of the network model can be made closer to the actual results, thereby reducing the target cost.

[0200] During the tuning process, the server will use appropriate optimization algorithms, such as gradient descent algorithm, Adam optimizer, etc., to update network parameters. These optimization algorithms can adjust network parameters according to the gradient information of the target cost, so that the network model gradually converges to the optimal solution.

[0201] As an implementation manner, the first facial identity information is used to indicate that the facial identity information is a first preset confidence level of a facial identity information sample, and the second facial identity information is used to indicate that the facial identity information is a second preset confidence level of the facial identity information sample. The step S50, determining the preset facial identity information through the first facial identity information and the second facial identity information, includes: determining a confidence mean of the first preset confidence level and the second preset confidence level; and determining the confidence mean as the preset facial identity information.

[0202] Optionally, the first preset confidence and the second preset confidence correspond to the same facial identity information; for example, the confidence mean of the first preset confidence output by the first component and the second preset confidence output by the second component is calculated to obtain the preset facial identity information; if there are multiple facial identity information samples, the first preset confidence and the second preset confidence corresponding to each facial identity information can be determined respectively; the average confidence corresponding to each facial identity information is obtained, and the facial identity information with the largest average confidence is determined as the preset facial identity information to obtain the preset facial identity information.

[0203] As an embodiment, the method further includes:

[0204] Obtaining environmental parameters corresponding to the face image learning sample; determining a first set influence coefficient of the first face identity information and a second set influence coefficient of the second face identity information through the environmental parameters; based on this,

[0205] The step S50, determining the preset facial identity information through the first facial identity information and the second facial identity information, includes: determining the first facial identity information through the multiplication result of the first facial identity information and the first set influence coefficient; determining the second facial identity information through the multiplication result of the second facial identity information and the second set influence coefficient; determining the preset facial identity information through the first facial identity information and the second facial identity information.

[0206] During the training process of the identity fusion verification neural network, the server will consider multiple factors to improve the accuracy and reliability of the verification. One of the important factors is the environmental parameters when the face image is collected, which can have a significant impact on the results of face recognition. This embodiment provides a method for adjusting the influence coefficient of face identity information according to environmental parameters, thereby optimizing the determination process of preset face identity information.

[0207] First, the server obtains the environmental parameters corresponding to the face image learning sample. These environmental parameters may include various factors such as lighting conditions, shooting angles, background complexity, etc., all of which may affect the accuracy of the face recognition algorithm. For example, in an environment with insufficient light, it may be difficult for the face recognition algorithm to accurately capture facial features, resulting in increased uncertainty in the recognition results. Next, the server determines the first set influence coefficient of the first face identity information and the second set influence coefficient of the second face identity information based on these environmental parameters. These influence coefficients are essentially a quantitative representation of the credibility of face recognition results in different environments. For example, in a frontal shooting environment with good light, the server may assign higher influence coefficients to the first face identity information and the second face identity information, respectively. In other words, the influence coefficient is the weight, because in such an environment, the accuracy of the face recognition algorithm is usually higher.

[0208] After determining the influence coefficients, the server uses these influence coefficients to adjust the weights of the first face identity information and the second face identity information. Specifically, the server multiplies the first face identity information by the first set influence coefficient to obtain the adjusted first face identity information; similarly, the server multiplies the second face identity information by the second set influence coefficient to obtain the adjusted second face identity information. The purpose of this is to dynamically and individually adjust the face recognition results according to environmental parameters, thereby improving the accuracy of verification. Finally, the server determines the preset face identity information based on the adjusted first face identity information and the second face identity information. This process may include a variety of methods such as calculating the weighted average of the two and fusing them according to confidence. The preset face identity information determined in this way not only takes into account the output results of the face recognition algorithm, but also takes into account the environmental factors during image acquisition, so it is more accurate and reliable.

[0209] For example, assuming that a facial image is collected in a dimly lit environment, the server determines that the recognition result in this environment may have a certain degree of uncertainty based on the environmental parameters. Therefore, the server reduces the influence coefficients of the first face identity information and the second face identity information, for example, setting them to 0.8 and 0.7 respectively. Then, the server multiplies the two face identity information with the corresponding influence coefficients to obtain the adjusted values. Finally, the server determines the preset face identity information based on the two adjusted values, such as by calculating their weighted average. In this way, even in a dimly lit environment, the server can obtain relatively accurate preset face identity information, thereby improving the training effect and recognition accuracy of the identity fusion verification neural network.

[0210] As an implementation mode, the pre-prepared neural network also includes a third neural network, and the method also includes: extracting a target face area in the face image learning sample; the target face area is a biometric core area; performing biometric extraction and debugging on the target face area through the third neural network to obtain third face identity information.

[0211] The step S50, determining preset facial identity information through the first facial identity information and the second facial identity information, includes: determining the preset facial identity information through the first facial identity information, the second facial identity information and the third facial identity information.

[0212] During the training process of the identity fusion verification neural network, the server uses multiple methods to improve the accuracy and reliability of verification. This implementation introduces a third neural network and focuses on the target face area in the face image, that is, the biometric core area, to further extract more refined biometric information.

[0213] First, the server extracts the target face region in the face image learning sample. This target face region mainly refers to the core part of the face that is rich in biometric information, such as the area where key feature points such as eyes, nose, and mouth are located. These areas contain a lot of key information for identity authentication. Next, the server uses the third neural network to perform biometric extraction debugging on this target face region. The third neural network can be a deep convolutional neural network (CNN), which is trained to accurately extract facial features. Through this process, the server can obtain the third face identity information, which is based on more detailed and specific biometric information of the target face region.

[0214] After obtaining the first face identity information, the second face identity information and the third face identity information, the server will use this information to determine the preset face identity information in step S50. Specifically, the server may use weighted average, voting mechanism or other fusion strategies to integrate the identity information from these three sources. For example, the server can assign a weight to each face identity information, and then calculate the weighted average to obtain a more robust and reliable preset face identity information.

[0215] For example, suppose the server obtains the first face identity information from the entire face image, the second face identity information from the local area of ​​the face image, and the third face identity information from the core area of ​​the biometric feature. These three identity information may have different focuses, but they are all descriptions of the same face. The server will fuse these three information according to certain strategies (such as weighted average) to obtain a more comprehensive and accurate preset face identity information. This preset face identity information will serve as an important reference for subsequent neural network training, which will help improve the performance and accuracy of the identity fusion verification neural network.

[0216] As an embodiment, the method also includes: extracting and debugging an image content representation vector of a face image learning sample through a first component to obtain a first biometric feature; for example, extracting an image content representation vector of the face image learning sample through the first component, the image content representation vector may include a region representation vector and an image block representation vector, to obtain the first biometric feature.

[0217] The second component extracts and debugs the focused image representation vector of the face image learning sample according to the above-mentioned selective focusing strategy to obtain the second biometric feature; for example, the second component extracts and debugs the focused image representation vector of the face image learning sample according to the selective focusing strategy to obtain the second biometric feature.

[0218] The first biometric feature and the second biometric feature are integrated to obtain an integrated biometric feature; for example, the features extracted from the two components can be integrated to obtain an integrated biometric feature, so as to perform network debugging based on the integrated biometric feature.

[0219] By integrating biometric features, the preset facial identity information is determined.

[0220] For example, the integrated biometric features are classified to obtain preset face identity information. Since the integrated biometric features are obtained by integrating the features extracted by the first component and the second component respectively, the preset face identity information obtained based on the integrated biometric features has higher accuracy.

[0221] It can be understood that if the technical solution of this application involves personal or private information, the product using the technical solution of this application has clearly informed the personal information processing rules before processing personal information, and obtained the individual's voluntary consent. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the requirements of "explicit consent", and collects within the scope of laws and regulations. For example, at personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed to agree to the collection of his or her personal information; or on the device for processing personal information, when the personal information processing rules are notified by obvious signs / information, the personal authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0222] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable a server (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0223] An embodiment of the present application provides a server, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the above method when executing the computer program.

[0224] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and the computer program implements the above method when executed by a processor. The computer-readable storage medium may be transient or non-transient.

[0225] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.

[0226] It should be noted that Figure 4 is a hardware entity diagram of a server provided in an embodiment of the present application, such as Figure 4 As shown, the hardware entity of the server 300 includes: a processor 310, a communication interface 320 and a memory 330, wherein: the processor 310 generally controls the overall operation of the server 300. The communication interface 320 enables the server to communicate with other terminals or servers through a network. The memory 330 is configured to store instructions and applications executable by the processor 310, and can also cache data to be processed or processed by the processor 310 and each module in the server 300 (for example, image data, audio data, voice communication data and video communication data), which can be implemented by flash memory (FLASH) or random access memory (Random Access Memory, RAM). Data can be transmitted between the processor 310, the communication interface 320 and the memory 330 through the bus 340. It should be pointed out here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0227] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0228] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0229] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0230] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0231] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be separately configured as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0232] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0233] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a server (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0234] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A self-service document signing method based on fusion verification, characterized in that: A server applied to a document self-service signing system based on fusion verification, wherein the document self-service signing system based on fusion verification further comprises a signing terminal communicatively connected to the server, and the method comprises: In response to a document signing request initiated by the signing terminal, the document signing request includes an identity ID of a target object; When the identity ID is consistent with the target identity ID of the document to be signed, an identity verification instruction is generated, and the identity verification instruction is sent to the signing terminal, so that the signing terminal can collect the face image of the target object according to the identity verification instruction; Receiving the facial image of the target object uploaded by the signing terminal; Using the facial image as a facial image to be verified; Loading the face image to be verified into the identity fusion verification neural network to perform face identity detection operation, and obtaining the target face identity information corresponding to the face image to be verified; The debugging process of the identity fusion verification neural network includes: Obtain learning samples of face images with prior information of face identity; Loading the face image learning sample into a pre-prepared neural network, the pre-prepared neural network comprising a first component and a second component; bidirectional iterative transmission is adopted between hidden layers of the first component, and the second component is established by a selective focusing strategy; Extracting and debugging the image content representation vector of the face image learning sample through the first component to obtain first face identity information; By using the second component and according to the selective focusing strategy, extracting and debugging the focused image representation vector of the face image learning sample, and obtaining the second face identity information; Determining preset facial identity information through the first facial identity information and the second facial identity information; By using the error between the preset face identity information and the face identity prior information, the pre-prepared neural network is tuned to obtain an identity fusion verification neural network; The first component includes a representation vector extraction module, a filtering module, a downsampling module and a dense connection module, and the first component is used to extract and debug the image content representation vector of the face image learning sample to obtain the first face identity information, including: Extracting the image content representation vector of the face image learning sample by the representation vector extraction module to obtain a set of region representation vectors and a set of image block representation vectors; Performing a sliding window dot product operation on the region representation vector set by the filtering module to obtain a region feature distribution set; Performing a sliding window dot product operation on the image block representation vector set by the filtering module to obtain an image block feature distribution set; Performing an integration operation on the regional feature distribution set and the image block feature distribution set through the downsampling module to obtain an integrated representation vector; Classifying the integrated representation vector by the dense connection module to obtain the first face identity information; The representation vector extraction module includes a visual word bag unit, a region description extraction unit and an image block description extraction unit. The image content representation vector of the face image learning sample is extracted by the representation vector extraction module to obtain a region representation vector set and an image block representation vector set, including: Extracting image features of the face image learning sample through the visual word bag unit to obtain regional image features and image block features; Extracting the region representation vector of the face image learning sample by the region description extraction unit to obtain a region descriptor; Extracting the image block representation vector of the face image learning sample by the image block description extraction unit to obtain an image block descriptor; Generate the region representation vector set by using the region image features and the region descriptor; Generate the image block representation vector set by using the image block features and the image block descriptor; The representation vector extraction module further includes an interactive long short-term memory network unit, and the generating of the region representation vector set by using the region image features and the region descriptor includes: Extracting the semantic representation vector of the region descriptor through the interactive long short-term memory network unit to obtain an initial region representation vector; Generate the region representation vector set by using the region image features and the initial region representation vector; The step of generating the image block representation vector set by using the image block features and the image block descriptors includes: Extracting the semantic representation vector of the image block descriptor through the interactive long short-term memory network unit to obtain an initial image block representation vector; Generate the image block representation vector set by using the image block features and the initial image block representation vector; The second component includes an embedding mapping module, a restoration mapping module and a classifier module, and the focused image representation vector represents the intermediate layer state of the target time step; the second component extracts and debugs the focused image representation vector of the face image learning sample according to the selective focusing strategy to obtain the second face identity information, including: Performing embedding mapping operations on the face image learning samples at each time step through the embedding mapping module to obtain a region representation vector sequence; Determine the hidden variable to be observed through the region representation vector sequence; Extracting semantic representation vectors from the region representation vector sequence by using the selective focusing strategy to obtain semantic embedding representation vectors; Performing restoration mapping on the latent variable to be observed and the semantic embedding representation vector through the restoration mapping module to determine the intermediate layer state at the target time step; Classifying the intermediate layer state through a classifier module to obtain the second face identity information; The step of determining the latent variable to be observed by using the region representation vector sequence includes: Generate an initial indefinite parameter set; Determining the expectation and discreteness corresponding to the region characterization vector sequence through the initial indefinite parameter set; According to the expectation and the discreteness, the hidden variable to be observed of any uncertain parameter in the initial uncertain parameter set is determined.

2. The method according to claim 1, characterized in that The step of integrating the regional feature distribution set and the image block feature distribution set through the downsampling module to obtain an integrated representation vector includes: Performing feature dimension reduction processing on the regional feature distribution set by the downsampling module to obtain a single region representation vector; Performing feature dimension reduction processing on the image block feature distribution set by the downsampling module to obtain a single image block representation vector; The single region representation vector and the single image block representation vector are integrated by the downsampling module to obtain the integrated representation vector.

3. The method according to claim 1 or 2, characterized in that: The method of tuning the pre-prepared neural network by the error between the preset face identity information and the face identity prior information to obtain the identity fusion verification neural network includes: Determining a target cost by using an error between the preset face identity information and the face identity prior information; The network parameters of the neural network prepared in advance are optimized through the target cost to obtain the identity fusion verification neural network.

4. The method according to claim 1 or 2, characterized in that: The first face identity information is used to indicate that the face identity information is a first preset confidence level of a face identity information sample, and the second face identity information is used to indicate that the face identity information is a second preset confidence level of the face identity information sample, and the determining of the preset face identity information through the first face identity information and the second face identity information includes: determining a confidence mean of the first preset confidence level and the second preset confidence level; and determining the confidence mean as the preset face identity information; Alternatively, the method further includes: obtaining environmental parameters corresponding to the face image learning sample; determining a first set influence coefficient of the first face identity information and a second set influence coefficient of the second face identity information through the environmental parameters; determining the preset face identity information through the first face identity information and the second face identity information, including: determining the first face identity information through the multiplication result of the first face identity information and the first set influence coefficient; determining the second face identity information through the multiplication result of the second face identity information and the second set influence coefficient; determining the preset face identity information through the first face identity information and the second face identity information; Alternatively, the pre-prepared neural network also includes a third neural network, and the method also includes: extracting a target face area in the face image learning sample; the target face area is a biometric core area; performing biometric extraction and debugging on the target face area through the third neural network to obtain third face identity information; determining the preset face identity information through the first face identity information and the second face identity information includes: determining the preset face identity information through the first face identity information, the second face identity information and the third face identity information.

5. The method according to claim 1 or 2, characterized in that: The method further comprises: Extracting and debugging the image content representation vector of the face image learning sample through the first component to obtain a first biometric feature; By using the second component and according to the selective focusing strategy, extracting and debugging the focused image representation vector of the face image learning sample to obtain a second biometric feature; Integrate the first biometric feature and the second biometric feature to obtain an integrated biometric feature; The preset facial identity information is determined through the integrated biometric feature.

6. A document self-service signing system based on fusion verification, characterized in that: It includes a server and a signing terminal communicatively connected to the server, the server includes a memory and a processor, the memory stores a computer program that can be run on the processor, and the processor implements the method described in any one of claims 1 to 5 when executing the computer program.