Face information processing method and device, electronic equipment, storage medium and product

Abstract

The embodiment of the invention provides a face information processing method and device, electronic equipment, a storage medium and a product, and relates to the technical field of face information processing. The method comprises the steps of receiving a voice input instruction of a user, inputting the voice input instruction into a voice recognition model to obtain corresponding instruction content, acquiring face information of the user in response to the instruction content, embedding the voice input instruction into the face information to obtain a face-voice embedded vector, and uploading the face-voice embedded vector to a block chain network. According to the embodiment of the invention, the face information is recorded through the voice instruction of the user, the face information and the voice content are stored in the block chain network together, the face recording process can be completed without input or selection of the user, the steps are simplified, and the face information processing efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of face information processing, in particular, the present application relates to a face information processing method and device, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the continuous maturity of face recognition technology, fast entry and recognition of face information can be realized. In the entry stage, the user usually needs to stand in front of the designated collection device, such as a computer or a special terminal with a camera, to capture the user's face image or video sequence using the camera, obtain the face feature vector and store it. In the recognition stage, when the user needs to verify the identity or unlock, the collected face features are compared and verified.

[0003] The current face collection method relies on manual selection or input of face image information for entry, which is complicated, time-consuming, low in face information processing efficiency, and poor in user experience. SUMMARY

[0004] The embodiments of the present application provide a face information processing method, device, electronic equipment, computer readable storage medium and computer program product, aiming at solving the technical problem of low efficiency caused by the complexity of the current face entry and recognition steps.

[0005] In a first aspect, a face information processing method is provided, applied to a node of a blockchain network, the blockchain network including a plurality of nodes including edge processing nodes, consensus nodes and storage nodes, the method comprising:

[0006] receiving a voice entry instruction of a user;

[0007] inputting the voice entry instruction into a voice recognition model to obtain corresponding instruction content;

[0008] in response to the instruction content, collecting face information of the user;

[0009] embedding the voice entry instruction into the face information to obtain a face-voice embedding vector;

[0010] uploading the face-voice embedding vector to the blockchain network.

[0011] Optionally, in response to the instruction content, collecting the face information of the user comprises:

[0012] in response to the instruction content, obtaining a face feature and a head position of the user;

[0013] generating and issuing an adjustment prompt for the head position based on the face feature;

[0014] In response to the adjustment action of the user based on the adjustment prompt, continue to collect the face information of the user; the adjustment action includes rotation or tilting of the head.

[0015] Optionally, the face information includes preliminary face features and supplementary face features.

[0016] In response to the instruction content, collect the face information of the user, including:

[0017] In response to the instruction content, collect the preliminary face features of the user.

[0018] Based on the number of preliminary face features, generate and issue an adjustment prompt.

[0019] In response to the adjustment action of the user based on the adjustment prompt, continue to collect the supplementary face features of the user.

[0020] Optionally, the speech recognition model includes a convolution layer, a recurrent layer, a transformer layer, and a decoding layer.

[0021] Input the speech input instruction into the speech recognition model to obtain the corresponding instruction content, including:

[0022] Input the speech input instruction into the convolution layer to obtain a convolution feature vector.

[0023] Input the convolution feature vector into the recurrent layer to obtain a time sequence feature vector.

[0024] Input the time sequence feature vector into the transformer layer to obtain an acoustic feature sequence.

[0025] Input the acoustic feature sequence into the decoding layer to obtain the corresponding instruction content.

[0026] Optionally, embed the speech input instruction into the face information to obtain a face-speech embedding vector, including:

[0027] Perform feature engineering on the speech input instruction to obtain a content vector.

[0028] Perform feature engineering on the face information to obtain a face vector.

[0029] Embed the content vector and the face vector to obtain the face-speech embedding vector.

[0030] Optionally, upload the face-speech embedding vector to a blockchain network, including:

[0031] Generate an instruction hash according to the instruction content.

[0032] Generate a vector hash according to the face-speech embedding vector.

[0033] The instruction hash and the vector hash are submitted to a consensus node through a smart contract module for multi-node consensus to form a target block.

[0034] The target block and the face-speech embedding vector are stored in association in a corresponding storage node.

[0035] Optionally, the vector hash is generated according to the face-speech embedding vector, including:

[0036] The face-speech embedding vector is homomorphically encrypted to obtain a desensitization vector.

[0037] The vector hash is generated according to the desensitization vector.

[0038] Optionally, the target block and the face-speech embedding vector are stored in association in a corresponding storage node, including:

[0039] The desensitization vector is sharded according to a preset sharding dimension to obtain a plurality of corresponding vector shards and corresponding storage nodes;

[0040] The vector shard is stored in the corresponding storage node to generate a corresponding storage hash;

[0041] The storage hash and the vector hash are stored in association in the corresponding storage node.

[0042] Optionally, the instruction hash and the vector hash are submitted to a consensus node through a smart contract module for multi-node consensus to form a target block, including:

[0043] The instruction hash and the vector hash are submitted to a consensus node through a smart contract module, so that the consensus node compares the instruction hash and the vector hash with a corresponding data hash to determine a similarity degree;

[0044] The target block is formed based on the similarity degree.

[0045] Optionally, the target block is formed based on the similarity degree, including:

[0046] If the similarity degree is lower than a preset threshold, the target block is generated based on the instruction hash and the vector hash.

[0047] Optionally, the method further includes:

[0048] A voice recognition instruction of a user is received; the voice recognition instruction includes to-be-recognized identity information;

[0049] The to-be-recognized identity information is checked in a user feature database;

[0050] If the checking is successful, a preset operation is triggered.

[0051] Optionally, if the checking is successful, the preset operation is triggered, including:

[0052] If the verification is successful, scene information in which the face recognition device is located is determined; the scene information includes a family scene, a community scene, or a commercial scene;

[0053] Based on the scene information and the to-be-identified identity information, a preset operation is triggered; the preset operation includes opening or closing a device in the corresponding scene, sending prompt information to an administrator, automatically opening an access control, pushing specified preferential information, or automatically handling check-in.

[0054] Optionally, in response to the instruction content, face information of the user is collected, including:

[0055] In response to the instruction content, verification information is randomly generated and displayed;

[0056] The user's voice input verification information is received, and it is determined whether the user is a living body;

[0057] If yes, the face information of the user is continuously collected;

[0058] Otherwise, an error prompt is generated and sent.

[0059] In a second aspect, a face information processing apparatus is provided, which is applied to a node of a blockchain network, the blockchain network including a plurality of nodes including an edge processing node, a consensus node, and a storage node. The apparatus includes:

[0060] A receiving module is configured to receive a voice input instruction of a user;

[0061] A voice recognition module is configured to input the voice input instruction into a voice recognition model to obtain corresponding instruction content;

[0062] A face collection module is configured to collect face information of the user in response to the instruction content;

[0063] An embedding module is configured to embed the voice input instruction into the face information to obtain a face-voice embedding vector;

[0064] An uploading module is configured to upload the face-voice embedding vector to the blockchain network.

[0065] In a third aspect, an electronic device is provided, which includes:

[0066] A memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the steps of any method of the first aspect.

[0067] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, the computer program being executed by a processor to implement the face information processing method according to any of the first aspect.

[0068] In a fifth aspect, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of any of the methods of the first aspect.

[0069] The technical scheme provided by the embodiments of the present application has the beneficial effects that:

[0070] The face information processing method provided by the present application simplifies the face input process. A user only needs to input a simple voice instruction to complete the synchronous input of identity information and face information, without the need for other input devices or complex operation processes, thereby greatly improving the input efficiency and user experience. BRIEF DESCRIPTION OF DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced.

[0072] Figure 1 An application scenario diagram of a face information processing method provided by the embodiments of the present application;

[0073] Figure 2 A flowchart of a face information processing method provided by the embodiments of the present application;

[0074] Figure 3 A schematic diagram of saving a vector to a blockchain in a face information processing method provided by the embodiments of the present application;

[0075] Figure 4 A flowchart of an example of a face information processing method provided by the embodiments of the present application;

[0076] Figure 5 A structural diagram of a face information processing device provided by the embodiments of the present application;

[0077] Figure 6 A structural diagram of an electronic device to which the face information processing method provided by the embodiments of the present application is applicable. DETAILED DESCRIPTION

[0078] The embodiments of the present application will be described below in conjunction with the drawings in the present application. It should be understood that the embodiments described below in conjunction with the drawings are exemplary descriptions of the technical solutions of the embodiments of the present application, and do not limit the technical solutions of the embodiments of the present application.

[0079] Those skilled in the art can understand that the singular forms "a", "an" and "the" used herein include plural forms unless expressly stated otherwise. It should be further understood that the terms "include" and "contain" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude other features, information, data, steps, operations, elements, components and / or their combinations supported by the present technology. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can mean that the element and the other element establish a connection relationship through an intermediate element. In addition, "connection" or "coupling" used herein can include wireless connection or wireless coupling. The terms "or", "and / or", "including at least one of the following" and the like used in the present application can be interpreted as inclusive or mean any combination.

[0080] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as processing circuitry or memory) or a combination thereof. Similarly, one processor (or multiple processors or memory) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0081] In the detailed description of the present application, any data related to the user object involved in the process of using the application program by the object, and the like, when the embodiments of the present application are applied to specific products or technologies, the permission or consent of the object needs to be obtained, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of the country and region. That is, if any data related to the object is involved in the embodiments of the present application, these data need to be obtained with the authorization and consent of the object, and in compliance with the relevant laws, regulations and standards of the country and region.

[0082] To make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0083] First, the technical terms involved in the present application are introduced and explained:

[0084] Blockchain network: data is packaged into blocks and linked into a chain through encryption algorithms, ensuring that data is tamper-proof, open and transparent, and traceable, and using consensus mechanisms to let network nodes verify together.

[0085] Smart contract module: a computer program based on blockchain technology, which realizes the automatic execution of contract clauses through preset code logic, and can automatically process data according to preset conditions in the blockchain network without human intervention, significantly improving efficiency and credibility.

[0086] In the prior art, the current face recognition system is often designed to be not simple enough in the user interaction link, resulting in that the user needs to spend a lot of effort to select and operate, the user may need to manually input information and pictures, or repeatedly confirm whether the input environment meets the requirements, adjust the face position to meet the system requirements, or when the recognition fails, judge and select the solution by himself / herself among a series of reasons such as insufficient light and incorrect angle. Such a cumbersome input and selection process not only increases the learning cost of the user, but also greatly reduces the fluency of the face input and recognition process, takes a long time, is low in efficiency, and is poor in user experience.

[0087] The face information processing method and device, the electronic device, the computer readable storage medium and the computer program product provided by the present application aim to solve at least one of the above technical problems in the prior art.

[0088] In view of the above at least one technical problem or the need for improvement in the related art, the present application provides a face information processing method, device, electronic device, computer readable storage medium and computer program product. The face information processing method provided by the scheme receives a voice input instruction of a user, triggers face information collection, obtains instruction content in the voice input instruction through a model, integrates the instruction content and the face information to obtain a face-voice embedding vector, and then uploads the face-voice embedding vector to a blockchain network and saves it. The face information processing method simplifies the face input process, and the user only needs to complete the synchronous input of identity information and face information through a simple voice instruction, without the aid of other input devices or complex operation processes, thereby greatly improving the input efficiency and user experience.

[0089] Further, the embodiments of the present application use a blockchain network to store the input face information, which improves the security and credibility of the stored data. When the face-voice embedding vector is uploaded to the blockchain network, the smart contract module automatically executes the data access permission, thereby completing the chaining of the target block and improving the efficiency of data processing.

[0090] In addition, when performing face recognition, a voice recognition instruction of a user is received, and recognition is triggered based on the voice. In the scenarios of home, community and business, the user's face is automatically recognized, thereby triggering the opening and closing of devices in different scenarios, sending prompt information to the administrator, automatically opening the access control, pushing specified preferential information, or automatically handling the check-in operation. Starting from the use scenario, the convenience and efficiency of user operation are fully considered, the efficiency of face recognition and operation is improved, and the user experience is improved.

[0091] The technical solutions of the embodiments of the present application and the technical effects generated by the technical solutions of the present application will be described below through the description of several exemplary embodiments. It should be pointed out that the following embodiments can be mutually referenced, borrowed or combined. For the same terms, similar features and similar implementation steps in different embodiments, they will not be described repeatedly.

[0092] Figure 1 The application scenario of the face information processing method provided by the embodiments of the present application is shown in the figure, wherein the application environment can include a face entry device 200 in a blockchain network node, and a user 100 can complete voice face entry through the face entry device 200. The above-mentioned blockchain network contains multiple nodes, including edge processing nodes 300, consensus nodes 400 and storage nodes 500.

[0093] Specifically, the face entry device 200 receives the voice entry instruction of the user, inputs the voice entry instruction into the voice recognition model to obtain the corresponding instruction content, responds to the instruction content, collects the face information of the user, embeds the voice entry instruction into the face information, obtains the face-voice embedding vector, and uploads the face-voice embedding vector to the blockchain network.

[0094] The above-mentioned application scenario is only an example, and does not limit the application scenario of the face information processing method of the present application.

[0095] Those skilled in the art can understand that the face entry device can include relevant hardware and software modules installed with the ability to collect, process and store face images or biometric data. The device can be a standalone physical terminal, such as a smart phone, tablet or dedicated face recognition collection terminal with a camera, or a networked system composed of multiple sensors or collection units, or a module integrated in a larger device (such as an access control machine, ATM machine). In addition, it can also be a cloud platform or edge computing node that provides basic services such as cloud services, edge computing, biometric databases, image processing APIs, data encryption, network transmission, security authentication, etc. The device can also serve as a node in a blockchain network, directly participating in the data chaining verification or consensus process, especially in a decentralized identity authentication system. In addition, a smart contract based on blockchain can be deployed on the device side or in the cloud to automatically manage the authorized access, usage rules and sharing permissions of face data, ensuring that only authorized entities can use the data under certain conditions. The distributed storage feature of blockchain can also be used to store part of the information or index of the face data on the chain, while the original data or complete features are stored in a secure off-chain area. The on-chain information is used to manage and access off-chain data, enhancing data security and privacy protection. The present embodiment can be applied to various scenarios that require high security and credibility, including but not limited to digital identity authentication, important area access control, hotel check-in, judicial evidence, supply chain traceability, etc. The specific association method can be determined based on the requirements for data sovereignty, privacy protection level and system credibility in actual application scenarios, which is not limited here.

[0096] In some possible implementations, taking the device in the blockchain network node as an example, the face information processing method provided by the present embodiment can include the following steps: Figure 2 as shown in the figure, which can include the following steps:

[0097] S210, receiving a voice entry instruction of a user.

[0098] The blockchain network includes multiple nodes, including edge processing nodes, consensus nodes and storage nodes.

[0099] The voice entry instruction can include user information.

[0100] Specifically, the voice receiving module in the device obtains the voice entry instruction of the user to obtain the user information. The user information can include destination information and identity information. The destination information indicates what function the user wants to complete, such as face entry or visiting the owner, etc. The identity information indicates the identity of the user, such as I am an employee of the company, etc. The voice entry instruction of the user can trigger the subsequent face entry function, which is convenient and efficient, effectively improving the user experience.

[0101] In the specific embodiment, when the face entry function and the face recognition function can be realized in one device, the mode switching instruction of the administrator can be received before the voice entry instruction of the user is received, the face entry mode is switched based on the mode switching instruction, or the voice input binds the device verification code or the entry password, and the permission of entering the face is controlled, so as to effectively prevent or limit the face entry of the person who does not meet the regulation, and the safety and stability of the system are improved.

[0102] In S220, the voice entry instruction is input into the voice recognition model to obtain the corresponding instruction content.

[0103] Specifically, the voice entry instruction is input into the voice recognition model to obtain the corresponding instruction content. The instruction content can be in the form of text, which is convenient for storage. Before the voice entry instruction is input into the voice recognition model, the voice entry instruction can be simply preprocessed, including noise reduction, encoding or conversion into an instruction vector. The instruction vector is input into the voice recognition model to obtain more accurate instruction content.

[0104] In the specific embodiment, the voice recognition module can be arranged in the device to accurately recognize the voice entry instruction of the user in real time. The voice recognition module can perform voice recognition based on a deep learning algorithm. The voice recognition model can be trained by a large amount of voice data. The voice recognition model can recognize voice content with different accents, speeds and tones, and has strong language understanding ability. Even if the voice content is in a dialect accent or out of order, the key information can also be accurately extracted, and the voice can be converted into text to provide a basis for subsequent matching with the face information.

[0105] In the specific embodiment, the training method of the voice recognition model includes: obtaining a voice sample set, the voice sample set containing a plurality of voice samples, each voice sample being marked with a corresponding text label, inputting the voice samples into an initial voice recognition model one by one to obtain a text result, updating the parameters of the initial voice recognition model according to the output text result and the text label, until a predetermined end condition is reached, and the training is ended to obtain a trained voice recognition model.

[0106] In the embodiment of the present application, during training, a voice sample set containing a plurality of voice samples can be obtained first, each voice sample containing audio data and being marked with a corresponding text label. Then, the plurality of voice samples are divided into a training set, a validation set and a test set according to a predetermined proportion. Then, the parameters of the acoustic model and the language model in the voice recognition model are adjusted and determined according to the voice samples included in the training set, the validation set and the test set, to obtain a trained voice recognition model.

[0107] It should be noted that the voice samples described above include voice samples with environmental noise and accents to improve the model's recognition ability for noise and various accents. When preprocessing the voice signal, the voice samples generated by environmental recordings with strong noise are appropriately processed to achieve a certain degree of noise suppression.

[0108] When training the model, the voice sample set can be divided into a training set, a validation set, and a test set. Then, based on the training set, the model is trained, based on the validation set, the model is validated, and based on the test set, the model is tested to obtain the trained voice recognition model.

[0109] Before training based on the training set, the audio samples in the training set can be preprocessed. The preprocessing includes sampling rate adjustment, framing, windowing, Fourier transform (such as MFCC feature extraction), normalization, and the like.

[0110] The specific method of data augmentation is to perform time domain transformation on the audio data (including adding noise, changing speed, changing pitch, reverberation, etc.), and also perform frequency domain transformation (such as filtering). These operations can better expand the performance of the voice in different environments.

[0111] In some embodiments, a generative adversarial network (GAN) can also be used to generate some voice samples that are difficult to recognize but not easy to obtain, such as voice samples with extreme noise, multiple people speaking, unclear pronunciation, and various interference sounds. This helps the algorithm model accurately recognize the voice content, reduces misrecognition and missed recognition, and improves the accuracy of the model in recognizing special voices.

[0112] After obtaining the augmented training set, the voice recognition model can be trained based on the augmented training set to update the parameters and weights in the network.

[0113] Specifically, the audio samples in the training set are input into the voice recognition model to obtain the text result output by the voice recognition model. The text result and the text label are compared to calculate the loss function (such as CTC loss or cross-entropy loss). Then, using the stochastic gradient descent method (or its variants) to minimize the loss function, the parameters and weights in the voice recognition model are updated by backpropagation until the loss function meets the predetermined condition, such as convergence or less than a predetermined threshold.

[0114] In some embodiments, the text result includes a recognized word sequence or a phoneme sequence, and the text label can include a corresponding word sequence label or a phoneme sequence label. When calculating the loss function, the comparison between the recognized sequence and the label sequence is obtained.

[0115] After training, the speech recognition model whose parameters are updated based on the training set can be verified based on the verification set. Specifically, the model is debugged based on the verification set data, and when the loss function meets the predetermined condition, the model parameters of this stage are output. If the loss function does not meet the predetermined condition, the learning rate and other hyperparameters are automatically adjusted, and the next round of network model training is performed.

[0116] When the loss function calculated on the verification set meets the predetermined condition, the parameters and weights can be retained, and then the retained parameters are tested based on the test set. Specifically, the speech recognition model with retained parameters and weights can output text results and model weights by inputting test set data. According to the comparison of the model loss and the corresponding weights of multiple rounds, the model weight with the minimum loss is output, and the trained speech recognition model is determined. After obtaining the trained speech recognition model, the speech-to-text conversion can be completed based on the speech recognition model.

[0117] In addition, data enhancement can not be performed for data input in the verification set and the test set.

[0118] S230, in response to the instruction content, collecting face information of the user.

[0119] The face information includes face features.

[0120] Specifically, in response to the instruction content, the face image of the user is collected through the face collection module, the face features are extracted from the face image to obtain the corresponding face information, and then the instruction content and the face information are stored in association. When face recognition is performed, the corresponding face data can be quickly found based on the instruction content or the face features, and the face recognition efficiency is improved.

[0121] In specific embodiments, the face feature extraction model can be used to extract the face features of the user. The collected face image is input into the face feature extraction model, and the convolutional neural network is used to accurately identify the facial features, facial contours, etc. of the face to obtain a unique face feature vector. Feature engineering can also be performed on the collected face image to obtain the corresponding face feature vector. Using a high-precision face recognition algorithm can quickly analyze and extract features from the collected face image, and speed up the face registration efficiency.

[0122] In the specific implementation, the training method of the face feature extraction model specifically includes: obtaining a face image sample set containing a plurality of face image samples, each face image sample being marked with a corresponding identity label, inputting the face image samples one by one into the face feature extraction model to obtain a feature vector result, updating parameters of the face feature extraction model according to the output feature vector result and the identity label, until a predetermined end condition is reached, ending the training, and obtaining the trained face feature extraction model.

[0123] In the embodiments of the present application, during training, a face image sample set containing a plurality of face image samples can be obtained first, each face image sample containing a face of a different individual and being marked with a corresponding identity label; then the plurality of face image samples are divided into a training set, a validation set and a test set according to a predetermined proportion, and then the parameters of the network layer in the face feature extraction model are adjusted and determined according to the face image samples included in the training set, the validation set and the test set, to obtain the trained face feature extraction model.

[0124] It should be noted that the face image samples described above include face image samples with pose changes, illumination changes, occlusions (such as glasses, masks) and different age stages, so as to improve the face representation ability of the model under different conditions. When post-processing the face image, the face region can be aligned and normalized to reduce the interference of irrelevant factors.

[0125] During model training, the face image sample set can be divided into a training set, a validation set and a test set, then training is performed based on the training set, validation is performed based on the validation set, and testing is performed based on the test set, to obtain the trained face feature extraction model.

[0126] Before training based on the training set, the face image samples in the training set can be preprocessed. The preprocessing includes image size adjustment, normalization, data enhancement and identity encoding.

[0127] The specific method of data enhancement is to perform geometric transformation (including flipping, rotation, translation, scaling, etc.) on the data image, and also perform color transformation (including brightness, contrast, saturation adjustment, noise addition, Gaussian blur, etc.). These operations can better expand the diversity of face images.

[0128] In some embodiments, a convolutional neural network can also be used to generate some face images, such as images containing extreme illumination, severe occlusion, different age stages, and images with noise, which are difficult to identify but not easy to obtain. This helps the algorithm model to accurately extract face features, reduces the phenomenon of false matching and missed matching, and improves the robustness of the model in feature extraction of special face images.

[0129] After obtaining the enhanced training set, the facial feature extraction model can be trained based on the enhanced training set to update the parameters and weights in the network.

[0130] Specifically, facial image samples from the training set are input into the facial feature extraction model, generating a feature vector output by the model. The resulting feature vector is then compared to the identity label (typically using a metric learning loss function, comparing it to other sample feature vectors or identity labels) to calculate a loss function. The loss function is then minimized using stochastic gradient descent (or its variants). The parameters and weights in the facial feature extraction model are updated using backpropagation until the loss function meets predetermined conditions, such as convergence or being less than a predetermined threshold.

[0131] In some embodiments, the resulting feature vectors are used to represent facial images, and identity labels are used to distinguish different individuals. When calculating the loss function, the distance between feature vectors (e.g., Euclidean distance, cosine similarity) is compared with the identity label (whether they are the same person) (for example, feature vectors of the same person should have smaller distances, while feature vectors of different people should have larger distances).

[0132] After training, the facial feature extraction model, whose parameters were updated based on the training set, can be verified using the validation set. Specifically, the facial feature extraction model is debugged based on the validation set data. When the loss function meets the predetermined conditions, the model parameters for that stage are output. If the loss function does not meet the predetermined conditions, hyperparameters such as the learning rate are automatically adjusted, and the next round of network model training is carried out.

[0133] When the loss function calculated on the validation set meets predetermined conditions, the parameters and weights are retained, and then the retained parameters are tested on the test set. Specifically, the test set data is input so that the facial feature extraction model with the retained parameters and weights can output feature vector results and model weights. Based on the comparison of the model losses and corresponding weights over multiple rounds, the model weight with the minimum loss is output to determine the trained facial feature extraction model.

[0134] After obtaining the trained facial feature extraction model, the feature vector of the facial image can be extracted based on the model for tasks such as face recognition and face verification.

[0135] In addition, no data augmentation is required for the data input in the validation set and the test set.

[0136] S240: embed the voice recording instruction into the face information to obtain a face-voice embedding vector.

[0137] Specifically, the process of embedding the voice input instruction into the face information and generating a face-voice embedding vector can include four steps of voice feature extraction, face information extraction, feature fusion, and embedding vector generation. The voice feature of the voice input instruction is extracted using Mel Frequency Cepstral Coefficients (MFCC) or Linear Predictive Coding (LPC), or using a deep learning model to obtain a voice feature vector. The face in the image is detected and located, and then a face feature extraction model is used to extract key features of the face to generate a face feature vector. The voice feature vector and the face feature vector are fused, including vector splicing, attention mechanism, or multi-modal neural network fusion. Then, the face-voice embedding vector is generated through the fused feature vector.

[0138] In the specific embodiment, after the voice recognition module converts the user's voice into text and the face recognition module extracts the face feature vector, the central processing unit of the device associates the two. By establishing an efficient data index and matching algorithm, a large amount of data can be compared in a short time when performing face recognition, ensuring that the face information and the corresponding identity information are accurately matched, and improving the face recognition efficiency.

[0139] For example, the user stands in front of the input device and introduces himself through voice, such as "I am Grandpa". The input system of the device automatically starts the face recognition program, scans the user's face information, and accurately matches the recognized face information with the voice content "Grandpa", and then completes the input process.

[0140] In the specific embodiment, by embedding the voice input instruction into the face information, the voice recognition and face recognition technologies are deeply integrated. The user only needs to use simple voice content, such as self-introduction, to complete the synchronous input of identity information and face information, without the need for other input devices or complex operation processes to achieve input, greatly improving the input efficiency and user experience, and providing a new, convenient and efficient face recognition input method for users.

[0141] S250, uploading the face-voice embedding vector to the blockchain network.

[0142] Specifically, the face-voice embedding vector is encrypted and compressed for preprocessing, then a suitable blockchain network is selected, a data packet containing the embedding vector is constructed, metadata is added, and the data packet is digitally signed using the user's private key to ensure that the data source is verifiable. The signed upload record is broadcast to the network nodes, the nodes verify the legality of the record, and through the consensus mechanism, the record is confirmed and recorded on the blockchain. After successful chaining, the data is permanently stored, and users can retrieve it through the blockchain browser. The smart contract can automatically perform related operations such as identity authentication, effectively ensuring the secure storage of data, and providing a foundation for identity authentication, access control, and other applications. The decentralized nature of the blockchain enhances the system's tamper resistance and credibility.

[0143] In the specific implementation, the feature vector corresponding to the face information and the corresponding voice input instruction can also be stored in the local storage device of the device or in the cloud server, ensuring the reliability and security of the data, protecting user data privacy with encryption technology, and supporting fast data retrieval and calling to quickly obtain matching information in subsequent face recognition processes.

[0144] In some possible implementations, the face information of the user is collected in response to the instruction content, including:

[0145] The face features and head position of the user are obtained in response to the instruction content.

[0146] An adjustment prompt for the head position is generated and issued based on the face features.

[0147] The face information of the user is continuously collected in response to the adjustment action of the user based on the adjustment prompt.

[0148] The adjustment action includes rotation or tilting of the head.

[0149] Specifically, the device collects face information, which can first detect whether the current frame contains a complete face. If it is not complete, the angle or direction of the device can be adjusted to fully expose the face, thereby collecting complete face information. In response to the instruction content, the face features of the user are obtained, and the current head position is determined. If the collected face features are incomplete or insufficient in number, an adjustment prompt for the head position can be generated and issued based on the current head position. In response to the adjustment action of the user based on the adjustment prompt, the face information of the user is continuously collected until complete face feature information is obtained.

[0150] In the specific embodiment, in the face registration process, the face collection unit of the device can guide the user to complete the collection of face images at different angles in real time according to the user's actions and postures through voice guidance. The voice guidance function generates corresponding adjustment prompts based on the intelligent analysis of the user's real-time actions. For example, the camera analyzes the user's current head position and angle, judges whether the user needs to further adjust the posture to obtain more complete face features, and dynamically adjusts the voice guidance content. Compared with the fixed collection angle of the traditional face recognition device or the way of relying only on the user's self-adjustment of the posture, this method can more accurately and comprehensively collect face information, significantly improve the accuracy and stability of face recognition, and the advantage is more obvious especially in complex environments and different user cooperation levels.

[0151] For example, the collection unit of the device, such as a camera, can guide the user to show the left and right sides of the face based on voice prompts, so as to obtain more accurate and comprehensive face recognition effect. For example, the camera issues a prompt sound: "Please slowly turn your head to the left" or "Please slowly turn your head to the right", guiding the user to complete the collection of face information at different angles. When the user shows the left and right sides of the face according to the guidance of the camera, the face recognition algorithm can perform fusion analysis on the face features at different angles to ensure accurate identification of the user's identity in various postures and improve the recognition accuracy. By setting such a function in the device, it can be ensured that in various complex scenes, the face information collected by the camera has enough feature points, thereby significantly improving the accuracy and stability of subsequent face recognition.

[0152] In some possible embodiments, the face information of the user is collected in response to the instruction content in the above steps, including:

[0153] The preliminary face features of the user are collected in response to the instruction content.

[0154] The adjustment prompt is generated and issued based on the number of preliminary face features.

[0155] The supplementary face features of the user are collected in response to the adjustment action of the user based on the adjustment prompt.

[0156] The face information includes the preliminary face features and the supplementary face features.

[0157] Specifically, in response to the instruction content of the user, the preliminary face features of the user are collected, the number of obtained preliminary face features is determined, if the number is less than a preset value, the adjustment prompt is generated and issued based on the number of preliminary face features, and the supplementary face features of the user are collected in response to the adjustment action of the user based on the adjustment prompt. The preliminary face features and the supplementary face features, that is, the complete face information, are obtained. The number of preliminary face features can include the number of key feature points contained in the preliminary face features.

[0158] In the specific embodiment, the key feature points can include specific positions of eyes, nose, mouth, eyebrows and the like, can be two-dimensional key points or three-dimensional key points, and in the implementation process, the key points can be labeled according to actual needs. When the number of preliminary facial features collected is small, a corresponding adjustment prompt is generated. After the user adjusts the position of the face or head, the facial features are continuously collected, and the feature points that have been collected can also be collected again. Then, the features of the same position collected at different angles are fused, and more accurate facial feature information can be obtained.

[0159] In some possible embodiments, the step of inputting the voice input instruction into the voice recognition model to obtain the corresponding instruction content includes:

[0160] inputting the voice input instruction into a convolution layer to obtain a convolution feature vector;

[0161] inputting the convolution feature vector into a recurrent layer to obtain a time sequence feature vector;

[0162] inputting the time sequence feature vector into a transformer layer to obtain an acoustic feature sequence;

[0163] inputting the acoustic feature sequence into a decoding layer to obtain the corresponding instruction content.

[0164] The voice recognition model includes the convolution layer, the recurrent layer, the transformer layer and the decoding layer.

[0165] Specifically, the voice input instruction is input into the convolution layer to extract acoustic features and obtain a convolution feature vector. The convolution feature vector is input into the recurrent layer to determine short-time dependence and generate a time sequence feature vector. The time sequence feature vector is input into the transformer layer to capture long-time dependence and complex acoustic relationship by using a self-attention mechanism, and generate an acoustic feature sequence. The acoustic feature sequence is input into the decoding layer to gradually predict text units by using an autoregressive manner, and obtain the corresponding instruction content.

[0166] In the specific embodiment, the training method of the voice recognition model specifically includes: obtaining a voice sample set, the voice sample set containing a plurality of voice samples, each voice sample being marked with a corresponding text label, inputting the voice samples into an initial voice recognition model one by one to obtain text results, updating parameters of the initial voice recognition model according to the output text results and the text labels, until a predetermined end condition is reached, ending the training, obtaining a trained voice recognition model, and the end condition can include that a loss function meets a preset condition, the loss function L CE may be a cross-entropy loss function, and the formula is as follows:

[0167]

[0168] where c is all possible character classes, y c is the true text label of character c, p c is the probability that the model predicts character c.

[0169] In some possible implementations, the embedding of the voice input instruction into the face information in the above step to obtain the face-voice embedding vector includes:

[0170] performing feature engineering on the voice input instruction to obtain a content vector;

[0171] performing feature engineering on the face information to obtain a face vector;

[0172] performing embedding processing on the content vector and the face vector to obtain the face-voice embedding vector.

[0173] The content vector can include MFCC (Mel Frequency Cepstral Coefficient), FBANK (Filter Bank Cepstral Coefficient), acoustic features (fundamental frequency, energy), and spectrogram features, and the face vector can include facial key point features and contour features.

[0174] Specifically, the feature engineering on the voice input instruction includes the steps of framing, windowing, denoising, and feature extraction, etc., to obtain the content vector, the feature engineering on the face information includes projecting high-dimensional face image data into a low-dimensional space to generate the face vector, and the embedding processing on the content vector and the face vector includes obtaining the face-voice embedding vector.

[0175] In some possible implementations, as shown in Figure 3 the above step of uploading the face-voice embedding vector to the blockchain network includes:

[0176] S310, generating an instruction hash according to the instruction content;

[0177] S320, generating a vector hash according to the face-voice embedding vector;

[0178] S330, submitting the instruction hash and the vector hash to the consensus node through the smart contract module for multi-node consensus to form a target block;

[0179] S340, storing the target block and the face-voice embedding vector in association to the corresponding storage node.

[0180] Specifically, determine the instruction hash algorithm for the instruction content, input the instruction content into the instruction hash algorithm, generate the corresponding instruction hash, determine the vector hash algorithm for the face-voice embedding vector, input the vector content into the vector hash algorithm, generate the corresponding vector hash, submit the instruction hash and the vector hash to the consensus node through the smart contract module for multi-node consensus, the consensus node verifies the submitted hash value to ensure the integrity and consistency of the data, and the hash value is verified and passed by a sufficient number of nodes, a consensus is reached, the consensus result is recorded on the blockchain to ensure that it is tamper-proof and traceable, and then the target block is formed and stored in the corresponding storage node in association with the face-voice embedding vector.

[0181] In the specific embodiment, during the subsequent face recognition process, the smart contract module can realize face comparison. After the smart contract is started, the system collects the face image of the user to obtain the face-voice embedding vector, and compares the face-voice embedding vector with the data stored on the blockchain. According to the comparison result, the smart contract will perform different operations. If the comparison is successful, the smart contract will store the target block and the face-voice embedding vector in association in the storage node on the blockchain. At the same time, the contract will record the timestamp and event identification and other key information of the operation to ensure the traceability of the verification process. If the comparison fails, the smart contract will trigger a re-verification process and prompt the user to re-collect the face image.

[0182] In some possible embodiments, the step of generating a vector hash according to the face-voice embedding vector includes:

[0183] Homomorphic encryption is performed on the face-voice embedding vector to obtain a desensitization vector.

[0184] The vector hash is generated according to the desensitization vector.

[0185] The homomorphic encryption includes partial homomorphic encryption and full homomorphic encryption, which can be selected based on actual needs.

[0186] Specifically, the homomorphic encryption method is determined, and the face-voice embedding vector is processed by homomorphic encryption: a key pair is generated, including a public key for encryption and a private key for decryption, the vector is mapped and converted into ciphertext form according to the encryption rule, and after the encryption process is completed, the ciphertext vector obtained is the required desensitization vector. The desensitization vector is hashed to obtain the vector hash. The above steps make the encrypted data appear random and unable to directly identify the original information, and only those who have the corresponding private key can decrypt, protecting data privacy.

[0187] In some possible embodiments, the step of storing the target block and the face-voice embedding vector in association in the corresponding storage node includes:

[0188] According to a preset sharding dimension, the desensitization vector is sharded to obtain a plurality of corresponding vector shards and corresponding storage nodes;

[0189] The vector shard is stored in the corresponding storage node to generate a corresponding storage hash;

[0190] The storage hash and the vector hash are stored in the corresponding storage node.

[0191] Specifically, the sharding strategy is determined, including the sharding dimension, the desensitization vector is divided into a plurality of vector shards based on the sharding dimension, each vector shard is assigned a storage node, the vector shard is stored in the corresponding storage node, the corresponding storage hash is generated, the storage hash and the vector hash are stored in the corresponding storage node, by storing each vector shard in its corresponding storage node, each node only stores a part of the original vector, and the complete vector is not stored, thereby improving the distribution of data and potential fault tolerance.

[0192] In the specific embodiment, the step of assigning a storage node can include assigning the vector shard to the node list in order, or performing a hash operation on the shard identifier (such as a shard number or a vector hash), determining the storage node according to the hash value, or using a consistent hash ring to assign, facilitating dynamic addition and reduction of nodes, or assigning the shard to a node with lower current load according to the node load.

[0193] In some possible embodiments, the instruction hash and the vector hash are submitted to the consensus node through the smart contract module in the above steps to perform multi-node consensus to form a target block, including:

[0194] The instruction hash and the vector hash are submitted to the consensus node through the smart contract module, so that the consensus node compares the instruction hash and the vector hash with the corresponding data hash to determine the similarity;

[0195] Based on the similarity, the target block is formed.

[0196] The data hash can include an instruction data hash and a vector data hash, and the data hash is a hash value saved in the storage node.

[0197] Specifically, the instruction hash and the vector hash are submitted to the consensus node through the smart contract module to obtain the corresponding data hash, the instruction hash, the vector hash and the data hash are converted into a string of the same format, a comparison method is selected, the similarity is determined based on the comparison method, and the target block is generated based on the similarity, so that the target block is chained, wherein the comparison method can include Hamming distance, cosine similarity, edit distance and similarity analysis model.

[0198] In the specific embodiment, if the comparison mode is Hamming distance, the strings corresponding to the instruction hash, the vector hash and the data hash can be compared bit by bit, the number of different bits between the instruction hash and the instruction data hash is calculated, and the number of different bits between the vector hash and the vector data hash is calculated to obtain the Hamming distance, and the similarity degree of the two hash values is obtained. The similarity degrees are comprehensively considered to determine the similarity degree with the consensus node. The smaller the Hamming distance is, the more similar the hash values are. If the comparison mode is cosine similarity, the instruction hash, the vector hash and the data hash are converted into vector values. The cosine similarity between the vectors of the instruction hash and the instruction data hash is compared, and the cosine similarity between the vectors corresponding to the vector hash and the vector data hash is calculated to obtain the similarity degree. If the comparison mode is edit distance, the number of characters that can be inserted, deleted or replaced to change one hash string into another is calculated. The instruction hash is changed into the instruction data hash, and the vector hash is changed into the vector data hash. The smaller the edit distance is, the higher the similarity degree is.

[0199] In the specific embodiment, if the comparison mode is the similarity analysis model, the instruction hash and the instruction data hash can be input into the similarity analysis model to obtain the instruction similarity, the vector hash and the vector data hash can be input into the similarity analysis model to obtain the vector similarity, and the instruction similarity and the vector similarity are weighted and considered to determine the similarity degree. The training process of the similarity analysis model can include: obtaining a plurality of hash sample pairs, each hash sample pair being marked with a corresponding similarity label, inputting the hash sample pairs into an initial analysis model one by one to obtain a similarity result, updating the parameters of the initial analysis model according to the output similarity result and the similarity label, until a predetermined end condition is reached, ending the training, and obtaining the trained similarity analysis model.

[0200] In some possible embodiments, the target block is formed based on the similarity degree in the above step, including:

[0201] If the similarity degree is lower than a preset threshold, the target block is generated based on the instruction hash and the vector hash.

[0202] Specifically, the similarity degree is compared with the preset threshold. If the similarity degree is lower than the preset threshold, it indicates that the face-voice embedding vector is not stored in the blockchain, that is, the face information and the instruction content of the related user are not stored in the blockchain, and the content needs to be added. The target block is generated based on the instruction hash and the vector hash, and the target block is added to the blockchain. If the similarity degree is equal to or greater than the preset threshold, it indicates that the corresponding face feature and instruction content already exist in the blockchain. The original face-voice embedding vector can be updated or the face can be re-entered.

[0203] In the specific embodiment, the instruction hash and the vector hash are packaged into a candidate block, the candidate block is verified by running a smart contract on a node in the blockchain network, and the consensus nodes reach consensus on the candidate block through a proof of work (PoW), a proof of stake (PoS), a practical byzantine fault tolerance (PBFT) protocol, or other mechanisms. After reaching consensus, the candidate block is added to the blockchain as a target block.

[0204] In some possible embodiments, the method further includes:

[0205] receiving a voice recognition instruction of the user;

[0206] checking the to-be-identified identity information in the user feature database;

[0207] if the checking is successful, triggering a preset operation.

[0208] The voice recognition instruction includes the to-be-identified identity information and destination information. The user feature database can include simple user features for preliminary user identification, such as features of community visitors, features of supermarket customers, and features of home visitors.

[0209] Specifically, when face recognition is needed, a voice recognition instruction of the user is received, and it is determined whether face recognition or other operations are needed based on the destination information in the voice recognition instruction. For example, if a visitor wants to access a resident, a reminder message is sent to the resident, and if the resident himself wants to open the door, face recognition is performed and the to-be-identified identity information of the user is verified. Specifically, the to-be-identified identity information can be checked in the user feature database to determine the user identity, or the face recognition information obtained and the to-be-identified identity information can be compared with information in a blockchain node to verify the user identity. If the checking is successful, a corresponding preset operation is triggered. The to-be-identified identity information can also be checked in the user feature database first. If the checking is passed, face information is continuously collected for face recognition verification. The convenience and efficiency of user operation are fully considered to improve the user experience.

[0210] In the specific embodiment, the device can receive a voice recognition instruction of the user and automatically perceive the user's needs to respond without complicated settings and operations. For example, when the user stands in front of the device and starts speaking, the device automatically starts the voice recognition and face recognition process, completes face verification, and reduces the user waiting time. During the identification process, if inaccurate identification or information missing occurs, a prompt is generated and sent to remind the user to re-operate or supplement information, thereby providing the user with an intelligent use experience throughout the process.

[0211] In some possible embodiments, if the checking is successful in the above steps, the preset operation is triggered, including:

[0212] If the verification is successful, scene information in which the face recognition device is located is determined;

[0213] Based on the scene information and the to-be-identified identity information, a preset operation is triggered.

[0214] The scene information includes a family scene, a community scene, or a commercial scene; and the preset operation includes opening or closing of a device in the corresponding scene, sending of prompt information to an administrator, automatic opening of an access control, pushing of specified preferential information, or automatic check-in.

[0215] Specifically, if the verification is successful, scene information in which the face recognition device is located is determined. The device can also collect image and voice information of the surrounding environment of a user to determine the scene. Based on the scene information and the to-be-identified identity information, a preset operation is triggered. Different scenes are set with different operations to meet the face verification needs of the user in different scenes.

[0216] In the specific implementation, in the family scene, face information of family members is uploaded to a blockchain through voice input and saved. A recognition device is arranged at the door of the house. When a person is identified to enter the door, the face of the person is collected to determine corresponding identity information. If it is identified that the person is a family member coming home, the family member can be associated with smart devices in the house to realize opening and closing of lighting, fans, air conditioners, and curtain devices, and the like, to provide a more convenient and comfortable living environment for the family member. If the family member is a child, when the child comes home after school, the device can send prompt information to a mobile phone of a parent (an administrator) to inform that the child has safely come home. If a stranger is identified to break in, a camera can immediately issue an alarm and notify the family member to ensure the safety of the family.

[0217] In the specific implementation, in the community scene, residents of the community input and save face information and identity information to the blockchain through voice instructions. A recognition device is installed at an entrance of the community. When a resident enters the community, the resident only needs to say his / her identity through voice, for example, I am Zhang San from 1# building 1 unit. The device can collect face information to complete face and identity recognition, and automatically open the access control of the community, without the need of the user to manually swipe a card or input a password, greatly improving the efficiency of passing through, and avoiding inconvenience caused by forgetting to bring a card, and the like. If a visitor wants to enter the community, the visitor's destination and identity information in the voice are received, for example, I want to visit Wang Wu from 1# building 1 unit Li Si. The device can save the visitor information based on the voice content and send prompt information to the corresponding visitor. After confirmation of the visitor, the visitor can remotely authorize the visitor to enter and open the access control. The scheme of the application can improve the security and intelligent level of the management of the access control of the community.

[0218] In the specific embodiment, in the commercial scene such as a shopping mall, a supermarket, etc., the device can be installed at the entrance or the cashier, and the member customers and ordinary customers can be identified by the device. The face information of the member customers has been recorded and saved to the blockchain. When the customers enter the store, the face information of the customers is collected, the collected information is compared with the data in the blockchain node, it is determined whether the customers are members, the exclusive preferential information is automatically pushed to the member customers, the shopping experience of the customers is improved, and if the customers are ordinary customers, the product information is pushed to improve the shopping experience of the users. If in the hotel scene, when the customers check in, the voice is collected and the face information of the customers is recorded and matched with the features in the user feature database, the identity is confirmed according to the matching result and the face information of the user is recorded, the check-in procedure of the customers is quickly handled, the waiting time of the customers is shortened, and the service efficiency is improved. The user feature database in the hotel scene can be the identity feature database of the public security system.

[0219] In some possible embodiments, the face information of the user is collected in response to the instruction content, including:

[0220] The verification information is randomly generated and displayed in response to the instruction content.

[0221] The verification information input by the user is received, and it is determined whether the user is a living body.

[0222] If yes, the face information of the user is continuously collected.

[0223] Otherwise, an error prompt is generated and sent.

[0224] The verification information can be a random string including numbers, letters and / or characters.

[0225] Specifically, the verification information is randomly generated and displayed on the display interface of the device in response to the instruction content, the verification information input by the user is received, it is determined whether the user is a living body, if yes, the face information of the user is continuously collected, otherwise, an error prompt is generated and sent, and the face recording or recognition is performed after the living body face is verified. This can effectively avoid the malicious recording operation of the picture or video, and improve the data security and stability of the system.

[0226] In the specific embodiment, the verification information input by the user is received to obtain audio data, and when the user inputs the verification information, the video data is collected by the camera in the device, and it is determined whether the user is a living body based on the audio data and the video data. This can improve the accuracy of the determination and maintain the safety and stability of the data of the device.

[0227] In the above embodiments, by receiving the voice input instruction of the user, the face information collection is triggered, the instruction content in the voice input instruction is obtained through the model, the face-voice embedding vector is obtained by integrating the instruction content and the face information, and then the face-voice embedding vector is uploaded to the blockchain network and saved, thereby simplifying the face input process. The user only needs to complete the synchronous input of the identity information and the face information through a simple voice instruction, without the help of other input devices or complex operation processes, thereby greatly improving the input efficiency and user experience.

[0228] Further, the embodiments of the present application use the blockchain network to store the input face information, thereby improving the security and credibility of the stored data. When the face-voice embedding vector is uploaded to the blockchain network, the data access permission is automatically executed through the smart contract module, thereby completing the chaining of the target block and improving the data processing efficiency.

[0229] In addition, when performing face recognition, the voice recognition instruction of the user is received, and the user's face is automatically recognized based on the voice trigger recognition in the scenes of home, community and business, thereby triggering the opening and closing of the device in different scenes, sending prompt information to the administrator, automatically opening the access control, pushing the specified preferential information or automatically handling the check-in operation. Starting from the use scene, the convenience and efficiency of user operation are fully considered, thereby improving the efficiency of face recognition and operation and improving the user experience.

[0230] In one example, the face information processing method of the present application, as shown in Figure 4 may include:

[0231] receiving a voice input instruction of a user;

[0232] inputting the voice input instruction into a voice recognition model to obtain corresponding instruction content;

[0233] collecting face information of the user, embedding the voice input instruction into the face information, and obtaining a face-voice embedding vector;

[0234] generating an instruction hash according to the instruction content, and generating a vector hash according to the face-voice embedding vector;

[0235] submitting the instruction hash and the vector hash to a consensus node through a smart contract module for multi-node consensus to determine a similarity degree;

[0236] determining whether the similarity degree is lower than a preset threshold;

[0237] if yes, generating a target block, and storing the target block and the face-voice embedding vector in a storage node in a blockchain network;

[0238] if no, updating the data stored in the blockchain network.

[0239] The face information processing method simplifies the face entry process. The user only needs to complete the synchronous entry of identity information and face information through a simple voice instruction, without the aid of other input devices or complex operation processes, greatly improving the entry efficiency and user experience.

[0240] Further, the face information entered by the embodiment of the application is stored in the blockchain network, improving the security and credibility of the stored data. When the face-voice embedding vector is uploaded to the blockchain network, the data access permission is automatically executed by the smart contract module, thereby completing the chaining of the target block, and improving the efficiency of data processing.

[0241] In addition, when performing face recognition, the voice recognition instruction of the user is received, and voice triggering recognition is performed. In home, community and commercial scenarios, the user's face is automatically recognized, thereby triggering the opening and closing of devices in different scenarios, sending prompt information to the administrator, automatically opening the access control, pushing specified preferential information or automatically handling the check-in operation. Starting from the use scenario, the convenience and efficiency of user operation are fully considered, the efficiency of face recognition and operation is improved, and the user experience is improved.

[0242] The embodiment of the application provides a face information processing device, as shown in Figure 5 The face information processing device 50 can include a receiving module 510, a voice recognition module 520, a face collection module 530, an embedding module 540 and an uploading module 550, wherein

[0243] The receiving module 510 is configured to receive a voice entry instruction of a user.

[0244] The voice recognition module 520 is configured to input the voice entry instruction into a voice recognition model to obtain corresponding instruction content.

[0245] The face collection module 530 is configured to collect face information of the user in response to the instruction content.

[0246] The embedding module 540 is configured to embed the voice entry instruction into the face information to obtain a face-voice embedding vector.

[0247] The uploading module 550 is configured to upload the face-voice embedding vector to a blockchain network.

[0248] As an optional embodiment, the face collection module 530 in the device is specifically configured to:

[0249] In response to the instruction content, the face feature and the head position of the user are acquired;

[0250] Based on the face feature, an adjustment prompt of the head position is generated and sent out;

[0251] In response to the adjustment action of the user based on the adjustment prompt, the face information of the user is continuously collected; the adjustment action includes rotation or inclination of the head.

[0252] As an optional embodiment, in the device, the face collection module 530 is specifically used for:

[0253] In response to the instruction content, the preliminary face feature of the user is collected;

[0254] Based on the number of the preliminary face feature, the adjustment prompt is generated and sent out;

[0255] In response to the adjustment action of the user based on the adjustment prompt, the supplementary face feature of the user is continuously collected.

[0256] As an optional embodiment, in the device, the voice recognition module 520 is specifically used for:

[0257] The voice input instruction is input into a convolution layer to obtain a convolution feature vector;

[0258] The convolution feature vector is input into a recurrent layer to obtain a time sequence feature vector;

[0259] The time sequence feature vector is input into a transformer layer to obtain an acoustic feature sequence;

[0260] The acoustic feature sequence is input into a decoding layer to obtain the corresponding instruction content.

[0261] As an optional embodiment, in the device, the embedding module 540 is specifically used for:

[0262] Feature engineering is performed on the voice input instruction to obtain a content vector;

[0263] Feature engineering is performed on the face information to obtain a face vector;

[0264] The content vector and the face vector are embedded to obtain a face-voice embedding vector.

[0265] As an optional embodiment, in the device, the uploading module 550 is specifically used for:

[0266] An instruction hash is generated according to the instruction content;

[0267] A vector hash is generated according to the face-voice embedding vector;

[0268] The instruction hash and the vector hash are submitted to the consensus node through the smart contract module for multi-node consensus to form a target block.

[0269] The target block and the face-speech embedding vector are stored in the corresponding storage node.

[0270] As an optional embodiment, the uploading module 550 is specifically configured to:

[0271] The face-speech embedding vector is homomorphically encrypted to obtain a desensitization vector.

[0272] The vector hash is generated according to the desensitization vector.

[0273] As an optional embodiment, the uploading module 550 is specifically configured to:

[0274] The desensitization vector is fragmented according to a preset fragmentation dimension to obtain a plurality of corresponding vector fragments and corresponding storage nodes.

[0275] The vector fragments are stored in the corresponding storage nodes to generate corresponding storage hashes.

[0276] The storage hashes and the vector hash are stored in the corresponding storage nodes.

[0277] As an optional embodiment, the uploading module 550 is specifically configured to:

[0278] The instruction hash and the vector hash are submitted to the consensus node through the smart contract module, so that the consensus node compares the instruction hash and the vector hash with the corresponding data hash to determine the similarity.

[0279] The target block is formed based on the similarity.

[0280] As an optional embodiment, the uploading module 550 is specifically configured to:

[0281] If the similarity is lower than a preset threshold, the target block is generated based on the instruction hash and the vector hash.

[0282] As an optional embodiment, the device further comprises an identification module, which is specifically configured to:

[0283] Receive a voice recognition instruction of a user, wherein the voice recognition instruction comprises to-be-identified identity information.

[0284] The to-be-identified identity information is checked in a user feature database.

[0285] If the check is successful, a preset operation is triggered.

[0286] As an optional embodiment, the device, the identification module is specifically used for:

[0287] If the verification is successful, scene information in which the face recognition device is located is determined; the scene information includes a family scene, a community scene or a commercial scene;

[0288] Based on the scene information and the to-be-identified identity information, a preset operation is triggered; the preset operation includes opening or closing of a device in a corresponding scene, sending of prompt information to an administrator, automatic opening of an access control, pushing of specified preferential information or automatic check-in.

[0289] As an optional embodiment, the device, the face collection module 530 is specifically used for:

[0290] In response to the instruction content, verification information is randomly generated and displayed;

[0291] The user's voice input verification information is received, and it is judged whether the user is a living body;

[0292] If yes, the user's face information is continuously collected;

[0293] Otherwise, an error prompt is generated and sent.

[0294] The face information processing device provided in the application simplifies the face entry process by receiving the user's voice input instruction, triggering face information collection, obtaining the instruction content in the voice input instruction through the model, integrating the instruction content and the face information to obtain a face-voice embedding vector, and uploading the face-voice embedding vector to a blockchain network and saving the face-voice embedding vector. The user only needs to complete the synchronous entry of the identity information and the face information through a simple voice instruction, without the help of other input devices or complex operation processes, greatly improving the entry efficiency and user experience.

[0295] Further, the face information entered by the application embodiment is stored in the blockchain network, improving the security and credibility of the stored data. When the face-voice embedding vector is uploaded to the blockchain network, the data access permission is automatically executed by the smart contract module, so that the target block is chained, and the data processing efficiency is improved.

[0296] In addition, when performing face recognition, the voice recognition instruction of the user is received, and the voice is triggered to recognize based on the voice. In the scenes of family, community and business, the user's face is automatically recognized, so as to trigger the opening and closing of the device in different scenes, send prompt information to the administrator, automatically open the access control, push the specified preferential information or automatically check in. Starting from the use scene, the convenience and efficiency of user operation are fully considered, the efficiency of face recognition and operation is improved, and the user experience is improved.

[0297] The device provided in the embodiments of the present application can perform the method provided in the embodiments of the present application, has similar implementation principles, and has corresponding technical effects. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed function description of each module of the device, refer to the description of the corresponding method in the foregoing description, which will not be repeated here.

[0298] An electronic device is provided in the embodiments of the present application, which includes a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the method provided in any optional embodiment of the present application. Compared with the prior art, the following can be achieved: by receiving a voice input instruction of a user, face information collection is triggered, the instruction content in the voice input instruction is obtained through a model, the instruction content and the face information are integrated to obtain a face-voice embedding vector, which is then uploaded to a blockchain network and saved. The face input process is simplified. The user only needs to complete the synchronous input of identity information and face information through a simple voice instruction, without the help of other input devices or complex operation processes, thereby greatly improving the input efficiency and user experience.

[0299] In an optional embodiment, an electronic device is provided, as shown in Figure 6 Figure 6 The electronic device shown in the foregoing embodiment can be a face input device, and the internal structure diagram thereof can be as shown in Figure 6 ​The electronic device includes a processor, a memory, an input / output interface, a communication interface, a voice unit, a face collection unit, a receiving frame and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the voice unit, the face collection unit, the receiving frame and the input device are connected to the system bus through the input / output interface. The processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the electronic device is configured to exchange information between the processor and external devices. The communication interface of the electronic device is configured to communicate with external terminals in a wired or wireless manner. The wireless manner can be achieved through WIFI, mobile cellular network, NFC (near field communication) or other technologies. The voice unit of the electronic device is configured to receive user voice instructions or send voice prompt information to the user. The face collection unit of the electronic device is configured to collect user face information based on instructions. The receiving frame of the electronic device is configured to receive the target light beam sent by the light emitter to complete the non-contact interaction positioning method of the present application. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the electronic device, or an external keyboard, touchpad, mouse, air mouse or remote control, etc.

[0300] Those skilled in the art can understand that, Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. A specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0301] The electronic device includes but is not limited to an electronic device or terminal capable of implementing the face information processing method.

[0302] The computer readable storage medium provided by the embodiment of the present application stores a computer program. When the computer program is executed by a processor, the steps and corresponding contents of the foregoing method embodiments can be implemented.

[0303] It should be noted that the computer-readable storage medium in the foregoing embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0304] The embodiments of the present application further provide a computer program product, comprising a computer program, which, when executed by a processor, can implement the steps and corresponding contents of the foregoing method embodiments.

[0305] The terms "first", "second", "third", "fourth", "1", "2", and the like (if any) in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described.

[0306] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a part of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than that noted in the drawings. For example, two blocks that are represented in succession can actually be executed substantially in parallel, and they can also be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0307] It should be understood that although each operation step in the flowchart of the embodiments of the present application is indicated by an arrow, the implementation order of the steps is not limited to the order indicated by the arrow. Unless explicitly stated herein, in some implementation scenarios of the embodiments of the present application, the implementation steps in each flowchart can be executed in other orders as required. In addition, part or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on the actual implementation scenario. Part or all of these sub-steps or stages can be executed at the same time, and each of these sub-steps or stages can also be executed at different times. In the scenario where the execution times are different, the execution order of these sub-steps or stages can be flexibly configured as required, and the embodiments of the present application do not limit this.

[0308] The above is only an optional implementation of some implementation scenarios of the present application. It should be pointed out that, for ordinary skilled persons in the technical field, other similar implementation means based on the technical idea of the present application without departing from the technical concept of the present application also belong to the protection scope of the embodiments of the present application.

Claims

1. A facial information processing method, characterized in that: Applied to a node of a blockchain network, the blockchain network includes multiple nodes, including edge processing nodes, consensus nodes, and storage nodes. The facial information processing method includes: Receive user's voice input command; Inputting the voice input instruction into a voice recognition model to obtain corresponding instruction content; In response to the instruction content, collecting the user's facial information; Embedding the voice recording instruction into the face information to obtain a face-voice embedding vector; Uploading the face-speech embedding vector to the blockchain network.

2. The facial information processing method according to claim 1, characterized in that: The collecting of the user's facial information in response to the instruction content includes: In response to the instruction content, obtaining the user's facial features and head position; Based on the facial features, generating and issuing a head position adjustment prompt; In response to the user's adjustment action based on the adjustment prompt, continue to collect the user's facial information; the adjustment action includes rotating or tilting the head.

3. The facial information processing method according to claim 1, characterized in that: The speech recognition model includes a convolutional layer, a recurrent layer, a transformer layer and a decoding layer; The step of inputting the voice input instruction into a voice recognition model to obtain corresponding instruction content includes: Inputting the voice recording instruction into the convolution layer to obtain a convolution feature vector; Inputting the convolution feature vector into the recurrent layer to obtain a time series feature vector; Inputting the time series feature vector into the transformer layer to obtain an acoustic feature sequence; The acoustic feature sequence is input into the decoding layer to obtain the corresponding instruction content.

4. The facial information processing method according to claim 1, characterized in that: The step of embedding the voice input instruction into the facial information to obtain a face-voice embedding vector includes: Performing feature engineering on the voice input command to obtain a content vector; Performing feature engineering on the facial information to obtain a face vector; The content vector and the face vector are embedded to obtain a face-speech embedding vector.

5. The facial information processing method according to claim 1, characterized in that: The uploading of the face-speech embedding vector to the blockchain network includes: Generate an instruction hash according to the instruction content; Generating a vector hash based on the face-speech embedding vector; Submitting the instruction hash and the vector hash to the consensus node through the smart contract module for multi-node consensus to form a target block; The target block and the face-speech embedding vector are associated and stored in corresponding storage nodes.

6. The facial information processing method according to claim 5, characterized in that: Generating a vector hash according to the face-speech embedding vector includes: Performing homomorphic encryption on the face-speech embedding vector to obtain a desensitized vector; A vector hash is generated according to the desensitized vector.

7. The facial information processing method according to claim 6, characterized in that: The associating and storing the target block and the face-speech embedding vector in corresponding storage nodes includes: Slice the desensitized vector according to a preset slicing dimension to obtain corresponding multiple vector slicings and corresponding storage nodes; Storing the vector slices in corresponding storage nodes and generating corresponding storage hashes; The storage hash and the vector hash are associated and stored in a corresponding storage node.

8. A facial information processing device, characterized in that: The nodes used in the blockchain network include multiple nodes, including edge processing nodes, consensus nodes, and storage nodes, including: A receiving module, configured to receive a user's voice input command; A speech recognition module is used to input the speech input instruction into a speech recognition model to obtain the corresponding instruction content; A face collection module, configured to collect the user's face information in response to the instruction content; An embedding module, configured to embed the voice input instruction into the face information to obtain a face-voice embedding vector; An uploading module is used to upload the face-speech embedding vector to the blockchain network.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the face information processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the face information processing method according to any one of claims 1 to 7 are implemented.

Images