Face recognition method and system in heterogeneous computing scenario

Abstract

The application discloses a face recognition method and system in a heterogeneous computing scenario, and the face recognition system comprises a central processor, an auxiliary processor and M computing nodes, wherein M is greater than or equal to 1 and is a positive integer; wherein the central processor and the M computing nodes are isolated and decoupled through the auxiliary processor; the central processor is used for performing system initialization operation, determining the program execution order of face recognition, and collecting a scene image frame sequence; the auxiliary processor is used for receiving the scene image frame sequence from the central processor; the M computing nodes are controlled to process the scene image frame sequence in parallel according to the program execution order, and face feature data corresponding to the scene image frame sequence is obtained; and the central processor is further used for comparing the face feature data with registered face data in a face database, and obtaining a face recognition result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of face recognition, in particular to a face recognition method and system in a heterogeneous computing scenario. BACKGROUND

[0002] Face recognition technology is a technology for recognizing a face target from a scene image and completing user identity registration and verification without contact, and is widely used in many fields.

[0003] To improve face recognition accuracy, a face recognition technology based on a deep neural network is generally used at present. The deep neural network has good effects on improving face recognition accuracy and reducing system time delay, but involves a large number of floating point value calculations, matrix multiplications, vector values and other operations, and the data scale to be processed is extremely large, and the calculation presents an exponential growth trend as the learning level deepens.

[0004] At present, face recognition technology has developed into a larger model scale and a more complex model structure, which requires huge computing and memory resources; but some application scenarios have strict power consumption limitations, such as smart cars, mobile terminals and the like. Therefore, face recognition technology has a difficult trade-off between accuracy requirements and power consumption limitations. SUMMARY

[0005] The present application provides a face recognition method and system in a heterogeneous computing scenario to solve or partially solve the technical problem that the accuracy requirements and power consumption limitation requirements of face recognition technology cannot be balanced.

[0006] To solve the above technical problem, a first aspect of the present application discloses a face recognition system in a heterogeneous computing scenario, which comprises a central processor, an auxiliary processor and M computing nodes, M≥1 and is a positive integer; wherein the central processor and the M computing nodes are isolated and decoupled through the auxiliary processor;

[0007] The central processor is configured to perform a system initialization operation, determine a program execution order of face recognition, and collect a scene image frame sequence.

[0008] The system initialization operation of the central processor specifically comprises:

[0009] A reading module configured to read model distribution information corresponding to the M computing nodes;

[0010] A segmented quantization module configured to split and segmentally quantize a face recognition network model according to the model distribution information to obtain N deployable sub-files; wherein 1≤N≤M, and N is a positive integer.

[0011] a model distribution module, configured to distribute the N deployable sub-files to corresponding computing nodes;

[0012] a fusion module, configured to fuse the flow order and the face recognition order of the M computing nodes to obtain the program execution order;

[0013] the auxiliary processor, configured to receive the sequence of scene image frames from the central processor, control the M computing nodes to process the sequence of scene image frames in parallel according to the program execution order, and obtain face feature data corresponding to the sequence of scene image frames;

[0014] the central processor, further configured to compare the face feature data with registered face data in a face database to obtain a face recognition result.

[0015] Optionally, the model distribution information is generated by referring to computing power information of the computing nodes.

[0016] Optionally, the segment quantization module is specifically configured to perform the following operations:

[0017] construct the face recognition network model;

[0018] split the face recognition network model into N sub-models according to the model distribution information; the N sub-models correspond to a processing order;

[0019] create a quantizer and a calibrator for each sub-model;

[0020] segmentally quantize the N sub-models based on the quantizers and the calibrators corresponding to the N sub-models to obtain N quantized sub-models;

[0021] post-process the N quantized sub-models according to a required format of the corresponding computing nodes to obtain N deployable sub-files.

[0022] Optionally, the segment quantization module is specifically further configured to:

[0023] load current calibration data by using the calibrator of the current sub-model; the current calibration data is output after quantization of a previous sub-model;

[0024] input the current calibration data into the current sub-model, adjust quantization parameters of the current quantizer according to a forward propagation result, obtain a current quantized sub-model, and input quantization output of the current quantizer as calibration data of a next sub-model into the next sub-model for processing, until all the sub-models are traversed.

[0025] Optionally, the system further comprises a splicing module configured to perform the following operations before splicing the quantizers and calibrators corresponding to the N sub-models in the corresponding sub-models to form N series models:

[0026] splicing the quantizers and calibrators corresponding to the N sub-models in the corresponding sub-models to form N series models;

[0027] splicing the N series models with each other according to the processing sequence to obtain a spliced model.

[0028] Optionally, the central processor specifically comprises an acquisition module configured to acquire the scene image frame sequence; and an encoding module configured to encode the scene image frame sequence in a binary number manner to obtain encoded data.

[0029] The auxiliary processor specifically comprises a receiving module configured to receive the encoded data from the central processor; and a decoding module configured to decode the encoded module to obtain the scene image frame sequence.

[0030] Optionally, the system further comprises a communication module configured to realize data interaction between the central processor and the auxiliary processor, and the data for interaction comprises instruction data and encoded data.

[0031] Optionally, the auxiliary processor specifically further comprises a control module configured to input each image frame in the scene image frame sequence in sequence in combination with a program execution sequence and a required processing time of each computing node, so that the M computing nodes process the each image frame in parallel to obtain face feature data corresponding to the each image frame.

[0032] Optionally, the system further comprises a public cache configured to store the scene image frame sequence transmitted by the central processor and the face feature data transmitted by the auxiliary processor.

[0033] A second aspect of the present application discloses a face recognition method in a heterogeneous computing scenario, characterized in that the face recognition method is applied to a face recognition system, and the face recognition system comprises a central processor, an auxiliary processor and M computing nodes, wherein M is a positive integer greater than or equal to 1; and the central processor and the M computing nodes are isolated and decoupled through the auxiliary processor.

[0034] The face recognition method comprises the following steps:

[0035] performing a system initialization operation by using the central processor to determine a program execution sequence of face recognition and acquire a scene image frame sequence;

[0036] The system initialization operation comprises the following steps:

[0037] reading model distribution information corresponding to the M computing nodes; segmenting and quantizing the face recognition network model into N deployable sub-files according to the model distribution information; distributing the N deployable sub-files to the corresponding computing nodes; wherein 1≤N≤M, N is a positive integer; fusing the flow order and the face recognition order of the M computing nodes to obtain the program execution order; wherein 1≤N≤M, N is a positive integer;

[0038] receiving the sequence of scene image frames from the central processor by using the auxiliary processor; controlling the M computing nodes to process the sequence of scene image frames in parallel according to the program execution order, to obtain face feature data corresponding to the sequence of scene image frames;

[0039] comparing the face feature data with registered face data in a face database by using the central processor, to obtain a face recognition result.

[0040] Optionally, the model distribution information is generated with reference to computing power information of the computing nodes.

[0041] Optionally, the segmenting and quantizing the face recognition network model into N deployable sub-files according to the model distribution information specifically includes:

[0042] constructing the face recognition network model;

[0043] splitting the face recognition network model into N sub-models according to the model distribution information; the N sub-models correspond to a processing order;

[0044] creating a quantizer and a calibrator for each sub-model;

[0045] segmenting and quantizing the N sub-models based on the quantizers and the calibrators corresponding to the N sub-models, to obtain N quantized sub-models;

[0046] post-processing the N quantized sub-models according to a required format of the corresponding computing nodes, to obtain N deployable sub-files.

[0047] Optionally, the segmenting and quantizing the N sub-models based on the quantizers and the calibrators corresponding to the N sub-models, to obtain N quantized sub-models specifically includes:

[0048] loading current calibration data by using the calibrator of the current sub-model; wherein the current calibration data is output after quantization of a previous sub-model;

[0049] Input the current calibration data into the current sub-model, adjust the quantization parameter of the current quantizer according to the forward propagation result, obtain the current quantization sub-model, and input the quantization output of the current quantizer as calibration data of a next-level sub-model into the next-level sub-model for processing until all the sub-models are traversed.

[0050] Optionally, before the N sub-models are segmented and quantized based on the quantizers and calibrators corresponding to the N sub-models to obtain N quantization sub-models, the method further comprises:

[0051] Splicing the quantizers and calibrators corresponding to the N sub-models in the corresponding sub-models to form N series models;

[0052] Splicing the N series models with each other according to the processing sequence to obtain a splicing model.

[0053] Optionally, in the central processor, the acquisition of the scene image frame sequence specifically comprises:

[0054] Acquiring the scene image frame sequence;

[0055] Encoding the scene image frame sequence in a binary number manner to obtain encoded data;

[0056] In the auxiliary processor, the receiving of the scene image frame sequence from the central processor specifically comprises:

[0057] Receiving the encoded data from the central processor;

[0058] Decoding the encoding module to obtain the scene image frame sequence.

[0059] Optionally, the method further comprises: utilizing a communication module to support data interaction between the central processor and the auxiliary processor, and the data for interaction comprises instruction data and encoded data.

[0060] Optionally, the control of the M computing nodes to process the scene image frame sequence in parallel according to the program execution sequence to obtain the face feature data corresponding to the scene image frame sequence specifically comprises: a control module configured to input each image frame in the scene image frame sequence in sequence in combination with the program execution sequence and the processing time required by each computing node, so that the M computing nodes process the each image frame in parallel to obtain the face feature data corresponding to the each image frame.

[0061] Optionally, the method further comprises: storing the scene image frame sequence transmitted by the central processor and the face feature data transmitted by the auxiliary processor by using a public cache.

[0062] By means of the one or more technical solutions of the present application, the present application has the following beneficial effects or advantages:

[0063] The technical solution disclosed by the present application utilizes a central processor, an auxiliary processor and M computing nodes to form a heterogeneous computing scenario; wherein, the auxiliary processor 102 can realize isolation decoupling between the central processor 101 and the M computing nodes, the central processor cannot directly access the M computing nodes, so that instruction execution and model calculation operation can realize separation operation at the hardware level, thereby improving hardware flexibility; on this basis, a floating point model used for face recognition is quantized into multiple quantized sub-models in a model segmentation quantization manner, compared with the original model, 65.71 % of storage space can be saved, and the accuracy only decreases by 0.71 %, which can better balance the accuracy requirement and power consumption limitation demand of face recognition technology. In addition, the auxiliary processor controls the M computing nodes to process the image frame sequence in parallel, which can accelerate the program running speed, improve the hardware resource utilization, and make the face recognition rate reach 14 frames / s.

[0064] The above description is only a summary of the technical solutions of the present application, in order to more clearly understand the technical means of the present application, the content of the specification can be implemented, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0065] By reading the detailed description of the preferred embodiments below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of illustrating the preferred embodiments and are not considered to be limiting on the present application. Moreover, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0066] Figure 1 A structure schematic diagram of a face recognition system in a heterogeneous computing scenario according to an embodiment of the present application is shown;

[0067] Figure 2 An implementation process diagram of a segmentation quantization module according to an embodiment of the present application is shown;

[0068] Figure 3 A schematic diagram of a spliced model according to an embodiment of the present application is shown;

[0069] Figure 4 A structure schematic diagram of encoded data according to an embodiment of the present application is shown;

[0070] Figure 5 An example diagram of a data frame according to an embodiment of the present application is shown;

[0071] Figure 6A diagram showing each computing node process parallel processing a sequence of scene image frames according to one embodiment of the present application is shown;

[0072] Figure 7 A diagram showing splitting of a face recognition network model according to one embodiment of the present application is shown;

[0073] Figure 8 A diagram showing structure deployment of a face recognition network model according to one embodiment of the present application is shown;

[0074] Figure 9 An implementation diagram showing parallel processing a sequence of scene image frames according to one embodiment of the present application is shown

[0075] Figure 10 A flowchart of a face recognition method according to one embodiment of the present application is shown. DETAILED DESCRIPTION

[0076] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it is understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0077] To balance the accuracy requirements and power consumption limit needs of face recognition technology, a face recognition system under a heterogeneous computing scenario is disclosed. "Heterogeneous computing" refers to a joint computing manner in which processors of different architectures are used in a system. Referring to Figure 1 is a structural diagram of a face recognition system under a heterogeneous computing scenario, including a central processor 101, an auxiliary processor 102, and M computing nodes, M≥1 and is a positive integer. The M computing nodes are in Figure 1 Examples in the figure are computing node 1, computing node 2, computing node 3, …, and computing node M. In addition, to support transmission and storage, the face recognition system is also configured with a communication module 103 and a storage medium 104.

[0078] The central processor 101 is responsible for segmenting and quantifying the face recognition network model, input data preprocessing, face comparison, and connection with peripherals such as cameras.

[0079] The communication module 103 supports at least two transmission modes of serial port and network, and is responsible for data interaction between the central processor 101 and the auxiliary processor 102. The data includes instruction data and image data. The communication module 103 is implemented by software programming at the central processor 101 end, and is implemented by hardware description language at the auxiliary processor 102 end, which is applicable at both software and hardware levels.

[0080] The storage medium 104 includes a memory and a cache. The storage medium 104 serves as a common cache area to store the sequence of scene image frames collected by the central processor 101 and the face feature data processed by the auxiliary processor 102.

[0081] The M computing nodes refer to computing devices for deploying the deployable sub-files obtained by segmenting and quantizing the face recognition network model and participating in the model calculation in the deployable sub-files. The computing nodes are specifically CPUs, GPUs, FPGA circuits, and store-compute integrated chips, etc. Each computing node needs to provide at least one driver program describing the hardware information of the node.

[0082] The auxiliary processor 102 is responsible for data reception and decoding, job distribution, collection, processing of the outputs of the computing nodes, and writing into the common cache area.

[0083] The auxiliary processor 102 of the embodiment can realize isolation and decoupling between the central processor 101 and the M computing nodes, that is, the central processor 101 cannot directly access the M computing nodes and needs to indirectly access through the auxiliary processor 102, and data interaction and data storage also rely on the communication module 103 and the storage medium 104 for transfer, and will not be directly transmitted between the auxiliary processor 102 and the central processor 101. Therefore, the central processor 101, the auxiliary processor 102, the M computing nodes, the communication module 103, and the storage medium 104 of the embodiment constitute a heterogeneous computing scene of face recognition, so that instruction execution, model calculation, storage, and other operations can be realized by separated operations.

[0084] In the hardware structure of the application, on the one hand, the technical concept of heterogeneous computing is borrowed, and the face recognition technology is distributed in the central processor 101, the auxiliary processor 102, and the M computing nodes for joint processing, and the auxiliary processor 102 is used to isolate the central processor 101 and the M computing nodes, thereby decoupling the system architecture from the hardware layer, so as to realize the separation of instruction execution, model calculation, storage, and other operations at the hardware layer, and improve the flexibility of the system architecture at the hardware layer. On the other hand, the model is segmented and quantized into N deployable sub-files, which can save 65.71% of the storage space compared to the face recognition network model before quantization, and the accuracy only decreases by 0.71% compared to the original model; the N deployable sub-files are distributed in M different computing nodes and are configured with a program execution sequence, the auxiliary processor 102 is used to control the M computing nodes to process the sequence of scene image frames in parallel, so as to extract the face feature data for recognition, which can well balance the accuracy requirement and power consumption limitation requirement of the face recognition technology.

[0085] In order to facilitate the description and explanation of the application, the following describes the application in the context of face recognition technology. Figure 1On this basis, the implementation principles of each component are specifically introduced.

[0086] The central processor 101 is configured to perform system initialization operation, determine the program execution sequence of face recognition, collect scene image frame sequence based on the connection with external devices such as camera, pre-process and encode the scene image frame sequence, and send the encoded data to the auxiliary processor 102 and the public cache area respectively by using the communication module 103. When the auxiliary processor 102 identifies the face feature data based on the encoded data and uploads the face feature data to the public cache area, the central processor 101 is further configured to extract the face feature data from the public cache area, read the face library data, compare the face library data with the registered face data in the face database, and obtain the face recognition result.

[0087] The central processor 101 performs system initialization according to the configurable file. When performing the system initialization operation, the reading module 1011, the segment quantization module 1012, the model distribution module 1013, and the fusion module 1014 are specifically included. Refer to Figure 1 .

[0088] The reading module 1011 is configured to obtain node information. The node information includes: the number M of computing nodes participating in distribution; the node types of the M computing nodes, such as ONNX format, C language format, etc.; and the model distribution information corresponding to the M computing nodes. The model distribution information is generated by referring to the computing power information of the computing nodes, determines the number M of computing nodes participating in distribution, the splitting mode of the face recognition network model, and the distribution mapping relationship of the N deployable sub-files obtained after the M computing nodes and the model segment quantization. The distribution mapping relationship between the deployable sub-files and the computing nodes is one-to-one or many-to-one. If the computing power of the computing nodes is sufficient, multiple deployable sub-files can be deployed in the same computing node.

[0089] The segment quantization module 1012 is configured to split and segment quantize the face recognition network model according to the model distribution information to obtain N deployable sub-files. Wherein, 1≤N≤M, N is a positive integer.

[0090] Refer to Figure 2 , the segment quantization module 1012 is specifically configured to perform the following operations:

[0091] S201, constructing a face recognition network model.

[0092] The construction process is implemented in the following manner: a base model for face recognition is constructed with reference to a deep learning network architecture. Relevant weights are loaded from a configuration file. Model optimization is performed, such as operator fusion and layer name simplification; operator fusion, also known as symbol operator fusion, reduces the overhead of computation and memory access by merging multiple operators into one larger operator, such as convolution computation and activation function fusion, convolution and batch normalization fusion, and the like. A model summary is obtained, i.e., a face recognition network model.

[0093] The constructed face recognition network model mainly includes two parts: a face detection model and a face feature extraction model. The two models can be independently split and quantized, or they can be jointly split and quantized.

[0094] S202, the face recognition network model is split into N sub-models according to the model distribution information.

[0095] The face recognition network model is mainly split according to modules, layers, and channels. The modules represent the face detection model and the face feature extraction model. The layers represent the relevant processing layers within the face recognition network model, such as convolution layers, pooling layers, and the like; splitting grouped convolution and layer-by-layer convolution can reduce the parameter quantity and computation quantity. The channels represent the input channel number within the relevant layer, and channel splitting refers to splitting the input channel within the relevant layer into multiple channels; for example, the input channel number of a convolution layer is split into multiple channels, each of which only processes a part of the input channel, thereby increasing the parallelism of the computation and reducing the latency. It is worth noting that the smallest splitting unit within the model is the weight value.

[0096] The splitting manner of the model depends on the computing power of the computing nodes, so as to fully utilize the computing power resources of the computing nodes. For example, if the computing power of each computing node is insufficient, the model can be split according to the "layer", and each computing node is assigned a convolution layer of the model. If the computing power of a computing node is sufficient, the face detection model can be assigned to the computing node as a whole, and the face feature extraction model can be adaptively split according to the computing power of the other computing nodes in terms of layers, channels, and weight values. The structure of the split model will be exemplarily illustrated later, and will not be described here.

[0097] After the model is split into N sub-models, the N sub-models have a processing order, and the model computation graph of the face recognition network model can be constructed accordingly. The model computation graph is a model structure file that configures the hierarchical structure of the face recognition network model, and the flow order between the M computing nodes can be obtained according to the model structure computation graph.

[0098] After the model is split, the subsequent segment quantization operation can be performed, as shown in S203-S204.

[0099] Segmented quantization refers to an operation mode in which each sub-model is sequentially quantized according to the processing order of the N sub-models obtained by splitting, and the quantization output of the previous sub-model is taken as the calibration input of the next sub-model.

[0100] S203, creating a quantizer and a calibrator for each sub-model.

[0101] In this embodiment, a static quantization mode is adopted, and the calibration data provided by the calibrator is used for quantization. Specifically, a quantizer and a calibrator are created for each sub-model. The quantizer refers to a program object for performing model quantization operations, which converts model-related data from floating-point representation to lower-precision representation, such as network parameters, activation functions, and other model-related data. The calibrator refers to a program object for performing calibration data loading operations. The quantizers of the N sub-models can be the same or different, and the calibrators of the N sub-models are related to the quantization output of the previous sub-model. For example, the calibrator of the first sub-model loads preprocessed image data, and the calibrator of the next sub-model loads feature map data quantized by the first sub-model.

[0102] In an alternative implementation process, referring to Figure 3 is a schematic diagram of a spliced model. After creating a quantizer and a calibrator for each sub-model, the quantizers and calibrators corresponding to the N sub-models are spliced into the corresponding sub-models to form N serial models; and the N serial models are spliced with each other according to the processing order to obtain a spliced model. For example, a face recognition network model is split into sub-model 1, sub-model 2, …, and sub-model N. Sub-model 1 has corresponding quantizer 1 and calibrator 1, sub-model 2 has corresponding quantizer 2 and calibrator 2, …, and sub-model N has corresponding quantizer N and calibrator N. When splicing, the above-mentioned splicing is performed to obtain the spliced model shown in Figure 3 The main purpose of splicing is to insert the quantizer and the calibration module into the corresponding sub-model, and the spliced model contains the processing order of each sub-model, and N quantized sub-models can be generated by one segmented quantization operation.

[0103] S204, based on the quantizers and calibrators corresponding to the N sub-models, performing segmented quantization on the N sub-models to obtain N quantized sub-models.

[0104] During quantization, the sub-models are sequentially quantized according to the processing order of the N sub-models, and the quantization output of the previous sub-model is taken as the quantization input of the next sub-model.

[0105] Specifically, the current calibration data is loaded using the calibrator of the current sub-model. This current calibration data is the output of the quantized version of the previous sub-model. The current calibration data is then input into the current sub-model, and the quantization parameters of the current quantizer are adjusted based on the forward propagation results to ensure that the output of the current quantized sub-model during image processing is as close as possible to the output of the floating-point model before calibration. Simultaneously, the quantized output of the current sub-model is used as the calibration data for the next-level sub-model, which is then processed to adjust the quantization parameters and output the next-level quantized sub-model… This process continues until all sub-models have been traversed.

[0106] Segmented quantization supports simultaneous quantization across multiple frameworks such as PyTorch, Xilinx Brevitas, and Vitis AI, allowing multiple quantized sub-models to be obtained with a single segmented quantization operation.

[0107] After all sub-models have been traversed, step 204 can be executed repeatedly until the number of iterations for segmented quantization is satisfied.

[0108] Furthermore, after quantization, the quantized models can be tested using a test dataset to improve the accuracy of the image data.

[0109] by Figure 3 For example, calibration data 1 is input into sub-model 1 in the splicing model. During quantization by quantizer 1, calibrator 1 adjusts the quantization parameters in quantizer 1 based on calibration data 1, using the forward propagation results: the quantization zero z and the quantization factor s, until the accuracy meets the requirements. After quantization, quantized sub-model 1 is obtained. The general theoretical formula for quantization is as follows: ,in, This indicates the quantized output result. Let z represent the calibration data, z represent the quantization zero point, and s represent the quantization factor s. The quantization output of sub-model 1 is used as the calibration data for sub-model 2, and input into sub-model 2 for similar processing... and so on until sub-model N is quantized and then the quantized sub-model N is output.

[0110] S205. Post-process the N quantization sub-models according to the format required by the corresponding computing nodes to obtain N deployable sub-files.

[0111] Post-processing refers to format conversion based on the requirements of different computing nodes. For example, if computing node 1 requires an ONNX file, the sub-model is converted to an ONNX file; if computing node 2 requires a C language file, the sub-model is converted to a C language-readable binary file.

[0112] The model distribution module 1013 is used to distribute N deployable sub-files to the corresponding computing nodes.

[0113] Specifically, after obtaining the N deployable sub-files, the N deployable sub-files are distributed to the corresponding computing nodes according to the distribution mapping relationship. After the distribution, the M computing nodes have the corresponding flow order according to the processing order of the N deployable sub-files. It is worth noting that if part of the weight values in the model calculation remain unchanged, they can be directly written in the computing nodes for use.

[0114] The fusion module 1014 is configured to fuse the flow order of the M computing nodes and the face recognition order to obtain a program execution order.

[0115] The face recognition order includes face detection, face alignment, feature extraction, and face comparison. The flow order between the M computing nodes is sorted according to the face recognition order, and the program execution order is obtained.

[0116] The above is a specific implementation process of system initialization.

[0117] The central processing unit 101, when collecting the sequence of scene image frames, specifically includes:

[0118] The collection module 1015 is configured to collect the sequence of scene image frames.

[0119] The encoding module 1016 is configured to encode the sequence of scene image frames in a base number manner to obtain encoded data.

[0120] Specifically, the sequence of scene image frames includes data frames of multiple real-time scene images, and the length depends on the collection capability of the collection module 1015, and is usually 30 frames / s. After the sequence of scene image frames is collected, gamma transformation, scale transformation, normalization, and data quantization (quantized to INT8) are performed on each scene image to obtain image data of a preset size.

[0121] After preprocessing, the sequence of scene image frames is encoded. Taking a single scene image as an example, the obtained encoded data includes a message header and a data bit. As shown in Figure 4 , it is a structure diagram of the encoded data. The message header includes the hardware address of the message receiving end, the hardware address of the message sending end, and the transmission protocol; the data bit is composed of the encoded value of the single scene image. The hardware address occupies 12 bytes, the transmission protocol occupies 2 bytes, and the size of the data bit is related to the length of the data to be transmitted.

[0122] During encoding, first, the single scene image is converted in a base number manner, for example, converted into a hexadecimal representation; the converted base number data is spliced with the message header to obtain a message, also known as a data frame; the data frame is converted into a preset format, for example, a Bytes type, to obtain the encoded data. As shown in Figure 5 , it is an example diagram of the data frame.

[0123] The encoding operation is performed on the sequence of scene image frames to obtain corresponding encoded data. The encoded data corresponding to the sequence of scene image frames is transmitted to the auxiliary processor 102 through the communication module.

[0124] In this embodiment, the communication module 103 is used to support the transmission of encoded data, and the length of the data frame is reduced through the encoding and decoding mode, so as to realize the rapid transmission of data.

[0125] The auxiliary processor 102 is configured to receive the sequence of scene image frames from the central processor 101, distribute jobs to different computing nodes, and control M computing nodes to process the sequence of scene image frames in parallel according to a program execution sequence to obtain face feature data corresponding to the sequence of scene image frames.

[0126] In the process of receiving the sequence of scene image frames, the auxiliary processor 102 specifically comprises:

[0127] The receiving module 1021 is configured to receive encoded data from the central processor 101.

[0128] The decoding module 1022 is configured to decode the encoding module 1016 to obtain the sequence of scene image frames. The decoding includes three processes of data filtering, data bit extraction and data translation.

[0129] The data filtering adopts a method combining character comparison and data frame length. Taking single needle data encoding as an example, after receiving the data frame, the first 6 bytes in the message header are read to determine whether it is the destination address; if so, it is determined whether the data bit length meets the requirements; if so, the data bit extraction is performed, otherwise the data frame is discarded. The data bit extraction is implemented by deleting the message header (i.e. the first 14 characters) to obtain the data corresponding to the data bit. The data translation is implemented by using a translation dictionary. The translation dictionary refers to a dictionary composed of Bytes data and hexadecimal data.

[0130] The jobs distributed by the auxiliary processor 102 contain weight values, which need to be distributed to the corresponding computing nodes in advance. It is worth noting that part of the weight values can be written in the computing nodes in advance if they remain unchanged.

[0131] The auxiliary processor 102 specifically further comprises a control module 1023 configured to control M computing nodes to process the sequence of scene image frames in parallel according to a program execution sequence.

[0132] Further, the control module 1023 is configured to control the input and output of each computing node according to the program execution sequence, and to upload the output result of the computing node responsible for the export to the public buffer area to interact with the central processor 101. The output result of the computing node responsible for the export has two types. One type is the output result of the computing node responsible for the export in the face detection model, which is face key points and face bounding box coordinates. The other type is the output result of the computing node responsible for the export in the face feature extraction model, which is face feature data.

[0133] The output result of the computing node responsible for the export in the face detection model, which is face key points and face bounding box coordinates, is uploaded to the public buffer area to interact with the central processor 101, and the interaction result is taken as the input of the next computing node.

[0134] On the side of the central processor 101, the face key point coordinates, the face bounding box coordinates, and the scene image are read from the public buffer area, the face image is intercepted from the scene image according to the bounding box coordinates, and the face image is subjected to affine transformation according to the key point coordinates to obtain a corrected face image as the input of the first computing node corresponding to the face feature extraction model.

[0135] The output result of the computing node responsible for the export in the face feature extraction model, which is face feature data, is uploaded to the public buffer area to interact with the central processor 101. On the side of the central processor 101, the face feature data is read from the public buffer area, the Euclidean distance with the registered face data is calculated, the minimum value is taken as the recognition result, if the minimum value is not greater than a preset threshold, the corresponding user label is output, and if the minimum value is greater than the preset threshold, “Unknown” is output.

[0136] Since the scene image frame sequence is composed of multiple scene image frames, in order to ensure the execution speed and processing accuracy of the model inference, the control module 1023 is configured to input each image frame in the scene image frame sequence in sequence according to the program execution sequence and the required processing time of each computing node, so that the M computing nodes process each image frame in parallel to obtain the face feature data corresponding to each image frame.

[0137] Specifically, the same scene image frame is circulated and processed among the M computing nodes according to the program execution sequence to obtain the corresponding face feature data. The input of each scene image frame in the same computing node has an interval time, which is the processing time required by the computing node to process the previous frame of scene image. In this way, the computing node can process the next frame of scene image after outputting the processing result of the previous frame of scene image, which can speed up the program running speed, improve the hardware resource utilization rate, and make the face recognition rate reach 14 frames / s.

[0138] For the convenience of description and explanation, the following refers to Figure 6is a schematic diagram of parallel processing of a scene image frame sequence by each computing node process.

[0139] Image data 1, image data 2, and image data 3 are sequentially input into the computing node 1 process for processing according to the required processing time corresponding to the computing node 1 process. While image data 1 is transferred from the computing node 1 process to the computing node 2, image data 2 is input into the computing node 1 process for processing. While image data 1 is transferred from the computing node 2 process to the computing node 3, image data 2 is transferred from the computing node 1 process to the computing node 2 process for processing, and image data 3 is input into the computing node 1 process for processing. In this way, the scene image frame sequence is processed in parallel between the M computing nodes, and the processing result of a single scene image is not affected.

[0140] Exemplary examples

[0141] This example shows four computing nodes and a face recognition network model. The face recognition network model is shown in Table 1.

[0142]

[0143] In Table 1, represents a convolution kernel size of convolution, represents a non-linear rectifier unit, represents a convolution kernel size of layer-by-layer convolution, represents a bottleneck layer, Residual Block represents a residual structure, FC represents a fully connected layer, and GDC represents a global layer-by-layer convolution, represents a convolution kernel size of convolution, represents a convolution kernel size of layer-by-layer convolution.

[0144] Model distribution information is determined according to the computing power of the four computing nodes. When determining the model distribution information, the face recognition network model is divided into an input layer, a backbone network, an output layer 1, and an output layer 2. The input layer is arranged in the computing node 1. The backbone network is Since the computing node 2 is a neural network accelerator provided by Xilinx, it has sufficient computing power, and to ensure running speed, the backbone network is placed entirely in the computing node 2. The FC is divided into the output layer 1 and the output layer 2, which are arranged in the computing node 3 and the computing node 4, respectively. Referring to Figure 7- Figure 8 , Figure 7 is a split schematic diagram of the face recognition network model, Figure 8 is a structure deployment schematic diagram of the face recognition network model.

[0145] The auxiliary processor 102 sends the single-frame scene image read from the off-chip DDR to the computing node 1, then sends the result of the computing node 1 to the computing node 2, and then uploads the output of the computing node 2 to the common buffer area and the central processor 101, interacts the result with the central processor 101, inputs the interaction result into the computing node 3 and the computing node 4, and outputs the output results of the computing node 3 and the computing node 4 in parallel to obtain the face feature data.

[0146] In Figure 8 Based on the deployment schematic diagram constructed, the input and output of each computing node are as follows:

[0147] Input of the computing node 1: a tensor with a dimension of [1, 3, 112, 112];

[0148] Output of the computing node 1: a tensor with a dimension of [1, 64, 56, 56];

[0149] Output of the computing node 2: a tensor with a dimension of [1, 512, 1, 1];

[0150] Output of the computing node 3: a vector with a dimension of [1, 64];

[0151] Output of the computing node 4: a vector with a dimension of [1, 448];

[0152] When uploading the face feature data, the auxiliary processor 102 transversely splices the data of the computing node 3 and the computing node 4 into a vector with a dimension of [1, 512] and uploads the vector to the common buffer area.

[0153] The processing precision of the segmented quantization model is compared with that of the original model, as shown in Table 2.

[0154] Table 2

[0155]

[0156] As shown in Table 2, the precision of the split model only decreases by 0.71%, but the model size is saved by 65.71%.

[0157] For a frame sequence with multiple-frame scene images, parallel processing is performed according to the processing mode shown in Figure 9 .

[0158] The central processor 101 extracts the face feature data from the common buffer area, performs Euclidean distance calculation on the face feature data and registered face data in a face database, and obtains a face recognition result according to the calculation result.

[0159] Based on the same inventive concept, the following embodiment discloses a face recognition method in a heterogeneous computing scenario, which is applied to a face recognition system, and includes a central processor 101, an auxiliary processor 102, and M computing nodes, where M≥1 and is a positive integer. The central processor 101 and the M computing nodes are isolated and decoupled through the auxiliary processor 102. For details of the specific structure of the face recognition system and the implementation principle thereof, refer to the description of the foregoing embodiment, which will not be repeated here.

[0160] Referring to Figure 10 , the face recognition method includes:

[0161] S1001, performing a system initialization operation by using the central processor to determine a program execution order of face recognition and collect a sequence of scene image frames.

[0162] The system initialization operation includes:

[0163] reading model distribution information corresponding to the M computing nodes; segmenting and quantizing a face recognition network model into N deployable sub-files according to the model distribution information; distributing the N deployable sub-files to the corresponding computing nodes; where 1≤N≤M and N is a positive integer; and fusing a flow order of the M computing nodes and a face recognition order to obtain the program execution order; where 1≤N≤M and N is a positive integer.

[0164] S1002, receiving the sequence of scene image frames from the central processor by using the auxiliary processor; controlling the M computing nodes to process the sequence of scene image frames in parallel according to the program execution order, and obtaining face feature data corresponding to the sequence of scene image frames.

[0165] S1003, comparing the face feature data with registered face data in a face database by using the central processor, and obtaining a face recognition result.

[0166] In an optional implementation, the model distribution information is generated with reference to computing power information of the computing nodes.

[0167] In an optional implementation, the segmenting and quantizing the face recognition network model into N deployable sub-files according to the model distribution information specifically includes:

[0168] constructing the face recognition network model;

[0169] splitting the face recognition network model into N sub-models according to the model distribution information; the N sub-models correspond to a processing order;

[0170] creating a quantizer and a calibrator for each sub-model;

[0171] segmentally quantize the N sub-models based on the quantizers and calibrators corresponding to the N sub-models to obtain N quantized sub-models;

[0172] post-process the N quantized sub-models according to a required format of the corresponding computing node to obtain N deployable sub-files.

[0173] In an optional implementation, the segmentally quantizing the N sub-models based on the quantizers and calibrators corresponding to the N sub-models to obtain N quantized sub-models specifically includes:

[0174] loading current calibration data by using the calibrator of the current sub-model; wherein the current calibration data is output after quantization of a previous sub-model;

[0175] inputting the current calibration data into the current sub-model, adjusting quantization parameters of the current quantizer according to a forward propagation result to obtain a current quantized sub-model, and inputting a quantization output of the current quantizer into a next sub-model as calibration data of the next sub-model for processing until all sub-models are traversed.

[0176] In an optional implementation, before the segmentally quantizing the N sub-models based on the quantizers and calibrators corresponding to the N sub-models to obtain N quantized sub-models, the method further includes:

[0177] splicing the quantizers and calibrators corresponding to the N sub-models in the corresponding sub-models to form N series models;

[0178] splicing the N series models according to the processing sequence to obtain a spliced model.

[0179] In an optional implementation, in the central processor, the collecting a sequence of scene image frames specifically includes:

[0180] collecting the sequence of scene image frames;

[0181] encoding the sequence of scene image frames in a base number manner to obtain encoded data;

[0182] In the auxiliary processor, the receiving the sequence of scene image frames from the central processor specifically includes:

[0183] receiving the encoded data from the central processor;

[0184] decoding the encoding module to obtain the sequence of scene image frames.

[0185] In an alternative embodiment, the method further comprises: supporting data interaction between the central processor and the auxiliary processor by using a communication module, and the data includes instruction data and encoding data.

[0186] In an alternative embodiment, the method further comprises: supporting data interaction between the central processor and the auxiliary processor by using a communication module, and the data includes instruction data and encoding data.

[0187] In an alternative embodiment, the method further comprises: storing the scene image frame sequence transmitted by the central processor and the face feature data transmitted by the auxiliary processor in a public cache.

[0188] It should be noted that the face recognition method provided by the embodiment of the present application, the specific implementation and the technical effects produced are the same as those of the foregoing system embodiment, and for brief description, the part not mentioned in the method embodiment can refer to the corresponding content in the foregoing system embodiment.

[0189] The algorithms and displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems can be used with these teachings, or with modifications that result from the application of the teachings of the present application. Structure for a variety of these systems will be apparent to those of skill in the art from the description above. In addition, the present application is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the application as described herein, and any particular language should not be construed as limiting the scope of the application.

[0190] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to not obscure the understanding of this description.

[0191] Similarly, it is to be understood that the embodiments of the present application can be alternately or additionally employed in combination with each other, where appropriate, even though not expressly stated in the above description. Although the present application has been described in accordance with the embodiments shown, one of ordinary skill in the art will readily recognize that there could be variations to the embodiments and modifications of the present application, such as: to the logical arrangement of the various elements composing it; to the materials and mechanisms used, as well as the methods used to assemble the devices; and to the computational procedures employed. Accordingly, the present application is not intended to be limited to that precisely as shown and described, and therefore various modifications and improvements, as well as combinations of elements of

[0192] Those skilled in the art will appreciate that the modules in the devices in the embodiments can be adapted and placed in one or more devices other than the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component and further can be divided into sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, any combination of all the features disclosed in the specification (including the accompanying claims, abstract and drawings), and all the processes or units of any of the methods or apparatuses disclosed so are encompassed. Unless explicitly stated, each feature disclosed in the specification (including the accompanying claims, abstract and drawings) can be replaced by alternative features that serve the same, equivalent or similar purpose.

[0193] Further, those skilled in the art will appreciate that the features of the different embodiments can be combined in any combination, where appropriate, to form different embodiments of the application. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0194] Various component embodiments of the present application can be implemented in hardware, or as software modules running in one or more processors, or in combinations thereof. As will be appreciated by those skilled in the art, microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functionality of some or all of the components of the gateway, proxy server, system according to embodiments of the present application. The present application can also be implemented as a program of instructions for performing part or all of the methods described herein, e.g., a computer program and a computer program product. Such program of the present application can be stored on a computer readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or in any other form.

[0195] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that one skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps other than those listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a unit claim, several devices can be listed with a conjunction like 'or', but it is to be understood that any of the devices can be used individually. The use of the words 'first','second', and 'third', etc. do not imply any order but rather are used for identification purposes only.

Claims

1. A face recognition system in a heterogeneous computing scenario, characterized in that, The face recognition system comprises a central processor, an auxiliary processor and M computing nodes, M≥1 and being a positive integer; wherein the central processor and the M computing nodes are isolated and decoupled through the auxiliary processor; The central processor is configured to perform system initialization operation, determine the program execution order of face recognition, and collect a scene image frame sequence; The central processor comprises the following during the system initialization operation: A reading module configured to read model distribution information corresponding to the M computing nodes; A segmented quantization module configured to split and segmentally quantize a face recognition network model according to the model distribution information to obtain N deployable sub-files; the segmented quantization module is specifically configured to perform the following operations: constructing the face recognition network model; splitting the face recognition network model into N sub-models according to the model distribution information; the N sub-models correspond to a processing order; creating a quantizer and a calibrator for each sub-model; segmentally quantizing the N sub-models based on the quantizers and calibrators corresponding to the N sub-models to obtain N quantized sub-models; post-processing the N quantized sub-models in a format required by a corresponding computing node to obtain N deployable sub-files; wherein 1≤N≤M, and N is a positive integer; A model distribution module configured to distribute the N deployable sub-files to the corresponding computing nodes; A fusion module configured to fuse the flow order of the M computing nodes and the face recognition order to obtain the program execution order; The auxiliary processor is configured to receive the scene image frame sequence from the central processor; control the M computing nodes to process the scene image frame sequence in parallel according to the program execution order, and obtain face feature data corresponding to the scene image frame sequence; The central processor is further configured to compare the face feature data with registered face data in a face database to obtain a face recognition result.

2. The facial recognition system of claim 1, wherein, The model distribution information is generated with reference to computing power information of the computing nodes.

3. The facial recognition system of claim 1, wherein, The segmented quantization module is further configured to: Load current calibration data by using the calibrator of the current sub-model; wherein the current calibration data is output after quantization of the previous sub-model; Input the current calibration data into the current sub-model, adjust the quantization parameters of the current quantizer according to the forward propagation result, obtain the current quantized sub-model, and input the quantization output of the current quantizer as the calibration data of the next sub-model into the next sub-model for processing, until all sub-models are traversed.

4. The facial recognition system of claim 1, wherein, The system further comprises a splicing module configured to perform the following operations before segmentally quantizing the N sub-models based on the quantizers and calibrators corresponding to the N sub-models to obtain N quantized sub-models: Splice the quantizers and calibrators corresponding to the N sub-models in the corresponding sub-models to form N series models; Splice the N series models with each other according to the processing order to obtain a spliced model.

5. The facial recognition system of claim 1, wherein, The central processor specifically comprises: an acquisition module, configured to acquire the sequence of scene image frames; and an encoding module, configured to encode the sequence of scene image frames in a binary number manner to obtain encoded data. The auxiliary processor specifically comprises: a receiving module, configured to receive the encoded data from the central processor; and a decoding module, configured to decode the encoded data to obtain the sequence of scene image frames.

6. The facial recognition system of claim 1 or 5, wherein, The system further comprises a communication module, configured to realize data interaction between the central processor and the auxiliary processor, and the data for interaction comprises instruction data and encoded data.

7. The facial recognition system of claim 1 or 5, wherein, The auxiliary processor specifically further comprises a control module, configured to sequentially input each image frame in the sequence of scene image frames in combination with a program execution sequence and processing time required by each computing node, so that the M computing nodes process the image frames in parallel to obtain face feature data corresponding to the image frames.

8. The facial recognition system of claim 1, wherein, The system further comprises a public cache, configured to store the sequence of scene image frames transmitted by the central processor and the face feature data transmitted by the auxiliary processor. 9.A face recognition method in a heterogeneous computing scenario, characterized in that, The face recognition method is applied to a face recognition system, and the face recognition system comprises a central processor, an auxiliary processor and M computing nodes, where M is greater than or equal to 1 and is a positive integer; the central processor and the M computing nodes are isolated and decoupled through the auxiliary processor. The face recognition method comprises: performing a system initialization operation by using the central processor to determine a program execution sequence of face recognition and acquire a sequence of scene image frames; The system initialization operation comprises: reading model distribution information corresponding to the M computing nodes; segmenting and quantizing a face recognition network model into N deployable sub-files according to the model distribution information, and specifically performing the following operations: constructing the face recognition network model; splitting the face recognition network model into N sub-models according to the model distribution information; the N sub-models correspond to a processing sequence; creating a quantizer and a calibrator for each sub-model; segmenting and quantizing the N sub-models based on the quantizers and the calibrators corresponding to the N sub-models to obtain N quantized sub-models; post-processing the N quantized sub-models in a format required by a corresponding computing node to obtain N deployable sub-files; and distributing the N deployable sub-files to the corresponding computing nodes; where 1≤N≤M, N is a positive integer; and fusing a flow sequence of the M computing nodes and a face recognition sequence to obtain the program execution sequence; where 1≤N≤M, N is a positive integer; receiving the sequence of scene image frames from the central processor by using the auxiliary processor; and controlling the M computing nodes to process the sequence of scene image frames in parallel according to the program execution sequence to obtain face feature data corresponding to the sequence of scene image frames; comparing the face feature data with registered face data in a face database by using the central processor to obtain a face recognition result.

Images