A method, device and equipment for identifying willingness to pay by face scanning

Abstract

The embodiments of this specification disclose a method, device and equipment for identifying willingness to pay by face recognition. The scheme includes: obtaining a face recognition 2D image and a face recognition 3D image corresponding to the face recognition 2D image; determining candidates to be identified in the face recognition 2D image, and generating corresponding mask images according to the first location of each candidate in the face recognition 2D image to distinguish the first location from other areas in the face recognition 2D image; extracting the features of the face recognition 2D image, and obtaining a first fusion feature according to the features of the face recognition 2D image and the mask image; extracting the features of the face recognition 3D image, and obtaining a second fusion feature according to the first fusion feature and the features of the face recognition 3D image; according to the second fusion feature, identifying whether each candidate has the willingness to pay by face recognition. It can improve the security of face recognition payment.

Description

Technical Field

[0001] This specification relates to the field of machine learning technology, and in particular to a method, device and equipment for identifying willingness to pay by face recognition. Background Art

[0002] With the development of computer and Internet technology, many businesses can be conducted online, which has promoted the development of various online business platforms. Among them, face payment refers to a new payment method based on artificial intelligence, machine vision, 3D sensing, big data and other technologies. By using face recognition as a payment method for identity authentication, it brings great convenience to users and is widely loved by users.

[0003] At present, in the face-scanning payment scenario, after the user to pay turns on the face-scanning payment, he needs to stand in front of the device with the face-scanning payment function to perform face recognition. However, during the face-scanning process, there may be multiple users standing in front of the device, which will cause multiple users to appear in the face-scanning image collected by the device. At this time, when the device performs face recognition on the face-scanning image, it is difficult to determine which user is the current user to pay, that is, which user has the willingness to pay by face-scanning. In other words, only the current user to pay has the willingness to pay by face-scanning, while other users do not have the willingness to pay by face-scanning.

[0004] Based on this, face-scanning payment intention recognition is an important part of face-scanning security in the payment system, which helps to improve the face-scanning security experience. However, if the device recognizes other users and identifies other users, mistaken face-scanning payment will occur, thereby reducing the security of face-scanning payment.

[0005] Based on this, a more secure identification solution is needed for face-scanning payment. Summary of the invention

[0006] One or more embodiments of this specification provide a method, device, equipment and storage medium for identifying willingness to pay by face recognition, so as to solve the following technical problem: a more secure identification solution is needed for face recognition payment.

[0007] To solve the above technical problems, one or more embodiments of this specification are implemented as follows:

[0008] One or more embodiments of this specification provide a method for identifying willingness to pay by face recognition, including:

[0009] Acquire a face scanning 2D image and a face scanning 3D image corresponding to the face scanning 2D image;

[0010] Determine candidates to be identified in the face scanning 2D image, and generate corresponding mask images according to first regions of each candidate in the face scanning 2D image to distinguish the first regions from other regions in the face scanning 2D image;

[0011] Extracting features of the face scanning 2D image, and obtaining a first fusion feature according to the features of the face scanning 2D image and the mask image;

[0012] Extracting features of the face scanning 3D image, and obtaining a second fusion feature according to the first fusion feature and the features of the face scanning 3D image;

[0013] According to the second fusion feature, it is identified whether each of the candidates is willing to pay by face recognition.

[0014] One or more embodiments of this specification provide a face-scanning payment willingness recognition device, including:

[0015] An acquisition module is used to acquire a face scanning 2D image and a face scanning 3D image corresponding to the face scanning 2D image;

[0016] A generation module, which determines candidates to be identified in the face scanning 2D image, and generates corresponding mask images according to the first regions of each candidate in the face scanning 2D image to distinguish the first regions from other regions in the face scanning 2D image;

[0017] A first extraction module extracts features of the face scanning 2D image, and obtains a first fusion feature according to the features of the face scanning 2D image and the mask image;

[0018] A second extraction module extracts features of the face scanning 3D image, and obtains a second fusion feature according to the first fusion feature and the features of the face scanning 3D image;

[0019] The identification module identifies, based on the second fusion feature, whether each of the candidates is willing to pay by face recognition.

[0020] One or more embodiments of this specification provide a face-scanning payment willingness recognition device, including:

[0021] at least one processor; and,

[0022] a memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0024] Acquire a face scanning 2D image and a face scanning 3D image corresponding to the face scanning 2D image;

[0025] Determine candidates to be identified in the face scanning 2D image, and generate corresponding mask images according to first regions of each candidate in the face scanning 2D image to distinguish the first regions from other regions in the face scanning 2D image;

[0026] Extracting features of the face scanning 2D image, and obtaining a first fusion feature according to the features of the face scanning 2D image and the mask image;

[0027] Extracting features of the face scanning 3D image, and obtaining a second fusion feature according to the first fusion feature and the features of the face scanning 3D image;

[0028] According to the second fusion feature, it is identified whether each of the candidates is willing to pay by face recognition.

[0029] One or more embodiments of this specification provide a non-volatile computer storage medium for face-scanning payment willingness recognition, storing computer executable instructions, wherein the computer executable instructions are configured as follows:

[0030] At least one of the above technical solutions adopted in one or more embodiments of this specification can achieve the following beneficial effects:

[0031] By generating a corresponding mask map for each candidate's first area in the face scanning 2D image, the candidate's feature information can be made more distinct, and the difference between those who have the willingness to pay by face scanning and those who do not is increased. The first fusion feature can be used to enhance the image contrast effect, thereby focusing attention on candidates who have the willingness to pay by face scanning, and the second fusion feature can be used to identify whether each candidate has the willingness to pay by face scanning. The features of the corresponding candidate in the face scanning 3D image and the features of the candidate in the face scanning 2D image complement each other's facial features, which can further improve the accuracy of face recognition, thereby more accurately distinguishing candidates with the willingness to pay by face scanning from candidates who do not have the willingness to pay by face scanning in the face scanning image, and more specifically identifying the willingness to pay by face scanning of candidates in the face scanning image, thereby enhancing the face scanning security experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0033] Figure 1 A flowchart of a method for identifying willingness to pay by face recognition provided in one or more embodiments of this specification;

[0034] Figure 2 A schematic diagram of a framework of a face-scanning payment willingness recognition system provided in one or more embodiments of this specification;

[0035] Figure 3 A flowchart of a method for recognizing willingness to pay by face recognition based on end-to-end learning of a deep convolutional neural network provided for one or more embodiments of this specification;

[0036] Figure 4 A schematic diagram of the structure of a device for identifying willingness to pay by face recognition provided in one or more embodiments of this specification;

[0037] Figure 5 A schematic diagram of the structure of a face-scanning payment willingness recognition device provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0038] The embodiments of this specification provide a method, device, equipment and storage medium for identifying willingness to pay by face recognition.

[0039] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0040] Figure 1 A flowchart of a method for identifying willingness to pay by face recognition provided in one or more embodiments of this specification. The process can be executed by an electronic device with a face recognition payment function, which can be a terminal with an image data processing function, for example, a mobile terminal such as a mobile phone, a tablet, a notebook, or a fixed terminal such as a desktop or a server. Some input parameters or intermediate results in the process allow manual intervention and adjustment to help improve accuracy.

[0041] In one or more embodiments of the present specification, the candidate to be identified refers to a user who needs to pay relevant fees. It should be noted that if the candidate wants to make a face payment, he or she needs to register his or her identity information on the corresponding client and enter the facial information. When the candidate turns on the face payment, the client will recognize that the candidate has the intention to make the face payment and then authenticate the candidate through the pre-registered facial information.

[0042] That is, the face image includes the candidate's facial information, and by identifying the facial information, it is possible to obtain whether the candidate has the willingness to pay by face scanning, and then the candidate can be authenticated by the facial information. It should be noted that the face image includes at least one of a face scanning 2D image and a face scanning 3D image.

[0043] Among them, after receiving the face-scanning payment instruction, the electronic device can obtain the face-scanning image through a pre-installed image acquisition device, or the electronic device can generate a face-scanning payment instruction according to the payment order and obtain the face-scanning image through the image acquisition device.

[0044] The number of candidates in front of the image acquisition device may be one or more. When the number of candidates is more than one, the face scan image includes multiple candidates. When the number of candidates is one, the face scan image includes one candidate. At the same time, the face scan image includes not only the facial information of the candidate, but also other feature information of the candidate, such as torso information and limb information, and may also include other objects that do not need to be identified, such as tables, chairs, hanging objects, etc. included in the environment where the candidate is located.

[0045] In addition, under normal circumstances, when an electronic device executes a single face-scanning payment instruction, it is to authenticate the identity of the specific candidate who currently enables face-scanning payment, and the specific candidate usually also has the willingness to pay by face-scanning. That is to say, when executing a single face-scanning payment instruction, even if the face-scanning image includes multiple candidates, not all of the multiple candidates have the willingness to pay by face-scanning. Only the specific candidate has the willingness to pay by face-scanning. It can be considered that the payment willingness of the specific candidate is safe, and the payment willingness of other candidates is unsafe.

[0046] For example, in public places, offline Internet of Things (IoT) face-scanning devices are usually used for face-scanning payment. Among them, face-scanning IoT devices in public places refer to devices with face-scanning functions set up in public consumption scenarios such as supermarkets / convenience stores / catering / hotels / campus education and medical care / campus education. Among them, face-scanning IoT devices generally integrate multiple sensor information, such as 2D vision and 3D vision, to support face-scanning payment systems to realize face recognition.

[0047] If candidate A clicks on face payment, face payment will be enabled, and the IoT face payment device will receive the face payment instruction and obtain the face image through the image acquisition device. Since it is in an open public place, there are multiple candidates queuing to pay, so the face image obtained by the IoT device is likely to include multiple candidates. However, among the multiple candidates, only candidate A is actually willing to pay by face. At this time, it is necessary to identify candidate A's willingness to pay by face and authenticate candidate A through his facial information, that is, other candidates do not actually have the willingness to pay by face.

[0048] Furthermore, if candidate B is behind candidate A or candidate B is lined up with candidate A, then even if candidate B has no intention to pay by face scan, during the face scan payment authentication process of candidate A, the image acquisition device may take pictures of candidate B at the same time as candidate A, resulting in the captured face scan image including both candidates A and B. If the electronic device does not identify candidate A as the face scan user during the face scan image recognition process, but mistakenly recognizes candidate B, then the electronic device will not identify whether candidate B has the intention to pay by face scan, but will directly authenticate candidate B. After the authentication is passed, payment will be made through candidate B's account, thereby mistakenly swiping candidate B's assets, resulting in candidate B's asset loss, which will cause user A to enable face scan and mistakenly swipe user B. Based on this, a more secure identification solution for face scan payment is provided to improve the face scan security experience. Next, through Figure 1 and related contents are described in detail.

[0049] Figure 1 The process in may include the following steps:

[0050] S102: Acquire a face scanning 2D image and a face scanning 3D image corresponding to the face scanning 2D image.

[0051] In one or more embodiments of this specification, the face scanning 2D image is a flat image, which does not carry three-dimensional information and has no stereoscopic effect. That is, the 2D image is on a plane, and the image on a plane is usually a two-dimensional image, while the face scanning 3D image carries three-dimensional information. It should be noted that the face scanning 3D image can directly or indirectly express a three-dimensional image. A typical face scanning 3D image is a depth image, which combines 2D and depth information to express 3D. Some of the following embodiments are mainly explained by taking the depth image as an example. The depth image is also called a distance image, which refers to an image that uses the distance (depth) from the image collector to each point in the scene as the pixel value, which directly reflects the geometric shape of the visible surface of the scene. In other words, the face scanning 2D image is equivalent to the projection of the candidate to be identified on the plane.

[0052] It should be noted that when collecting 2D face images, the main thing is to obtain the RGB color image of the candidate to be identified. Only the planar RGB image information is needed, and no depth information is needed. When collecting 3D face images, one more dimension of depth information is needed than 2D face images, that is, an RGB image plus a depth image D is needed, which together form an RGBD image.

[0053] However, since the 2D face image is equivalent to the projection of the candidate to be identified on a plane, when the face is recognized by the 2D face image, the recognition result will be affected by external factors such as lighting, posture, expression, etc., which will affect the accuracy of face recognition and thus affect the accuracy of face payment intention recognition. The 3D face image is more three-dimensional. Therefore, it is necessary to make full use of the two-dimensional and three-dimensional information of the face. Multimodal recognition refers to the fusion of multiple modal information for recognition. Therefore, the recognition accuracy can be improved through multimodal visual information, that is, 2D face image and 3D face image.

[0054] Based on this, an image acquisition device including color and depth data is used to collect facial information, and the image acquisition device can output color images and depth data at the same time.

[0055] That is to say, each face-scanning 2D image has a face-scanning 2D image taken at the same time, that is, each face-scanning 2D image has its own corresponding face-scanning 3D image.

[0056] Among them, after receiving the face-scanning payment instruction, the electronic device can obtain the face-scanning 2D image and the face-scanning 3D image through a pre-installed image acquisition device, or the electronic device can generate a face-scanning payment instruction according to the payment order and obtain the face-scanning 2D image and the face-scanning 3D image through the image acquisition device.

[0057] S104: Determine candidates to be identified in the face scanning 2D image, and generate corresponding mask images according to the first area of ​​each candidate in the face scanning 2D image to distinguish the first area from other areas in the face scanning 2D image.

[0058] In one or more embodiments of the present specification, the area where the candidate is located in the face scanning 2D image is referred to as the first area, and the first area may include the candidate's appearance feature information, such as face information, torso information, and limb information, but in order to increase the accuracy of the recognition result, the first area mainly includes the candidate's face information. At the same time, the first area can be determined according to the position information of the candidate in the face scanning 2D image.

[0059] Since the mask operation of an image refers to recalculating the value of each pixel in the image through a mask kernel operator, the mask kernel operator describes the degree of influence of the domain pixel points on the new pixel value, and performs weighted averaging of the pixel points according to the weight factor in the mask operator, the image mask operation is often used in image smoothing, edge detection, feature analysis and other areas. Therefore, the mask operation can be used to distinguish the first area where the candidate is located in the face scanning 2D image and other areas in the face scanning 2D image.

[0060] It should be noted that the first area where a single candidate is located in the face-scanning 2D image corresponds to a single mask image. That is to say, if there are multiple candidates in the face-scanning 2D image, a corresponding mask image is generated for each candidate, and multiple mask images are finally obtained.

[0061] That is, in a single mask image, the first area can be distinguished from other areas, for example, the first area is filled with a value of 1, and other areas are filled with values ​​of 0. That is, by generating a mask image corresponding to each candidate, the characteristic information of the candidate can be more distinct, thereby increasing the difference between those who are willing to pay by face scanning and those who are not willing to pay by face scanning.

[0062] S106: extracting features of the face scanning 2D image, and obtaining a first fusion feature according to the features of the face scanning 2D image and the mask image.

[0063] In one or more embodiments of this specification, how to extract the features of the face-scanning 2D image is not limited here. For example, the features of the face-scanning 2D image are extracted by a first feature extraction model. The features of the face-scanning 2D image may include facial features, torso features, and limb features of each candidate. Among them, the facial features may be global features of the candidate's face. Recognizing the face through global features can improve the accuracy of the recognition result.

[0064] Of course, after obtaining the mask image, the first fusion feature can be obtained by inputting the features of the face scanning 2D image and the mask image into the first fusion feature extraction model.

[0065] By extracting the features of the face scanning 2D image and combining the features of the face scanning 2D image with the mask image, it is equivalent to adding a new channel to the features of the face scanning 2D image, that is, increasing the number of channels of the features of the face scanning 2D image, thereby obtaining the first fusion feature. In the first fusion feature, more attention is paid to the facial features of the corresponding candidates in the face scanning 2D image, so that the candidates with the willingness to pay by face scanning and the candidates who do not have the willingness to pay by face scanning in the face scanning 2D image can be more accurately distinguished.

[0066] S108: Extracting features of the face-scanning 3D image, and obtaining second fusion features according to the first fusion features and the features of the face-scanning 3D image.

[0067] In one or more embodiments of this specification, how to extract the features of the face-scanning 3D image is not limited here. For example, the features of the face-scanning 3D image can be extracted by a second feature extraction model. Alternatively, the features of the face-scanning 2D image corresponding to the face-scanning 3D image can be detected by performing feature detection on the face-scanning 2D image, and then the detection result of the face-scanning 2D image is matched to the face-scanning 3D image, thereby extracting the features of the face-scanning 3D image.

[0068] Since the face scanning 3D image corresponds to the face scanning 2D image, the features of the face scanning 3D image may also include the facial features, torso features, and limb features of each candidate. The facial features may be global features of the candidate's face. Recognizing the face through global features can improve the accuracy of the recognition result.

[0069] Of course, the second fusion feature can be obtained by inputting the first fusion feature and the feature of the face scanning 3D image into the second fusion feature extraction model.

[0070] By combining the first fusion feature with the feature of the face scanning 3D image, it is equivalent to adding a new channel to the feature of the first fusion feature, that is, increasing the number of channels of the first fusion feature, thereby obtaining the second fusion feature. In the second fusion feature, the spatial difference between the face scanning 2D image and the face scanning 3D image can be combined, and the facial features of the corresponding candidate in the face scanning 2D image can be supplemented by the features of the face scanning 3D image, or the facial features of the corresponding candidate in the face scanning 3D image can be supplemented by the features in the face scanning 2D image, that is, the two complement each other's facial features of the same candidate, thereby improving the accuracy of face recognition, and realizing multimodal recognition of face scanning payment willingness, and finally being able to accurately distinguish candidates with the willingness to pay by face scanning from candidates with the willingness to pay by face scanning.

[0071] S110: Identify, based on the second fusion feature, whether each of the candidates is willing to pay by face recognition.

[0072] In one or more embodiments of the present specification, the characteristic information of the candidate is determined based on the second fusion feature, thereby identifying whether the candidate is willing to pay by face recognition based on the characteristic information of the candidate.

[0073] Among them, since there are multiple candidates in the face scanning 2D image, a corresponding mask image is generated for each candidate, and finally multiple mask images are obtained. Therefore, when identifying whether a single candidate has the willingness to pay by face scanning, when a mask image is generated, a recognition process is performed to identify whether the candidate corresponding to the mask image has the willingness to pay by face scanning. In other words, when there are multiple mask images, multiple face scanning willingness to pay recognition processes will be performed.

[0074] It should be noted that the preset rules can be used to identify whether the corresponding candidate has the willingness to pay by face recognition. For example, if the candidate's facial area is identified to be located in the middle area, the candidate is considered to have the willingness to pay by face recognition; or, if the candidate's facial area is identified to occupy most of the face recognition image, the candidate is considered to have the willingness to pay by face recognition; or, if the candidate's facial area is identified to occupy most of the face recognition image and the face angle meets the preset angle threshold, the candidate is considered to have the willingness to pay by face recognition.

[0075] Furthermore, the second fused feature can be input into the willingness to pay identification model, the characteristic information of the candidate can be identified by the willingness to pay identification model, and the processing result can be output according to the preset rules. Then, the probability value of willingness to pay by face recognition can be generated according to the processing result, and the willingness to pay by face recognition can be used to judge whether the candidate has the willingness to pay by face recognition.

[0076] The processing result may be a vector generated by the willingness to pay recognition model and used to represent the probability value of the willingness to pay by face recognition.

[0077] For example, if the probability value is greater than a preset probability threshold, it can be considered that the candidate has the willingness to pay by face recognition, that is, the face recognition payment instruction of the electronic device is generated after the candidate turns on the face recognition payment. If the probability value is less than or equal to the preset probability threshold, it can be considered that the candidate does not have the willingness to pay by face recognition. That is, the face recognition payment instruction of the electronic device is not generated after the candidate turns on the face recognition payment, but is generated after other candidates turn on the face recognition payment.

[0078] Furthermore, if there are multiple candidates with probability values ​​greater than the preset probability, it means that the result of this face-scanning payment willingness recognition is not credible, and the authentication will fail. On the contrary, if there is no candidate with a probability value greater than the preset probability, it also means that the result of this face-scanning payment willingness recognition is not credible, and the authentication will fail.

[0079] pass Figure 1The method can generate a corresponding mask map for each candidate's first area in the face scanning 2D image, so as to make the candidate's feature information more distinct, increase the difference between those with the willingness to pay by face scanning and those without the willingness to pay by face scanning, and enhance the image contrast effect through the first fusion feature, thereby focusing attention on the candidates with the willingness to pay by face scanning, and identify whether each candidate has the willingness to pay by face scanning through the second fusion feature, combine the features of the corresponding candidate in the face scanning 3D image with the features of the candidate in the face scanning 2D image, and the two complement each other's facial features of the same candidate, which can further improve the accuracy of face recognition, so as to more accurately distinguish the candidates with the willingness to pay by face scanning and the candidates without the willingness to pay by face scanning in the face scanning image, and more specifically identify the candidates' willingness to pay by face scanning in the face scanning image, thereby enhancing the face scanning security experience.

[0080] based on Figure 1 This specification also provides some specific implementation plans and extension plans of the method, which will be described below.

[0081] In one or more embodiments of the present specification, after the candidate is determined, the face area of ​​the candidate in the face scanning 2D image is extracted. For example, the face of the candidate in the face scanning 2D image is first extracted by a face extraction model, and then the face area of ​​the candidate is determined by the position information of the face. The face area is then processed to determine the face area selection box. The face area selection box can be displayed in a variety of ways, such as a circular box, a rectangular box, an irregular polygonal box, etc. However, there is a prerequisite that in order to ensure the accuracy of the recognition result, the face area selection box must completely surround the face area of ​​the candidate.

[0082] After obtaining the face area selection frame, the first filling area of ​​the mask image corresponding to the candidate will be determined according to the candidate's face area selection frame. The shape of the first filling area can be displayed in a variety of ways, which are not limited here, such as a circular area, a rectangular area, an irregular polygonal area, etc. However, there is a prerequisite that in order to ensure the accuracy of the recognition result, when determining the first filling area, it is necessary to combine the face area selection frame and be as close to the actual face area as possible.

[0083] After the first filling area is determined, a second filling area other than the first filling area is further determined in the face scanning image, and different filling values ​​are assigned to the first filling area and the second filling area.

[0084] It should be noted that in order to make the first filling area in the mask image match the face area in the face-scanning 2D image as much as possible, after assigning different filling values ​​to the first filling area and the second filling area, a mask image with a resolution consistent with the resolution of the face-scanning 2D image is generated.

[0085] Furthermore, since most of the face region is a circular region or an elliptical region, in order to make the first filling region more closely fit the candidate's face region, the first filling region is used as a circular region.

[0086] Specifically, when processing the face region and determining the face region selection frame, the face region selection frame is determined as a rectangular frame. After obtaining the face region of the rectangular frame, the face frame width and the face frame height are calculated according to the position of the rectangular frame in the face scan image, and the radius of the circular region is calculated according to the face frame width and the face frame height.

[0087] Among them, when calculating the radius of the circular area, since the face area is a circular area or an elliptical area, when the rectangular frame is initially generated, the face area is similar to the inscribed circle of the rectangular frame. Therefore, in order to restore the face area as much as possible and ensure that the first filling area includes the entire face area as much as possible, the maximum value between half the width of the rectangular frame and half the height of the frame is taken as the radius of the circular area.

[0088] Therefore, the center of the rectangular box is taken as the center of the circle, half the length of the longest side of the rectangular box is determined as the radius, and then the circular area formed by the center of the circle and the radius is determined as the first filling area of ​​the mask map corresponding to the candidate.

[0089] For example, suppose the position of the candidate's face rectangle in the face scan image is (x1, y1, x2, y2), where x1 and x2 are the position coordinates of the rectangle width on the x-axis, and y1 and y2 are the position coordinates of the rectangle height on the y-axis.

[0090] Then the expression for calculating the face frame width is w=x2-x1, where w is the face frame width. It should be noted that the position coordinate of x1 is smaller than the position coordinate of x2.

[0091] The expression for calculating the face frame height is h=y2-y1, where h is the face frame height. It should be noted that the position coordinate of y1 is smaller than the position coordinate of y2.

[0092] The expression for determining the radius of the circular area is Where R is the radius of the circular area.

[0093] Based on this, the position coordinates of the center of the rectangular box are R is max

[0094] According to the previous description, more intuitively, combined with Figure 2-Figure 3 , a more specific description of this scheme is given.

[0095] Figure 2 A schematic diagram of the framework of a face-scanning payment willingness recognition system provided for one or more embodiments of this specification.

[0096] In one or more embodiments of this specification, in order to more accurately identify the willingness of candidates to pay by face recognition, it is proposed to use a deep convolutional neural network to perform end-to-end learning of multimodal information on the 2D face recognition images and 3D face recognition images obtained during the face recognition payment process, so as to achieve security detection of willingness to pay by face recognition, and in the end-to-end learning network, the learning features of the 2D face recognition images introduce the candidate area attention mechanism so as to integrate the mask map into the network learning, so that the network can more specifically identify the willingness of candidates to pay by face recognition in the 2D face recognition images, thereby enhancing the face recognition security experience. Furthermore, the 3D face recognition images are processed based on the depth value of the face area of ​​the 3D face recognition images, so that the network can effectively perceive the face area of ​​the 3D face recognition images. Thereby, the network features of the 2D face recognition images and the network features of the 3D face recognition images are integrated to achieve multimodal recognition of willingness to pay by face recognition.

[0097] like Figure 2 As shown, the face-scanning payment willingness recognition system is implemented by using a deep convolutional neural network to perform end-to-end learning of multimodal information. The system includes a face-scanning 2D image, a mask map generated by the first area of ​​the candidate in the face-scanning 2D image, a third convolutional network module, an attention mechanism implementation module for the first area of ​​the candidate, a face-scanning 3D image, a processed face-scanning 3D image, a first convolutional network module, a multimodal feature fusion module, a fifth convolutional network module, and a network output.

[0098] It should be noted that the third convolutional network and the fifth convolutional network have a specific corresponding relationship, that is, the network types involved in the third convolutional network and the fifth convolutional network are not limited here, but the two are not separated. For example, the third convolutional network and the fifth convolutional network belong to different parts of the same recognition network type.

[0099] Among them, the face scanning 2D image, the generated mask image and the processed face scanning 3D image are used as input data of the deep convolutional neural network, and the network output is the probability value of willingness to pay by face scanning, that is, the willingness safety probability and the willingness non-safe probability.

[0100] It should be noted that since the depth map refers to an image that uses the distance (depth) from the image collector to each point in the scene as the pixel value, the farther each pixel in the face-scanning 3D image is from the image acquisition device, the greater the depth value of the corresponding pixel. When a candidate is performing face-scanning payment willingness recognition, he or she is usually closest to the image acquisition device relative to other candidates.

[0101] Therefore, in the case where there are multiple candidates in the face-scanning 3D image, at this time, the candidate who does not have the willingness to pay by face-scanning is usually farther away from the image acquisition device, so the depth value of the corresponding pixel point of the candidate who does not have the willingness to pay by face-scanning in the face-scanning 3D image is greater than the depth value of the corresponding pixel point of the candidate who has the willingness to pay by face-scanning in the face-scanning 3D image. Then, the face-scanning 3D image can be processed before extracting the features of the face-scanning 3D image to filter the pixels whose distance from the image acquisition device exceeds the preset threshold. It should be noted that the filtering here may not simply abandon these pixels completely, but may generalize these pixels uniformly, such as unifying the values ​​used to represent these pixels into a specified same value. The purpose is to reduce the differences between these pixels and reduce the contribution of these pixels to the model training and use process, so that the model can focus more on the pixels within the preset threshold, concentrate the computing power on more valuable pixels, improve efficiency, and reduce interference. Therefore, for the candidates who start this face-scanning payment, it is also helpful to improve the accuracy of face-scanning payment willingness recognition.

[0102] Based on this, in the process of face-scanning payment willingness recognition, the features of the face-scanning 2D image are extracted through the third convolutional network. The features and mask map of the face-scanning image are input into the candidate's first area attention mechanism implementation module, which processes the features and mask map of the face-scanning 2D image and outputs the first fusion feature. In addition, the features of the face-scanning 3D image are extracted through the first convolutional network and the processed face-scanning 3D image.

[0103] Among them, Figure 2 As shown, the attention mechanism implementation module of the candidate's first area includes the features of the face 2D image, the mask image after resolution reduction processing, the fourth convolutional network module, and the first fusion feature.

[0104] When the attention mechanism implementation module in the first candidate region processes the features and mask map of the face scanning 2D image, the mask map is down-sampled to obtain a mask map after down-sampled processing to adapt to the features of the face scanning 2D image. Then, the features of the face scanning 2D image and the mask map after down-sampled processing are fused through the fourth convolutional network to output the first fused features.

[0105] Then, the features of the face scanning 3D image and the first fusion features are input into the multimodal feature fusion module, and the multimodal feature fusion module processes the features of the face scanning 3D image and the first fusion features, and outputs a second fusion feature.

[0106] Among them, Figure 2 As shown, the multimodal feature fusion module includes a first fusion feature, a feature of a face-scanning 3D image, and a second convolutional network module.

[0107] When the multimodal feature fusion module processes the features of the face-scanning 3D image and the first fused features, the features of the face-scanning 3D image and the first fused features are fused through a second convolutional network to output a second fused feature.

[0108] Finally, the second fused features are input into the fifth convolutional network, and the second fused features are processed by the fifth convolutional network to obtain a processing result.

[0109] Next, we will continue to explain how to process the face-scanning 3D image before extracting its features, so as to filter out the pixels whose distance from the image acquisition device exceeds a preset threshold.

[0110] Specifically, firstly, the depth value of the second region of each candidate in the face scanning 3D image is calculated. It should be noted that the region where the candidate is located in the face scanning 3D image is called the second region. The second region includes the face region.

[0111] Then, among all the pixels in the second area, the pixels whose distance from the image acquisition device exceeds a preset threshold are filtered out according to the depth value of the second area. When setting the preset threshold, it is necessary to effectively ensure through experience that the depth value of the pixel corresponding to the candidate who is willing to pay by face recognition is less than the preset threshold.

[0112] Furthermore, since it is necessary to obtain the second area of ​​each candidate in the 3D face-scanning image when calculating the depth value of the second area of ​​each candidate in the 3D face-scanning image, the face-scanning payment willingness recognition system will obtain the 2D face-scanning image and extract the features of the 2D face-scanning image. Therefore, in order to reduce the computational pressure of the system, the detection results of the 2D face-scanning image are corresponded to the 3D face-scanning image to obtain the second area, rather than directly extracting the features of the 3D face-scanning image through the second feature extraction model.

[0113] Specifically, firstly, the face area of ​​each candidate in the face scanning 3D image is determined according to the face area selection box of each candidate in the face scanning 2D image. For example, the position coordinates of the face area selection box in the face scanning 2D image are extracted, and then according to the position coordinates, the face area selection box is mapped to the face scanning 3D image, so as to obtain the face area of ​​each candidate in the face scanning 3D image.

[0114] Then, multiple depth values ​​of all pixels in the face area are calculated, and finally the depth value of the face area of ​​each candidate in the face scanning 3D image is determined based on the average value of the multiple depth values. For example, the average value of the multiple depth values ​​is used as the depth value of the face area. In other words, the average value of the depth measurement results in the face area is taken as the depth value of the face.

[0115] Furthermore, when filtering pixels whose distance from the image acquisition device exceeds a preset threshold according to the depth value of the second area, for the convenience of data processing, the depth value of the second area is used as a reference, and the multiple depth values ​​of all pixels are processed and mapped accordingly to the required specified range to better reflect the difference between the pixels and facilitate subsequent calculations. The processed multiple depth values ​​can be filtered to filter out pixels whose distance from the image acquisition device exceeds the preset threshold.

[0116] Specifically, first, the ratios between the multiple depth values ​​of all pixels and the depth value of the face area are calculated respectively. Since the depth value of the face area is obtained based on the average value of the multiple depth values ​​of all pixels, the multiple depth values ​​of all pixels are not much different from the depth value of the face area. In other words, if all the multiple depth values ​​are positive numbers, the data distribution of the ratio result will mostly be between 0 and 2.

[0117] Then, according to the reference value and the ratio of the depth value of the face area, the multiple depth values ​​of all the pixels are processed to be within the vicinity of the reference value. The reference value of the depth value of the face area can be set according to actual needs. For example, the reference value of the depth value of the face area is set to 127, that is, if the multiple depth values ​​are all positive numbers, then the data distribution of the multiple depth values ​​of all the pixels after processing is, for example, mostly between 0-254.

[0118] For example, in order to reduce the calculation pressure of the system, the product of the reference value and the ratio of the depth value of the face area can be calculated first, and then based on the product, the multiple depth values ​​of all pixels are processed to be within the range near the reference value.

[0119] Finally, all the pixels whose multiple depth values ​​after processing are greater than the preset reference threshold are filtered. The larger the depth value of the processed pixel, the farther the pixel is from the image acquisition device. The preset reference threshold can be set according to actual needs. For example, the reference threshold is set to 127+30=157, where 30 is an exemplary value and can be adjusted according to actual needs. The larger the value, the more pixels at farther distances are considered.

[0120] Furthermore, when the candidate is very close to the image acquisition device, the candidate's face area may be incomplete, that is, the image acquisition device does not fully capture the candidate's face area, which will also affect the recognition result of the candidate's willingness to pay by face recognition. Therefore, in order to ensure the accuracy of the recognition result, the pixels far away from the image acquisition device can be filtered, and the pixels very close to the image acquisition device can also be filtered at the same time.

[0121] Specifically, a first preset reference threshold and a second preset reference threshold are preset, wherein the first preset reference threshold is less than the reference value, and the second preset threshold is greater than the reference value. When setting the second preset threshold, the adjustable interval can be set based on the reference value.

[0122] When filtering the pixels corresponding to the multiple depth values ​​of all processed pixels greater than the preset reference threshold, it is equivalent to filtering the pixels corresponding to the second preset reference threshold, which means that the pixels very far from the image acquisition device are filtered, and when filtering the pixels corresponding to the depth values ​​less than the first preset reference threshold, it means that the pixels very close to the image acquisition device are filtered. That is, it can be considered that the pixels corresponding to the depth values ​​greater than the first preset reference threshold and less than the second preset reference threshold are retained as realistically as possible, and other pixels can be generalized or ignored.

[0123] Based on this, the maximum value between multiple depth values ​​of all processed pixel points and the first preset reference threshold can be extracted respectively, and the minimum value between the maximum value and the second preset reference threshold can be extracted. According to the maximum value and the minimum value, the processed pixel points whose depth values ​​are greater than the second preset reference threshold are filtered, and the processed pixel points whose depth values ​​are less than the first preset reference threshold are filtered.

[0124] For example, the following expression is used to filter out pixel points corresponding to a value less than the first preset reference threshold value through the maximum value, and to filter out pixel points corresponding to a value greater than the second preset reference threshold value through the minimum value.

[0125] For example, the expression min Among them, D is the depth value of the pixel point in the face-scanning 3D image, d is the depth value of the face area, 127 is the benchmark value of the depth value of the face area, 0 is the first preset benchmark threshold, 30 is the adjustable value of the pixel point far away from the image acquisition device, and an interval including 30 can be set as the above-mentioned adjustable interval, 127+30 is the second preset benchmark threshold, min is the minimum value, and max is the maximum value.

[0126] pass Figure 2 The system utilizes multimodal information collected during face-scanning payment, that is, multimodal information includes 2D face-scanning images and 3D face-scanning images, and adopts a deep convolutional neural network end-to-end learning approach. The attention mechanism implementation module of the candidate's region enables the network to learn more specifically to judge the candidate's willingness to pay by face-scanning in the face-scanning image, and introduces a multimodal fusion mechanism in the end-to-end network, so that the network can make full use of multiple modal information to effectively judge the face-scanning willingness of the identified candidate in the face-scanning image, thereby realizing the multimodal willingness recognition of the candidate's face-scanning payment, especially in public places, which can effectively prevent the technical effect of candidate A enabling face-scanning and mistakenly swiping the assets of candidate B, thereby enhancing the safe face-scanning experience.

[0127] based on Figure 2 system, more intuitively, Figure 3 A flowchart of a method for recognizing willingness to pay by face recognition based on end-to-end learning of a deep convolutional neural network is provided for one or more embodiments of this specification.

[0128] Figure 3 The process in may include the following steps:

[0129] S302: Acquire a face scanning 2D image and a face scanning 3D image corresponding to the face scanning 2D image.

[0130] S304: Calculate the depth value of the second area of ​​each candidate in the face scanning 3D image, and filter out the pixel points whose distance from the image acquisition device exceeds a preset threshold according to the depth value of the second area among all the pixel points in the second area.

[0131] S306: extracting features of the face scanning 2D image through a third convolutional network, and performing resolution reduction processing on the mask image to adapt to the features of the face scanning 2D image. It should be noted that the third convolutional network is obtained in advance through supervised training.

[0132] For example, by reducing the resolution of the mask image through nearest neighbor sampling, a mask image with the same resolution as the face-scanning 2D image is generated, which can adapt to the features of the face-scanning 2D image so that the fourth convolutional network can process it.

[0133] S308: Using a fourth convolutional network, the features of the face scanning 2D image and the mask image after the resolution reduction process are fused to obtain a first fused feature. It should be noted that the fourth convolutional network is obtained in advance through supervised training.

[0134] In one or more embodiments of the present specification, in the process of obtaining the first fusion feature, the features of the face scanning 2D image and the mask image after resolution reduction processing are connected according to the channel dimension, and the connected features are input into the fourth convolutional network for processing to obtain the first fusion feature.

[0135] The number of convolutional layers of the fourth convolutional network is not specifically limited herein. That is, the number of convolutional layers of the fourth convolutional network can be composed of one or more convolutional layers.

[0136] S310: Extracting features of the face scanning 3D image through a first convolutional network. It should be noted that the first convolutional network is obtained in advance through supervised training.

[0137] The number of convolutional layers of the first convolutional network is not specifically limited herein. That is, the number of convolutional layers of the first convolutional network can be composed of one or more convolutional layers.

[0138] S312: Connect the first fused feature and the feature of the face scanning 3D image according to the channel dimension; input the connected feature into the second convolutional network for processing to obtain a second fused feature. It should be noted that the second convolutional network is obtained in advance through supervised training.

[0139] The second fusion feature has the same number of feature channels and resolution as the first fusion feature.

[0140] S314: Inputting the second fused feature into a fifth convolutional network corresponding to the third convolutional network for processing to obtain a processing result, wherein the third convolutional network and the fifth convolutional network are pre-split from the same convolutional network.

[0141] In one or more embodiments of the present specification, the third convolutional network and the fifth convolutional network are pre-split from the same convolutional network, for example, the convolutional network is resent, ShUffleNet V2, etc.

[0142] During the splitting process, the third convolutional network and the fifth convolutional network can be respectively regarded as the front and back parts of the same convolutional network.

[0143] The position of the split can be determined by the resolution of the features of the face scanning 2D image. Therefore, it is necessary to determine the target resolution so as to serve as the resolution of the features of the face scanning 2D image.

[0144] When determining the target resolution, it is necessary to consider the resolution of the face scanning 2D image. For example, the resolution of the features of the face scanning 2D image is set to 1 / 2 of the resolution of the face scanning 2D image. The target resolution is the resolution of the face scanning 2D image.

[0145] Then, among the convolutional layers in the same convolutional network, determine the convolutional layer that matches the target resolution. Finally, using the matching convolutional layer as the split point, split the same convolutional network into the first part and the second part, with the first part serving as the third convolutional network and the second part serving as the fifth convolutional network.

[0146] S316: Generate a probability value according to the processing result to indicate whether the corresponding candidate is willing to pay by face recognition.

[0147] For example, the probability value is compared with the set threshold probability. If the probability value is greater than the set threshold probability, it is determined that the candidate's willingness is safe, that is, the candidate has the willingness to pay by face recognition. If the probability value is less than or equal to the set threshold probability, it is determined that the candidate's willingness is unsafe, that is, the candidate does not have the willingness to pay by face recognition.

[0148] Based on the previous explanation, the following will further explain how to perform supervised training on the first convolutional network, the second convolutional network, the third convolutional network, the fourth convolutional network, and the fifth convolutional network before identifying whether each candidate has the willingness to pay by face recognition.

[0149] In one or more embodiments of the present specification, a training data set needs to be established first, and then network training is performed using the training data set.

[0150] Specifically, when establishing a training data set, first obtain a face-scanning 2D sample image that contains a confirmed face-scanning user and a face-scanning 3D sample image corresponding to the face-scanning 2D sample image, that is, the candidate enables face-scanning payment, and the face-scanning 2D image and the face-scanning 3D sample image are collected by an image acquisition device. Regardless of whether there are multiple candidates to be identified in the face-scanning 2D image, in the face-scanning 2D image, the candidate is regarded as a face-scanning user, and the face-scanning 2D image is regarded as a face-scanning 2D sample image. Similarly, regardless of whether there are multiple candidates to be identified in the face-scanning 3D image, in the face-scanning 3D image, the candidate is regarded as a face-scanning user, and the face-scanning 3D image is regarded as a face-scanning 3D sample image.

[0151] For example, face scanning images are collected by the camera on an offline IoT face scanning device. If candidate A enables face scanning payment, candidate A will be used as the face scanning user in the face scanning 2D image, and the face scanning 2D image will be used as the face scanning 2D sample image.

[0152] It should be noted that each time face-scanning payment is enabled, a corresponding face-scanning 2D image and face-scanning 3D image will be collected. For example, if candidate A enables face-scanning payment, the IoT face-scanning device will collect images of nearby users to obtain a single face-scanning 2D image and a single face-scanning 3D image. If candidate B enables face-scanning payment, the IoT face-scanning device will collect images of nearby users again to obtain a single face-scanning 2D image and a single face-scanning 3D image.

[0153] After obtaining the face scanning 2D sample image, the positive sample is obtained by marking the face scanning user as having the willingness to pay by face scanning and generating a corresponding mask image, that is, the positive sample includes the face scanning 2D sample image, the corresponding mask image, and the corresponding face scanning 3D sample image.

[0154] When generating the corresponding mask image, the position of the face area selection box of the face scanning user in the face scanning 2D image is selected from the face scanning 2D image, and then the corresponding mask image is generated through the position.

[0155] In addition, when filtering the pixels in the face-scanning 3D image whose distance from the image acquisition device exceeds a preset threshold, the face-scanning 3D depth image is corresponded according to the position of the face area selection box in the face-scanning 2D image, and then the average value of the depth measurement results in the face area of ​​the face-scanning 3D depth image is taken as the depth value of the face area, so that the pixels whose distance from the image acquisition device exceeds the preset threshold are filtered according to the depth value of the face area.

[0156] In the face scanning 2D sample image and the face scanning 3D sample image, when the face scanning user is marked as having the willingness to pay by face scanning, the willingness label can be used for labeling. For example, the willingness label is {0,1}, where 1 represents the willingness to pay by face scanning and 0 represents the lack of willingness to pay by face scanning.

[0157] After obtaining the face scanning 2D sample image and the face scanning 3D sample image, since there may be multiple candidates to be identified in the face scanning sample image, if the face scanning 2D sample image and the face scanning 3D sample image also contain other users who were photographed by the way, the other users are marked as not willing to pay by face scanning, and the corresponding mask image is generated to obtain negative samples. For example, if candidate A enables face scanning payment, candidate B is marked as not willing to pay by face scanning in the face scanning 2D sample image and the face scanning 3D sample image.

[0158] Finally, according to the obtained positive samples and negative samples, supervised training is performed on the first convolutional network, the second convolutional network, the third convolutional network, the fourth convolutional network, and the fifth convolutional network.

[0159] In the case of supervised training, firstly, the positive samples and negative samples are randomly sampled in the training data set to generate training batches and their corresponding labeled labels. Then, the training batches and their corresponding labeled labels are input into the initial deep convolutional neural network. The initial deep convolutional neural network includes the untrained first convolutional network, the second convolutional network, the third convolutional network, the fourth convolutional network, and the fifth convolutional network.

[0160] Then, the initial deep convolutional neural network outputs the probability value, calculates the loss function through the probability value and the corresponding label, and performs network training without interrupting the optimization of the loss function through the gradient descent method, thereby completing the supervised training and obtaining the deep convolutional neural network. Among them, the rules for identifying whether the candidate has the willingness to pay by face recognition can be obtained through network training. For example, if the deep convolutional neural network recognizes that the candidate's face area is located in the middle area, it is considered that the candidate has the willingness to pay by face recognition.

[0161] Based on the same idea, one or more embodiments of this specification also provide devices and apparatuses corresponding to the above methods, such as Figure 4 , Figure 5 shown.

[0162] Figure 4 A schematic diagram of a device for identifying willingness to pay by face recognition provided in one or more embodiments of this specification, the device comprising:

[0163] An acquisition module 402 acquires a face scanning 2D image and a face scanning 3D image corresponding to the face scanning 2D image;

[0164] A generating module 404 determines candidates to be identified in the face scanning 2D image, and generates corresponding mask images according to the first regions of each candidate in the face scanning 2D image to distinguish the first regions from other regions in the face scanning 2D image;

[0165] A first extraction module 406 extracts features of the face scanning 2D image, and obtains a first fusion feature according to the features of the face scanning 2D image and the mask image;

[0166] A second extraction module 408 extracts features of the face scanning 3D image, and obtains a second fusion feature according to the first fusion feature and the features of the face scanning 3D image;

[0167] The identification module 410 identifies whether each of the candidates is willing to pay by face recognition according to the second fusion feature.

[0168] Optionally, a filtering module is also included to calculate the depth value of the second area of ​​each candidate in the face scanning 3D image, and filter out the pixel points whose distance from the image acquisition device exceeds a preset threshold according to the depth value of the second area among all the pixel points in the second area.

[0169] Optionally, the filtering module determines the face area of ​​each candidate in the face scanning 3D image according to the face area selection box of each candidate in the face scanning 2D image;

[0170] Calculating multiple depth values ​​of all pixels in the face area;

[0171] Determine the depth value of the face area of ​​each candidate in the face scanning 3D image according to the average value of the multiple depth values.

[0172] Optionally, the filtering module calculates ratios between the multiple depth values ​​of all the pixel points and the depth value of the face area respectively;

[0173] According to the reference value of the depth value of the face area and the ratio, the multiple depth values ​​of all the pixels are processed to be within a range near the reference value;

[0174] Filter the pixel points corresponding to the multiple depth values ​​of all the processed pixel points that are greater than a preset reference threshold; wherein, the larger the depth value of the processed pixel point is, the farther the pixel point is from the image acquisition device.

[0175] Optionally, the filtering module calculates the product of a reference value of the depth value of the face area and the ratio;

[0176] According to the product, the multiple depth values ​​of all the pixels are respectively processed to be within a range near the reference value.

[0177] Optionally, a first preset reference threshold and a second preset reference threshold are provided, the first preset reference threshold is less than the reference value, and the second preset reference threshold is greater than the reference value; the filtering module respectively extracts the maximum value between the multiple depth values ​​of all the processed pixel points and the first preset reference threshold;

[0178] Extracting a minimum value between the maximum value and a second preset reference threshold value;

[0179] According to the maximum value and the minimum value, the processed pixel points whose depth values ​​are greater than the second preset reference threshold are filtered, and the processed pixel points whose depth values ​​are less than the first preset reference threshold are filtered.

[0180] Optionally, the second extraction module 408 extracts features of the face scanning 3D image through a first convolutional network;

[0181] According to the channel dimension, the first fusion feature and the feature of the face scanning 3D image are connected; the connected feature is input into the second convolutional network for processing to obtain the second fusion feature.

[0182] Optionally, the first extraction module 406 extracts features of the face scanning 2D image through a third convolutional network;

[0183] Performing resolution reduction processing on the mask image to adapt to the features of the face scanning 2D image;

[0184] The features of the face scanning 2D image and the mask image after the resolution reduction processing are fused through the fourth convolutional network to obtain a first fused feature.

[0185] Optionally, the first extraction module 406 connects the features of the face scanning 2D image and the mask image after the resolution reduction processing according to the channel dimension;

[0186] The connected features are input into the fourth convolutional network for processing to obtain the first fusion features.

[0187] Optionally, the recognition module 410 inputs the second fused feature into a fifth convolutional network corresponding to the third convolutional network for processing to obtain a processing result, wherein the third convolutional network and the fifth convolutional network are pre-split from the same convolutional network;

[0188] A probability value is generated according to the processing result to indicate whether the corresponding candidate is willing to pay by face recognition.

[0189] Optionally, the recognition module 410 determines a target resolution so as to serve as a resolution of features of the face scanning 2D image;

[0190] Among the convolutional layers in the same convolutional network, determining a convolutional layer that matches the target resolution;

[0191] Taking the matched convolution layer as a splitting point, the same convolution network is split into a front part and a rear part, the front part serves as the first convolution network, and the rear part serves as the third convolution network.

[0192] Optionally, a supervised training module is further included to obtain a face scanning 2D sample image of a confirmed face scanning user and a face scanning 3D sample image corresponding to the face scanning 2D sample image;

[0193] The positive sample is obtained by marking the face-scanning user as having the willingness to pay by face-scanning and generating a corresponding mask map;

[0194] If the face-scanning 2D sample image and the face-scanning 3D sample image also include other users who are photographed incidentally, then the other users are marked as not willing to pay by face-scanning and a corresponding mask image is generated to obtain a negative sample;

[0195] According to the obtained samples, supervised training is performed on the first convolutional network, the second convolutional network, the third convolutional network, the fourth convolutional network, and the fifth convolutional network.

[0196] Optionally, the generating module 404 executes, for each of the determined candidates, the following steps:

[0197] Determine, according to the face region selection box of the candidate, a first filling region of the mask image corresponding to the candidate, and a second filling region outside the first filling region;

[0198] By assigning different filling values ​​to the first filling area and the second filling area, the mask image having a resolution consistent with a resolution of the face scanning image is generated.

[0199] Optionally, the face region selection frame is a rectangular frame; the generating module 404 determines the center of the rectangular frame as the center of the circle, and determines half the length of the longest side of the rectangular frame as the radius;

[0200] A circular area formed by the center of the circle and the radius is determined as the first filling area of ​​the mask image corresponding to the candidate.

[0201] Optionally, the face scanning 2D image contains at least two human faces.

[0202] Optionally, the device is applied to an offline IoT face recognition device, and the face recognition image is collected by the IoT face recognition device for nearby users.

[0203] Figure 5 A schematic diagram of a face-scanning payment willingness recognition device provided in one or more embodiments of this specification, the device comprising:

[0204] at least one processor; and,

[0205] a memory communicatively connected to the at least one processor; wherein,

[0206] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0207] Acquire a face scanning 2D image and a face scanning 3D image corresponding to the face scanning 2D image;

[0208] Determine candidates to be identified in the face scanning 2D image, and generate corresponding mask images according to first regions of each candidate in the face scanning 2D image to distinguish the first regions from other regions in the face scanning 2D image;

[0209] Extracting features of the face scanning 2D image, and obtaining a first fusion feature according to the features of the face scanning 2D image and the mask image;

[0210] Extracting features of the face scanning 3D image, and obtaining a second fusion feature according to the first fusion feature and the features of the face scanning 3D image;

[0211] According to the second fusion feature, it is identified whether each of the candidates is willing to pay by face recognition.

[0212] Based on the same idea, one or more embodiments of this specification also provide a non-volatile computer storage medium for face-scanning payment willingness recognition corresponding to the above method, storing computer executable instructions, and the computer executable instructions are set to:

[0213] Acquire a face scanning 2D image and a face scanning 3D image corresponding to the face scanning 2D image;

[0214] Determine candidates to be identified in the face scanning 2D image, and generate corresponding mask images according to first regions of each candidate in the face scanning 2D image to distinguish the first regions from other regions in the face scanning 2D image;

[0215] Extracting features of the face scanning 2D image, and obtaining a first fusion feature according to the features of the face scanning 2D image and the mask image;

[0216] Extracting features of the face scanning 3D image, and obtaining a second fusion feature according to the first fusion feature and the features of the face scanning 3D image;

[0217] The above description is only one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of this specification.

Claims

1. A method for identifying willingness to pay by face recognition, comprising: Acquire a face scanning 2D image and a face scanning 3D image corresponding to the face scanning 2D image; Determine candidates to be identified in the face scanning 2D image, and generate corresponding mask images according to first regions of each candidate in the face scanning 2D image to distinguish the first regions from other regions in the face scanning 2D image; Extracting features of the face scanning 2D image, and obtaining a first fusion feature according to the features of the face scanning 2D image and the mask map, specifically comprising: obtaining the first fusion feature by inputting the features of the face scanning 2D image and the mask map into a first fusion feature extraction model, wherein the first fusion feature extraction model includes a convolutional network; Extracting features of the face scanning 3D image, and obtaining a second fusion feature according to the first fusion feature and the features of the face scanning 3D image; According to the second fusion feature, it is identified whether each of the candidates is willing to pay by face recognition.

2. The method according to claim 1, before extracting the features of the face scanning 3D image, the method further comprises: Calculate the depth value of the second area of ​​each candidate in the face scanning 3D image, and filter out the pixel points in the second area whose distance from the image acquisition device exceeds a preset threshold according to the depth value of the second area.

3. The method according to claim 2, wherein the calculating the depth value of the second region of each candidate in the face scanning 3D image comprises: Determine the face area of ​​each candidate in the face scanning 3D image according to the face area selection box of each candidate in the face scanning 2D image; Calculating multiple depth values ​​of all pixels in the face area; Determine the depth value of the face area of ​​each candidate in the face scanning 3D image according to the average value of the multiple depth values.

4. The method according to claim 3, wherein filtering the pixels whose distance from the image acquisition device exceeds a preset threshold according to the depth value of the second area specifically comprises: Calculating the ratios of the multiple depth values ​​of all the pixel points to the depth value of the face area respectively; According to the reference value of the depth value of the face area and the ratio, the multiple depth values ​​of all the pixels are processed to be within a range near the reference value; Filter the pixel points corresponding to the multiple depth values ​​of all the processed pixel points that are greater than a preset reference threshold; wherein, the larger the depth value of the processed pixel point is, the farther the pixel point is from the image acquisition device.

5. The method according to claim 4, wherein the processing of the multiple depth values ​​of all the pixels to be within a range near the reference value comprises: Calculate the product of the reference value of the depth value of the face area and the ratio; According to the product, the multiple depth values ​​of all the pixels are respectively processed to be within a range near the reference value.

6. The method according to claim 4, wherein a first preset reference threshold and a second preset reference threshold are set, wherein the first preset reference threshold is smaller than the reference value, and the second preset reference threshold is larger than the reference value; The filtering of the pixel points corresponding to the plurality of depth values ​​of all the processed pixel points being greater than a preset reference threshold specifically includes: Respectively extracting the maximum values ​​between the multiple depth values ​​of all the processed pixel points and the first preset reference threshold; Extracting a minimum value between the maximum value and a second preset reference threshold value; According to the maximum value and the minimum value, the processed pixel points whose depth values ​​are greater than the second preset reference threshold are filtered, and the processed pixel points whose depth values ​​are less than the first preset reference threshold are filtered.

7. The method according to claim 1, wherein extracting the features of the face-scanning 3D image and obtaining the second fusion feature according to the first fusion feature and the features of the face-scanning 3D image comprises: Extracting features of the face scanning 3D image through a first convolutional network; According to the channel dimension, the first fusion feature and the feature of the face scanning 3D image are connected; the connected feature is input into the second convolutional network for processing to obtain the second fusion feature.

8. The method according to claim 1, wherein extracting the features of the face scanning 2D image and obtaining the first fusion feature according to the features of the face scanning 2D image and the mask image specifically comprises: Extracting features of the face scanned 2D image through a third convolutional network; Performing resolution reduction processing on the mask image to adapt to the features of the face scanning 2D image; The features of the face scanning 2D image and the mask image after the resolution reduction processing are fused through the fourth convolutional network to obtain a first fused feature.

9. The method according to claim 8, wherein the fourth convolutional network is used to fuse the features of the face scanning 2D image and the mask image after the resolution reduction process to obtain the first fused feature, specifically comprising: According to the channel dimension, the features of the face scanning 2D image and the mask image after the resolution reduction processing are connected; the connected features are input into the fourth convolutional network for processing to obtain the first fusion feature.

10. The method according to claim 8, wherein the step of identifying whether each candidate has a willingness to pay by face recognition based on the second fusion feature comprises: Inputting the second fused feature into a fifth convolutional network corresponding to the third convolutional network for processing to obtain a processing result, wherein the third convolutional network and the fifth convolutional network are pre-split from the same convolutional network; A probability value is generated according to the processing result to indicate whether the corresponding candidate is willing to pay by face recognition.

11. The method according to claim 10, wherein the splitting comprises: Determine a target resolution so as to serve as a resolution of features of the face scanning 2D image; Among the convolutional layers in the same convolutional network, determining a convolutional layer that matches the target resolution; Taking the matched convolution layer as a splitting point, the same convolution network is split into a front part and a rear part, the front part serves as the third convolution network, and the rear part serves as the fifth convolution network.

12. The method according to claim 10, before identifying whether each of the candidates has a willingness to pay by face recognition based on the second fusion feature, the method further comprises: Acquire a face scanning 2D sample image including a confirmed face scanning user and a face scanning 3D sample image corresponding to the face scanning 2D sample image; The positive sample is obtained by marking the face-scanning user as having the willingness to pay by face-scanning and generating a corresponding mask map; If the face-scanning 2D sample image and the face-scanning 3D sample image also include other users who are photographed incidentally, then the other users are marked as not willing to pay by face-scanning and a corresponding mask image is generated to obtain a negative sample; According to the obtained samples, supervised training is performed on the first convolutional network, the second convolutional network, the third convolutional network, the fourth convolutional network, and the fifth convolutional network.

13. The method according to claim 1, wherein generating a corresponding mask map according to the first region of each candidate in the face scanning 2D image comprises: For each of the determined candidates, execute: Determine, according to the face region selection box of the candidate, a first filling region of the mask image corresponding to the candidate, and a second filling region outside the first filling region; By assigning different filling values ​​to the first filling area and the second filling area, the mask image having a resolution consistent with a resolution of the face scanning 2D image is generated.

14. The method according to claim 13, wherein the face area selection frame is a rectangular frame; The step of determining the first filling area of ​​the mask image corresponding to the candidate according to the face area selection box of the candidate specifically includes: The center of the rectangular box is determined as the center of the circle, and half the length of the longest side of the rectangular box is determined as the radius; a circular area formed by the center of the circle and the radius is determined as the first filling area of ​​the mask image corresponding to the candidate.

15. The method according to any one of claims 1 to 14, wherein the face scanning 2D image contains at least two human faces.

16. The method according to any one of claims 1 to 14, wherein the method is applied to an offline IoT face recognition device, and the face recognition 2D image and the face recognition 3D image are collected by the IoT face recognition device for nearby users.

17. A device for identifying willingness to pay by face recognition, comprising: An acquisition module is used to acquire a face scanning 2D image and a face scanning 3D image corresponding to the face scanning 2D image; A generation module, which determines candidates to be identified in the face scanning 2D image, and generates corresponding mask images according to the first regions of each candidate in the face scanning 2D image to distinguish the first regions from other regions in the face scanning 2D image; The first extraction module extracts features of the face scanning 2D image, and obtains a first fusion feature according to the features of the face scanning 2D image and the mask map, specifically comprising: obtaining the first fusion feature by inputting the features of the face scanning 2D image and the mask map into a first fusion feature extraction model, wherein the first fusion feature extraction model includes a convolutional network; A second extraction module extracts features of the face scanning 3D image, and obtains a second fusion feature according to the first fusion feature and the features of the face scanning 3D image; The identification module identifies, based on the second fusion feature, whether each of the candidates is willing to pay by face recognition.

18. A face-scanning payment willingness recognition device, comprising: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Acquire a face scanning 2D image and a face scanning 3D image corresponding to the face scanning 2D image; Determine candidates to be identified in the face scanning 2D image, and generate corresponding mask images according to first regions of each candidate in the face scanning 2D image to distinguish the first regions from other regions in the face scanning 2D image; Extracting features of the face scanning 2D image, and obtaining a first fusion feature according to the features of the face scanning 2D image and the mask map, specifically comprising: obtaining the first fusion feature by inputting the features of the face scanning 2D image and the mask map into a first fusion feature extraction model, wherein the first fusion feature extraction model includes a convolutional network; Extracting features of the face scanning 3D image, and obtaining a second fusion feature according to the first fusion feature and the features of the face scanning 3D image; According to the second fusion feature, it is identified whether each of the candidates is willing to pay by face recognition.

Images