Face-scanning payment willingness identification method, device and equipment

Abstract

The embodiment of the invention discloses a face-scanning payment willingness identification method, device and equipment, and belongs to the technical field of electronic payment. The scheme comprises the following steps: acquiring a face scanning 2D image; candidate persons to be recognized are determined in the face-scanning 2D image, and corresponding mask graphs are generated according to first areas of the candidate persons in the face-scanning 2D image so as to distinguish the first areas from other areas in the face-scanning 2D image; extracting features of the face-scanning 2D image, and obtaining a first fusion feature according to the features of the face-scanning 2D image and the mask pattern; judging whether the 3D mode is applicable currently; if yes, acquiring a face-scanning 3D image corresponding to the face-scanning 2D image, extracting features of the face-scanning 3D image, obtaining a second fusion feature according to the first fusion feature and the features of the face-scanning 3D image, and identifying whether each candidate has the willingness of face-scanning payment according to the second fusion feature; and otherwise, according to the first fusion feature, identifying whether each candidate has the willingness of face-scanning payment.

Description

[0001] This application is a divisional application of the invention patent application submitted to the China Patent Office on May 10, 2022, with application number 202210504933.9 and invention name "A method, device and equipment for identifying willingness to pay by face recognition". The full text of the application is incorporated into this application by reference. Technical Field

[0002] This specification relates to the field of electronic payment technology, and in particular to a method, device and equipment for identifying willingness to pay by face recognition. Background Art

[0003] With the development of computer and Internet technology, many businesses can be conducted online, which has promoted the development of various online business platforms. Among them, face payment refers to a new payment method based on artificial intelligence, machine vision, 3D sensing, big data and other technologies. By using face recognition as a payment method for identity authentication, it brings great convenience to users and is widely loved by users.

[0004] Currently, in the face-scanning payment scenario, after the user to pay turns on the face-scanning payment, he needs to stand in front of the face-scanning payment device for face recognition. However, during the face-scanning process, there may be multiple users standing in front of the device, which will cause multiple users to appear in the face-scanning image collected by the device. At this time, when the device performs face recognition on the face-scanning image, it is difficult to determine which user is the current user to pay, that is, which user has the willingness to pay by face-scanning. In other words, only the current user to pay has the willingness to pay by face-scanning, while other users do not have the willingness to pay by face-scanning.

[0005] Based on this, face-scanning payment intention recognition is an important part of face-scanning security in the payment system, which helps to improve the face-scanning security experience. However, if the device recognizes other users and identifies other users, mistaken face-scanning payment will occur, thereby reducing the security of face-scanning payment. Not only that, with the popularization of face-scanning payment function devices, the impact of device diversity and differences on recognition also needs attention.

[0006] Based on this, a more secure and more widely applicable identification solution is needed for face-scanning payment. Summary of the invention

[0007] One or more embodiments of the present specification provide a method, device, equipment and storage medium for identifying willingness to pay by face recognition, so as to solve the following technical problem: a safer and more widely applicable identification solution is needed for face recognition payment.

[0008] To solve the above technical problems, one or more embodiments of this specification are implemented as follows:

[0009] One or more embodiments of this specification provide a method for identifying willingness to pay by face recognition, including:

[0010] Get the face scan 2D image;

[0011] Determine candidates to be identified in the face scanning 2D image, and generate corresponding mask images according to first regions of each candidate in the face scanning 2D image to distinguish the first regions from other regions in the face scanning 2D image;

[0012] Extracting features of the face scanning 2D image, and obtaining a first fusion feature according to the features of the face scanning 2D image and the mask image;

[0013] Determine whether the 3D mode is currently applicable;

[0014] If yes, then obtain a 3D face recognition image corresponding to the 2D face recognition image, extract features of the 3D face recognition image, obtain a second fusion feature based on the first fusion feature and the features of the 3D face recognition image, and identify whether each of the candidates has a willingness to pay by face recognition based on the second fusion feature;

[0015] Otherwise, based on the first fusion feature, identify whether each of the candidates is willing to pay by face recognition.

[0016] One or more embodiments of this specification provide a face-scanning payment willingness recognition device, including:

[0017] 2D image acquisition module, to obtain the 2D image of the face scanned;

[0018] a mask map generating module, which determines candidates to be identified in the face scanning 2D image, and generates corresponding mask maps according to the first regions of each candidate in the face scanning 2D image to distinguish the first regions from other regions in the face scanning 2D image;

[0019] A 2D modality fusion module extracts features of the face scanning 2D image, and obtains a first fusion feature according to the features of the face scanning 2D image and the mask image;

[0020] Modality adaptability judgment module, judging whether the current 3D modality is applicable;

[0021] The multimodal fusion module, if yes, obtains the face-scanning 3D image corresponding to the face-scanning 2D image, extracts features of the face-scanning 3D image, obtains a second fusion feature based on the first fusion feature and the features of the face-scanning 3D image, and the first recognition module recognizes whether each of the candidates has a willingness to pay by face-scanning based on the second fusion feature;

[0022] The second identification module, otherwise, identifies whether each of the candidates is willing to pay by face recognition according to the first fusion feature.

[0023] One or more embodiments of this specification provide a face-scanning payment willingness recognition device, including:

[0024] at least one processor; and,

[0025] a memory communicatively connected to the at least one processor; wherein,

[0026] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0027] Get the face scan 2D image;

[0028] Determine candidates to be identified in the face scanning 2D image, and generate corresponding mask images according to first regions of each candidate in the face scanning 2D image to distinguish the first regions from other regions in the face scanning 2D image;

[0029] Extracting features of the face scanning 2D image, and obtaining a first fusion feature according to the features of the face scanning 2D image and the mask image;

[0030] Determine whether the 3D mode is currently applicable;

[0031] If yes, then obtain a 3D face recognition image corresponding to the 2D face recognition image, extract features of the 3D face recognition image, obtain a second fusion feature based on the first fusion feature and the features of the 3D face recognition image, and identify whether each of the candidates has a willingness to pay by face recognition based on the second fusion feature;

[0032] Otherwise, based on the first fusion feature, identify whether each of the candidates is willing to pay by face recognition.

[0033] One or more embodiments of this specification provide a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to:

[0034] Get the face scan 2D image;

[0035] Determine candidates to be identified in the face scanning 2D image, and generate corresponding mask images according to first regions of each candidate in the face scanning 2D image to distinguish the first regions from other regions in the face scanning 2D image;

[0036] Extracting features of the face scanning 2D image, and obtaining a first fusion feature according to the features of the face scanning 2D image and the mask image;

[0037] Determine whether the 3D mode is currently applicable;

[0038] If yes, then obtain a 3D face recognition image corresponding to the 2D face recognition image, extract features of the 3D face recognition image, obtain a second fusion feature based on the first fusion feature and the features of the 3D face recognition image, and identify whether each of the candidates has a willingness to pay by face recognition based on the second fusion feature;

[0039] Otherwise, based on the first fusion feature, identify whether each of the candidates is willing to pay by face recognition.

[0040] At least one of the above technical solutions adopted in one or more embodiments of this specification can achieve the following beneficial effects: by generating a corresponding mask map for each candidate's first area in the face-scanning 2D image, the feature information of the candidate can be more distinct, and the difference between those who are willing to pay by face-scanning and those who are not willing to pay by face-scanning is increased. Through the first fusion feature, the image contrast effect can be enhanced, thereby focusing attention on the candidates who are willing to pay by face-scanning; compared with the 2D mode, the 3D mode is less supported in different face-scanning payment devices. Therefore, by judging whether the current face-scanning device is in the current recognition scenario, Whether the 3D modality is applicable, if applicable, multimodal recognition is prioritized. Through the second fusion feature, the features of the corresponding candidate in the 3D face image are combined with the features of the candidate in the 2D face image. The two complement each other's facial features of the same candidate, which can further improve the object accuracy of face recognition. If it is not applicable, 2D modality recognition can be used alone, which is low-cost and helps support more devices to realize face payment. In summary, it also helps to accurately distinguish candidates who are willing to pay by face payment from candidates who are not willing to pay by face payment in the same face image, thereby improving the security of face payment. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 A flowchart of a method for identifying willingness to pay by face recognition provided in one or more embodiments of this specification;

[0043] Figure 2 A schematic diagram of a framework of a face-scanning payment willingness recognition system provided in one or more embodiments of this specification;

[0044] Figure 3 A flowchart of a method for recognizing willingness to pay by face recognition based on end-to-end learning of a deep convolutional neural network provided for one or more embodiments of this specification;

[0045] Figure 4 A schematic diagram of the structure of a device for identifying willingness to pay by face recognition provided in one or more embodiments of this specification;

[0046] Figure 5 A schematic diagram of the structure of a face-scanning payment willingness recognition device provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0047] The embodiments of this specification provide a method, device, equipment and storage medium for identifying willingness to pay by face recognition.

[0048] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0049] At present, the face-scanning payment function is not only popular on personal terminals, but also on devices in public places, such devices mainly include offline IoT machines. In the scenario of face-scanning payment using offline IoT machines in public places, there are many scenarios where users have to queue up to pay. Because it is in an open public place, if there are multiple people in the image collected by the machine, there may be a situation where user A activates face-scanning and mistakenly scans user B. When user A starts face-scanning payment, if the machine mistakenly recognizes and pays for user B's face, it will cause property loss to user B and affect the security of face-scanning payment. Based on this, for face-scanning payment devices, especially offline machines, this solution identifies the face-scanning payment intention of users who are currently suspected of making face-scanning payments, so as to find the correct payment object and avoid wrong deductions.

[0050] Based on the mask map, this solution distinguishes multiple suspected users in the same image of the face-scanning payment device, and then combines 2D and 3D modalities to identify the payment willingness of each suspected user, thereby improving the security of face-scanning payment. On the other hand, offline IoT devices will provide different hardware and software configurations in different usage scenarios. For example, in small store scenarios where the merchant can accept low device costs, low-cost devices often only include 2D visual sensors of the main mode, but not other modality sensors, in order to reduce costs. In some cases, the use conditions of the functional modules of the 3D modality are more stringent than those of the functional modules of the 2D modality, so their service life is relatively shorter and they are more likely to be unavailable. In order to solve the above problems, make the willingness algorithm independent of the influence of software and hardware conditions, and enhance equipment management and high availability, this solution aims to be able to fully adapt to the available modal states of the machine, automatically and fully utilize multimodal information under multimodality, and automatically utilize 2D modal information alone when some modalities (mainly including 3D modality) are not applicable, and realize secure recognition of facial payment willingness through end-to-end learning, so that it can flexibly respond to different scenarios and the solution has a wider applicability.

[0051] Based on this idea, the following is further explained in detail.

[0052] Figure 1 A flow chart of a method for identifying willingness to pay by face recognition provided for one or more embodiments of this specification. The method can be applied to different business fields, such as e-commerce business field, financial business field, electronic payment business field, instant messaging business field, robot business field, video service business field, game business field, official business field, etc. The process can be executed by a computing device in the corresponding field (such as an operation and maintenance server corresponding to the e-commerce business, etc.), and some input parameters or intermediate results in the process allow manual intervention and adjustment to help improve accuracy.

[0053] Figure 1 The process in may include the following steps:

[0054] S102: Acquire a 2D face scan image.

[0055] The face scanning 2D image is collected by the device for nearby users. Generally, the device will collect the face scanning 2D image only when triggered by a user who currently requests face scanning.

[0056] In one or more embodiments of this specification, the face-scanning 2D image is a flat image, which does not carry three-dimensional information and has no stereoscopic effect. In comparison, the face-scanning 3D image carries three-dimensional information and can directly or indirectly express a three-dimensional image. A typical face-scanning 3D image is a depth image, which combines 2D and depth information to express 3D. The following embodiments are mainly described by taking the depth image as an example. The depth image is also called a distance image, which refers to an image that uses the actual distance (depth) from the image collector to each point in the scene as the pixel value, which directly reflects the geometric shape of the visible surface of the scene.

[0057] When performing face recognition on a person using a 2D face image, the recognition result will be affected by external factors such as lighting, posture, and expression, which will affect the accuracy of face recognition and thus the accuracy of face payment intention recognition. A 3D face image is more three-dimensional. Therefore, it is necessary to make full use of the two-dimensional and three-dimensional information of the face for multimodal recognition and improve the recognition accuracy based on multimodal visual information (2D face images and 3D face images, etc.).

[0058] In one or more embodiments of this specification, if the current device supports 3D mode, the face scanning 3D image corresponding to each face scanning 2D image can be further obtained when executing S102. For example, after receiving the face scanning payment instruction, the device obtains the face scanning 2D image and the face scanning 3D image through a pre-installed image acquisition device, or the electronic device generates a face scanning payment instruction according to the payment order, and obtains the face scanning 2D image and the face scanning 3D image through the image acquisition device.

[0059] S104: Determine candidates to be identified in the face scanning 2D image, and generate corresponding mask images according to the first area of ​​each candidate in the face scanning 2D image to distinguish the first area from other areas in the face scanning 2D image.

[0060] In one or more embodiments of the present specification, the candidates to be identified are the suspected users mentioned above, including at least users who need to pay relevant fees through the current device. These users may want to use face payment or other payment methods, such as password payment, fingerprint payment, etc. Candidates to be identified may also include some irrelevant persons captured by the device lens, such as passers-by who flashed past the lens, passers-by who stood far away and did not pay attention to the device, etc. Ideally, if there is only one user in the face-scanning 2D image, it can be assumed that the user is the user who currently wants to pay by face. However, in actual applications, the face-scanning 2D image may contain more than two users, and only one of them is the user who currently wants to pay by face (i.e., the user who has the intention to pay by face), while other users are either waiting in line to pay or are not ready to pay at all.

[0061] It should be noted that, assuming that the degree of automatic triggering of face-scanning payment is high (for example, there is no need for the user or power source to actively trigger face-scanning payment by clicking), then even if there is only one user in the face-scanning 2D image, for safety reasons, the user's willingness to pay by face-scanning can be identified.

[0062] In one or more embodiments of this specification, when the device executes a single face-scanning payment instruction, it is to authenticate the identity of a specific candidate who currently enables face-scanning payment, and the specific candidate currently has the willingness to pay by face-scanning, that is, the specific candidate is considered to have a safe payment willingness. Other candidates do not currently have the willingness to pay by face-scanning, and their payment willingness is not safe. Even if one of these candidates is waiting in line for the next face-scanning payment, it is considered that the candidate does not currently have the willingness to pay by face-scanning, but will have the willingness to pay by face-scanning when it is his turn in the line.

[0063] In one or more embodiments of the present specification, at least a portion of the area where the candidate is located in the face-scanning 2D image is referred to as a first location area, and the first location area includes appearance feature information of the candidate, at least includes face information, and if necessary, may also include torso information, limb information, etc. The first location area may be determined by performing face area detection in the face-scanning 2D image.

[0064] Since the face scanning 2D image may contain multiple candidates' faces, this solution specifically identifies the willingness to pay for each face. The mask operation can be used to distinguish the first area of ​​a single candidate in the face scanning 2D image from other areas in the face scanning 2D image. Under the mask operation, the value of each pixel in the image can be recalculated through the mask kernel operator. The mask kernel operator characterizes the degree of influence of the domain pixel point on the new pixel value, and at the same time, the pixel points are weighted averaged according to the weight factor in the mask operator, so that the single face area of ​​focus can be highlighted.

[0065] In one or more embodiments of the present specification, the first area where a single candidate is located in the face-scanning 2D image corresponds to a single mask image. That is, if there are multiple candidates in the face-scanning 2D image, a corresponding mask image is generated for each candidate, and ultimately multiple mask images are obtained.

[0066] In a single mask image, the first area can be distinguished from other areas, for example, the first area is filled with a value of 1, and other areas are filled with values ​​of 0. That is, by generating a mask image corresponding to each candidate, the characteristic information of the candidate can be more distinct, thereby increasing the difference between those who are willing to pay by face recognition and those who are not willing to pay by face recognition.

[0067] S106: extracting features of the face scanning 2D image, and obtaining a first fusion feature according to the features of the face scanning 2D image and the mask image.

[0068] In one or more embodiments of the present specification, for example, the features of the face scanning 2D image are extracted through a pre-trained neural network model. The features of the face scanning 2D image may include facial features, torso features, limb features, etc. The facial features may be global features of the candidate's face, and recognizing the face through global features can improve the accuracy of the recognition result.

[0069] New channels can be added for the features of the face scanning 2D image, and the features of the mask image can be expressed through the newly added channels, thereby achieving feature fusion. This method can not only utilize these two types of features at the same time, but also clearly distinguish these two types of features, so as to facilitate separation when necessary, and to be able to more specifically learn the area of ​​concern in the face scanning 2D image.

[0070] S108: Determine whether the 3D mode is currently applicable. If yes, execute S110, otherwise, execute S112.

[0071] In one or more embodiments of the present specification, S108 may also be executed in advance, and has no time sequence dependency with the aforementioned steps.

[0072] Regarding whether the 3D modality is applicable, this question can be analyzed at multiple levels and granularities.

[0073] As mentioned above, offline IoT devices will provide different software and hardware configurations in different usage scenarios. If a device has a relatively low configuration, it may not support 3D mode at all, which means that 3D mode is not applicable. Based on this, the software status information and / or hardware status information of the current device can be obtained, and whether the software status information and / or hardware status information supports 3D mode can be used to determine whether 3D mode is currently applicable. At least it must be supported to be applicable.

[0074] In one or more embodiments of this specification, the reason for considering combining 3D modality is to improve security. The current improvement in security also comes at a price. On the one hand, the cost of software and hardware increases accordingly. On the other hand, it may also have an adverse effect on recognition efficiency. This solution is based on actual application scenarios and uses 3D modality more flexibly from another perspective to balance security and recognition efficiency. The following is an analysis and explanation.

[0075] In private settings, users usually use personal devices (such as smartphones) to pay by face recognition. In this case, if the user wants to do something malicious, for example, using someone else's photo or mask to impersonate another customer to scan the face in an attempt to deceive the device, then because the setting is relatively private, it is easier for the user to commit malicious acts.

[0076] However, in public places, users use public devices to pay by face recognition, such as queuing in supermarkets to check out by face recognition. In such cases, if the user wants to do evil, he or she needs to pay more attention to the surrounding environment. The surrounding environment may bring certain obstacles to the user's active evil. These obstacles may be intentional (for example, the store's surveillance cameras, other users waiting in line behind the user to pay attention to the user's checkout progress, etc.), or they may be unintentional (for example, other users around may accidentally glimpse the user, etc.). This scheme considers measuring the degree of such obstacles, and uniformly refers to the measurement results as the degree of public supervision. The reason is that this is essentially some factors in the public environment that restrict the possibility of users doing evil to a certain extent, which is equivalent to supervising the user.

[0077] Based on this idea, environmental information can be obtained, and the current level of public supervision can be determined based on the environmental information. The level of public supervision reflects the degree of obstacles that the surrounding environment brings to the current face-swiping payment object's active evil. Based on the level of public supervision, it is determined whether the 3D modality is currently applicable. If the level of public supervision exceeds the set level, it indicates that it is relatively difficult to commit evil at present, and it can be determined that the 3D modality is not currently applicable. For example, the 2D modality can be used alone, which helps to improve recognition efficiency and ensures security. On the contrary, if the level of public supervision does not exceed the set level, it indicates that it is relatively easy to commit evil at present, and it can be determined that the 3D modality is currently applicable. The 2D modality and the 3D modality can be combined for multi-modal recognition to ensure security.

[0078] Environmental information, such as information about some of the above factors, can be obtained from the face scan 2D image or other means such as pre-collected environmental images. For example, the number of users that can be detected from the face scan 2D image, the more users there are, the higher the degree of public supervision; for another example, whether there is a camera can be detected from the environmental image, and if there is a camera, the degree of public supervision is relatively higher; for another example, voice communication information that may exist between candidates can be detected through voice recognition, etc. This situation is relatively complicated, and if there is communication, it may mean that the degree of public supervision is relatively lower.

[0079] Furthermore, since the current candidates are relatively close to each other, it is easier to deter malicious behavior. Based on this, some candidates are regarded as powerful public representatives, and the current candidate status is considered to more reliably and timely measure the degree of public supervision.

[0080] For example, based on environmental information, the attention level between candidates is determined, and based on the attention level, the current public supervision level is determined according to the positive correlation between the public supervision level and the attention level. The attention level reflects the current level of attention of a candidate to another candidate, such as whether the candidate is looking at the other candidate from the side or directly, whether the candidate is not paying attention to the other candidate but staring at his or her mobile phone, whether the candidate is communicating with the other candidate, whether the candidate has physical contact with the other candidate, etc.

[0081] In most cases, the higher the attention, the higher the degree of public supervision. It should be noted that there is another situation to consider, that is, the two candidates who pay attention to each other may be accomplices in committing evil. For example, as mentioned above, if there is communication between the candidates, it may indicate that the degree of public supervision is relatively low. This is because strangers generally do not communicate. In addition to voice communication, some more intimate actions or even physical contact actions (actions can be recognized by images) may also indicate accomplices in committing evil. Based on this, if the attention is high to a certain level (called the basic level), it may reflect a relatively high degree of public supervision, and if it is high to a certain unusual level (called the suspected acquaintance level), it may instead reflect a relatively low degree of public supervision. Multiple thresholds can be set accordingly to make more reliable judgments.

[0082] In actual applications, especially in public places, there are often multiple users waiting in line to pay, so recognition efficiency is more likely to affect user experience than in private places. Security can be adjusted dynamically according to the current actual situation. For example, consider whether the current global business status is urgent enough and whether users can tolerate it. This solution comprehensively considers and measures these listed situations, which are called global business urgency.

[0083] The global here means that at least multiple candidates are considered. In addition, other environmental factors can also be considered, such as time, weather, etc. The business here is related to multiple candidates. For example, these candidates are currently waiting in line to complete payment services. The global business urgency can be determined based on environmental information, and whether the 3D modality is currently applicable can be determined based on the business urgency. If the business urgency is high, it can be considered that the 3D modality is not currently applicable, and recognition efficiency is prioritized. For example, assuming that the candidate queue is long or dense, it reflects that the global business urgency is high; for another example, in sudden rainy weather, multiple candidates who want to buy raincoats or umbrellas are detected, which reflects that the global business urgency is high.

[0084] S110: Obtain a 3D face-scanning image corresponding to the 2D face-scanning image, extract features of the 3D face-scanning image, obtain a second fusion feature based on the first fusion feature and the features of the 3D face-scanning image, and identify whether each of the candidates has a willingness to pay by face-scanning based on the second fusion feature.

[0085] In one or more embodiments of the present specification, features of the face scanning 3D image can be extracted through a pre-trained neural network model. Corresponding features can be extracted from the face scanning 3D image based on the detection results of the face scanning 2D image, for example, key point mapping, contour alignment and other processing are performed to determine the corresponding relationship, so as to facilitate feature fusion.

[0086] A new channel can be added to the features of the first fusion feature to combine the features of the face scanning 3D image to obtain the second fusion feature. In the second fusion feature, the complementarity of the two types of features allows the corresponding candidates to be represented more accurately, which helps to improve the accuracy of face recognition.

[0087] In one or more embodiments of the present specification, it is possible to identify whether the corresponding candidate has the willingness to pay by face recognition in combination with preset conditions. In the case of low security requirements and high efficiency requirements, simpler conditions can be set. For example, if the candidate's face area is identified to be in the middle area, the candidate is considered to have the willingness to pay by face recognition, or if the candidate's face area is identified to occupy most of the face recognition image, the candidate is considered to have the willingness to pay by face recognition, or if the candidate's face area is identified to occupy most of the face recognition image and the face angle meets the preset angle threshold, the candidate is considered to have the willingness to pay by face recognition.

[0088] In practical applications, in order to ensure security, this solution is more inclined to identification based on machine learning models. For example, the second fusion feature can be input into a willingness-to-pay identification model that has been previously supervised, and the candidate's feature information can be mapped in a high-dimensional manner through the willingness-to-pay identification model, and the mapping result can be output. Then, the mapping result is converted into a probability value of willingness-to-pay by face recognition, and the willingness-to-pay probability value can be used to determine whether the candidate has the willingness to pay by face recognition.

[0089] For example, if the probability value is greater than a preset probability threshold, it can be considered that the candidate has the willingness to pay by face recognition, that is, the face recognition payment instruction of the electronic device is generated after the candidate turns on the face recognition payment. If the probability value is less than or equal to the preset probability threshold, it can be considered that the candidate does not have the willingness to pay by face recognition. That is, the face recognition payment instruction of the electronic device is not generated after the candidate turns on the face recognition payment, but is generated after other candidates turn on the face recognition payment.

[0090] In one or more embodiments of this specification, if there are multiple candidates with probability values ​​greater than the preset probability, the result of this face-scanning payment willingness recognition may be considered unreliable, and the authentication failure may be prompted. Conversely, if there is no candidate with a probability value greater than the preset probability, it also means that the result of this face-scanning payment willingness recognition may be unreliable, and the authentication failure may be prompted.

[0091] S112: Identify, based on the first fusion feature, whether each of the candidates is willing to pay by face recognition.

[0092] In one or more embodiments of the present specification, if the 3D modality is not currently applicable, recognition can be performed based on a simpler single modality (eg, 2D modality), thereby improving recognition efficiency and being applicable to more devices.

[0093] In the case of S112, the device is able to realize lack of modality (for example, lack of 3D modality, using only 2D modality) recognition. Not only that, even in the case of no lack of modality, it can still intelligently and dynamically choose to use multi-modal recognition to compensate for safety, or directly use lack of modality recognition to improve efficiency based on the inherent safety level of the on-site environmental conditions.

[0094] pass Figure 1 The method can generate a corresponding mask map for each candidate's first location in the face-scanning 2D image, so that the candidate's feature information can be more distinct, and the difference between those with the willingness to pay by face-scanning and those without the willingness to pay by face-scanning is increased. The first fusion feature can enhance the image contrast effect, thereby focusing attention on the candidates with the willingness to pay by face-scanning. Compared with the 2D modality, the 3D modality is less supported in different face-scanning payment devices. Therefore, by judging whether the current face-scanning device is applicable to the 3D modality in the current recognition scenario, if applicable, multi-modal recognition is prioritized. By combining the features of the corresponding candidate in the face-scanning 3D image with the features of the candidate in the face-scanning 2D image through the second fusion feature, the two complement each other's facial features of the same candidate, which can further improve the accuracy of face recognition objects. If it is not applicable, it can be recognized based on the 2D modality alone, which is low-cost and helps to support more devices to realize face-scanning payment. In summary, it is also helpful to accurately distinguish candidates with the willingness to pay by face-scanning from candidates without the willingness to pay by face-scanning in the same face-scanning image, thereby improving the security of face-scanning payment.

[0095] based on Figure 1 This specification also provides some specific implementation plans and extension plans of the method, which will be described below.

[0096] In one or more embodiments of the present specification, after the candidate is determined, the face area of ​​the candidate in the face scanning 2D image is extracted. For example, the face of the candidate in the face scanning 2D image is first extracted by a face extraction model, and then the face area of ​​the candidate is determined by the position information of the face. The face area is then processed to determine the face area selection box. The face area selection box can be displayed in a variety of ways, such as a circular box, a rectangular box, an irregular polygonal box, etc. However, there is a prerequisite that in order to ensure the accuracy of the recognition result, the face area selection box at least contains all the face areas of the current candidate.

[0097] After obtaining the face region selection frame, the first filling region of the mask image corresponding to the candidate will be determined according to the face region selection frame of the candidate. The shape of the first filling region can be various. In order to ensure the accuracy of the recognition result, when determining the first filling region, it can be made to fit the actual face contour as much as possible.

[0098] After the first filling area is determined, a second filling area other than the first filling area is further determined in the face scanning image, and different filling values ​​are assigned to the first filling area and the second filling area.

[0099] In order to make the first filling area in the mask image match the face area in the face-scanning 2D image as much as possible, after assigning different filling values ​​to the first filling area and the second filling area, a mask image with a resolution consistent with the resolution of the face-scanning 2D image is generated.

[0100] Furthermore, since most human faces are circular or elliptical, in order to make the first filling area more closely fit the candidate's facial area, the first filling area may be determined as a circular area or an elliptical area, and a circular area is relatively easier to handle.

[0101] Based on this, for example, when processing the face area and determining the face area selection frame, the face area selection frame is determined as a rectangular frame. After obtaining the face area of ​​the rectangular frame, the face frame width and the face frame height are calculated by the position of the rectangular frame in the face scan image, and the radius of the circular area is calculated by the face frame width and the face frame height.

[0102] Among them, when calculating the radius of the circular area, since the face area is a circular area or an elliptical area, when the rectangular frame is initially generated, the face area is similar to the inscribed circle of the rectangular frame. Therefore, in order to restore the face area as much as possible and ensure that the first filling area includes the entire face area as much as possible, the maximum value between half the width of the rectangular frame and half the height of the frame is taken as the radius of the circular area.

[0103] Therefore, the center of the rectangular box is taken as the center of the circle, half the length of the longest side of the rectangular box is determined as the radius, and then the circular area formed by the center of the circle and the radius is determined as the first filling area of ​​the mask map corresponding to the candidate.

[0104] For example, suppose the position of the candidate's face rectangle in the face scan image is (x 1 ,y 1 ,x 2 ,y 2 ), where x 1 With x 2 are the x-axis coordinates of the width of the rectangle, and y-axis coordinates of the 1 With y 2 They are respectively the position coordinates of the rectangular frame height on the y-axis.

[0105] Then the expression for calculating the face frame width is w = x 2 -x 1 , where w is the face frame width, x 1 <x 2 .

[0106] Then the expression for calculating the face frame height is h = y 2 -y 1 , where h is the face frame height, y 1 <y 2 .

[0107] The expression for determining the radius of the circular area is

[0108] Then the position coordinates of the center of the rectangular box are For example, the filling value of the circular area is determined to be 1, and the filling value of the background area outside the circular area is determined to be 0.

[0109] In one or more embodiments of the present specification, a multi-prediction head deep convolutional neural network is pre-constructed as a recognition model, and the recognition system at least includes the model. The multi-prediction head deep convolutional neural network includes two end-to-end network prediction channels trained according to at least partially different modalities, one of which is used for multi-modal recognition and the other is used for modality-free recognition. Each channel corresponds to one or more different prediction heads, and the mutual independence of the two channels helps to better train to complete its own prediction task. The multi-prediction head deep convolutional neural network is pre-learned for multi-modal information (at least including face scanning 2D image samples and face scanning 3D image samples) end-to-end, and a candidate region attention mechanism is introduced for face scanning 2D images to integrate mask maps into learning, so that the model can more specifically identify the face scanning willingness to pay of candidates in face scanning 2D images. The system framework is given as an example below.

[0110] Figure 2A schematic diagram of the framework of a face-scanning payment willingness recognition system provided for one or more embodiments of this specification.

[0111] exist Figure 2 In the figure, a part of the structure is selected with a large solid box, which is used for multimodal recognition, and another part of the structure is selected with a large solid box, which is used for non-modal recognition. These two parts have a common part, which is used to process the 2D modality alone to obtain the first fusion feature, and then generate branches based on the judgment result of whether the 3D modality is applicable. One branch belongs to the above-mentioned multimodal recognition prediction channel, and the other branch belongs to the above-mentioned non-modal recognition prediction channel. The face-swiping 2D image, the mask map generated for the candidate, and the optional face-swiping 3D image are used as the input data of the system, and the output is the probability value of the willingness to pay by face-swiping, that is, the probability of willingness to be safe and the probability of willingness to be unsafe. The working process of the system is illustrated by way of example.

[0112] For the 2D modality, the features of the 2D face image are extracted through the first convolutional network, which are referred to as 2D features in the figure. The 2D features and the mask map corresponding to the current candidate are input into the attention mechanism implementation module, and the first fusion features are output. For example, the mask map can be down-sampled to adapt to the features of the 2D face image, and then the 2D features and the mask map after down-sampled are connected according to the channel dimension, and the connected features are input into the second convolutional network for fusion processing.

[0113] The branch selection of the prediction channel is performed based on whether the 3D mode is currently applicable.

[0114] If the 3D mode is applicable, the relevant business logic can be executed.

[0115] Since the depth map refers to an image that uses the real distance (depth) from the image collector to each point in the scene as the pixel value, the farther the distance (here the distance also includes the depth) of each pixel in the face-scanning 3D image is from the image-scanning device, the greater the depth value of the corresponding pixel. When a candidate is performing face-scanning payment willingness recognition, he or she is usually closest to the image-scanning device relative to other candidates.

[0116] Therefore, in the case where there are multiple candidates in the face-scanning 3D image, at this time, the candidate who does not have the willingness to pay by face-scanning is usually farther away from the image acquisition device, so the depth value of the corresponding pixel point of the candidate who does not have the willingness to pay by face-scanning in the face-scanning 3D image is greater than the depth value of the corresponding pixel point of the candidate who has the willingness to pay by face-scanning in the face-scanning 3D image. It is possible to perform fine processing on the face-scanning 3D image before extracting the features of the face-scanning 3D image to filter out the pixels whose distance from the image acquisition device exceeds a preset threshold. It should be noted that the filtering here may not simply abandon these pixels completely, but may perform unified generalization processing on these pixels, such as unifying the values ​​used to represent these pixels into a specified same value. The purpose is to reduce the differences between these pixels and reduce the contribution of these pixels to the model training and use process, so that the model can focus more on the pixels within the preset threshold, concentrate the computing power on more valuable pixels, improve efficiency, and reduce interference. Therefore, for the candidates who start this face-scanning payment, it is also helpful to improve the accuracy of face-scanning payment willingness recognition.

[0117] Through the fourth convolutional network, features are extracted from the refined 3D face scanned image, which are referred to as 3D features in the figure.

[0118] The 3D feature and the first fused feature are input into the multimodal feature fusion module, and the multimodal feature fusion module processes the 3D feature and the first fused feature and outputs a second fused feature. Specifically, for example, the first fused feature and the 3D feature are connected according to the channel dimension, and the connected feature is input into the fifth convolutional network for fusion processing.

[0119] The second fused feature is then input into the sixth convolutional network for processing to obtain the probability value of willingness to pay by face recognition. There are two possible situations: willingness to be safe and willingness to be unsafe, which are represented by the full-modal safety probability 1 and the full-modal unsafe probability 2 in the figure respectively.

[0120] If the 3D modality is not applicable, prediction can be made based on the 2D modality alone.

[0121] The first fused feature is input into the third convolutional network for processing to obtain the probability value of willingness to pay by face recognition. There are two possible situations: willingness to be safe and willingness to be unsafe, which are represented by the lack of modality safety probability 1 and the lack of modality unsafe probability 2 in the figure respectively.

[0122] In one or more embodiments of the present specification, the first convolutional network and the third convolutional network can be pre-split from the same convolutional network, and the first convolutional network and the sixth convolutional network can be pre-split from the same convolutional network. The third convolutional network has the same structure as the sixth convolutional network but does not share, and is a separate module. The same convolutional network, for example, uses resnet, ShuffleNetV2, etc.

[0123] The splitting process, for example, includes: determining a target resolution so as to serve as the resolution of the features of the face scanning 2D image, determining a convolution layer that matches the target resolution in the convolution layer of the same convolution network, using the matching convolution layer as a split point, and splitting the same convolution network into a front part and a back part, with the front part serving as the first convolution network and the back part serving as another convolution network required. When determining the target resolution, for example, the resolution of the 2D features is set to the resolution of the face scanning 2D image. The target resolution is the resolution of the face scanning 2D image.

[0124] Further supplementary explanation of the business logic related to the 3D modality. Specifically introduce how to process the face scanning 3D image before extracting the features of the face scanning 3D image, so as to filter out the pixels whose distance from the image acquisition device exceeds the preset threshold.

[0125] Calculate the depth value of the second region of each candidate in the face scanning 3D image. The region where the candidate is located in the face scanning 3D image is called the second region. The second region includes the face region.

[0126] Among all the pixels in the second area, the pixels whose distance from the image acquisition device exceeds a preset threshold are filtered according to the depth value of the second area. When setting the preset threshold, it is necessary to effectively ensure through experience that the depth value of the pixel corresponding to the candidate who is willing to pay by face recognition is less than the preset threshold.

[0127] Furthermore, since it is necessary to obtain the second area of ​​each candidate in the 3D face-scanning image when calculating the depth value of the second area of ​​each candidate in the 3D face-scanning image, the face-scanning payment willingness recognition system will obtain the 2D face-scanning image and extract the features of the 2D face-scanning image. Therefore, in order to reduce the computational pressure of the system, the detection results of the 2D face-scanning image are corresponded to the 3D face-scanning image to obtain the second area, rather than directly extracting the features of the 3D face-scanning image through the second feature extraction model.

[0128] Specifically, firstly, the face area of ​​each candidate in the face scanning 3D image is determined according to the face area selection box of each candidate in the face scanning 2D image. For example, the position coordinates of the face area selection box in the face scanning 2D image are extracted, and then according to the position coordinates, the face area selection box is mapped to the face scanning 3D image, so as to obtain the face area of ​​each candidate in the face scanning 3D image.

[0129] Then, multiple depth values ​​of all pixels in the face area are calculated, and finally the depth value of the face area of ​​each candidate in the face scanning 3D image is determined based on the average value of the multiple depth values. For example, the average value of the multiple depth values ​​is used as the depth value of the face area. In other words, the average value of the depth measurement results in the face area is taken as the depth value of the face.

[0130] Furthermore, when filtering pixels whose distance from the image acquisition device exceeds a preset threshold according to the depth value of the second area, for the convenience of data processing, the depth value of the second area is used as a reference, and the multiple depth values ​​of all pixels are processed and mapped accordingly to the required specified range to better reflect the difference between the pixels and facilitate subsequent calculations. The processed multiple depth values ​​can be filtered to filter out pixels whose distance from the image acquisition device exceeds the preset threshold.

[0131] Specifically, first, the ratios between the multiple depth values ​​of all pixels and the depth value of the face area are calculated respectively. Since the depth value of the face area is obtained based on the average value of the multiple depth values ​​of all pixels, the multiple depth values ​​of all pixels are not much different from the depth value of the face area. In other words, if all the multiple depth values ​​are positive numbers, the data distribution of the ratio result will mostly be between 0 and 2.

[0132] Then, according to the reference value and the ratio of the depth value of the face area, the multiple depth values ​​of all the pixels are processed to be within the vicinity of the reference value. The reference value of the depth value of the face area can be set according to actual needs. For example, the reference value of the depth value of the face area is set to 127, that is, if the multiple depth values ​​are all positive numbers, then the data distribution of the multiple depth values ​​of all the pixels after processing is, for example, mostly between 0-254.

[0133] For example, in order to reduce the calculation pressure of the system, the product of the reference value and the ratio of the depth value of the face area can be calculated first, and then based on the product, the multiple depth values ​​of all pixels are processed to be within the range near the reference value.

[0134] Finally, all the pixels whose multiple depth values ​​after processing are greater than the preset reference threshold are filtered. The larger the depth value of the processed pixel, the farther the pixel is from the image acquisition device. The preset reference threshold can be set according to actual needs. For example, the reference threshold is set to 127+30=157, where 30 is an exemplary value and can be adjusted according to actual needs. The larger the value, the more pixels at farther distances are considered.

[0135] Furthermore, when the candidate is very close to the image acquisition device, the candidate's face area may be incomplete, that is, the image acquisition device does not fully capture the candidate's face area, which will also affect the recognition result of the candidate's willingness to pay by face recognition. Therefore, in order to ensure the accuracy of the recognition result, the pixels far away from the image acquisition device can be filtered, and the pixels very close to the image acquisition device can also be filtered at the same time.

[0136] Specifically, a first preset reference threshold and a second preset reference threshold are preset, wherein the first preset reference threshold is less than the reference value, and the second preset threshold is greater than the reference value. When setting the second preset threshold, the adjustable interval can be set based on the reference value.

[0137] Then, when filtering the pixels corresponding to the multiple depth values ​​of all processed pixels greater than the preset reference threshold, it is equivalent to filtering the pixels corresponding to the second preset reference threshold, which means that the pixels very far from the image acquisition device are filtered, and when filtering the pixels corresponding to the depth values ​​less than the first preset reference threshold, it means that the pixels very close to the image acquisition device are filtered. That is, it can be considered that the pixels corresponding to the depth values ​​greater than the first preset reference threshold and less than the second preset reference threshold are retained as realistically as possible, and other pixels can be generalized or ignored.

[0138] Based on this, the maximum value between multiple depth values ​​of all processed pixel points and the first preset reference threshold can be extracted respectively, and the minimum value between the maximum value and the second preset reference threshold can be extracted. According to the maximum value and the minimum value, the processed pixel points whose depth values ​​are greater than the second preset reference threshold are filtered, and the processed pixel points whose depth values ​​are less than the first preset reference threshold are filtered.

[0139] For example, the following expression is used to filter out pixel points corresponding to a value less than the first preset reference threshold value through the maximum value, and to filter out pixel points corresponding to a value greater than the second preset reference threshold value through the minimum value.

[0140] For example, the expression is Among them, D is the depth value of the pixel point in the face-scanning 3D image, d is the depth value of the face area, 127 is the benchmark value of the depth value of the face area, 0 is the first preset benchmark threshold, 30 is the adjustable value of the pixel point far away from the image acquisition device, and an interval including 30 can be set as the above-mentioned adjustable interval, 127+30 is the second preset benchmark threshold, min is the minimum value, and max is the maximum value.

[0141] based on Figure 2 More intuitively, some key steps of training and launching the system are summarized as follows: Figure 3 shown.

[0142] S302: Establish a multimodal training dataset.

[0143] Offline IoT devices collect 2D and 3D face scanning images. Assume that the image shows user A using face scanning to pay. Regardless of whether there are multiple people in the image, user A is selected and marked as a positive sample (i.e., intentional safety). If there are multiple people in the image, other user B is selected and marked as a negative sample (i.e., intentional unsafe).

[0144] Each annotation data includes: the captured image; the position of the face frame of the user who can be a candidate on the face scanning 2D image (x 1 ,y 1 ,x 2 ,y 2 ); a label {1,0} indicating whether the user is willing to pay by face recognition, where 1 represents yes, i.e. willingness and safety, and 0 represents no, i.e. no willingness and safety. The face frame is located at the position (x 1 ,y 1 ,x 2 ,y 2 ) corresponds to the face scanning 3D image to determine the face area, and the average value of the depth measurement results in the face area in the face scanning 3D image is taken as the depth value d of the face.

[0145] S304: Perform two-stage network training.

[0146] In the first stage, we train the network for end-to-end learning of multimodal (2D and 3D) information. We sample from the established multimodal training dataset, obtain the training batch and its corresponding label, and input it into Figure 2 In the multimodal information end-to-end learning network (solid line frame part), the probability value related to willingness is output. The loss function is calculated by the label corresponding to the training batch output by the network, and the network is trained by the gradient descent method.

[0147] The second stage is to train the end-to-end learning network for modality-deficient (only 2D modality) information. The weights of the first convolutional network and the attention mechanism implementation module are fixed. This part is shared with the multimodal information end-to-end learning network. The training has been completed in the first stage. Sample from the established multimodal training data set to obtain the training batch and its corresponding label, and input it into Figure 2 In the end-to-end learning network without modal information (the dotted line frame), the probability value related to willingness is output. The network training is carried out based on this.

[0148] S306: Model deployment reasoning.

[0149] When it is actually used online, you can determine whether the current device is currently applicable to the 3D mode. If applicable, choose to use the multimodal information end-to-end learning network to predict the probability value of whether the intention is safe, and then compare it with the set threshold to obtain the recognition result. If not applicable, choose the end-to-end learning network without modal information for prediction, which will not be repeated.

[0150] Based on the same idea, one or more embodiments of this specification also provide devices and apparatuses corresponding to the above methods, such as Figure 4 , Figure 5 shown.

[0151] Figure 4 A schematic diagram of a device for identifying willingness to pay by face recognition provided in one or more embodiments of this specification, the device comprising:

[0152] 2D image acquisition module 402, acquires face scanning 2D image;

[0153] The mask map generating module 404 determines candidates to be identified in the face scanning 2D image, and generates corresponding mask maps according to the first location area of ​​each candidate in the face scanning 2D image to distinguish the first location area from other areas in the face scanning 2D image;

[0154] The 2D modality fusion module 406 extracts the features of the face scanning 2D image, and obtains a first fusion feature according to the features of the face scanning 2D image and the mask image;

[0155] A modality adaptability determination module 408 determines whether the 3D modality is currently applicable;

[0156] The multimodal fusion module 410, if yes, obtains the face-scanning 3D image corresponding to the face-scanning 2D image, extracts the features of the face-scanning 3D image, obtains the second fusion feature according to the first fusion feature and the features of the face-scanning 3D image, and the first recognition module 412 recognizes whether each of the candidates has the willingness to pay by face-scanning according to the second fusion feature;

[0157] The second identification module 414, otherwise, identifies whether each of the candidates has the willingness to pay by face recognition according to the first fusion feature.

[0158] Optionally, the modality adaptability judgment module 408 acquires environmental information;

[0159] Determine the current public supervision level based on the environmental information, where the public supervision level reflects the degree of obstacles that the surrounding environment brings to the current face-scanning payment object's active evil deeds;

[0160] Based on the degree of public supervision, it is determined whether the 3D mode is currently applicable.

[0161] Optionally, the modality adaptability determination module 408 determines the attention degree between the candidates according to the environmental information;

[0162] Based on the attention level, the current public supervision level is determined according to the positive correlation relationship set between the public supervision level and the attention level.

[0163] Optionally, the modality adaptability determination module 408 determines the global urgency of a business according to the environmental information, the business being related to the plurality of candidates;

[0164] According to the business urgency, it is determined whether the 3D mode is currently applicable.

[0165] Optionally, the modality adaptability determination module 408 obtains software status information and / or hardware status information of the current device;

[0166] Whether the 3D mode is currently applicable is determined according to whether the software status information and / or the hardware status information supports the 3D mode.

[0167] Optionally, after determining whether the 3D mode is currently applicable, the modality adaptability determination module 408 further executes:

[0168] Determining a pre-built multi-prediction head deep convolutional neural network, the multi-prediction head deep convolutional neural network comprising two end-to-end network prediction channels trained according to at least partially different modalities;

[0169] According to the result of the judgment, a corresponding one of the two end-to-end network prediction channels is selected to identify whether each of the candidates has the willingness to pay by face recognition.

[0170] Optionally, the device is applied to an offline IoT face-scanning device, and the face-scanning 2D image is collected by the IoT face-scanning device for nearby users, and the face-scanning 2D image contains at least two faces.

[0171] Figure 5 A schematic diagram of a face-scanning payment willingness recognition device provided in one or more embodiments of this specification, the device comprising:

[0172] at least one processor; and,

[0173] a memory communicatively connected to the at least one processor; wherein,

[0174] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0175] Get the face scan 2D image;

[0176] Determine candidates to be identified in the face scanning 2D image, and generate corresponding mask images according to first regions of each candidate in the face scanning 2D image to distinguish the first regions from other regions in the face scanning 2D image;

[0177] Extracting features of the face scanning 2D image, and obtaining a first fusion feature according to the features of the face scanning 2D image and the mask image;

[0178] Determine whether the 3D mode is currently applicable;

[0179] If yes, then obtain a 3D face recognition image corresponding to the 2D face recognition image, extract features of the 3D face recognition image, obtain a second fusion feature based on the first fusion feature and the features of the 3D face recognition image, and identify whether each of the candidates has a willingness to pay by face recognition based on the second fusion feature;

[0180] Otherwise, based on the first fusion feature, identify whether each of the candidates is willing to pay by face recognition.

[0181] Based on the same idea, one or more embodiments of this specification also provide a non-volatile computer storage medium for face-scanning payment willingness recognition corresponding to the above method, storing computer executable instructions, and the computer executable instructions are set to:

[0182] Get the face scan 2D image;

[0183] Determine candidates to be identified in the face scanning 2D image, and generate corresponding mask images according to first regions of each candidate in the face scanning 2D image to distinguish the first regions from other regions in the face scanning 2D image;

[0184] Extracting features of the face scanning 2D image, and obtaining a first fusion feature according to the features of the face scanning 2D image and the mask image;

[0185] Determine whether the 3D mode is currently applicable;

[0186] If yes, then obtain a 3D face recognition image corresponding to the 2D face recognition image, extract features of the 3D face recognition image, obtain a second fusion feature based on the first fusion feature and the features of the 3D face recognition image, and identify whether each of the candidates has a willingness to pay by face recognition based on the second fusion feature;

[0187] Otherwise, based on the first fusion feature, identify whether each of the candidates is willing to pay by face recognition.

[0188] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0189] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0190] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0191] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0192] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification may be in the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of this specification may be in the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0193] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0194] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0195] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0196] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0197] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0198] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0199] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0200] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0201] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0202] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0203] The above description is only one or more embodiments of this specification and is not intended to limit this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of this specification shall be included in the scope of the claims of this specification.

Claims

1. A method for identifying willingness to pay by face recognition, comprising: Obtaining a 2D face scan image, which is collected by the device for nearby users; Determining candidates to be identified in the face scanning 2D image; Determining whether the 3D mode is currently applicable, specifically comprising: obtaining environmental information, and determining the current public supervision level based on the environmental information, wherein the public supervision level reflects the degree of obstacles brought by the surrounding environment to the current face-scanning payment object's active evil deeds, and determining whether the 3D mode is currently applicable based on the public supervision level; If so, a face scanning 3D image corresponding to the face scanning 2D image is obtained to identify whether each of the candidates is willing to pay by face scanning.

2. The method according to claim 1, wherein determining the current level of public supervision based on the environmental information specifically comprises: Determining the attention degree between the candidates according to the environmental information; Based on the attention level, the current public supervision level is determined according to the positive correlation relationship set between the public supervision level and the attention level.

3. The method according to claim 1 or 2, wherein the determining whether the 3D mode is currently applicable further comprises: Determining, based on the environmental information, a global business urgency, the business being related to the plurality of candidates; According to the business urgency, it is determined whether the 3D mode is currently applicable.

4. The method according to claim 1 or 2, wherein determining whether the 3D mode is currently applicable comprises: Obtain software status information and / or hardware status information of the current device; Whether the 3D mode is currently applicable is determined according to whether the software status information and / or the hardware status information supports the 3D mode.

5. The method of claim 1, further comprising: According to the first regions of each candidate in the face scanning 2D image, respectively generate corresponding mask images to distinguish the first regions from other regions in the face scanning 2D image; Extracting features of the face scanning 2D image, and obtaining a first fusion feature according to the features of the face scanning 2D image and the mask image; A second fusion feature is obtained according to the first fusion feature and the feature of the face-scanning 3D image, and according to the second fusion feature, it is identified whether each of the candidates has the willingness to pay by face-scanning.

6. The method according to claim 1, wherein extracting the features of the face scanning 2D image and obtaining the first fusion feature according to the features of the face scanning 2D image and the mask map specifically comprises: Extracting features of the face scanning 2D image through a first convolutional network; Performing resolution reduction processing on the mask image to adapt to the features of the face scanning 2D image; The features of the face scanning 2D image and the mask image after the resolution reduction processing are fused through the second convolutional network to obtain a first fused feature.

7. The method according to claim 6, wherein the step of identifying whether each candidate has a willingness to pay by face recognition based on the first fusion feature comprises: Inputting the first fused feature into a third convolutional network corresponding to the first convolutional network for processing to obtain a first processing result, wherein the first convolutional network and the third convolutional network are pre-split from the same convolutional network; A probability value is generated according to the first processing result to indicate whether the corresponding candidate is willing to pay by face recognition.

8. The method according to claim 6, wherein extracting the features of the face-scanning 3D image and obtaining the second fusion feature according to the first fusion feature and the features of the face-scanning 3D image comprises: Extracting features of the face scanning 3D image through a fourth convolutional network; According to the channel dimension, the first fusion feature and the feature of the face scanning 3D image are connected; the connected feature is input into the fifth convolutional network for processing to obtain the second fusion feature.

9. The method according to claim 8, wherein the step of identifying whether each candidate has a willingness to pay by face recognition based on the second fusion feature comprises: Inputting the second fused feature into a sixth convolutional network corresponding to the first convolutional network for processing to obtain a second processing result, wherein the first convolutional network and the sixth convolutional network are pre-split from the same convolutional network; A probability value is generated according to the second processing result to indicate whether the corresponding candidate is willing to pay by face recognition.

10. The method according to claim 1, after determining whether the 3D mode is currently applicable, the method further comprises: Determining a pre-built multi-prediction head deep convolutional neural network, the multi-prediction head deep convolutional neural network comprising two end-to-end network prediction channels trained according to at least partially different modalities; According to the result of the judgment, a corresponding one of the two end-to-end network prediction channels is selected to identify whether each of the candidates has the willingness to pay by face recognition.

11. The method according to claim 5, wherein generating a corresponding mask map according to the first region of each candidate in the face scanning 2D image comprises: For each of the determined candidates, execute: Determine, according to the face region selection box of the candidate, a first filling region of the mask image corresponding to the candidate, and a second filling region outside the first filling region; By assigning different filling values ​​to the first filling area and the second filling area, the mask image having a resolution consistent with a resolution of the face scanning 2D image is generated.

12. The method according to claim 11, wherein the face area selection frame is a rectangular frame; The step of determining the first filling area of ​​the mask image corresponding to the candidate according to the face area selection box of the candidate specifically includes: The center of the rectangular box is determined as the center of the circle, and half the length of the longest side of the rectangular box is determined as the radius; a circular area formed by the center of the circle and the radius is determined as the first filling area of ​​the mask image corresponding to the candidate.

13. The method according to claim 7 or 9, wherein the splitting comprises: Determine a target resolution so as to serve as a resolution of features of the face scanning 2D image; Among the convolutional layers in the same convolutional network, determining a convolutional layer that matches the target resolution; Taking the matched convolution layer as a splitting point, the same convolution network is split into a front part and a rear part, the front part serves as the first convolution network, and the rear part serves as another required convolution network.

14. The method according to claim 1, wherein the method is applied to an offline IoT face recognition device, the face recognition 2D image is collected by the IoT face recognition device for nearby users, and the face recognition 2D image contains at least two faces.

15. A device for identifying willingness to pay by face recognition, comprising: 2D image acquisition module, which acquires face scanning 2D images. The face scanning 2D images are collected by the device for nearby users; A mask image generation module is used to determine candidates to be identified in the face scanning 2D image; The modality adaptability judgment module judges whether the 3D modality is currently applicable, specifically including: obtaining environmental information, and determining the current public supervision level based on the environmental information, wherein the public supervision level reflects the degree of obstacles brought by the surrounding environment to the current face-scanning payment object's active evil deeds, and judging whether the 3D modality is currently applicable based on the public supervision level; The multimodal fusion module obtains the face recognition 3D image corresponding to the face recognition 2D image to identify whether each of the candidates is willing to pay by face recognition.

16. The apparatus of claim 15, wherein the modality adaptability determination module determines the attention level between the candidates based on the environmental information; Based on the attention level, the current public supervision level is determined according to the positive correlation relationship set between the public supervision level and the attention level.

17. The apparatus according to claim 15 or 16, wherein the modality adaptability determination module determines the global urgency of a business according to the environmental information, the business being related to a plurality of the candidates; According to the business urgency, it is determined whether the 3D mode is currently applicable.

18. The device according to claim 15 or 16, wherein the modality adaptability judgment module obtains software status information and / or hardware status information of the current device; Whether the 3D mode is currently applicable is determined according to whether the software status information and / or the hardware status information supports the 3D mode.

19. The device according to claim 15, wherein the modality adaptability determination module, after determining whether the 3D modality is currently applicable, further executes: Determining a pre-built multi-prediction head deep convolutional neural network, the multi-prediction head deep convolutional neural network comprising two end-to-end network prediction channels trained according to at least partially different modalities; According to the result of the judgment, a corresponding one of the two end-to-end network prediction channels is selected to identify whether each of the candidates has the willingness to pay by face recognition.

20. The device as claimed in claim 15, wherein the device is applied to an offline IoT face recognition device, the face recognition 2D image is collected by the IoT face recognition device for nearby users, and the face recognition 2D image contains at least two faces.

21. A face-scanning payment willingness recognition device, comprising: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Get the face scan 2D image; Determining candidates to be identified in the face scanning 2D image; Determining whether the 3D mode is currently applicable, specifically comprising: obtaining environmental information, and determining the current public supervision level based on the environmental information, wherein the public supervision level reflects the degree of obstacles brought by the surrounding environment to the current face-scanning payment object's active evil deeds, and determining whether the 3D mode is currently applicable based on the public supervision level; If so, a face scanning 3D image corresponding to the face scanning 2D image is obtained to identify whether each of the candidates is willing to pay by face scanning.

Images